COVID forecasting failed
Failure in epidemic forecasting is an old problem. In fact, it is surprising that epidemic forecasting has retained much credibility among decision-makers, given its dubious track record. Modeling for swine flu predicted 3,100-65,000 deaths in the UK . Eventually only 457 deaths occurred . The prediction for foot-and-mouth disease expected up to 150,000 deaths in the UK and led to slaughtering millions of animals. However, the lower bound of the prediction was as low as only 50 deaths, a figure close to the eventual fatalities. Predictions may work in “ideal”, isolated communities with homogeneous populations, not the complex current global world.
Forecasting for COVID-19 has failed
Authors: John P.A. Ioannidis, Sally Cripps, Martin A. Tanner
From: Stanford University, University of Sydney, Northwestern University
Science is about understanding properties, not forecasting single outcomes
In this article, Nicholas Taleb argues that as pandemics are extremely fat tailed, point forecasts are close to useless. So, risk management should be concerned with tail properties and distribution of extrema, not averages or survival functions.
At the onset of the COVID-19 pandemic, many researcher groups and agencies produced single point “forecasts” for the pandemic — most relied on the compartmental SIR model, sometimes supplemented with cellular automata. The prevailing idea is that producing a numerical estimate is how science is done, and how science-informed decision-making ought to be done: bean counters producing precise numbers.
Well, no. That’s not how “science is done”, at least in this domain, and that’s not how informed decision-making ought to be done. Furthermore, subsequently, many criticized the predictions because these did not play out (no surprise there). This is also wrong. Both forecasters (who missed) and their critics were wrong — and the forecasters would have been wrong even if they got the prediction right.
On single point forecasts for fat tailed variables
Author: Nassim Nicholas Taleb
From: New York University
How is public trust in science affected by COVID?
In this paper Aksoy and al. use a 2018 Wellcome Trust survey of more than 70,000 individuals in 160 countries with data on global epidemics since 1970 to test the hypothesis of whether COVID-19 will reverse the secular trend toward questioning the value of scientific research and expertise. They find:
- It is unclear whether the authority of science and scientists will be enhanced or diminished by COVID-19;
- Past pandemics did not have an impact on the public’s perception of the importance of science;
- But, past pandemics did reduce confidence in individual scientists, worsened perceptions of their honesty, and weakened the belief that their activities benefit the public!
Revenge of the Experts: Will COVID-19 Renew or Diminish Public Trust in Science?
Authors: Cevat Giray Aksoy, Barry Eichengreen, Orkun Saka
From: European Bank for Reconstruction and Development, University of California, Berkeley, University of Sussex
