Skip to content
 

(Some) forecasting for COVID-19 has failed: a discussion of Taleb and Ioannidis et al.

Nassim Taleb points us to this pair of papers:

On single point forecasts for fat tailed variables, by Nassim Taleb

Forecasting for COVID-19 has failed, by John Ioannidis, Sally Cripps, and Martin Tanner

The two articles agree in their mistrust of media-certified experts.

Here’s Taleb:

Both forecasters and their critics are wrong: At the onset of the COVID-19 pandemic, many researcher groups and agencies produced single point “forecasts” for the pandemic — most relied on the compartmental SIR model, sometimes supplemented with cellular automata. The prevailing idea is that producing a numerical estimate is how science is done, and how science-informed decision-making ought to be done: bean counters producing precise numbers.

Well, no. That’s not how “science is done”, at least in this domain, and that’s not how informed decision-making ought to be done. Furthermore, subsequently, many criticized the predictions because these did not play out (no surprise there). This is also wrong. Both forecasters (who missed) and their critics were wrong — and the forecasters would have been wrong even if they got the prediction right. . . .

Here are Ioannidis et al.:

COVID-19 is a major acute crisis with unpredictable consequences. Many scientists have struggled to make forecasts about its impact. However, despite involving many excellent modelers, best intentions, and highly sophisticated tools, forecasting efforts have largely failed. . . . Despite these obvious failures, epidemic forecasting continued to thrive, perhaps because vastly erroneous predictions typically lacked serious consequences. . . .

But Taleb makes a point that the others miss:

What’s relevant is the distributional forecast, not the point forecast. This came up last month, when political columnist John Fund criticized Imperial College epidemiologist Neil Ferguson for getting a bunch of forecasts wrong—but it turned out that what Fund was doing was taking the upper bounds of Ferguson’s forecasts of past public health crises and pointing out that they were overestimates of the actual number of deaths in each case. That was just wrong on Fund’s part: it’s the nature of upper bounds that they will generally too high.

Ioannidis et al. do show problems with the distributional forecasts from the University of Washington IHME, and indeed Cripps and Tanner, the two coauthors of the recent article with Ioannidis, are also coauthors of an earlier paper pointing out problems with the IHME forecasts. The IHME model has lots of problems, and I don’t think it’s right to take its flaws as representative of more serious statistical models for epidemic progression. It’s more fair to take this as a flaw of the news media for sometimes presenting IHME forecasts uncritically.

One thing that bothers me about the Ioannidis et al. article is that it does not at all address the previous statistical failures of Ioannidis’s work in this area.

1. In their above-linked 11 June article, Ioannidis et al. write:

Despite these obvious failures, epidemic forecasting continued to thrive, perhaps because vastly erroneous predictions typically lacked serious consequences. Actually, erroneous predictions may have been even useful. . . .

But just two months earlier, on 9 Apr, Ioannidis said:

If I were to make an informed estimate based on the limited testing data we have, I would say that covid-19 will result in fewer than 40,000 deaths this season in the USA.

That’s fine—Ioannidis was careful at the time to condition his estimate based on the limitations of available data, and you can learn a lot in two months. Still, 40,000 was an erroneous prediction, and at the very least this error should cause him to reassess his assumptions. And if he’s gonna write about erroneous predictions, he could mention his own.

2. In their above-linked 11 June article, Ioannidis et al. list the following problems with forecasts: “Wrong assumptions in the modeling,” “High sensitivity of estimates,” “Lack of incorporation of epidemiological features,” “Lack of transparency,” “Errors”, “Lack of expertise in crucial disciplines,” “Groupthink and bandwagon effects,” and “Selective reporting.”

All of these problems arose with the much discussed Stanford antibody study. Ioannidis was only the 16th of 17 authors on this study, so I’m not blaming him for the wrong assumptions, lack of transparency, errors, etc., but I don’t think he’s disavowed that paper either. My point is not to use this as a “gotcha” on Ioannidis but rather to say that it’s hard to know what to make of these criticisms given that they all apply to work that he stands by. Maybe his article should’ve been titled, “Our forecasting for COVID-19 has failed,” and he could’ve criticized the errors and lack of expertise in that Stanford study.

I don’t think the Imperial College models (for example, here) are so flawed. They’re not perfect—wrong assumptions and high sensitivity are unavoidable!—but they are transparent and I think they’re a way forward. Full disclosure: I’ve worked with the first author of that paper, and my colleagues and I helped him with some of the modeling.

The sad truth, I’m afraid, is that Taleb is right: point forecasts are close to useless, and distributional forecasts are really hard. We have to try our best and use all available resources.

P.S. Maybe we could get law professor Richard Epstein to weigh in on this one. He’s the real expert here.

224 Comments

  1. zbicyclist says:

    I note that the purpose of epidemic forecasting is often to make your predictions wrong — i.e. as a guide to take actions that will make your “predictions” too high.

    Any predictions for COVID-19 are likely to depend on how well we control superspreading occasions, since these seem to be much more dangerous than, say, taking a bike ride.

    • Ryan says:

      Absolutely. In this case with such a strong feedback loop between policy decisions and the outcome of the forecast, I think that forecasts need to carry a tag to describe them as either “forecasts conditional on no policy change” or “forecasts anticipating actual policy changes”. And changes in human behaviour in the voluntary adoption of social distancing measures as the pandemic progresses fits somewhere between those two categories as well.

      There’s been a long list of critics arguing that the early pandemic forecasts in many countries were horribly wrong because some countries (like Australia or NZ) didn’t experience terrible outbreaks despite early forecasts saying they would. But this criticism is rather like saying that your doctor advised you to wear a seatbelt or you might die in a car crash – but then saying that since you wore your seatbelt and didn’t crash the doctor’s advice was worthless. That’s clearly a false statement, and doesn’t invalidate the policy recommendation, though clearly it makes testing the accuracy of the forecast after the fact near impossible because we can’t observe a “no policy change” actual outcome.

  2. A similar point is made here (in that prediction intervals rely on distributional forecasts):
    https://twitter.com/josephnwalker/status/1256767436512129026

  3. Don says:

    Both criticisms still miss another important point. Exponential growth is like chaos: in the long term it is intrinsically unpredictable.

    Nobody seems to be paying attention to this obvious fact (see this preprint
    https://arxiv.org/abs/2004.08842

    • Andrew says:

      Don:

      Yes, good math point. It’s a stochastic differential equation with noisy inputs.

    • Zhou Fang says:

      I do love the pithy abstract on this paper!

      Still I think it’s not quite like chaos in some respects. Specifically, you’d expect to be able to make conditional assertions like “if you don’t social distance cases would increase”. Whereas in a pure chaotic system you wouldn’t even have that monotonicity.

      • Don says:

        Good point. Still, chaotic systems display have bifurcations (kinda like confining until you get R0=1).

        Anyway, I guess that appealing to chaos in that preprint was a matter of analogy.

        By the way they should have used Stan instead of jags :-)

      • confused says:

        You’d think. But in the US at least there seems to be very little correlation between measures in place vs. what the outbreak is doing. The states that look like they’re getting worse right now are mostly neither among the most “open” or the most “closed”.

        Now maybe that’s just because official measures are poor predictors of / ineffective at controlling actual behavior at the individual level.

        • Zhou Fang says:

          That just means there’s a lot of confounding.

          • confused says:

            It might.

            Or it might mean that the dynamics of this disease (highly dependent on super-spreaders, large difference between congregate settings and elsewhere) lead it to behave radically differently in different places that have different behaviors/contact-patterns/living conditions/etc… some of those differences coming from social distancing, but some of them coming from pre-existing differences.

            IE – South Dakota, outside meatpacking plants, might be as “socially distanced” with just banning mass gatherings as a dense urban area with only essential businesses open. This could explain why exponential growth appeared to stop after the initial big outbreak without new measures being imposed, and it now seems to be growing

            There are other states that seem anomalous.

            IE, is it just confounding of a relationship which holds universally, or are pre-existing behaviors/contact-patterns/living conditions/etc different enough across the US that the effects of different levels of measures can’t be extrapolated usefully between dissimilar areas?

            • Anoneuoid says:

              Arent around half the US deaths still in nursing homes? Did we ever find out why nursing homes were ordered by some states but not others to take covid patients?

              Especially in NYC when there were 3 empty hospitals at the time. I doubt any modeller would have included a scenario of sending infected people into nursing homes. Maybe in the future it should be standard to include policies like that.

              • Martha (Smith) says:

                ” I doubt any modeller would have included a scenario of sending infected people into nursing homes. Maybe in the future it should be standard to include policies like that.”

                Good point.

              • Mendel says:

                What *evidence* is there that this is an effect that needs to be modeled?

              • Anoneuoid says:

                What *evidence* is there that this is an effect that needs to be modeled?

                That half the deaths came from nursing homes. And most of the deaths are from states who sent patients into nursing homes. Also, just collective prior knowledge and reason (ie, science).

              • Anoneuoid says:

                Apparently it was NY, NJ, CA, PA, and MI that issued such orders: https://twitter.com/SteveScalise/status/1272661471382700032

                I didn’t find better info, maybe that list is incomplete or inaccurate? For example how long did each state do this for, etc.

                Here is covid deaths according to covidtracking and population according to the R usmap package. You can see that, besides California, all the states are in the top ten in deaths per 100k pop:
                https://pastebin.com/PjBq750F

                I get deaths per 100k pop to be 60 vs 24 in the ones that issued the order vs those that didnt. The total deaths were ~55k for each but the 5 states in question make up only 28% of the population (16% without California, which reported “only” 5 out of those 55k deaths).

    • David Young says:

      Yes Don, It’s a critical point. The range of results for these problems is huge. And it also shows I think that scientific advice in this area should depend more on experience and data about fatality and hospitalization rates than on modeling. In this sense, criticizing the whole modeling enterprise is fully justified.

    • jim says:

      Well we can all be thankful that public officials seem to have given up on forecasting fallen back on talking about how to behave to reduce risk of infection, informed by a more realistic concept of how the virus spreads.

    • M3 says:

      I believe this is exactly Talebs point and why he is using EVT instead of using SIR models. He is actually saying something even more profound, that SIR models even with stochastic (and noisy) inputs will underestimate the fat tail behaviour of pandemics. So your posterior distributions in your paper still underestimate the worst case scenarios.

      “Second, epidemiological models such as the susceptible–infectious–recovered (SIR) differential equations11, sometimes supplemented with simulation experiments12, while useful for scientific discussions for the bulk of the distributions of infections and deaths, or for understanding the dynamics of events after they have happened, should not be used for precautionary risk management, which should focus on maxima and tail exposures instead. It is not rigorous to use naive (but reassuring) statistics, such as the expected average outcome of compartmental models, or one or more point estimates, as a motivation for policies. Owing to the compounding effect of parameter uncertainty, the tail-wags-the-dog effect easily invalidates both point estimates and scenario analyses. However, it is encouraging to note that the impact of parameter uncertainty on the scenarios generated by epidemiological models has recently started to be examined more carefully13.

      Extreme value theory is a natural candidate to handle pandemics. It was developed as a means to cope with maxima14, and it has subsequently evolved to deal with tail risk in a robust way, even with a limited number of observations and their associated uncertainty3. In the Netherlands, for example, EVT has been used to get a handle on the distribution of the maxima—and not the average—of sea levels in order to build dams and dykes high and strong enough for the safety of its citizens2.”
      https://www.nature.com/articles/s41567-020-0921-x

  4. Peter says:

    Sage and the UK government made much of Ferguson’s “reasonable worst -case scenario” and seems to have based policy on that. Who is responsible for that? Do modelled advising on policy have a responsibility to make sure they are not misunderstood? Especially when there maybe conflicts of interest when lockdown can help pharmaceutical business models.

  5. Joseph Candelora says:

    Ioannidis is a hypocrite who seems insistent on getting things wrong. Taleb is arguing against a strawman.

    Worse than Ioannidis’s 40,000 death prediction was this line in the March STAT article that rocketed him to Covid fame:
    “If we assume that case fatality rate among individuals infected by SARS-CoV-2 is 0.3% in the general population — a mid-range guess from my Diamond Princess analysis — and that 1% of the U.S. population gets infected (about 3.3 million people), this would translate to about 10,000 deaths.”
    This suggestion that we might end up with 1% prevalence in an unmitigated epidemic made me think he wasn’t arguing in good faith on this stuff. And that was proven for me in an interview he claimed that Ferguson and the ICL revised their estimated IFR downward significantly after issuing their March 17th paper (didn’t happen) in service of walking back and reducing their prediction of 500,000 UK deaths (also didn’t happen): https://youtu.be/cwPqmLoZA4s?t=2609

    Taleb complains that “many researcher groups and agencies produced single point ‘forecasts’ for the pandemic”. But none of the serious/influential forecasts that I remember were single point forecasts, and he doesn’t deign to actually give us an example of one.

    Not a fan of either paper.

    • Phil says:

      Ionnidis’s “10,000” was itself a point forecast, based on at least three of the things he criticizes: “Wrong assumptions in the modeling,” “High sensitivity of estimates,” “Lack of incorporation of epidemiological features”. And it was probably fairly influential; certainly it got a lot of press coverage. So there’s one example.

      • Joseph Candelora says:

        But it wasn’t a serious forecast.

        • confused says:

          Yeah, I think it was a bad idea to come out with that number, but I don’t think it was meant to be a *prediction* of what would actually occur as much as an illustration of how different assumptions can give very different results, demonstrating how large the uncertainty was (or at least how large Ioannidis thought that it was).

          But the only way I can see 1% infection being even vaguely defensible is if the idea was that there might be *such* a strong dependence on super-spreading events that the disease basically couldn’t spread outside population density levels comparable to European/Chinese cities and cruise ships (IE – not in car-dependent US cities). Which seems pretty extreme.

          • Joshua says:

            confused –

            > but I don’t think it was meant to be a *prediction* of what would actually occur as much as an illustration of how different assumptions can give very different results, demonstrating how large the uncertainty was (or at least how large Ioannidis thought that it was).

            I don’t entirely disagree. I mostly disagree in that I think that Ioannidis has made a series of projections that are all to support his policy advocacy. They’ve all been low (some by a way wide margin and some arguably within a reasonable range if on the low side). But I agree a bit in that it’s better to think of these estimates as conditional projections rather than predictions *if the analysts are clear that their projections are conditional*. I don’t think that Ioannidis has done that well, however.

            Nonetheless, my sense is that don’t extend the same generosity to those that you think made higher projections, that caused in your estimation an “overreaction.”

            • Joshua says:

              For example, when Ioannidis said that he thinks that “there’s no reason to fear,” that wasn’t aimed at stressing uncertainty.

              It was aimed at projecting certainty and minimizing uncertainty.

            • confused says:

              >>I don’t think that Ioannidis has done that well, however.

              >>Nonetheless, my sense is that don’t extend the same generosity to those that you think made higher projections, that caused in your estimation an “overreaction.”

              I probably come across that way, but that’s not my actual position.

              I think the actual Imperial College London model was not terribly unreasonable at the time it was published. The model itself did not really predict 2.2 million deaths in the US. I think the 1.1 million deaths for “limited mitigation” was still too high, but for reasons that probably weren’t clear in mid-March, especially for someone in London (who may not have personal experience of how different contact and travel patterns are in say 75% of the US vs. the UK).

              It’s the way it was presented in the media etc. that I think was utterly irresponsible. The 2.2 million deaths was based on “no action” – on March 16th, a lot of action had already been taken, though much of it was voluntary (businesses going to teleworking) or local (school closures driven by school districts) and thus not captured in the dates of statewide orders.

              It might have been better not to publish the 2.2 million estimate at all, though, since it was based on a scenario that was already known to not be the case.

          • Yes, I agree with you in the main. I did not interpret the 10,000 as a prediction nation-wide. It was based on the Princess Cruise Ship situation, as I remember. Let’s be mindful of the fact that many of these predictions are basically guesses that are being modeled statistized as I like to inject a bit of humor.

            • Carlos Ungil says:

              > I did not interpret the 10,000 as a prediction nation-wide.

              Maybe a it was a non-prediction nation-wide. But when you say that “if we assume that case fatality rate among individuals infected by SARS-CoV-2 is 0.3% in the general population (…) and that 1% of the U.S. population gets infected (about 3.3 million people) this would translate to about 10,000 deaths” definitely you’re definitely calculating a nation-wide something.

              • Hi Carlos,

                I heard John Ioannidis give a larger figure of deaths before I read this 10,000 deaths figure. I did read the article when it came out. But heard a video earlier. I did not record the days. Yes, that seems like a nation-wide something. In the context of John’s entire article, I didn’t interpret the calculation to mean nationwide. I should reconstruct the timeline of his analysis to see why I did not interpret the 10,000 deaths nationwide.

                Wellll…

                Had it not been for the shortages of medical supplies and equipment, we might have had far fewer deaths. I am basing this on the measures that several countries took. Namely South Korea, Taiwan, China, New Zealand, and Germany. Of course, this is now hindsight. Massive testing took place in most of the countries listed. We had shortages I gather 1.3% of the US pop is infected.

                It is noted that we have not had a handle on the numbers of asymptomatic carriers either. A limitation of all efforts to model the IFR it seems.

        • Phil says:

          No True Scotsman…

          • Joseph Candelora says:

            You’re telling me that garbage like Ioannidis’s back-of-the envelope 10,000 point estimate in a STAT article justifies Taleb railing against “many reasearcher groups and agencies”?

            Taleb asserts* that “Both forecasters and their critics are wrong… many reasearcher groups and agencies produced single point ‘forecasts’ for the pandemic…”. And goes on to say “Both forecasters (who missed) and their critics were wrong — and the forecasters would have been wrong even if they got the prediction right.”

            Ioannidis here falls into the “critics” bucket that Taleb is criticizing.

            My question remains — who were the many researcher groups and agencies that produced single point forecasts?

            *Full excerpt from Taleb’s paper:
            “Both forecasters and their critics are wrong: At the onset of the COVID-19 pandemic, many researcher groups and agencies produced single point “forecasts” for the pandemic — most relied on the compartmental SIR model, sometimes supplemented with cellular automata. The prevailing idea is that producing a numerical estimate is how science is done, and how science-informed decision-making ought to be done: bean counters producing precise numbers.

            Well, no. That’s not how “science is done”, at least in this domain, and that’s not how informed decision-making ought to be done. Furthermore, subsequently, many criticized the predictions because these did not play out (no surprise there). This is also wrong. Both forecasters (who missed) and their critics were wrong — and the forecasters would have been wrong even if they got the prediction right.”

            • Phil says:

              Joseph,
              You said “But none of the serious/influential forecasts that I remember were single point forecasts, and [Taleb] doesn’t deign to actually give us an example of one.” So I gave you an example. You don’t like this example, but I think that’s a No True Scotsman issue.

              Ionnidis made an influential point forecast that had at least three of the characteristics he (Ionnidis himself) says are problems with forecasts. I find this kind of amusing — physician, heal thyself!

              • Joseph Candelora says:

                Did you not read Taleb’s paper? And also not read the section of the paper I quoted directly before my comment about serious/influential forecasts?

                Here, I’ll reproduce it again.

                —-
                Taleb complains that “many researcher groups and agencies produced single point ‘forecasts’ for the pandemic”. But none of the serious/influential forecasts that I remember were single point forecasts, and he doesn’t deign to actually give us an example of one.
                —–

                You’re telling me that Ioannidis’s back-of-the-envelope trash inhis Stay article was a serious/influential single point forecast by a researcher groups and/or agency? Come on man.

                That’s not what I was asking for.

        • Zad says:

          Not a serious forecast, eh? Then why write it on a serious medical news publication (STAT News) read by many serious clinicians and medical researchers? Surely someone as smart as Ioannidis would know that writing such statements would result in very serious interpretations by very serious policy makers

          • Joseph Candelora says:

            Of course Ioannidis has been trying to pain the discussion by throwing out garbage numbers like that.

            But an obviously-garbage-on-its-face estimate like that does not rise to the level of a serious/influential forecast by a researcher group or agency.

            And if trash like that _is_ what Taleb was referring to when he coyly asserted “many researcher groups and agencies produced single point ‘forecasts’ for the pandemic”, then his paper is an even worse strawman beating than I thought.

            And it’s even worse malpractice to tar the legitimate researcher groups and agencies doing much more serious work with this blanket condemnation based on crap like that Ioannidis estimate.

  6. Joshua says:

    Not precisely on point to the main thrust of your post, but I would also note from the Ionnaides et al. article:

    > Predictions for hospital and ICU bed requirements were also entirely misinforming. Public leaders trusted models (sometimes even black boxes without disclosed methodology) inferring massively overwhelmed health care capacity (Table 1) [3]. However, eventually very few hospitals were stressed, for a couple of weeks. Most hospitals maintained largely empty wards, waiting for tsunamis that never came. The general population was locked and placed in horror-alert to save the health system from collapsing. Tragically, many health systems faced major adverse consequences, not by COVID-19 cases overload, but for very different reasons. Patients with heart attacks avoided visiting hospitals for care [4], important treatments (e.g. for cancer) were unjustifiably delayed [5], mental health suffered [6]. With damaged operations, many hospitals started losing personnel, reducing capacity to face future crises (e.g. a second wave). With massive new unemployment, more people may lose health insurance. The prospects of starvation and of lack of control for other infectious diseases (like tuberculosis, malaria, and childhood communicable diseases for which vaccination is hindered by the COVID-19 measures) are dire [7,8].

    As seems all to (unfortunately) typical of Ioannidis’ work on the pandemic, this paragraph just flat out ignores important uncertainties. For example, the projections about the pressure on hospitals were made based on existing resources, not based on changes that would take place in the resources available.

    Now it’s certainly reasonable to criticize the forecasters for building into their projections a belief that hospital resources would remain static, but Ioannidis et al.’s criticism isn’t entirely reasonable in the sense that the projections themselves could have been accurate based on the assumption that you made. That consideration doesn’t necessarily make such projections more useful…it’s complicated. But there is a lack of comprehensiveness in Ionannidis et al.’s critique, IMO.

    Similarly:

    > Blindly acting based on extreme value theory alone would be sensible if we lived in the times of the Antonine plague or even in 1890[…] Upon acquiring solid evidence about the epidemiological features of new outbreaks, implausible, exaggerated forecasts [19] should be abandoned. Otherwise, they may cause more harm than the virus itself.

    “Blindly acting” looks awfully strawmanish – not particularly useful, IMO. And “implausible, exaggerated” characterizations can always be made in hindsight, but as a practical matter when confronted with an immediate concern, hindsight isn’t available. If we used the same rhetorical form as they use, we’d say that they are arguing that we could never act to address a crisis because there’s a chance that our projections might be wrong.

    • confused says:

      Maybe it was sensible preparedness vs. over-reaction. But in the western US (and I think everywhere except a few Northeast/Midwest urban areas) the difference between what was predicted about hospital capacity vs. what happened is pretty stark. I definitely think missed cancer screenings and such are a real and significant problem, and were an unnecessary harm in all but about 2-3 states.

      Yes, there are definitely risks from under-reaction as well as over-reaction. But I think it is true that we could set an upper bound on how bad this could plausibly get even back in March, maybe even February once the WHO report from China came out. It was pretty clear this was not going to be the next 1918 Flu even then.

      • Joshua says:

        confused –

        > Maybe it was sensible preparedness vs. over-reaction. But in the western US (and I think everywhere except a few Northeast/Midwest urban areas) the difference between what was predicted about hospital capacity vs. what happened is pretty stark.

        Again, as with Ioannidis et al., this is a bit unfair. The projections were made in a time that was, necessarily, ignorant to some degree of the subsequent changes that would take place – such as shelter in place orders and generally behavioral changes among the public. You go to war with the evidence you have. In hindsight, it may well turn out to be wrong. The ways in which it was wrong should be acknowledged and incorporated into future modeling.

        The alternative to acting with incomplete information is to say, well, we might be wrong about making projections so we should just throw up our hands and do nothing. Other than that, there is no alternative to acting with incomplete information – so unless that’s your argument then you need to not play games with unfair standards.

        And what makes the rhetorical positioning of Iannidis et al. even more troubling is that he frequently, if not necessarily uniformly, treats the uncertainties selectively – such as when he talks about the uncertainties in “died with” verses “died from” categorization but fails to acknowledge the potential undercounting due to deaths at home or at care facilities where no tests are performed.

        Likewise, he frequently talks about economic harm of government mandated shelter in place orders – but ignores the uncertainties of how much economic harm would have been incurred absent such orders as the potential result of far greater deaths, sickness, perhaps panic from deaths and sickness, people not being able to stay home from work and collect unemployment or not get fired, etc.

        > I definitely think missed cancer screenings and such are a real and significant problem, and were an unnecessary harm in all but about 2-3 states.

        Sure. Problems exist. But when you say “unnecessary” you are embedding a counterfactual belief that ignores uncertainties. You actually don’t know what would have been unnecessary had shelter in place orders, and behavioral changes, not manifest.

        > Yes, there are definitely risks from under-reaction as well as over-reaction. But I think it is true that we could set an upper bound on how bad this could plausibly get even back in March, maybe even February once the WHO report from China came out.

        Doing so requires counterfactual thinking: what would have happened had things been different? The problem is that counterfactual thinking, to really be supportable, requires an incredibly high standard of carefully calibrated evidence. We are not able to reach that standard at this point, ever we ever reach that stage.

        So you have to decide in the face of uncertainty. And you have to incorporate your values accordingly. My own values are largely informed by a high prioritization of the welfare of healthcare workers – because they’re on the front line risking their lives for the sake of others. It is their work, along with essential workers, that allow people who are more vulnerable to stay safer. IMO, we should multiply the welfare of each healthcare worker and essential worker into your calculations. You might have other factors that you’d assign more value to – say each cancer diagnosis not made. Those values need to be made explicit – and not dealt with as some kind of putatively unsubjective component of a pure science.

        • confused says:

          >>You actually don’t know what would have been unnecessary had shelter in place orders, and behavioral changes, not manifest.

          Well, not all US states had shelter-in-place orders. There were *some* behavioral changes even in those states, but nothing like what was expected to be necessary in mid-March to prevent unrestrained exponential growth.

          It really does seem fairly clear to me that extrapolating from dense regions like Wuhan and Lombardy produced results that were OK for the Northeast megalopolis (also very dense, high mass-transit, etc.) but not representative at all of the rest of the US.

          >>Likewise, he frequently talks about economic harm of government mandated shelter in place orders – but ignores the uncertainties of how much economic harm would have been incurred absent such orders

          True to a degree, but if the main government message had been “think of this like the 1957/1968 flu pandemics” I find it very hard to believe we would have seen anything like the economic harm we’ve seen. Industries like airlines and tourism, sure, but…

          >>And you have to incorporate your values accordingly. My own values are largely informed by a high prioritization of the welfare of healthcare workers – because they’re on the front line risking their lives for the sake of others.

          See, I would strongly disagree with weighting some lives as “worth more” just because of their job.

          I can see an argument for doing a life-year adjustment, because otherwise you are arguably undervaluing the young (who have maybe 60 years to lose rather than 1, or 5, or 10), IE more actual “life” (though not “lives”) are lost – but even that can get tricky.

          I definitely wouldn’t use any other adjustment. Valuing healthcare workers’ lives more because of their value to society then opens questions of the flip side… what about people who don’t work? Carving out ad-hoc categories is really shaky ground.

          • Joshua says:

            confused –

            > Well, not all US states had shelter-in-place orders. There were *some* behavioral changes even in those states, but nothing like what was expected to be necessary in mid-March to prevent unrestrained exponential growth.

            I don’t know on what basis you’re making that claim. Even in some places where there weren’t SIP orders, schools were closed. People’s mobility decreased markedly. People stopped going to theaters, to bars, etc. We don’t know precisely what changes had what kind of impact, but the kinds of changes that likely had the biggest impact were pretty widespread, at least for a while. The point being, we just don’t have enough info for making highly certain counterfactual arguments as Ioannidis does. Again, his argument is that there is a huge differential and negative impact from SIP orders. In point of fact, we just don’t know that’s the case. Again, if there had been none, many of the same negative effects would likely have occurred, and some would likely have been worse. Imagine the huge impact of employees not feeling safe to go into work, but also not being able to stay home and collect unemployment. What do they do? Stay home and get fired? Same with parents sending their kids to school. Suppose they felt unsafe doing so? Should they stay home from work to take care of their kids and get fired and not be able to collect unemployment? Would that have been worse than the SIP orders? We just don’t know. And it’s sloppy to make an assumption.

            > It really does seem fairly clear to me that extrapolating from dense regions like Wuhan and Lombardy produced results that were OK for the Northeast megalopolis (also very dense, high mass-transit, etc.) but not representative at all of the rest of the US.

            Sure. But that’s a separate issue.

            > True to a degree, but if the main government message had been “think of this like the 1957/1968 flu pandemics” I find it very hard to believe we would have seen anything like the economic harm we’ve seen. Industries like airlines and tourism, sure, but…

            You find it hard to believe. That’s fine. But don’t pretend that you’re making some kind of empirical analysis as Ioannidis seems to me to be doing.

            > See, I would strongly disagree with weighting some lives as “worth more” just because of their job.

            I’m not suggesting that someone’s life is worth more because of their job. Not at all. I’m suggesting that if someone sacrifices in order to enable other people to stay safe, if someone sacrifices in order for other people to be able to buy food, they their welfare (and their ability to make those sacrifices) deserves extra consideration when you’re making policy decisions.

            > I definitely wouldn’t use any other adjustment. Valuing healthcare workers’ lives more because of their value to society then opens questions of the flip side… what about people who don’t work? Carving out ad-hoc categories is really shaky ground.

            See above.

            • confused says:

              >>I don’t know on what basis you’re making that claim. Even in some places where there weren’t SIP orders, schools were closed. People’s mobility decreased markedly. People stopped going to theaters, to bars, etc.

              Sure, but in some states, those changes have largely reversed. Schools wouldn’t be in session in mid-June anyway, in a normal year. Yet cases aren’t resurging in South Dakota and Wisconsin (probably the most “open” states in the US – South Dakota did the least all along, and Wisconsin’s order was totally struck down rather than gradually relaxed as elsewhere).

              Which states are doing badly doesn’t really seem to correlate very strongly with any of the things we’d *expect* to be predictive. It mostly seems to correlate with geography (Northeast and Midwest originally, now Southwest and some of the South), for unclear reasons.

              >>Again, if there had been none, many of the same negative effects would likely have occurred, and some would likely have been worse.

              I can accept “possibly have been worse”, but not “likely have been worse”. I just don’t think history bears that out (in an assumed case where basically no governments took strong actions, probably leading to a rather different media and public response — IE a more restrained “public image of the disease” more like 1957/1968.

              Accurate messaging about risks early on would have helped a lot.

              >>Imagine the huge impact of employees not feeling safe to go into work, but also not being able to stay home and collect unemployment. What do they do? Stay home and get fired? Same with parents sending their kids to school. Suppose they felt unsafe doing so?

              If the public messaging had been clearer on the differential risk with age, and the fact that it’s clearly less dangerous than regular flu for school-age populations, there probably would have been much less fear among parents and the majority of the workforce.

              >>Sure. But that’s a separate issue.

              Not really, I don’t think. The models assumed rapid exponential growth for “the US”, as if the US was one homogeneous population — people were quoting R0 values for “the US” or even for “the disease”, as if it would be comparable in New York City and South Dakota.

              If that underlying assumption is wrong, the models are basically worthless.

              >>I’m not suggesting that someone’s life is worth more because of their job. Not at all. I’m suggesting that if someone sacrifices in order to enable other people to stay safe, if someone sacrifices in order for other people to be able to buy food, they their welfare (and their ability to make those sacrifices) deserves extra consideration when you’re making policy decisions.

              How is that different in practice from valuing their lives more, when it comes to making actual decisions?

      • In my opinion what happened around the US was that stay at home orders worked, and then around the country people pretended that things would have worked out that way even if they hadn’t stayed at home, and all the public health people were full of it.

        So, they started reopening around late May, and of course in the reopening people are still doing social distancing and masks and things, somewhat, so it’s not growing as crazy fast as it was. But basically it’s growing, in many places. Such as:

        https://nbviewer.jupyter.org/github/dlakelan/JuliaDataTutorials/blob/master/COVID-monitoring.ipynb

        AK, AL, AR, AZ, CA, FL, GA, NC, NV, OK, OR, SC, TX, UT, WA

        So having bought ourselves about 3 months of time to prepare for this pandemic by ramping up case contact tracing into the hundreds of thousands of workers, and manufacture N95 masks at a couple million a day for months on end, and do many other things that were called “the dance” by Tomas Pueyo on his second medium.com post (see Mendel’s reference below)… Instead we did none of it, we’re pretending everything is all done and there was no problem, and most people are going to get caught off guard when around July 4th the extent of the insufficiently mitigated proper first wave becomes very obvious in their region

        Sure, it won’t look like Lombardy throughout every county of the US… but that doesn’t matter, because we’ll have some large metropolitan regions where we will have at least some significant hospital overload. The growth rate is now closer to doubling every 10-15 days instead of every 3. But at this rate for example South Carolina will have 2000 cases a day by July 1 in mid July they’ll have death rates in the hundreds per day.

        Of course it’s always possible the virus has mutated, or that summertime makes the virus less virulent due to say vitamin D or temperature effects or whatever. But looking at the obvious exponential growth in SC it’s hard to understand how they’ll be anything but in big trouble by early July. Similarly in NC, and CA, and NV where the casinos are all operating …

        • Chris Wilson says:

          +1 to everything Daniel said.
          Unfortunately, I don’t see the political will being mustered to do anything like the shutdowns again until there is obvious catastrophe playing out. And even there, it will have to be at the local level. And even there, look at Montgomery Alabama as an example of a city failing to pass a mask ordinance that their doctors are pleading for. Of course, the vote broke down on racial lines and 80% of the Covid patients there are African American. Sigh.
          The other thing to note is that CARES act unemployment benefits run out end of July. Moratoriums on evictions and foreclosures are also set to start phasing out. So, lots of economic pain and uncertainty to throw into the mix! Fun times :)

          • Martha (Smith) says:

            Yes. And then there’s what was discussed in the news this morning: That people in the upper income brackets have been cutting their spending, which has reduced employment/ income for the people in the lower income brackets. We need to do something about this — like lowering income taxes for the lower income brackets, while raising income taxes for the upper brackets to more than make up for the reduced taxes from the lower brackets, and allocating the extra revenue to essential services (including health care) for the lower income brackets. (But I’m skeptical that there is the political will to do this more than a token amount.)

            • Martha, your solution might seem logical, but I would argue that it would continue the practice of complex and gameable taxation which has led to enormous problems that are generally unacknowledged. For example, when poor people are able to afford to live on their low wages because they are taxed very little and have subsidized housing and such… and then they work hard to make a little more money… the *marginal* taxes on those extra dollars can be enormous… 50% for example. Combined with loss of qualification for services/subsidies, it can easily become a net negative, even DRAMATICALLY negative outcome to earn more. This traps people in poverty. I’m sure that’s an outcome you wouldn’t want.

              A reasonable alternative which is also incredibly simple: Eliminate income and payroll taxes as they exist today in favor of a flat tax of about 30% on income and a universal basic income of about 10% of GDP/capita. Even those who are not working would take in $6000/yr per person, and every dollar you earn, you take home 70% of it, regardless of how much you earn. Everyone could easily determine whether some new job opportunity was a good idea for them etc. way WAY better.

              • Martha (Smith) says:

                Daniel,
                Your example seems to be supposing that what I was proposing would be some kind of permanent change; I was thinking of something appropriate for the current situation, which would be modified as the situation changes. (Although perhaps the idea of modification as the situation changes might be pie-in-the-sky.)

              • Even for “temporary” situations, the perverse incentives can be very real. There’s a big issue right now with unemployment insurance providing more income than the work people are unemployed from. This means if they go back to work they lose income.

                A UBI and flat tax would ensure that *whatever* your situation, you have some reasonable level of income, enough to keep you from being utterly homeless and starving… while also ensuring that whenever you provide products or services your take home income always increases.

              • Chris Wilson says:

                I’m going to channel my inner Georgist and say we should go after wealth and unimproved land value taxes first :) A flat 2% wealth tax on all American households funds your UBI for instance. We could be even slicker and exempt the first million, which still leaves ~70% of the tax base (IIRC), and go a bit higher, say 3%. Arguably, a higher 5-6% bracket on wealth over say 50 million wouldn’t be hard to justify.
                Unimproved land value taxes remain the perfect swish. Economically efficient. No perverse disincentives anywhere. This is basically socializing a portion of value that is absolutely 100% socially created by anyone’s definition.
                Basically, I think we should soak the rent-seeking class and get real about the implications of non-ergodic, multiplicative accumulation of wealth (https://ergodicityeconomics.com/2017/08/14/wealth-redistribution-and-interest-rates/)
                If we get creative, I think we could fund our entire government at all scales, and a robust social safety net, with nothing but wealth tax, unimproved land value taxes, a modest VAT, and some assorted taxes on a couple key resources (carbon tax) and any resource rents.

              • Martha (Smith) says:

                Chris said,
                “I’m going to channel my inner Georgist and say we should go after wealth and unimproved land value taxes first”

                I’ve got a problem with the. concept of “unimproved land value tax” — namely, that what the real estate system calls “unimproved land” is actually land that is serving a purpose — in particular, providing ecosystem services.

              • Anoneuoid says:

                Unimproved land value taxes remain the perfect swish. Economically efficient. No perverse disincentives anywhere

                Like Martha said (I think), wont this incentivize people building unneccesary structures on that land?

              • Mendel says:

                Wikipedia:

                A land value tax or location value tax (LVT), also called a site valuation tax, split rate tax, or site-value rating, is an ad valorem levy on the unimproved value of land. Unlike property taxes, it disregards the value of buildings, personal property and other improvements to real estate.[1] A land value tax is generally favored by economists as (unlike other taxes) it does not cause economic inefficiency, and it tends to reduce inequality.[2]

                Building a structure on land does not change its valuation for land tax purposes; the tax is levied on the value of the land that it would have if it was not improved, usually by multiplying its size with a tax value for the general location.

                This tax burdens land owners, and has the potential to hit farmers disproportionally hard.

              • Chris Wilson says:

                No, farmers would get the usual agricultural exemptions, etc. A properly implemented land valuation tax would hit landowners who hang onto urban land and extract rent…which in turn is a big part of why small/mid-size cities often have downtowns that struggle. The valuation per sq ft of land increases exponentially with density and location. As noted, no one is penalized for making improvements – in fact, improvements are encouraged under this kind of scheme!

              • Anoneuoid says:

                farmers would get the usual agricultural exemptions, etc.

                So there is incentive to qualify as “farmer” in the eyes of the government. Actually, did you know that depending where they are cows, chicken, and rats are not considered “animals” by the federal government? Instead it’s livestock, poultry, and research organism.

                As noted, no one is penalized for making improvements – in fact, improvements are encouraged under this kind of scheme!

                Can’t you see this is a perverse incentive?

              • Chris Wilson says:

                Anoneuoid et al,

                I think this topic requires more careful analysis and discussion, and I may have contributed to over-simplifying. Yes, I am generally familiar with ag policies – I work in an Agronomy department and have been a farmer myself :) It is pretty easy already to qualify for ag exemptions, but this is regulated at county level – which it probably should be.

                So, what I should have said is, this kind of tax is designed to *disincentive* deadweight. The extent of this disincentive is directly proportional to desirability of location so will be much higher in urban areas than rural areas. You are not *incentivized* to make improvements that *don’t generate value*. And you are only incentivized to make improvements in direct proportion to their value. So, if you have a house, and you make improvements to it, your tax bill would not go up. Was it worth it? That’s entirely up to you. The tax is neutral with respect to this kind of thing. If you have a business, and you make improvements, and business improves you *WIN* under this kind of taxation, rather than incur additional tax liabilities.

                To be super clear, the goal of this kind of taxation is to tax purely speculative land holding and unearned rents – accrued due to the general improvements surrounding land and thus not one’s own industry in any sense. The holders of land are actually incentivized directly or indirectly to either be getting personal value or creating social value from it, or renting it to others who are creating value – precisely so they can pay their taxes! This actually means that incentives of landholders and their business-running tenants come into closer alignment AFAICT.

                I admit, this is a counter-intuitive proposal. We are so ingrained to think of labor and the fruits of labor as the natural target of taxation, but this is quite odd if you pull back for a second.

                All that said, I am not convinced 100% by classical Georgism. I don’t think we should make this kind of taxation do all the lifting, hence wealth + VAT also make sense to me. I am not opposed to income tax overall, but I think we have allowed the goalposts to shift way too much in favor of unearned wealth accumulation and rent-seeking and our society basically cannot get on a better path until we break up this unholy alliance of monopolists, rent-seekers and politicians.

              • Anoneuoid says:

                I’m just saying that all taxes lead to some form of perverse incentive. What is wrong with conserving your land? It is basically saving resources for a rainy day.

        • dhogaza says:

          “In my opinion what happened around the US was that stay at home orders worked, and then around the country people pretended that things would have worked out that way even if they hadn’t stayed at home, and all the public health people were full of it.”

          Well, that’s how I read this statement by Ioannidis:

          “Predictions for hospital and ICU bed requirements were also entirely misinforming. Public leaders trusted models (sometimes even black boxes without disclosed methodology) inferring massively overwhelmed health care capacity (Table 1) [3]. However, eventually very few hospitals were stressed, for a couple of weeks. Most hospitals maintained largely empty wards, waiting for tsunamis that never came. The general population was locked and placed in horror-alert to save the health system from collapsing. “

          He could be saying that hey, the lockdown measures worked, hooray! But … no, that’s not where he’s coming from. He’s essentially arguing that it wasn’t necessary and that things would’ve worked out the same if we’d done nothing, as you put it.

          Then we move to sheer hyperbole:

          “The prospects of starvation and of lack of control for other infectious diseases (like tuberculosis, malaria, and childhood communicable diseases for which vaccination is hindered by the COVID-19 measures) are dire”

          I believe he’s still talking about the US here. I’ve been hearing people complain about gaining wait during lockdown … and no one complaining about starving.

          • dhogaza says:

            Gaining weight … though there’s been a lot of waiting, too :)

          • confused says:

            The starvation / communicable diseases bit is I think more about how the shutdown of the US economy affects the world economy and thus poverty in less developed countries. Not starvation in the US itself. (I think the mention of malaria is key there…)

            • Stephen Cooper says:

              confused – it is kind of you to bother.

              Instruction of the ignorant is a virtue.

              For those people reading this who are not trying to score infighting statistician points, please reflect on this.

              The effects of individual human beings – Cuomo and his kulakization of the low-income elderly in nursing homes being a good example – had an effect on human misery much larger, over the course of the three months, than any measured or measurable effect caused by the unconscious, amoral, and unpredictable nature of our little fellow creature (some say organic, some say not quite organic) whose closest recognizable forebear was born somewhere last year somewhere in China.

              When I was young, people like the people super-invested in the types of arguments foregrounded here were called eggheads – the fragility of their ego, the instability of their passionate search for the truth and the limited goals they set themselves were the reason for that name.

              • Stephen Cooper says:

                By the way, Andrew G., if you are reading this, I was not including you among the eggheads.

                Not that you should care one way or the other. I disagree with you on a lot – not that you should care – but I have no desire to insult you in any way on your own website. You seem like an honest person who does not get super-invested in your own stances.

          • I am not sure whether John is referring to the US exclusively. But I do think that if it were not for that ‘10,000’ figure John’s analysis more broadly is more correct than the other analyses that came out. Specifically, on the basis of his analysis of Italy’s experience, the disease had a disproportionate impact on the elderly with and without co-morbidities. The real problem was that we lagged in massive testing, which is a theme over which there is considerable agreement. We do need better numbers of asymptomatic and pre-symptomatic carriers. That requires massive testing. Oxford noted that 5% to 80% of carriers were asymptomatic or pre-symptomatic. Huh? Okkkkkk.

            • Carlos Ungil says:

              > John’s analysis more broadly is more correct than the other analyses that came out. Specifically, on the basis of his analysis of Italy’s experience, the disease had a disproportionate impact on the elderly with and without co-morbidities.

              Do you think that nobody had noticed that? The following was published more than four months ago, before there was even a single death in Italy (only three cases had been detected in at that point in Rome and there had been only three deaths reported outside of China/Taiwan):

              “The ≥80 age group had the highest case fatality rate of all age groups at 14.8%. Case fatality rate for males was 2.8% and for females was 1.7%. By occupation, patients who reported being retirees had the highest case fatality rate at 5.1%, and patients in Hubei Province had a >7-fold higher case fatality rate at 2.9% compared to patients in other provinces (0.4%). While patients who reported no comorbid conditions had a case fatality rate of 0.9%, patients with comorbid conditions had much higher rates—10.5% for those with cardiovascular disease, 7.3% for diabetes, 6.3% for chronic respiratory disease, 6.0% for hypertension, and 5.6% for cancer.”

            • Joseph Candelora says:

              Ignoring the parts that Ioannidis got wrong, he was roughly on par with the analyses of others, not better. This is not an impressive outcome.

              The age profile of the disease had been studied, published and was in heavy use before Ioannidis published anything.

              Not sure what your Oxford non-sequitur is about.

              • Hi Carlos,

                Thanks for responding. I was referring back to this article co-authored by Johh, which I should have posted also. Apologies.

                During the COVID-19 Pandemic

                https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2764369

                I am not sure that all that many here in the States knew much of anything until February. My own speculation has been that the virus was here in the US perhaps as early as mid-December. Again, just speculation > a hunch. We tend to entertain hunches in these situations. I think there is pressure to provide forecasts and analysis. So I don’t fault experts on that score. We seem to have been ill-prepared on many fronts. Particularly in New York.

                I have followed nearly every expert weighing in on the COVID-19 situation. What many of us non-statisticians/consumers of statistics have noted is that it is exceedingly difficult to discern the statistical assumptions made, notwithstanding the fact that a good deal of analyses is quite technical and obscurely quantitative.

                There are a few highly competent wordsmiths in my observation. And journalistic accounts, even the most accurate and eloquent, do not get the acclaim that they deserve.

                think that ultimately citizens have to be able to evaluate the merit and demerits of the analyses.

                It’s worth reading Frank Von Hippel’s book on the role of the Citizen-Scientist and the extent to which such a role is hindered by scientific advisory boards/commissions who largely provide technical advice pertaining to public policy to the Executive branches of the Federal and State Governments. Thus marginalizing the necessity for citizens to weigh-in. They bear the costs and consequences of opaque data/research methods, and the commercialization of public health guidance.

                Oh that Oxford study? Here ya go.
                https://www.cebm.net/covid-19/covid-19-what-proportion-are-asymptomatic/

              • Joseph Candelora says:

                @Sameera

                You’ve linked to a study published April 7 recurving data through April 1.

                Verity et al posted the pre-print containing their age-stratification estimates on March 9: https://www.medrxiv.org/content/10.1101/2020.03.09.20033357v1.

                Those mortality estimates are what Imperial College London used in their March 16 study.

                All of this cwoek was well completed before Ioannidis published his Stat article on March 17, or his paper in early April.

                What is the relevance of the Oxford article you linked?

            • Joshua says:

              > Ignoring the parts that Ioannidis got wrong, he was roughly on par with the analyses of others, not better.

              Heh. In other words, “Other than that, how was the play Mrs. Lincoln?”

              Or

              The operation was a success but the patient died.

              • Well, everyone makes mistakes, right? It should be urged that before we lambast someone for his mistakes that we list the ones we have made. It would amount to a penance maybe. But that is asking too much.

              • Joshua says:

                Sameera –

                > Well, everyone makes mistakes, right? It should be urged that before we lambast someone for his mistakes that we list the ones we have made. It would amount to a penance maybe. But that is asking too much.

                I completely agree with you! And I applaud you for reminding people of such. It is a hugely important thing to keep in mind. I was writing as much to you in another comment.

                It isn’t the mistakes for which I criticize Ioannidis, but other aspects of what he’s been presenting.

                My point is that that same approach should be extended to everyone attempting to analyze the evidence, and that Ioannidis should do that as well. One of my primary criticisms of Ioannidis is exactly that he hasn’t done that. For example, he called the Imperial College projections “science fiction.”

                (I’m also critical of certain aspects of his methodology, and because I feel that he’s engaging in a political process while portraying himself as merely engaged in science even as he throws shade at others doing science. If you engage in a political policy discussion, just be straight forward that’s what you’re doing).

                And to be clear, I am not defending personal attacks against him. I consider those to be counterproductive.

              • confused says:

                I’m not defending Ioannidis’ approach in general – it definitely seems “one-sided”.

                But was the “science fiction” comment in regard to the ICL model itself, or the 2.2 million deaths number that was being quoted widely in the media? It’s arguably an accurate description of the latter, since that number was contingent on a situation (absolutely no mitigation action, even on an individual or local level) which was already known to be untrue on March 16.

              • Joshua says:

                confused –

                > But was the “science fiction” comment in regard to the ICL model itself, or the 2.2 million deaths number that was being quoted widely in the media? It’s arguably an accurate description of the latter, since that number was contingent on a situation (absolutely no mitigation action, even on an individual or local level) which was already known to be untrue on March 16.

                He and Katz giggled about how the “predictions” of 2 million were “science fiction.” And exactly in one with Sameera’s comment that bugged me because it want a prediction, it was a conditional projection.

                They completely ignored that aspect.

                That is exactly the kind of rhetorical posting that I find disappointing, and unscientific, and counterproductive towards the goal of reasoned exchange of view that I assume both aspire to achieve.

              • Joshua says:

                confused –

                > It’s arguably an accurate description of the latter, since that number was contingent on a situation (absolutely no mitigation action, even on an individual or local level) which was already known to be untrue on March 16

                I think this is problematic also.

                I see this happening with climate science all the time. People take high range projections thst have confidence intervals attached, act as if they were predictions of what would happen with no uncertainty, and then complain that the “predictions” were wrong and therefore the whole scientific endeavor is fatally flawed and biased.

                There is nothing wrong with making a worst case projection to inform the cost/benefit analysis – as long as it is made clear that it is a low probability/worst case scenario. If that is not communicated properly, than sure that communication effort deserves criticism. But so does it deserve criticism when people leverage rhetorical spin and conflate conditional projections with “predictions” of what’s going to happen so they can discredit modelers for producing models that are wrong but useful.

              • confused says:

                My point is that the 2.2 million wasn’t a high range (involving various forms of uncertainty) projection from an existing starting point, it was a projection from a counterfactual starting point. It wasn’t posed as “this is a low-probability outcome”, it was posed as “this is the likely outcome if X is done” but we already were not doing X.

                This is a criticism of PR/reporting, not of the model itself.

              • Mendel says:

                The nonpharmaceutical interventions do not change the 2.2 million number, isn’t that Taleb’s point?
                If we assume 1% lethality and 70% infected at least once until we achieve population immunity, then we’re going to have 2.2 million deaths in the US no matter what (but see below), the nonpharmaceutical interventions just shift them back into the tail instead of having them all in 3 months.

                From the situation we’re in, back to the normal we had before, the only way is across these deaths unless
                a) we deploy a vaccine, or
                b) we develop a cure (or simply get better at treatment), or
                c) we succeed in shifting infections away from the vulnerable people, or
                d) we contain the virus like New Zealand, but globally.

                If you find this number too big and want to do some psychological manipulation, you come up with unrealistic assumptions to field a number that is way too low, and you’re anchoring the perception elsewhere where people don’t take this as seriously as they should.

              • Joshua says:

                Mendel –

                > the nonpharmaceutical interventions just shift them back into the tail instead of having them all in 3 months.

                I think that there’s an argument to be made that the degree of fatality is, to some extent at least, a function of how much pressure there is on the medical system. Likewise, better tracing and isolation can, at least potentially, affect the fatality rate by virtue of something like reducing viral load – even if theoretically, ultimately the same number of people would be infected.

                The same works for the economic costs. There is a complex interaction between the rate of spread and the degree of impact. A faster spread that leads to a higher fatality rate might be less costly because it could lead to a shorter term of impact. Or it could be more costly because the depth of the impact would be more dramatic, and it might not diminish rapidly enough, relatively, to equalize the greater costs of a more intense spread.

              • Joseph Candelora says:

                Counterfactual is a bit strong. The results were made available on March 16, but certainly they had been working on it for weeks. And the steps taken we’re all temporary anyway — e.g. HISD just gave the week before spring break off, with the stated intention of reopening after spring break.

                So the unmitigated epidemic is no more a counterfactual than a scenario where people practice 100% compliance with distancing. They give a range of potential outcomes. These were illustrations where individual behavior is the independent variable; you obviously confound things by feeding in your personal opinions of how people will behave.

                And bear in mind that people like Ioannidis himself were then (and are still) making the argument that Covid would simply be noise against the backdrop of the flu season, so assuming that people would do anything differently than they do for the flu (i.e. nothing) was not justified for the unmitigated scenario.

              • confused says:

                >>Counterfactual is a bit strong. The results were made available on March 16, but certainly they had been working on it for weeks.

                Sure; again, the model itself is not the problem, the PR/media reporting is. That scenario may have been realistic when the model was run, but not when the results were published and the

                >>And the steps taken we’re all temporary anyway — e.g. HISD just gave the week before spring break off, with the stated intention of reopening after spring break.

                School closures in TX were originally very short-term, but not necessarily the businesses going to telework etc. And I think the latter is probably more significant for COVID.

                >>you obviously confound things by feeding in your personal opinions of how people will behave.

                I’d say that’s a prior, not a confounder ;)

                It’s not wrong for the modelers to show a “no action at all” result as one extreme of the possibilities.

                It is wrong for media etc. to report this as a prediction that “2.2 million Americans will die if we don’t lock down”.

                >> assuming that people would do anything differently than they do for the flu (i.e. nothing) was not justified for the unmitigated scenario.

                I disagree, because if the scenario was going to 2.2 million deaths, it would become obvious much earlier than that that what was actually happening was worse than flu, and behavior would change.

        • confused says:

          I think some of those states are not actually growing, just finding cases more effectively.

          And even if they all are, what about South Dakota and Wisconsin, which are more “open” than any of those?

          I really doubt we will have overloaded hospitals in multiple metro areas in the US in a month from now. I guess we’ll see…

        • Joseph Candelora says:

          I’ve looked at new hospitalization counts in a few of those places.

          In Florida, it’s essentially level since late March. I don’t see any reason to be concerned about unconstrained growth there.

          In Arizona, it’s been growing exponentially since early May, but with a doubling time of about 4 weeks.

          I couldn’t find data for all of Texas, but in the Tx Medical Center (major hospital system in Houston), it’s been growing exponentially since mid/late May, but with a doubling time of 3 weeks.

          Contrast that with NY in late March, which had a doubling time of 3 or 4 days.

          So those states (not FL so much) have something they need to watch out for, but it’s nothing like when it blew up in Italy or NYC. In the growth phases there, if you wanted to take an action that wouldn’t have effect for about 2 weeks, you’d be faced with a full order of magnitude more cases before you change took effect. In TX/AZ, you’re looking at less than double the case count under same circumstances.

          My view is that our distancing measures have been sticky enough that they’re still having significant effect. I mean, my company is still work at home countrywide.

          There’s definitely a possibility that even further opening could cause transmission rates to spoke to the point that things could totally get away from us, but given the testing capacity in place and overall monitoring resources, I think we’ll be able to push the limit of resource exhaustion and then choose at that point whether we want to lock down and cool it off or if we’re just going to accept the overload.

          • confused says:

            >>My view is that our distancing measures have been sticky enough that they’re still having significant effect. I mean, my company is still work at home countrywide.

            Definitely this. My workplace is also work at home, in TX, which is relatively open (though not as much as South Dakota or Wisconsin).

            I also think it matters that most of the US is lower population-density and lower mass-transit use than NYC or European cities. (And American demographics skew younger than European ones, as well – and some of those apparently growing states are among the youngest. UT is the youngest state by a large margin, but AK and TX are also very low.)

        • Hi Daniel or anyone else here:

          It is astounding to me that the Oxford study estimates that there are between 5 to 80% asymptomatics. What kind of help is that?

        • confused says:

          >>But at this rate for example South Carolina will have 2000 cases a day by July 1 in mid July they’ll have death rates in the hundreds per day.

          That doesn’t seem consistent.

          If IFR is 1%, hundreds of deaths (say 200-500) in mid-July would imply tens of thousands of infections (say 20,000-50,000) in late June.

          I don’t think testing *now* is so bad that we’d only measure 2000 cases per day if there are 20,000-50,000 infections per day. In April we were probably detecting less than 1/10 of infections, but now?

          Also, IFR for infections happening in late June is probably not 1%, since supportive care for COVID is understood better, and there is some evidence that the age profile of people being infected may be skewing younger.

          2000 cases/day seems plausible, but I really don’t expect a state the size and density of South Carolina to have 200+ deaths/day.

      • Mendel says:

        confused wrote: “I definitely think missed cancer screenings and such are a real and significant problem, and were an unnecessary harm in all but about 2-3 states.”

        People miss cancer screening because they are afraid to visit the doctor. They are afraid to visit the doctor because they think they might get infected there. Interventions that reduce the risk to get infected (that “empty” the hospitals) and efforts to communicate the actual risk (compared to the rrisk of not detecting a detectable tumor) help here. Combating the virus 2aggressively” should lead to more patients getting screened.

        Hospital capacity being reserved for Covid-19 is mostly ICU beds (up to 30% depending on location), and this does not affect cancer screening or radiation therapy, since these don’t require hospital beds.

        Reference 5 in the paper states, “This represents on average 18·1 LYG per patient, which markedly reduces to 17·2 with three months delay and to 15·9 with six months delay.” And that is across all cancer types; if surgeons manage operations by forecast outcome, the number is even lower. The German surgeon Markus Albertsmeier said in an interview: „Die deutschen Kapazitäten werden allgemein so eingeschätzt, dass keine dringende Krebsoperation hinausgezögert werden muss.“ The German capacities are estimated such that no urgent cancer operation needs to be postponed.
        Attacking the virus aggressively, bringing hospitalisation numbers down quickly, should shorten delays for cancer patients and save their lives.
        The reference also shows that not treating a Covid-19 patients leads to more loss of LYG than delaying a cancer operation.

        It seems prudent to not consider health effects one-dimensionally: just because there is a possible negative effect with the current strategy, that doesn’t mean that another strategy fares better. Comparing the current state of affairs to “no epidemic at all” is living in denial.

        • confused says:

          >>Combating the virus 2aggressively” should lead to more patients getting screened.

          Only if people’s fear of the virus is correlated with their actual risk from the virus. I think it’s often not — most people are either overly unafraid or overly afraid.

          What I was talking about is that stronger measures may convey an impression that the crisis is worse than weaker measures do – thus increasing fear despite reducing actual risk.

          >>Hospital capacity being reserved for Covid-19 is mostly ICU beds (up to 30% depending on location), and this does not affect cancer screening or radiation therapy, since these don’t require hospital beds.

          Now, yes. Two months ago, all ‘non-essential’ procedures (including things like this that are critical but less urgent) were not performed in much (most?) of the US. That’s what I was complaining about — not the measures in place at most hospitals now.

          The comparison IMO should be (risk of getting COVID x risk of dying assuming getting COVID) vs (increased risk of death from not getting screened).

          And the people who get the best benefit from cancer screening (if you have only a couple years of life expectancy left anyway, catching cancer early doesn’t help you much) don’t have *that* high of a risk of dying of COVID.

          • Joshua says:

            confused –

            > What I was talking about is that stronger measures may convey an impression that the crisis is worse than weaker measures do – thus increasing fear despite reducing actual risk.

            This is very speculative. It’s ok to speculate. But I think that you’re wrong. I think that there would likely have been a worse effect, and a longer effect, if there had been a higher amount of spread, more people dying, more overloaded medical facilities, less access to PPE, etc.

            Certainly, these effects would have been regionally variable – but we don’t know how people might have traveled if had really gotten totally out of hand in a place like NY.

            The point I’m making isn’t that I know, but that these issues need to be taken into consdieration when someone goes on to make counterfactual arguments about how much economic pain has been caused, differentially, by SIP orders as opposed the potential havoc that might have occurred without them. If you want to make an argument that the SIP orders cause harm differentially, then you need to lay out the argument for why things wouldn’t have been worse, economically, without them. You can’t just say that shutting things down caused harm and then say that therefore, the harm is differentially worse because of them.

            • confused says:

              Yes, it’s definitely uncertain.

              Here’s why I think the economic effects of a less controlled epidemic would have been less severe than they might at-first-glance seem:

              -mortality rate of COVID is much lower than that of 1918 H1N1 pandemic*, so large-scale social breakdown and economic collapses are probably unlikely, since they didn’t occur in the far worse 1918 pandemic.

              -“direct” losses to the economy from sickness will be limited by the fact that the vast majority of deaths are outside the workforce (due to the usual US retirement age), and a lockdown means people aren’t working for a couple of months anyway. A third or half of a company’s employees being off work sick for a month vs. 100% of them being off work for lockdown.

              And a third or half being off work sick for a month is probably a high end estimate.

              -“indirect” losses (due to people’s fear of activities) in my hypothetical will be limited by the fact that the effects would still be very regional. People don’t have a strong “gut feeling” for large numbers, e.g. 120,000 deaths vs. 300,000 deaths. If you don’t know someone personally, it’s not entirely real to you.

              And an uncontained epidemic might end rather quickly (due to herd immunity) and the economy could then return basically to normal.

              *1918 H1N1 killed something like 2.5% of the *entire world population*. ~50 million deaths out of ~1.8 billion. IFR probably something like 8-10% if 1/3 of world population was infected. COVID IFR is probably ~1% at least in places like New York, Spain, etc. Seems to be lower in some places due to what age-groups are infected.)

              I am not at all saying this would be ethically acceptable. But I think history shows (18th and 19th century yellow fever epidemics; 1918, 1957, 1968 flus) that epidemic and pandemic diseases (short of something like New World smallpox or the Black Death) don’t have nearly the economic, social, and political effects that would be expected given the number of deaths. People I know who were around in 1957 and 1968 don’t even remember that pandemics happened then.

              • Andrew says:

                Confused:

                I’ve been thinking about this too, and one thing that puzzles me is why more places didn’t just say, What the hell, we’ll just live our life like normal. It can’t all be blamed on Neil Ferguson or whatever. It does seem that people will put in some effort to avoid spreading the disease.

              • Joshua says:

                confused –

                > -“direct” losses to the economy from sickness will be limited by the fact that the vast majority of deaths are outside the workforce (due to the usual US retirement age),

                But what you’re describing is what took place during a lockdown where most people weren’t going to work. The loses would have been huge if infections were spreading oh necked I workplaces. People would have not wanted to go to work. Does an employer fire them? Then they can’t collect UE.

                > and a lockdown means people aren’t working for a couple of months anyway.

                Maybe the workplace would have been massively disrupted for a longer period with a higher rate of spread.

                Keep in mind, deaths is out the only relevant issue. Sickness is also important. And people getting infected in a highly infectious workplace then spread the virus to family members back home.

                > A third or half of a company’s employees being off work sick for a month vs. 100% of them being off work for lockdown.

                This is a purely hypothetical scenario that seems extremely unlikely to me. In the very least it is the best case scenario at one end of a spectrum of possibilities. It looks to me like you’re reverse engineering from the conclusion you want to draw.

                I think thar cross-country comparisons are fraught, but the economy in Sweden does not show a massive advantage to “locked down” economies in similar countries. And then you need to factor in the “costs” of more deaths. And if a vaccine is developed and widely distributed in a fairly short time frame, I’d say that there would be significantly greater economic cost Sweden, differentially – but the point is its too early to tell. There is a point at which perhaps the economic advantages of a faster spread equalize with those of a slower spread. Much could depend on where you are w/r/t that equilibrium point.

                > And a third or half being off work sick for a month is probably a high end estimate.

                ? If there is widespread sickness and death I’d say thst is a massive underestimate of what rhe work hours lost would look like.

                > And an uncontained epidemic might end rather quickly (due to herd immunity) and the economy could then return basically to normal.

                Purely hypothetical. It might take a long time. Again, awden might be instructive, and there is much I Sweden to make a no “lockdown” approach more viable than here

                > I am not at all saying this would be ethically acceptable.

                I’m not brining ethics up as an issue.

                I think that your are creating a false dividing like between the economic outcomes of the two conditions.

                > But I think history shows (18th and 19th century yellow fever epidemics; 1918, 1957, 1968 flus) that epidemic and pandemic diseases (short of something like New World smallpox or the Black Death) don’t have nearly the economic, social, and political effects that would be expected given the number of deaths. People I know who were around in 1957 and 1968 don’t even remember that pandemics happened then

                I think that trying to use history as a way to work through the variety of uncertainties on the different side so the line only add to the problem. Our society is vastly different than 100 years ago. Our economics are vastly different. Out age stratification is vastly different (we’re quite a bit older). It’s a useful exercise, but I think that a tone of work would have to be done before it really becomes instructive.

                Counterfactual arguments are really, really hard, and they require a very high quality of fats to be valid, IMO.

              • Joshua says:

                confused –

                Imagine trying to run a business with a large % of the the employees sick. Another large percentage wouldn’t want to come to work. Many might be key employees because they’d be older and more senior.

                But the overhead would likely not decrease in proportion to the loss of productivity. So l you’re running your business, and carrying payroll. Even though tons of people are sick or not coming in, and you’re not selling much, but your overhead is high. And you have nothing much to so with your inventory. And you probably can’t get supplies. And morale sucks because your employees don’t want to be there and because you’re not shut they can’t stay home with their kids and collect unemployment. And they’re pissed at you because they have to send their kids to school at the risk of them getting sick or infected (remember that back a bit there was far less verso th about kids getting infected), but giub won’t shit down so that they can stay home.

                Sounds like a nightmare to me. And I imagine that it would be nuts with a lot more people getting infected, and a lot more people getting sick. Differentially worse than shutting down? I don’t know. In Sweden a much larger % of people live in single-person households. They have a functional healthcare system. They have incredible benefits for people on leave. A very high % can work from home. Their population density is relatively concentrated in just one city. And they still took a big economic hit. It’s not clear yet what the economic impact will be relative to other countries.

                Maybe you’re right, but I think you’re taking a lot on faith about the relative economic benefits of not locking down. And that’s the problem I have with Ioannidis’ arguments as well. He doesn’t even make a reasonable attempt at due diligence to obvious potential counter-arguments.

                He just acts like because we know that there are costs associated with “locking down,” the differential economic costs would be greater from “locking down”, as if there there wouldn’t be costs to not “locking down.”. Why do we know that? Becsuee we can’t see them directly like we can see the costs of “locking down?” It’s so obviously, in my opinion, a false choice that’s being presented. And it’s disappointing to me that he’s engaged at such a shallow level in these issues.

                Just as other scientists should consult with statisticians in their research, perhaps so should Ioannidis consult with some economists. His understanding seems to completely lack consideration of basic considerations – akin to externalities or opportunity costs.

                Yes arguing about economic policy without really taking the issue seriously.

              • confused says:

                >>I’ve been thinking about this too, and one thing that puzzles me is why more places didn’t just say, What the hell, we’ll just live our life like normal. It can’t all be blamed on Neil Ferguson or whatever. It does seem that people will put in some effort to avoid spreading the disease.

                Yeah, I’ve been wondering about this too. Specifically, why are Americans reacting to this so differently than to the 1957 and 1968 pandemics? Those caused something like 120k and 100k US deaths, in a population more like 200M – so 1957 might be the equivalent per-capita of maybe 200,000 US deaths today.

                1968 is especially interesting as a comparison, because that was a very politically divided era as well.

                It’s also interesting that many tropical countries with a much higher ‘baseline risk’ from infectious disease took strong measures. Most Americans under 70 or so have never had to worry about infectious disease (except maybe AIDS, which is a very different thing). But in countries where malaria, yellow fever, dengue, etc. are prevalent, and where median age is much lower, COVID might not change the overall risk picture much.

                I think it has a lot to do with “immediacy” of modern media and a more connected world. In the 50s and 60s, places like Italy — or even New York City, if you lived in Texas or California or Colorado — were “a lot farther away”.

                So what is happening there now seems more relevant, and so there’s political pressure to adopt strategies that are (or seem to be) working in other countries.

                And I think back in March the world started to divide into “containment” strategies (East Asia and much of Western Europe) and “limited mitigation/this is no big deal” strategies (the US, Brazil, UK, Sweden). The Imperial College model and political pressure / fear of what was happening in Italy pushed the US and UK into changing their strategies. I think that movement of the US and UK created a much stronger impression of a ‘global consensus’.

              • Martha (Smith) says:

                confused said,
                “Here’s why I think the economic effects of a less controlled epidemic would have been less severe than they might at-first-glance seem:

                -mortality rate of COVID is much lower than that of 1918 H1N1 pandemic*, so large-scale social breakdown and economic collapses are probably unlikely, since they didn’t occur in the far worse 1918 pandemic.”

                This is comparing apples and oranges (or maybe even apples and zucchini?). In particular, the economy and the social factors in 1918 were very different than they are now. In 1918, a much larger portion of the population lived in rural areas or small towns. There was also not as much travel — and especially not the kind of long-distance traveling in a few hours that we have today. Also, health care was not as advanced then — treatment for the 1918 flu was pretty much limited to bed rest and chicken soup (and maybe prayer).

              • Martha (Smith) says:

                confused said,

                “Yeah, I’ve been wondering about this too. Specifically, why are Americans reacting to this so differently than to the 1957 and 1968 pandemics? Those caused something like 120k and 100k US deaths, in a population more like 200M – so 1957 might be the equivalent per-capita of maybe 200,000 US deaths today.
                .
                .
                .
                I think it has a lot to do with “immediacy” of modern media and a more connected world. In the 50s and 60s, places like Italy — or even New York City, if you lived in Texas or California or Colorado — were “a lot farther away”.

                So what is happening there now seems more relevant, and so there’s political pressure to adopt strategies that are (or seem to be) working in other countries.”

                Good point. I was alive in (and old enough to remember events from) 1957 and 1968 — but I don’t remember any talk of pandemics then.

              • Martha (Smith) says:

                However, I do remember talk of the polio epidemic in the late 1940’s, and especially in the early 1950’s — polio was really scary.

              • confused says:

                I feel like I am coming across as saying “we shouldn’t have locked down”. I’m not, I am rather undecided on that.

                But I think the economic/social effects of pandemics seem from history to be surprisingly small short of a Black Death/New World Smallpox type scenario.

                >>> -“direct” losses to the economy from sickness will be limited by the fact that the vast majority of deaths are outside the workforce (due to the usual US retirement age),

                >>But what you’re describing is what took place during a lockdown where most people weren’t going to work.

                No, the vast majority of deaths would still have been outside the workforce, because the age-effect on IFR is *so* strong. There’s a huge difference between being 60 and being 80 for example.

                >>Maybe the workplace would have been massively disrupted for a longer period with a higher rate of spread.

                I don’t see how. How long would it have taken for an uncontained epidemic to infect everyone it was going to infect?

                >>Keep in mind, deaths is out the only relevant issue. Sickness is also important. And people getting infected in a highly infectious workplace then spread the virus to family members back home.

                Sure, but I was speaking strictly of economics, not ethics/suffering.

                >> A third or half of a company’s employees being off work sick for a month vs. 100% of them being off work for lockdown.

                >This is a purely hypothetical scenario that seems extremely unlikely to me. In the very least it is the best case scenario at one end of a spectrum of possibilities.

                I don’t see how, because many infections are asymptomatic, and not all symptomatic infections are subjectively significant enough to act on. People come to work with colds, flu, etc. (which I hate, but it is an observed fact).

                A third to a half of a *working* population (therefore excluding the oldest and least healthy – “healthy worker effect”) being out sick for a month actually seems very pessimistic to me, given the asymptomatic proportion observed in prisons, homeless shelters, the USS Theodore Roosevelt, etc.

                >>I think thar cross-country comparisons are fraught, but the economy in Sweden does not show a massive advantage to “locked down” economies in similar countries.

                Isn’t that largely because Sweden is rather dependent on the rest of the world? I think if *no one* (except maybe China, given that I’m talking about decisions made in March) had locked down, that would be very different.

                In 6 to 12 months we should know more. Maybe Andrew could do a ‘looking back on models and projections’ post then?

              • Carlos Ungil says:

                > why are Americans reacting to this so differently than to the 1957 and 1968 pandemics? Those caused something like 120k and 100k US deaths, in a population more like 200M – so 1957 might be the equivalent per-capita of maybe 200,000 US deaths today.

                I guess the reactions in 1957 and 1968 would also have been different if there had been 15’000 deaths in one month in NYC.

                > But in countries where malaria, yellow fever, dengue, etc. are prevalent, and where median age is much lower, COVID might not change the overall risk picture much.

                Brazil has around one million cases of dengue per year and less than one thousand deaths. In a few weeks it has reported already one million of cases of covid-19 and 50’000 deaths.

              • Zhou Fang says:

                I think comparing to 1957/1968 is an apples and oranges comparison. Firstly, the mortality in those pandemics are calculated based on subsequent analysis, it was not based on a direct count *during* the pandemic. Hence the figures cannot be directly compared, it could be the case that a subsequent analysis years after (especially including second waves!) would put the covid19 death toll as far higher. Secondly the 1957/1968 death toll took place in the absence of lockdown and social distancing, whereas we are seeing similar death tolls to those *despite* some lockdown and social distancing etc.

                Even the argument that Covid19 is less lethal than 1918 Spanish flu is risky – consider that we define cases of Covid19 based on genetic or antibody testing. This is much more sensitive than the diagnosis methods available in 1918, with the result that the usual virulence estimates for 1918 can be overestimates due to many cases of mild infection being missed.

                I think it’s very plausible that if handled the same way as 1918 or 1957-1968, we could see from covid19 levels of death equal or even exceeding that of 1918 spanish flu.

              • confused says:

                >>I think it’s very plausible that if handled the same way as 1918 or 1957-1968, we could see from covid19 levels of death equal or even exceeding that of 1918 spanish flu.

                I don’t think so. Missed mild cases lowering the IFR don’t make that much of a difference here; the percentage of the *entire world population* dying of 1918 flu was significantly larger than any of the reasonable IFR estimates for COVID.

                The CDC estimates 50 million deaths; the world population then was 1.8 billion – so ~2.8% of everyone on Earth.

                So the IFR of 1918 flu, no matter how many mild cases were missed, cannot be less than that, since more than 100% of the world cannot be infected.

                COVID’s IFR is debatable, but clearly not >2%. The higher plausible estimates are a bit over 1%.

              • confused says:

                And yes, comparing numbers directly to 57/68 is somewhat apples and oranges. (Most importantly because this pandemic is not over… reporting is improving drastically as this develops. While there have clearly been missed deaths from looking at CDC excess death data, from that same data I don’t think there is room for the numbers to double or anything close to that. I don’t think it will change the qualitative picture of “much worse than 2009 pandemic or seasonal flu, much less severe than 1918 pandemic” – that’s why I chose 57/68 as comparisons.

                Sure, this is with some degree of social distancing and shelter-in-place orders. But I am not convinced that the way the US did shelter-in-place orders is going to have more than a marginal effect on total deaths, since those orders are no longer in effect most places. If we get a vaccine distributed to the public by November (as in 2009), maybe, but that was working from a known flu vaccine.

              • Zhou Fang says:

                confused:
                > The CDC estimates 50 million deaths; the world population then was 1.8 billion – so ~2.8% of everyone on Earth.

                But how big is the uncertainty around that estimate?

                This most recent study estimates the 1918 death toll to be 15 million (+2 million the next year), implying IFR is only >0.7%.

                https://academic.oup.com/aje/article/187/12/2561/5092383

                This is based off of a multilevel regression model that I haven’t looked into the details of. The uncertainties in that, plus the fact that we still don’t have a clear idea what covid IFR looks like in the third world, leads me to argue we shouldn’t directly compare.

                Instead, more useful is to look at 1918 US statistics, where we estimate 500k-800k dead out of 29.4 million infected. Assuming undercounting of deaths is less likely than undercounting of cases, this bounds IFR in the 1918 US above, so IFR is 0.4-0.8%.

                Put all this together with our current uncertainty on covid19 and I don’t think we can rule out these intervals overlapping.

              • Zhou Fang says:

                (reposting because sigh, html)

                confused:
                > The CDC estimates 50 million deaths; the world population then was 1.8 billion – so ~2.8% of everyone on Earth.

                But how big is the uncertainty around that estimate?

                This most recent study estimates the 1918 death toll to be 15 million (+2 million the next year), implying IFR is only bounded below by 0.7%.

                https://academic.oup.com/aje/article/187/12/2561/5092383

                This is based off of a multilevel regression model that I haven’t looked into the details of. The uncertainties in that, plus the fact that we still don’t have a clear idea what covid IFR looks like in the third world, leads me to argue we shouldn’t directly compare.

                Instead, more useful is to look at 1918 US statistics, where we estimate 500k-800k dead out of 29.4 million infected. Assuming undercounting of deaths is less likely than undercounting of cases, this bounds IFR in the 1918 US above, so IFR is less than 1.7 to 2.7%. Bring in the population total of 105 million and spanish flu IFR is greater than 0.4-0.8%.

                Put all this together with our current uncertainty on covid19 and I don’t think we can rule out these intervals overlapping.

              • confused says:

                Yeah, the 50 million is not set in stone. I’ve seen estimates of ~18 million, but also of 100 million.

                But even 18 million is 1% of the total world population then – *not* a 1% IFR! Probably would be more like 3%, flu pandemics don’t tend to infect the majority of people IIRC. COVID definitely doesn’t have 3% IFR.

                Yeah, COVID IFR could be higher in countries not hard hit yet — but I am very skeptical of this, as nations with poorer medical care tend to have much younger demographics. Though the joker in the deck there is HIV/AIDS prevalence in sub-Saharan Africa.)

                If anything, I think the end-of-pandemic overall IFR will be significantly lower than 1%, since treatments will improve in the wealthier countries at least (already are improving), median ages of those infected seem to be dropping in the US at least*, and demographics are far younger elsewhere.

                *Early hot-spots (Lombardy, Madrid, NYC) seem to have infected the elderly fairly ‘efficiently’.

              • confused says:

                I read that article with the 15-17 million more closely, and I don’t find it terribly convincing. They are working from Europe and India data… but they are more or less making an argument that Europe (with lower death rates) is a better model for the parts of the world we don’t have data on than India (with higher death rates).

                This seems pretty implausible to me, as in 1918 the effects (economic, demographic, etc.) of the Industrial Revolution had not spread very far beyond Europe and the US.

                I think they are right that 100 million is far too high and that 50 million is likely too high, but 15 million?

      • Clyde Schechter says:

        “I definitely think missed cancer screenings and such are a real and significant problem”

        I strongly doubt it. I’ll speak first of breast cancer screening, since it’s the area I work in and know best. Numerous analyses going back decades have suggested that the lead time gained from mammography screening is around 2 years on average, and longer in older women who are the age group that accounts for the bulk of breast cancers. With newer technologies such as digital mammography and digital breast tomosynthesis, the lead time is, if anything, longer still (though probably not by much). The shutdowns have lasted only a few months. Only a small fraction of women who ordinarily have mammograms every 18-30 months will actually transition an occult, but radiologically detectable, breast cancer from treatable with good outcome to lethal during that short interval. Perhaps the greatest danger will be the small group of women who carry very high risk due to BRCA-1 mutations: for them the lead time is much shorter. Evidence that delay of a few months is probably irrelevant can be seen by comparing breast cancer mortality in the US and the UK. In the UK, regular mammography screening is scheduled at 3 year intervals, and their breast cancer outcomes are very similar to ours.

        What may become a problem is diminished capacity for screening as radiology practices reopen. Because the epidemic remains active, and is still in its exponential growth phase in many parts of the country, it is likely that breast imaging centers will have to reduce the number of mammograms that can be performed due to decreased capacity in limited physical space and perhaps equipment downtime needed to disinfect rooms and equipment between patients. A sustained and substantial reduction in capacity could lead to much longer intervals between screenings that do result in non-trivial increases in breast cancer mortality later on. So, the fact that we have not suppressed the virus, rather than the temporary interruptions due to shutdown, may well ultimately lead to an uptick in breast cancer deaths. Time will tell.

        While I am less familiar with the details of colon and prostate cancer screening, if anything these should be less problematic than breast cancer because the lead times provided by screening are much longer, typically a decade or more.

        I am surprised to see people speak of our response to the epidemic as an over-reaction on a site like this where people are pretty well-informed. The US death toll stands today at over 120,000. While the epidemic has crested in the northeast and Midwest, it is now growing exponentially in some parts of the country that previously saw little activity, and it’s anybody’s guess where it will end. Health care people in Tucson and Houston are saying that the hospitals and ICUs are now nearly full and if things keep going, in a few days they will be overwhelmed. What would it take for you to believe that we did not overreact?

        In my view, none of this had to happen. But the only countries that have had good results were ones that acted very early and took the epidemic seriously from the day the first case was identified in their territory. We frittered away precious weeks doing nothing, and then bungled our early efforts at responding. Now, we have simply given up on containing it. Pretty much all policy measures against the epidemic are being dismantled across the country, even in places where it is still on the upswing. People are socializing in bars and restaurants, an mass gatherings are ongoing every day. We will be very lucky to come out the other side of this with only 200,000 deaths.

        • Martha (Smith) says:

          Thank you very much. I hope others with knowledge of specific areas of medicine can give input as well; you effort sets a good example for them.

        • confused says:

          Good points about the screening. I am likely wrong about that.

          >>I am surprised to see people speak of our response to the epidemic as an over-reaction on a site like this where people are pretty well-informed.

          Well, my prior on “overreaction” is past US pandemics. We did very little for the 1957 or 1968 pandemics. Our reaction now is possibly stronger than our reaction to the 1918 pandemic.

          It’s not clear to me why COVID requires an order of magnitude stronger response than 1957/1968. (Those pandemics, per-capita, would be equivalent to a bit under 200,000 deaths.)

          >> What would it take for you to believe that we did not overreact?

          Well, basically, evidence from other countries that our reactions made a dramatic difference to the progress of the disease in the US. (Action taken at the stage we actually did act – back in January would have been very different!)

          It’s impossible for me to believe that every nation on earth is handling this very well. So if the original projections of what an unmitigated pandemic or one with limited mitigation would do were correct, then death rates in the range projected would happen *somewhere*.

          • Anoneuoid says:

            Maybe because it was hyped to be a distraction from the collapse of the multilevel ponzi scheme that began last September. So people blame a virus and racists instead of the banks, politicians, and corporations responsible who made out like bandits.

            I mean people are pretty distracted from the bailouts going on and like half are clamoring for the government to become even more powerful.

    • dhogaza says:

      “Upon acquiring solid evidence about the epidemiological features of new outbreaks, implausible, exaggerated forecasts [19] should be abandoned.”

      Note that back in April when the IHME model had its greatest influence (IMO, at least), the IHME model was forecasting total deaths in the US by early August to be most likely 60,000-70,000.

      Welcome to 120,000 deaths and still counting, people.

      • Indeed. I keep being impressed at how vehemently people are saying that all the stuff predicted by public health isn’t happening… even as it is in fact obviously happening.

        • confused says:

          I feel like there were kind of three phases:

          -second half of March, when the forecast that was getting the most attention was the Imperial College London’s 2.2 million US deaths.

          The actual report said that applied in an implausible scenario where no action was taken, even on an individual, company, or local level — which was already not true on March 16, some businesses had already gone to teleworking, school districts in my area had closed at least in the short term, etc. The forecast for limited mitigation was 1.1 million. But the media largely ignored that in favor of ‘2.2 million deaths’ headlines.

          -April, when the overly-optimistic early version of the IHME model was getting the most media attention and government attention.

          -The last six weeks or so, when more models are available and models have incorporated data from how the outbreak evolved in areas other than the very dense ones that provided the original data (Hubei, China/Lombardy, Italy). These models seem more realistic than either set of the other ones, but we won’t really know for a while…

          (Also, what is actually happening is *dramatically* different across different parts of the US. If you live in a state that has — so far — seen seasonal-flu levels of deaths or less, then it seems like e.g. New York was happening in an entirely different world that has no relevance to your community.)

          • “If you live in a state that has — so far — seen seasonal-flu levels of deaths or less, then it seems like e.g. New York was happening in an entirely different world that has no relevance to your community”

            This is an important point, especially when the downsides of shutdowns are very evident locally — joblessness, poor education, mental stress and depression, etc. Not only is it easy to think of New York, etc., as irrelevant, it becomes even easier to tune out when the implication of a lot of commentary is that residents of these states are too dumb to understand public health, or that the costs they experience aren’t worth considering.

            • confused says:

              Yes, exactly. The effects of government actions are immediately obvious, but outside of specific hotspots, the effects of the disease aren’t (in the central/interior-west US). Obviously that could change.

      • Carlos Ungil says:

        They were also forecasting that there would be no deaths anymore in August. Now, their prediction/projection is that by the end of September there will be 1400 deaths per day (95% confidence interval 600-3300) and the cumulative deaths will be 200,000 (170,000-270,000).

      • CFD says:

        What I find surprising, and going back to the original Imperial College report for the UK, is that it actually UNDER-performed. Ferguson stated, at a parliamentary hearing at the end of March, that his model forecasted around 30,000 dead with the measures in place. That value was surpassed way before the restrictions were lifted and in subsequent hearing for the House of Lords, he actually apologised for the optimism of his forecast.

    • confused says:

      >>As seems all to (unfortunately) typical of Ioannidis’ work on the pandemic, this paragraph just flat out ignores important uncertainties. For example, the projections about the pressure on hospitals were made based on existing resources, not based on changes that would take place in the resources available

      Sure, but isn’t the idea of static resources obviously wrong? I’m not sure that’s really an argument against the idea that predictions of widespread hospital overload were irresponsible.

      And I’m not saying they were irresponsible. I just don’t know. Maybe hospitals would have been overloaded all over the place if the US had followed a Sweden/South Dakota model. I don’t think we know enough right now to determine where the truth falls on the spectrum between ‘hospitals would have been overloaded in a lot of places on a Sweden/South Dakota model’ and ‘only a few high-density parts of the US were ever at risk for overloaded hospitals, even if total deaths over the entire span of the pandemic are high’.

  7. Mendel says:

    Sidenote: I just re-read Tomas Pueyo’s “Coronavirus: Why You Must Act Now” on medium.com that I thought was fairly i fluential, as it made a co pelling argument to “flatten the curve” on March 10th. I was surprised to see that this essay did not use forecasts at all. It used up-to-date data and an uncalibrated model to show the effects of interventions. We did not need correct forecasts to make correct public health decisions.

    An incorrect forecast may impart the same message as an uncalibrated model, and have more political power. How we evaluate the utility of these forecasts then becomes a matter of whether we agree with the political decisions that were based on them.

  8. Chris Wilson says:

    Taleb’s point about point forecasts is blindingly obvious, at least to my Bayesian indoctrinated brain :) Are there serious epidemiologists who think that point forecasts are the way to go?

    • Jon says:

      It is easier to tackle strawmen than to make serious improvements in the field.

    • Clyde Schechter says:

      I can only speak for myself, but I and the other epidemiologists I work with would love to do our forecasts with full posterior distributions. Now, I gave up infectious disease modeling back in the early days of the AIDS epidemic because the computational power needed to simulate a model with enough detail to have some reasonable hope of verisimilitude far exceeded the computational resources I could have gotten my hands on.

      My work in chronic disease epidemiology is much simpler and quicker. But even so, when I run my breast cancer simulations, just a single draw from the posterior for a single scenario will typically take 24-48 hours of runtime on a high-level desktop computer using compiled code with lots of optimization. So to do this kind of thing properly requires either a pretty extensive distributed computing network or access to a supercomputer. To my knowledge, nobody in this line of work has been able to secure funding for that, though I know some of us have tried.

      • Hmmm. A *single draw*? What kind of model is that?

      • Chris Wilson says:

        Wow, that’s intense! Lots of parameters? Thrown HMC/Stan at it? I will preface my next comments by saying I’m not an epidemiologist, but I have close friends who are and I understand some of the basic theory and math decently well. My sense is that almost any compartment model could easily be fit in Stan even with granular county time series’ and full posterior predictions pushed out. I did a little playing around with that on the early county-level data. Same goes for all the phenomenological curve-fitting junk that is floating out there, although the ROI of getting a full predictive distribution there is maybe not great :) I’m going to ask around about the more sophisticated movement-based models and such.

      • Anoneuoid says:

        Is it intel with all the meltdown, etc bugfixes?

      • Ben McMillan says:

        Do epidemiologists really struggle to get access to computing time? And why, when other disciplines have heaps?

        24 hours of desktop computing time is just so little money compared to 24 hours of employing an epidemiologist.

        The university I am familiar with, you can get 100,000 core-hours with a very light-touch one-page application. Mostly the physicists+engineers asking for it.

        • dhogaza says:

          We know from the physicist who runs the “And Then There’s Physics” blog that Ferguson’s Imperial College agent-based model takes about 20 minutes to run on a 24-core processor, using the UK population dataset.

          That’s not bad at all. It will run on a desktop in a matter of several hours, apparently, though obviously this is going to vary a lot depending on clock speed and the number of cores your machine has.

  9. Tom says:

    With respect to the missed forecasts and distributional assumptions, the specific mechanism that led to all those nursing home deaths are not really in these models, i.e., state policy to send sick patients to cohabitate with vulnerable populations. You then get the aggregate (distribution and point estimate) wrong. If you actually compared to the flu where we don’t purposefully send recovering flu patients to nursing homes then perhaps some of those early estimates of relatively low deaths would not be so absurd (although probably still low). And the papers looking at the efficacy of social distancing completely ignores the fact that it doesn’t affect the nursing home policy.

    And that doesn’t even get into measurement problems (what precisely is a coronavirus death and how is that defined across 50 states and however many individual reporting units…and internationally?). Usually measurement is a larger part of these discussions rather than getting lost in the details of complicated models (or lamenting the simplicity of some models).

    • Mendel says:

      “state policy to send sick patients to cohabitate with vulnerable populations.” — Gov. Cuomo and his staff spoke on that in his press briefing today. They made the points that
      a) state policiy was based on White House/CDC/NIH guidance
      b) other states instituted teh same policy
      c) NYC currently ranks 35th among the states on nursing home deaths
      d) nursing home deaths are not correlated with readmissions; cause-effect has not been established
      e) in most known cases, infections have been spread from external sources (vistors, staff)
      f) the state policy stipulated that hopitals accepting readmissions must be able to properly care for/isolate readmitted residents
      g) old folks staying in a hospital for an extra 2 weeks run a risk of acquiring a hospital infection

      The point you’re making is a political point, and I don’t believe it is supported by evidence.

      • Mendel says:

        P.S.: “And the papers looking at the efficacy of social distancing completely ignores the fact that it doesn’t affect the nursing home policy.” — in Germany and other countries, “nursing home policy” forbade (and with opening, closely regulated) visits to nursing home residents. This is a social distancing measure, and it does affect how many infections reach the nursing home. The health officials also imposed additional requirements, such as segmenting the population and staff to limit the potential spread, and others.

        “what precisely is a coronavirus death” — a death where the medical examiner lists “coronavirus” as cause of death on the death certificate.

        • confused says:

          >>“what precisely is a coronavirus death” — a death where the medical examiner lists “coronavirus” as cause of death on the death certificate.

          It’s actually not that simple… reported coronavirus deaths on e.g. state dashboards don’t necessarily correspond to death-certificate cause-of-death. Death certificate data arrives later.

          Early on, this probably led to reported deaths being underestimated because testing was unavailable so people who obviously most likely had it didn’t get tested and thus didn’t become ‘confirmed’.

          Now, it’s not clear which way the effect goes, because states may report on dashboards ‘all deaths of people with confirmed COVID+’ while the death certificate is based on actual cause of death determined, and if everyone coming into hospitals etc. gets tested, there are going to be people who are COVID+ but it’s irrelevant to their cause of death.

          • Mendel says:

            “Now, it’s not clear which way the effect goes” — you only say this because you haven’t looked at the mortality data, haven’t actually multiplied Covid incidence and overall death rate (adjusted for risk groups ideally) adjusted for duration, haven’t seen physician testimony (“of 75 patients I’ve treated or Covid-19, I’ve yet to see one die from another cause”), and don’t use statistics that get updated with death certificate data when it arrives (as many health offices do). There is NO evidence for Covid-19 death numbers being overcounted.

            • Anoneuoid says:

              There is NO evidence for Covid-19 death numbers being overcounted.

              If you count everyone who dies *with covid* as a covid death, then you will definitely be over-counting the covid deaths. And that was apparently the official policy in the US at least into mid-April. So if there is lack of evidence for this happening I would say there is a problem with the evidence.

              https://www.realclearpolitics.com/video/2020/04/08/dr_birx_unlike_some_countries_if_someone_dies_with_covid-19_we_are_counting_that_as_a_covid-19_death.html#!

              • Carlos Ungil says:

                > If you count everyone who dies *with covid* as a covid death, then you will definitely be over-counting the covid deaths

                Not *definitely*, because the number of “false negatives” may be (and is likely to be) higher than the number of “false positives”. You could count people dying in traffic accidents *with covid* as covid deaths and still be under-counting the covid deaths.

              • Anoneuoid says:

                Undercounting isnt mutually exclusive with overcounting.

              • confused says:

                There are clearly at least a few ‘mis-coded’ deaths, people who test positive for COVID but die of an unrelated cause. A few of those have been reported (recreational drug overdose, alcohol poisoning, shooting, etc.)

                But those sorts of things are not numerous enough to impact the overall statistics.

                Until recently, deaths were clearly undercounted significantly due to lack of tests.

                But that may have changed now that testing is much more available… and hospitals tend to test every patient for COVID regardless of what their actual complaint/reason for admission is. So if the truly ‘missed’ deaths (never tested) are now small, hospital patients dying of other causes while incidentally COVID-positive might become a factor.

                But probably now the reported deaths are a very good measure of real COVID deaths.

              • Anoneuoid says:

                About 87% of the extra deaths reported as of May 20th were apparently counted as covid.

              • Anoneuoid says:

                But are covid deaths reported faster than other deaths? So many details about how this data is generated are missing.

            • confused says:

              >>There is NO evidence for Covid-19 death numbers being overcounted.

              That’s not what I’m saying.

              I agree that *total* COVID deaths are undercounted (more people have died of COVID during the course of the epidemic than are currently reported).

              That doesn’t necessarily mean that COVID deaths *occurring once testing became good* are undercounted. There were so many missed/undercounted deaths in late March and the first half of April that they drive the statistics.

              >>because you haven’t looked at the mortality data,

              I look at the CDC death data regularly, both national and for my state and other large states. The fraction of “excess deaths not explained by COVID” seems to be decreasing since week-ending April 11, and will probably increase further as CDC processes more death certificates with COVID.

    • Joseph Candelora says:

      “All those nursing home deaths”.

      All what nursing home deaths?

  10. Mendel says:

    The first reference in the Ioannidis/Cripps/Tanner article seems to contradict that article, with its title of “Wrong but Useful — What Covid-19 Epidemiologic Models Can and Cannot Tell Us”. “useful” and “failed” are opposites.

    Five Questions to Ask about Model Results.

    1. What is the purpose and time frame of this model? For example, is it a purely statistical model intended to provide short-term forecasts or a mechanistic model investigating future scenarios? These two types of models have different limitations.

    2. What are the basic model assumptions? What is being assumed about immunity and asymptomatic transmission, for example? How are contact parameters included?

    3. How is uncertainty being displayed? For statistical models, how are confidence intervals calculated and displayed? Uncertainty should increase as we move into the future. For mechanistic models, what parameters are being varied? Reliable modeling descriptions will usually include a table of parameter ranges — check to see whether those ranges make sense.

    4. If the model is fitted to data, which data are used? Models fitted to confirmed Covid-19 cases are unlikely to be reliable. Models fitted to hospitalization or death data may be more reliable, but their reliability will depend on the setting.

    5. Is the model general, or does it reflect a particular context? If the latter, is the spatial scale — national, regional, or local — appropriate for the modeling questions being asked and are the assumptions relevant for the setting? Population density will play an important role in determining model appropriateness, for example, and contact-rate parameters are likely to be context-specific.

    These answers to these questions determine the limitations of that model. Conclusion:

    Unlike other scientific efforts, in which researchers continuously refine methods and collectively attempt to approach a truth about the world, epidemiologic models are often designed to help us systematically examine the implications of various assumptions about a highly nonlinear process that is hard to predict using only intuition. Models are constrained by what we know and what we assume, but used appropriately and with an understanding of these limitations, they can and should help guide us through this pandemic.

    In the light of this, I struggle to understand why the paper argues that the “forecasting has failed”.

    One failure they point at are empty hospital beds. Accurate forecasts would allow better use of resources. I know that some of the “empty” capacity in the German hospital system are provisional beds (for example, in my community, an empty schoolhouse was converted to be an overflow hospital, it hasn’t been used). The question is, can you make the forecasts more accurate than the ones we have, and not run the risk of running out of beds? The cost of running out is not considered in the paper. What is the alternative to the best (but “wrong”) forecast we have, and how much does it cost? What alternatives do we have in dealing with forecast uncertainty except to keep a number of free hospital beds in reserve?

    “However, with COVID-19, espoused wrong predictions can devastate billions of people in terms of the economy, health, and societal turmoil at-large.” & “Civilization is threatened from epidemic incidentalomas.” — might this be an exaggerated forecast? nah.

    “we have little evidence that aggressive measures which focus only on few dimensions of impact actually reduce death toll and do more good than harm.” — we do have mortality figures that we can use to compare countries that reacted early (“aggressively”) to countries that reacted late, e.g. via Euromomo. This data reflects the acute overall death toll and excludes potential long-term effects. We have evidence that these measures, applied aggressively, reduced the death toll in the short term, shortened the period of lockdown (and hence, the associated psychological effects), and presumably reduced the number of hospitalizations. To forego this short-term gain in fear of a long-term loss that we have no evidence for seems unwarranted.

    “We need models which incorporate multicriteria objective functions. Isolating infectious impact, from all other health, economy and social impacts is dangerously narrow-minded.” — does this quote advocate for making a model with a large degree of uncertainty even more complex? To me, it doesn’t seem to be a criticism of the epidemiological model per se as a criticism of how its output is used as an input into the political process: but the article doesn’t actually deal with how the economic and social impacts are considered in that process. I feel that an epidemiogical model shouldn’t be extended until it yields a political decision: it’s probably better to keep epidemiological science and decision making separate, with one informing the other. What the paper seems to call for is not an epidemiological model, but a political model: but that contravenes the democratic process that posits that interested groups negotiate politicial decisions according to their respective interests.

    “Upon acquiring solid evidence about the epidemiological features of new outbreaks, implausible, exaggerated forecasts should be abandoned.” — and implausible evidence should not be trusted blindly. The paper does not support the implied claim — that this demand has not been met in this pandemic — with evidence.

    My take on this paper is this: epidemiological models *fail* to produce political advice. They *fail* to transcede their inherent limitations. But they’re useful for understanding the epidemic and inform the political debate in useful ways.

  11. Jon says:

    Unless you consider journalists, wanna-be epidemiologists, and opinion writers who distort model projections, I think Taleb is shooting straw men he created. The major epidemiological modelers acknowledge the huge uncertainties in the projections and run their models with different assumptions.

    Ioannidis seems to have wrecked his credibility. It seems he and Taleb are engaged in the opposite of group-think; trying to sell themselves as being the mavericks who are right when everyone else is wrong.

    • somebody says:

      I tend to find the same thing about Taleb. He tends to construe one opinion column in the NY Times as “the media” or one person giving a forecast as “epidemiologists as a discipline.” It came to a pretty confusing head when he was lambasting “the media” for downplaying the threat of Covid when all the news I was following had treated it seriously since January, while at the same time personalities from the right were lambasting the media for apparently constructing the threat of Covid from thin air and generating a panic.

      But I think he’s still great to read because he’s usually right, and the other guy might not be everyone he says they are but they exist, and have some influence.

    • Re: That’s fine—Ioannidis was careful at the time to condition his estimate based on the limitations of available data, and you can learn a lot in two months. Still, 40,000 was an erroneous prediction, and at the very least this error should cause him to reassess his assumptions. And if he’s gonna write about erroneous predictions, he could mention his own.
      —-

      It looks to me that nearly every estimate has been wrong for the US.

      So it’s easy for any of us to absolve ourselves of the biases if we don’t weigh in.

      Plus I don’t think John ever said that it was wrong to have an initial lockdown. Rather his focus was on the effects of a ‘prolonged’ lockdown. At least that is how I read it.

      Most experts don’t go out on a limb and admit that they were wrong. Cognitive dissonance at play.

      • To qualify further, in a couple of Youtube presentations, John admits that he has been wrong. My view is that John doesn’t seem to respond to an outright confrontation. Not his style. Nassim is full frontal. LOL

      • Carlos Ungil says:

        > Plus I don’t think John ever said that it was wrong to have an initial lockdown.

        Exactly three months ago he wrote: “If we decide to jump off the cliff, we need some data to inform us about the rationale of such an action and the chances of landing somewhere safe.”

        The cliff had been metioned before in that article: “That huge range markedly affects how severe the pandemic is and what should be done. A population-wide case fatality rate of 0.05% is lower than seasonal influenza. If that is the true rate, locking down the world with potentially tremendous social and financial consequences may be totally irrational. It’s like an elephant being attacked by a house cat. Frustrated and trying to avoid the cat, the elephant accidentally jumps off a cliff and dies.”

        Reading things like the above and “In the absence of data, prepare-for-the-worst reasoning leads to extreme measures of social distancing and lockdowns. Unfortunately, we do not know if these measures work.” people who doesn’t know him well may think that he was kind of opposed to the lockdowns. He didn’t even suggested that they could be the right approach at least in some places.

        “Some worry that the 68 deaths from Covid-19 in the U.S. as of March 16 will increase exponentially to 680, 6,800, 68,000, 680,000 … along with similar catastrophic patterns around the globe.” He was not among the worriers, prefering to “assume that case fatality rate among individuals infected by SARS-CoV-2 is 0.3% in the general population — a mid-range guess from my Diamond Princess analysis — and that 1% of the U.S. population gets infected (about 3.3 million people), this would translate to about 10,000 deaths.”

        • ”Those mortality estimates are what Imperial College London used in their March 16 study.

          All of this work was well completed before Ioannidis published his Stat article on March 17 or his paper in early April.

          What is the relevance of the Oxford article you linked?’
          ———–
          Joseph,

          The 1st article link refers to an analysis conducted by John Ioannidis was about Sars-CoV-2 in Italy. I don’t get the impression that John relied on the Imperial College estimates. Why should he? This is about Italy. Pointing out that in Italy the disease hit the elderly doesn’t need an Imperial College citation or reference nor the claim that Imperial College’s analysis came out 1st. Maybe I misunderstand your question.

          The Oxford article indicates to me that we needed to do massive testing in February or earlier. So we are lagging in data pertaining to asymptomatic and pre-symptomatic spread. That has implications for the reopening of schools in the Fall and raises so many unanswered questions. Particularly regarding appropriate testing practices.

  12. paul alper says:

    Few if any of the bloggers on this site listen and watch Alex Jones. He and his followers simultaneously maintain that Covid-19 is really
    a. Nothing more that the common cold
    b. The death count is fictitious because anyone who dies of anything is listed as a Covid-19 victim
    c. A Soros/Gates Globalist plot to promote insidious, sex-altering vaccinations
    d. This is the first stage of a ChiComs weaponized assault

  13. IoannidisAgain says:

    Andrew you are too kind to Ioannidis. It’s clear that he is pushing an agenda. His predictions of death rates were way off, 10,000 in his first article. The Stanford antibody study was highly flawed. And now he dismisses the possibility that the extensive social distancing measures that have been implemented (not to mention possible seasonal effects) could be playing a part in the current death counts.

    He is arguing in bad faith. His position is that drastic measures like lockdowns are damaging, which well may be reasonable.

    But instead of making that case with quantitative comparisons on the impact of policy decisions, he has resorted to presenting misleading, cherry picked, and often wrong information about the severity of Coronavirus.

    If he instead made the case along the lines of, “If there are no interventions than 200k-400k people might die… but with interventions, X% of people will lose their jobs which will lead to Y”, I, for one, would be more likely to consider his position.

    • Joshua says:

      +1

      Again –

      > It’s clear that he is pushing an agenda.

      I have long been engaged in arguments online to defend climate scientists’ right to advocate for policy. I certainly think that Ioannidis has that right as well. He has a right to step outside his scientific lane to give his input to policy development he considers extremely important.

      But I think there’s a problem here for two basic reasons: (1) he’s doing so under a cloak of empirical analysis. If you want to engage in political debates about policy, that’s fine. But don’t do so and then claim that you aren’t being political (which Ioannidis has done), and (2) engage in quality advocacy. That means don’t be selective in how you treat uncertainty. At least take on obvious counterarguments. Ioannidis has consistently failed to do so during the course of his policy advocacy.

      • Clyde Schechter says:

        +1

        Over his career, Ioannidis has made many important contributions to the medical literature. He is a person whom I, until now, respected enormously. But he has lost credibility in my eyes with his clearly political-agenda-based stance on the pandemic. Now I put him in the same (trash) basket as all those public health figures who said that the anti-lockdown protests were a public health danger but mass demonstrations over police murders aren’t.

        • Mendel says:

          Who are the “public health figures” you are talking about? The most prominent ones are probably Fauci, Redfield and Birx, and they’re on record saying it is a danger:

          “Dr. Anthony Fauci, the director of the National Institute of Allergy and Infectious Diseases, expressed concern for people attending mass protests across the country, touched off by the death of George Floyd, a black man who died after a white Minnesota police officer knelt on his neck for nearly nine minutes. Fauci said that while he understands why people feel compelled to attend protests, spreading the contagious respiratory disease at such large-scale events poses a real threat. ” (USA Today)

          (CNN)If you’ve marched in any recent protests, you may want to consider getting tested for the coronavirus.

          That’s the recommendation of the director of the US Centers for Disease Control and Prevention, who said during a House Appropriations hearing on Thursday that anyone who participated should “highly consider” getting tested.

          “I do think there is a potential, unfortunately, for this to be a seeding event,” Dr. Robert Redfield said during Thursday’s hearing on the coronavirus response. He said the risk of infection is higher in major cities where there’s been significant transmission. To prevent transmission, Redfield suggested people who attend protests tell their loved ones that they were out in public and get tested within 3 to 7 days.

          “The White House coronavirus task force spoke with governors Monday on a conference call and discussed concerns over a second wave of coronavirus infections due to the nationwide protests against systemic racism sparked by the death of George Floyd at the hands of Minneapolis police.
          During the call, Deborah Birx, a key member of the task force, as well as Vice President Pence said they are monitoring coronavirus cases, expressing concerns that activities like demonstrating in close quarters and yelling without masks could enable the spread of the virus.” (The Hill)

      • Anonymous says:

        “I have long been engaged in arguments online to defend climate scientists’ right to advocate for policy. “

        Everyone has the right to advocate for policy. Of course they do. The question in science – not just climate science – is: how much does one’s advocacy bias and therefore compromise one’s scientific results and credibility? Unfortunately a scientist isn’t in control of people’s perception of their work, and as we’ve seen time and time again: a) bias isn’t so easy to detect; and b) proclamations that look obvious today can look stupid tomorrow. (Recall a number of prominent scientists signed on to the Peak Oil and the more general resource depletion meme, advocating with gusto). Wisdom should council that they consider this problem before they take up the bullhorn and start digging a trench that has only two directions. Unfortunately many scientists aren’t so humble.

        It’s not the right that’s in question. It’s the wisdom.

  14. A Country Farmer says:

    > Still, 40,000 was an erroneous prediction, and at the very least this error should cause him to reassess his assumptions.

    Isn’t there a difference between over-predicting vs. under-predicting? Isn’t there a burden of proof on predictions that lead to policy changes (i.e. coercion)?

    > Maybe his article should’ve been titled, “Our forecasting for COVID-19 has failed,” and he could’ve criticized the errors and lack of expertise in that Stanford study.

    Where in his article did he say he was going to canvas all forecasts? Why don’t you discuss the other points in the article?

    Maybe all models should be put into a meta-model with the bayesian prior of: don’t collapse the world unless the evidence is really, really, really good; if there’s cause for concern, gather better data.

    • Andrew says:

      Country:

      1. Yes, there’s a difference between over-predicting vs. under-predicting. I don’t think there’s a burden of proof in either direction: A prediction is a prediction, and it’s the policymakers who need to balance uncertainties and risks.

      2. I would not expect Ioannidis to canvas all forecasts, but I would expect him to comment on forecasts and models that he himself has put his name to.

    • IoannidisAgain says:

      > Isn’t there a difference between over-predicting vs. under-predicting? Isn’t there a burden of proof on predictions that lead to policy changes (i.e. coercion)?

      Some comments.

      You raise a good point, here. What we are really talking about is decision theory and utility functions. Even in that framework though, we should strive to model a distribution as accurately as possible. In other words, we shouldn’t alter our distributions and priors because of our utility functions.

      The second related point is that each person has their own utility function. If I have diabetes or if I am old, I may not want to go out during the pandemic, which is why it doesn’t make sense to put a prior that artificially lowers predictions.

      The last thing I will say is even without formal policy interventions, many would have taken actions like sheltering in place. As I see it, the only way to prevent “collapsing the world” would have been to lie or withhold information. Or, alternatively, to have taken quick and drastic measures like in Taiwan with mandatory hospital quarantines for infected person and people entering the country.

      • Clyde Schechter says:

        +1 +1 +1……

      • confused says:

        >>As I see it, the only way to prevent “collapsing the world” would have been to lie or withhold information.

        I don’t know. I mean, if there had been accurate messaging on the differential risk with age, schools wouldn’t have to have been closed, many employers could have gone on basically as usual, etc.

        The severity of this seems closer to 1957/1968 flu than to the 1918 flu.

        • IoannidisAgain says:

          It’s hard to argue convincingly about a counterfactual. My recollection is that teachers and their union in my district essentially demanded that their schools close.

          This same district just emailed all parents asking if they prefer to continue distance learning or some hybrid where kids go to school on different days to reduce classroom sizes.

          Several local colleges have stated that they will not be reopening campuses in Fall.

          Anecdotally, many companies that I and friends work at voluntarily switched to working remotely and plan to do so for the foreseeable future.

          You are obviously correct that some restaurants and other businesses would have remained open and similarly some schools systems, but I would argue that voluntary changes in behavior still would have resulted in a large economic impact.

          I also argue that the differential risks by age were accurately messaged. Very early on, China stated that no children had died from Covid.

          Even with differential risks by age, how many teachers fall in a relatively high risk group because of age or comorbidities? How many children have relatives that fall into these categories? I don’t know how these populations break down along these axes so it’s hard for me to have an opinion whether they should have remained open or not.

          The sanest approach would have been to adopt policies similar to those in Taiwan and Korea with extensive testing, contact tracing, and strict quarantining of the infected.

          For multiple reasons, we seem to be unable to implement that in the US. So we are left to choose between open everything or close everything which feels like a damned if you do, damned if you don’t kind of situation.

        • Zhou Fang says:

          Schools absolutely should be closed unless you’re proposing kids never see their parents and grandparents again until the pandemic ends. Inferences that kids cannot transmit the virus are extremely unsafe.

          • confused says:

            >>Inferences that kids cannot transmit the virus are extremely unsafe.

            Hmm, ok. I had heard reports out of Europe that this didn’t seem to happen (or at least the risk was low), and places that had reopened schools (like Norway) hadn’t seen problems. Were these wrong?

            • Zhou Fang says:

              If you look at the actual studies they are usually based on extremely small sample sizes (the most often quoted one had something like 20 households, and most of the children they looked at were under 1 year old, so obviously they aren’t the index case). In a lot of other stats you also see an extremely sharp drop off from say, 10-19 year olds (almost none) from 20-29 year olds (basically the same as older age groups) which just doesn’t feel biologically plausible to me.

              • Joshua says:

                Zhou –

                I have heard epidemiological modelers saying that closing schools for young children doesn’t help after maybe 10? days or so. But they also couldn’t explain the counterintuitiveness of that finding, and they couldn’t explicate a causal mechanism (they noted that they couldn’t apply what they new to a context where a variable mix of other interventions were taking place simultaneously) . So that makes me kind of dubious. I read your comment about the available evidence, but I’m not inclined to just dismiss what they were saying. Do you know of evidence that says that opening schools will lead to worse outcomes, rather than just the weakness of the evidence that says it won’t? It certainly is my intuitive sense that opening schools isn’t a great idea – but they’re doing it in places that have been very careful – so maybe the outcomes would be context specific?

              • Zhou Fang says:

                Joshua:

                No, I don’t think I have any strong positive proof that opening schools would lead to worse outcomes, sorry. Also, there’s some fuzziness in the definition of “young children”, which doesn’t help.

              • confused says:

                >>which just doesn’t feel biologically plausible to me.

                I am pretty skeptical about arguments about COVID transmission on this basis, since its spread seems to be highly dependent on super-spreaders and therefore pretty non-intuitive.

                There seem to be people who are hyper-infectious and people who are non-infectious though infected. I don’t see why it’s biologically implausible for people under a certain age to be overwhelmingly in the latter category.

                I mean, I am sure there is not a magic switch that turns on your ability to spread COVID on your 20th birthday.

                But I can certainly see a plausible scenario for it. Much more of the 10-19 age group are going to school than the 20-29 age group; not everyone goes to college, and relatively few do past 23 or so.

                If there is really a degree of cross-immunity from ‘common cold’ coronaviruses (as some admittedly tentative — preprint stage — studies of T-cell response have suggested), then going to school (greatly increasing risk of getting common cold coronaviruses) might make a large difference.

                Combine that with generally increasing susceptibility with age… (IE maybe many very young people get infected, kind of, but clear it so fast they don’t really have a chance to spread any virus).

                And since the US is not going to eradicate COVID short of a really good vaccine, it doesn’t have to be zero-risk to reopen schools. Just relatively low risk. I mean, if a teacher/parent/older relative is 100 times more likely to catch flu from a student, but 100 times more likely to die of COVID, the total risk is the same.

    • Zhou Fang says:

      I would have suggested that the default pre-existing policy would have been to do exactly what the South Koreans did given they have experience with pandemics and a method that works quite well to minimise both economic loss and loss of life. If Ioannidis wants to come up with a new and different way of handling pandemics, the burden of proof should be on him to justify this methodology.

      • confused says:

        That policy simply can’t be implemented in the US, though. (And probably many other countries too.)

        It might be a viable model for Hawaii, but definitely not for the US as a whole.

        • Martha (Smith) says:

          Good points.

        • Zhou Fang says:

          I think it’s still reasonable to try and go as close as possible to the South Korean model though.

          • Mendel says:

            https://doi.org/10.1080/14719037.2020.1766266

            Europe and the United States are enforcing a much stronger containment policy than Korea, but these policies were put in place late. Korea’s quick response can be attributed to the lessons learned during previous epidemics and accidents. Such lessons led to a firm belief that testing and contact tracing should form a central part of any response to a viral epidemic. Consequently, when COVID-19 arrived in Korea, the effective testing, tracing, and treating of infected people became the course of action that seems to have set Korea apart from other nations.

            Identify cases, trace contacts, isolate infectious is the established response to any outbreak involving human-to-human transmission. South Korea isn’t special in the methods they applied, but in how quickly they were able to apply them compared to other countries and particularly the US, with the CDC fumbling the testing rollout and the political resistance to acknowledging the issues.

            It’s probably fair to say that if the US had implemented their own pandemic plans with vigour, the situation would be much better than it is.

          • confused says:

            I don’t think so, in the US specifically. The US is very individualistic and has little trust in government, especially centralized government. Strong measures may lead to ‘backlash’ and thus worse outcomes overall than milder measures that would actually be followed by most of the population.

            If we had had good testing and tracing in January, yes definitely! But bureaucratic issues in CDC and FDA messed up the testing.

            But in mid-March when we were actually making these decisions, any attempt at eradicating the virus would have required far stronger measures than the US would ever accept.

            It’s just part of the cost of a more individualistic/non-conformist society with a federal rather than unitary structure. Whether that cost (increased risk) is worth it is obviously very debatable. (I tend to think it is, but then, I grew up and live in one of the more “small-government” US states that has a fairly strong cultural memory of its frontier history.)

            • Once you get below the threshold, the virus prevalence decays exponentially, and things get easier and easier. We could still TODAY spend 3 weeks doing community surveys across the country in states where cases are growing. Do case contact tracing, and conservative quarantining. Within 2 months cases would be minimal in those areas. We just don’t have the will and leadership to do it.

              • confused says:

                >>Do case contact tracing, and conservative quarantining.

                I just don’t think you are going to get particularly great compliance with those things in the US, especially in the parts of the US where things seem to be getting worse now (AZ, TX, AR, etc.) These are southern and/or interior-west states, a very different culture with much more suspicion of government and individualism than

                It might well have worked in January or February. But that was lost when there were all those media claims that TX/FL were going to be “the next Italy” in 2-3 weeks after Spring Break, WI was going to see mass deaths 2-3 weeks after the election, GA was going to see mass deaths 2-3 weeks after reopening, etc. When that clearly failed to happen, a lot of people decided the whole thing was overblown/crying wolf, sure NYC was bad but NYC is nothing like TX/AR/AZ/GA/etc. I think the chance to get real public consensus for this was gone by mid-April if not earlier.

                I think most of these places will not get bad enough for that mindset to chance (because I think the threshold for “bad enough” is pretty high, and residual social distancing + lower density + younger people getting infected more vs. older people will mean that NYC-scale effects don’t happen.)

    • Andreas says:

      > Isn’t there a difference between over-predicting vs. under-predicting? Isn’t there a burden of proof on predictions that lead to policy changes (i.e. coercion)?

      No, IMO the other way around. Scientists and policy makers have had to deal with a new virus where not all aspects are known from the beginning. Decisions must be made quickly and based on partly known data. The original models were not based on wild guesses, but based on data from China, South Korea, Italy etc. And looking backward: They have not been that off.

      As responsbile adviser and policy maker you cannot wait with a warning and steps till there is hard evidence for a problem when you see many ṕeople infected and many people dying in other countries. Then it does not matter if the IFR is 0.6 as originally estimated *under optimal conditions and a not overwhelmed health care system* or just 0.4. And as long there is no hard evidence for cross-immunity it’s strange to guess that it may end after 1 % infected.

      I think as adviser and as policy maker you cannot lean back and say “Ooops… yes, many have died and we thought this could have happened – but we had no hard evidence that it really happens, so we decided to do nothing”.

      Italy, France and Spain probably did not notice the spread early enough and got into big trouble. UK noticed it but listened to irresponsbile advice first to just let it run to reach herd immunity quickly. Look at the result.

      It is always not optimal if advisers/modellers over or understimate something. But: underestimating something *esp* just because of making wild optimistic guesses is much worse.

      • confused says:

        I really think that goes too far. It’s risk/benefit or cost/benefit either way, but I have never liked the idea that strong action is “automatically” favored if we can’t make reasonable estimates of risks/costs.

        Advisors/modelers shouldn’t lean either way, IMO. It’s responsible to say “under these assumptions we might have 50,000 deaths, under these assumptions we might have 2 million deaths and massive overload of hospitals”. Once you start saying “therefore we should act as though the second set of assumptions are true” that gets questionable.

        As for policy makers’ responsibilities… there are of course disputed issues of responsibility here. Many people in the US, especially the south/interior west, would tend to assign less responsibility to the government (especially national government) and more to individuals, businesses, and smaller-scale governments (state/county/city).

        The CDC, for example, doesn’t have much power to mandate things. The primary responsibility for public health is with the states. Different states have done radically different things – and this is not an example of individual politicians doing random things, but often is a pretty good reflection of that state’s culture (though maybe not so much in some states with a R governor in an often D-leaning state, or vice versa).

        • Chris Wilson says:

          confused, there are basically two formalized ways of handling this kind of situation: 1. Bayesian decision theory, and 2. minimax. In either case, done right, you are going to end up with recommendations that “look like” over-reacting, and are deliberately calibrated higher than the level of response required for the “most likely” scenario.

          • confused says:

            Why will it always “look like” over-reacting?

            I can see that that would be true in a situation where there are high-probability mild effects and low-probability really severe effects, so (severity x probability) of the really severe effects is much greater than their probability.

            But I don’t think that’s an accurate description of the situation with COVID when US states were making these sorts of decisions (mid-March to beginning of April). At that point there were clear upper and lower limits (enough people had recovered that this was clearly not the Black Death or probably even comparable to 1918 flu, but by that point it was clearly pandemic, so really mild effects were also off the table.)

            If you’re talking about decisions being made in January, though, absolutely.

            • Andreas says:

              I disagree with the assumption “or probably even comparable to 1918 flu”.

              It very much depends on the availability of treatment. When you look at the situation in hot-spots where hospitals are overwhelmed IFR goes up steeply. But that depends on how well the outbreak is mitigated.

              There has been the risk of overestimating IFR (for example using data from Italy where testing was limited), but also the risk of underestimation (when using data from a small region with limited outbreak and good healthcare conditions).

              • confused says:

                I agree that IFR will be higher where hospitals are overwhelmed.

                But I don’t think that changes the picture that (over large populations – I’m not ruling out some very local discrepancies) it is still significantly less than 1918 flu.

                Estimates for the global deaths from 1918 flu range from 17 to 50 million, even 100 million. CDC says 50 million. 17 million is ~1% of the world population then (1.8 billion), 50 million is ~2.8%.

                It probably infected something like 30% of world population (CDC estimates 500 million). So IFRs were probably 3%-10% or so.

                I don’t think COVID gets that high even in the worst hotspots (except nursing homes, which are solely a very vulnerable population).

  15. Ron Kenett says:

    Andrew concludes his blog with “The sad truth, I’m afraid, is that Taleb is right: point forecasts are close to useless, and distributional forecasts are really hard. We have to try our best and use all available resources.”

    This ignores the goal of a study, forecast, model…. In assessing information quality, the first step is to disclose the goal. Are you focused on the healthcare system and the number of ventilators it needs, the economy and the impact of restrictions or society and the well being and psychological impact of social distancing?

    In Israel, the models used by the ministry of health were making forecasts mainly focused on predictions affecting the hospitals. The state of the economy and society in general where lower in priority. If you think that way you consider worst case scenarios.

    Neither Nassim Taleb, nor John Ioannidis consider information quality, To do so requires first stating the goal. The Santa Clara study was linked to a possible conflict of interest with funding of the Jet Blue owner who has interest in downplaying the gravity of the situation. I suggest that one should consider several goals and then proceed with the discussion on fat tails and worst case scenarios.

    Once you do that your first step is to evaluate the data resolution in the context of the goal. The information quality framework has 8 dimensions. https://www.wiley.com/en-us/Information+Quality%3A+The+Potential+of+Data+and+Analytics+to+Generate+Knowledge-p-x000754836

    The last one is about communication of findings. The COVID19 related indicators have to be communicated in the right way, at the right time, to the right person. The media has made a bog mess of all this. David Spieglehalter is trying hard to improve this with limited success because he does not state the goal of the communication https://wintoncentre.maths.cam.ac.uk/coronavirus/covid-excess/

    • Andrew says:

      Ron:

      Excellent points. We should not think of data as being static.

      • Ron Kenett says:

        Andrew – also the goals are not static. You collect data for goal 1 and then move on to use it to address goal 2. This is the difference between primary and secondary data. You analysis should be sensitive to this. In considering goal 2 some decisions made in collecting towards goal 1 might need to be accounted for, as a minimum disclosed.

        Adopting the information quality perspective allows you to address all this. For example, matching data resolution to the goal might yield different solutions in considering goal 1 as opposed to goal 2.

        Going back to COVID19. A goal focused on avoiding saturation of hospitals will get you to look at numbers infected. A goal towards reopening the economy would look at social movements and business activity.

        DIscussing the information quality with David Cox about a year ago, he immediately pointed out that goals are not static. The implications of this raises interesting questions. Same data different goals – how do you account for that….

  16. Mendel says:

    Andrew:
    nobody responded to that above, so I plucked it out, although I’m not sure that it isn’t a rhetorical question.

    Confused:

    I’ve been thinking about this too, and one thing that puzzles me is why more places didn’t just say, What the hell, we’ll just live our life like normal. It can’t all be blamed on Neil Ferguson or whatever. It does seem that people will put in some effort to avoid spreading the disease.

    The thing was “flatten the curve”: the evidence of quick exponential growth, the idea that our hospital systems would get overwhelmed as seen in Lombardy and elsewhere, and the knowledge that if we can’t provide oxygen to people who need it, the death rate will go up.
    “Flatten the curve” messaging spread very widely from many sources.

  17. gofx says:

    Andrew,

    As an experienced forecaster (not in epidemiology) I have observed that there is a tendency to understate the uncertainty in our forecasts lest they be thought of as useless as per the discussion of Taleb.

    On top of the non-linearities etc. that have been noted, there are other sources of uncertainty. Good ole measurement error in the dependent and independent variables contribute to uncertainty. Also, forecast errors regarding any inputs or independent variables also increases uncertainty.

    From my modest review of some of these epidemiological models I think that they just aren’t that good at times series. I don’t know if it is because these modelers come more out of a world of effect testing, parameter estimation, survival curves and p-values rather than true forecasting of dependent variables. I just don’t know.

    Economics went through a similar situation where large-scale structural econometric models that were built to explain the economy in detail and forecast the economy in detail were many times out-forecasted by simple time series models. Econometric modelling evolved–rational expectations models were developed where agents anticipated model-based policy changes (hint to epidemiologists), VAR modeling was introduced as a bridge between structural and time-series modeling, GARCH models were developed, cointegration was recognized, etc. Whether this has enhanced forecasting accuracy—I hope so.

    I think the main complaint with the current situation is the lack of humility of the forecaster. Forecasting an observable, that can be observed regularly in your lifetime should make you extremely humble about the certainty of your prediction and its use for telling the rest of the world what to do. Same goes for climate modeling except there, the observable keeps getting pushed farther into the future. I recognize that the policymaker, the user of the forecast has a role here too. But the influence of the forecast seems outsized.

    If I were modeling this epidemic, I would have several forecasting models. One of them would NOT be that huge code-dependent simulation model (for lack of a better description) that started all this. I would generally make them Bayesian since we have had epidemics before.
    Start with some simpler, even univariate time series models for countries or the globe in total. Graduate to single equation model(s) with a small number of explanatory variables (use a hierarchical model for more richness if needed, likely with time-varying parameters), then possibly some VARs or BVARS or State Space representations. Maybe even some smaller scale, multi-equation structural models. (But remember, any variable used to drive the dependent variable forecast, must be itself forecast!) Examine the results from the suite of models and see if the ensemble “tells me anything”.

    If measurement error is extremely pervasive, I might switch to “all cause deaths” as a dependent variable, although this would entail some modeling changes.

    From an interval forecasting point of view, even with my lack of knowledge about epidemiology, I’m not sure I’d do that much worse than what has been put out so far.

  18. Colonel Alexsey Potemkin says:

    I would be interested in your take on Dexamethasone. Sensibly you are probably waiting for the full data rather than the press releases

    https://www.recoverytrial.net/results

    For HCQ there control group mortality was 23.5%. For Dexamethasome – with largely the same control group (30% bigger) – the mortality is much higher.

    The number I have been trying to replicate is towards the end. Overall Hazards ratio of 0.83 [0.74 to 0.92] p value of 0.0007. Treatment ~ 2100 and control ~4300
    To get close to that p value I have to my mortality in the control group as 1500 out of 4300, around 35% (645 out of 2100 in the treatment group).

    Unless I have messed up – entirely possible – the people in the control group suddenly started dying in far greater numbers for the dexamethasone trial

    • Mendel says:

      I’m quoting from the two press releases available via your link:

      A total of 1542 patients were randomised to hydroxychloroquine and compared with 3132 patients randomised to usual care alone. There was no significant difference in the primary endpoint of 28-day mortality (25.7% hydroxychloroquine vs. 23.5% usual care; hazard ratio 1.11 [95% confidence interval 0.98-1.26]; p=0.10). There was also no evidence of beneficial effects on hospital stay duration or other outcomes.

      A total of 2104 patients were randomised to receive dexamethasone 6 mg once per day (either by mouth or by intravenous injection) for ten days and were compared with 4321 patients randomised to usual care alone. Among the patients who received usual care alone, 28-day mortality was highest in those who required ventilation (41%), intermediate in those patients who required oxygen only (25%), and lowest among those who did not require any respiratory intervention (13%).

      Dexamethasone reduced deaths by one-third in ventilated patients (rate ratio 0.65 [95% confidence interval 0.48 to 0.88]; p=0.0003) and by one fifth in other patients receiving oxygen only (0.80 [0.67 to 0.96]; p=0.0021). There was no benefit among those patients who did not require respiratory support (1.22 [0.86 to 1.75]; p=0.14).

      Based on these results, 1 death would be prevented by treatment of around 8 ventilated patients or around 25 patients requiring oxygen alone.

      [..]

      Overall dexamethasone reduced the 28-day mortality rate by 17% (0.83 [0.74 to 0.92]; P=0.0007) with a highly significant trend showing greatest benefit among those patients requiring ventilation (test for trend p<0.001). But it is important to recognise that we found no evidence of benefit for patients who did not require oxygen and we did not study patients outside the hospital setting. Follow-up is complete for over 94% of participants.

      83% of p=23.5% of N=2104 = 410 deaths
      For that p and N, observing an outcome of 410 deaths or less has a probability of 0.00000532.
      I have to lower p to under 19% to get up to 0.0007 (or presumably use a more sophisticated method).

    • Colonel Alexsey Potemkin says:

      Paper out now, I think they just massively over-stated the p value. it should be 0.007 not 0.0007

      https://www.medrxiv.org/content/10.1101/2020.06.22.20137273v1.full.pdf

  19. Jazi Zilber says:

    Ioannidis “forecast” from March was never a forecast.

    Explicitly an “informed guess”

    “If I were to make an informed estimate based on the limited testing data we have, I would say that covid-19 will result in fewer than 40,000 deaths this season in the USA.”

    Guesses with such heavy caveating are very much ok to end up wrong.

    Modellers never hinted “our models are very very noisy, no better that an informed guess”.

    • Andreas says:

      > Modellers never hinted “our models are very very noisy, no better that an informed guess”.

      I think they hinted that very well. And I think one may not forget is, that a model is a model. Esp. in the case of an epidemic a model never can be absolutely accurate because behavior of the people also influences the outcome. Humans are not stupid robots who just act in a certain way till they are ordered to do something different.

      If a model estimates 2 mio death in case of no mitigation it already can be a bit less or more. But even worse: It’s unlikely that people won’t react if they see a growing number of ill people. Even without an ordered lockdwon many people would stay at home, avoid crowded places, which would also influence the outcome.

      Models cannot be perfect and are hardly usable for point-estimates. But I really have a hard time to believe the “informed” in the “informed guess” in the 40000 estimate. To me it reads like a optimstic guess not backed by data.

  20. Anoneuoid says:

    The most intriguing finding of this
    study is that only 4% of healthcare providers without confirmed COVID-19 diagnosis
    had IgG antibodies to SARS-CoV-2 in their blood. Most of the healthcare providers were
    exposed to SARS-CoV-2 during the first few months of the outbreak when use of
    personal protection equipment was sparse as person-to-person transmission was not
    suspected. COVID-19 IgM/IgG tests in the US and around the world as reported in the
    news constantly showed that the true infection rate would be 10 to 80 times higher than
    that had been confirmed by RT-PCR tests for SARS-CoV-2.

    Seroprevalence of
    antibodies to SARS-CoV-2 in 1021 people before resuming work from April 3 to 15,
    2020 in Wuhan was reported to be ~10%, 29 about 20 times higher than the infection
    attack rate calculated from the confirmed COVID-19 cases. In New York City, a 21.2%
    positive rate was reported with young and middle-aged people having the highest
    positive rate. The proportion of people infected with SARS-CoV-2 who have no
    symptom or only mild symptoms that do not need medical attention or hospitalization
    may account for the majority of SARS-CoV-2 infections.

    […]

    In Zhongnan Hospital of Wuhan University, 2.88% (118/4099) healthcare
    workers were diagnosed with COVID-19 before March 16, 2020. With a moderate
    estimation, the true infection rate would be ten times that had been confirmed, i.e., >25%
    of those healthcare providers without diagnosed COVID-19 had been infected. However,
    only 4% of those infected healthcare workers without confirmed COVID-19 still had IgG
    antibodies to SARS-CoV-2. They just got infected with SARS-CoV-2 and cleared the
    virus by their own immune systems. No long-lasting protective antibodies against
    SARS-CoV-2 were produced in these healthcare providers. Our observed high
    prevalence of IgG antibodies to SARS-CoV-2 in older groups (60-69 years old and ≥ 70
    years old) among health care workers and general worker in Wuhan also raised the
    concern that IgG antibodies to SARS-CoV-2 would be lost some time after the infection
    was cleared, as young or middle-aged people usually took more social responsibilities
    and had higher chances to get infected during lockdown of the city. We also found
    that >10% of confirmed COVID-19 cases had no detectable serum levels of IgG
    antibodies to SARS-CoV-2 after 21 days post symptom onset.

    […]

    In conclusion, very few healthcare providers without confirmed COVID-19 diagnosis in
    Wuhan have IgG antibodies to SARS-CoV-2, though a substantial portion of them had
    been infected with the virus. More than 10% of COVID-19 patients did not have those
    antibodies after 21 days post symptom onset. After SARS-CoV-2 infection, people are
    unlikely to produce long-lasting protective antibodies against this virus.

    https://www.medrxiv.org/content/10.1101/2020.06.13.20130252v1

    • Anoneuoid says:

      The main finding was that “very few healthcare providers without confirmed COVID-19 diagnosis in
      Wuhan have IgG antibodies to SARS-CoV-2″ despite no use of PPE early on in the pandemic. They jump to the conclusion this is because they were infected but did not develop antibodies, ie seroprevelance data will underestimate by ~5x the number of mild/asymptomatic infections that have occurred.

      Another explanation is that the virus is only actually infectious for about 1 week after symptoms first appear (even if they keep testing positive on PCR) and it takes about a week to get sick enough to end up in the hospital. Thus all the PPE precautions are not really necessary for most severe covid patients.

      That has been previously reported here: https://www.nature.com/articles/s41591-020-0869-5

      There is another paper that reports the same but I don’t have it at the moment.

      • Anoneuoid says:

        But then there is this paper:

        We also followed 37 asymptomatic individuals and 37 symptomatic patients into the early convalescent phase (8 weeks after they were discharged from the hospital). The IgG levels in the symptomatic group were still significantly higher than those in the asymptomatic group in the early convalescent phase (P = 0.002) (Fig. 3b). Surprisingly, the IgG levels in 93.3% (28/30) of the asymptomatic group and 96.8% (30/31) of the symptomatic group declined during the early convalescent phase (Fig. 3c). The median percentage of decrease was 71.1% (range, 32.8–88.8%) for IgG levels in the asymptomatic group, whereas the median percentage of decrease was 76.2% (range, 10.9–96.2%) in the symptomatic group. Using a pseudovirus-based neutralization assay (Methods), we also observed a decrease in neutralizing serum antibodies levels in 81.1% (30/37) of the asymptomatic group and in 62.2% (23/37) of the symptomatic group. The median percentage of decrease was 8.3% (range, 0.5–22.8%) for neutralizing serum antibodies in the asymptomatic group, whereas the median percentage of decrease was 11.7% (range, 2.3–41.1%) in the symptomatic group (Fig. 3d). Moreover, 40.0% (12/30) of asymptomatic individuals, but only 12.9% (4/31) of symptomatic individuals, became seronegative for IgG (Fig. 3e).

        https://www.nature.com/articles/s41591-020-0965-6

      • Mendel says:

        I see a problem with their assumption of the true infection rate undercount. Healthcare workers are sensitive to symptoms and have ready access to testing, so the chance that a mildly symptomatic case goes untested (by PCR) in this population is probably lower than it is in the general population.

        When you find fewer cases you expected, you can either look for an explanation why there are fewer cases, or look for an explanation why your expectation is wrong. Confirmation bias favors the former over the latter.

        • Anoneuoid says:

          They say:

          There were 4099 healthcare providers working in Zhongnan Hospital of Wuhan University, of whom 118 were diagnosed of COVID-19 before March 16, 2020 and 3835 healthcare providers without diagnosed COVID-19 received both tests before resuming normal clinical services. Three healthcare providers who were tested positive for SARS-CoV-2 by RT-PCR tests in their throat swabs were also excluded from the analyses.

          But its not clear what happened with those 118 who were diagnosed.

        • Anoneuoid says:

          And keep in mind:

          Most of the healthcare providers were exposed to SARS-CoV-2 during the first few months of the outbreak when use of personal protection equipment was sparse as person-to-person transmission was not suspected.

          So wouldnt we expect higher transmission rates than in general?

        • Joseph Candelora says:

          Quite possibly the dumbest study I’ve read this whole pandemic.

          Their key finding was premised on this chain of logic:
          1. Wuhan had a cumulative PCR positive percentage of about 0.5%, but a seroprevalence rate of 10%
          2. Other places have shown seroprevalence rates 10 to 80 times the cumulative PCR positive percentage
          3. Healthcare workers in Wuhan had a PCR positive rate of about 2.8%
          4. Thus, since Wuhan healthcare workers were exposed early the pandemic before PPE use was prevalent and since other places have seen at least a factor of 10 greater seroprevalence rate than cumulative PCR positive rate, a conservative estimate of seroprevalence rate in Wuhan healthcare workers must be 10x 2.8% = 28%
          5. The non-PCR positive Wuhan healthcare workers only exhibited a seroprevalence rate of 4%
          6. So since 28%+ “must” have been infected at some point, and we only caught 4% on antibody testing, it must be the case that antibody protection disappears over time.

          I’ll quote the conclusion from the abstract:
          “Conclusions Very few healthcare providers had IgG antibodies to SARS-CoV-2, though a significant proportion of them had been infected with the virus. After SARS-CoV-2 infection, people are unlikely to produce long-lasting protective antibodies against this virus.”

          If step 4 in the chain above isn’t the most ass-backwards thing I’ve ever read, I’ll eat my hat. Mendel put it perfectly in his comment above.
          6. So since 28%+ “must” have been infected at some point, and we only caught 4% on antibody testing, it must be the case that antibody protection disappears over time.

          I’ll quote the conclusion from the abstract:
          “Conclusions Very few healthcare providers had IgG antibodies to SARS-CoV-2, though a significant proportion of them had been infected with the virus. After SARS-CoV-2 infection, people are unlikely to produce long-lasting protective antibodies against this virus.”

          If step 4 in the chain above isn’t the most ass-backwards thing I’ve ever read, I’ll eat my hat. Mendel, you’ve described the issue perfectly.

          Now that said, it is surprising that healthcare workers are showing a seroprevalence lower than the general population, given the likely higher exposure. But 4% isn’t the right number, since they excluded from the cohort of healthcare workers the 2.8% who were PCR-positive. So let’s call its 6%-7%, still lower than 10%.

          There may be something to their inference that antibody levels diminish over time, but this sure seems like the wrong way to support such a claim. How about you just do a longitudinal study of antibody levels in known positives?

          All that said, isn’t diminution of antibody levels the normal course of things? And doesn’t our immune system have mechanisms for ramping up antibody production upon re-exposure? (My understanding is that’s the role of the white blood cells known as “memory B cells”.)

          • Mendel says:

            Thank you!
            I concur with your last paragraph.
            Health care workers are trained in hygiene and effective use of PPE, I’d expect them to be less susceptible to Covid-19 infection generally as long as they don’t work with Covid-19 patients.

            The interesting bit about the study data is this:

            IgG prevalence increased significantly by age among healthcare providers, and was 2.8% in those <30 years old, 9.6% in those 60-69 years old and 10.0% in those 70 years old (p<0.001 for trend). IgG prevalence also increased significantly by age among general workers.

            The data (Table 2) shows a similar trend, but not as pronounced, for general workers.
            I see several possible explanations:
            — hospitals use older, more experienced staff to care for Covid-19 patients, so older staff are more likely to get infected
            — IgG levels decline faster in younger people
            — younger people are more likely to overcome the infection without developing specific antibodies
            I have no idea if any of these hypotheses is true.

          • Anoneuoid says:

            How about you just do a longitudinal study of antibody levels in known positives?

            This was also published recently. I saw it just after I saw the paper you are commenting on:

            https://statmodeling.stat.columbia.edu/2020/06/17/some-forecasting-for-covid-19-has-failed-a-discussion-of-taleb-and-ioannidis-et-al/#comment-1363192

            tl;dr: The antibodies of about half the people who are asymptomatic and 10% of those with mild illness are undetectable after about 2 months.

            • Joseph Candelora says:

              Yes, that’s a much better paper.

              It has some issues with the charts A and B in Figure 3 — they did something really weird with their Y axis, where the appear to be plotting logbase2 of the S/CO ratio (as indicated on the axis) for all observations with S/CO values above 2 or so, but using the actual S/CO (no log) for observations with values below 2 or so. And some other small stuff. I sent an email to the authors earlier. Since they published their full dataset (awesome!) I was able to put together an illustration:
              https://i.redd.it/cgvk8pk6pw551.png

              And to clarify, the numbers you referenced aren’t people whose antibodies were undetectable; these people had antibody levels below the seropositive threshold set by the manufacturer (which, as we know, represents a tradeoff between specificity and sensitivity). E.g, the highest negative individual was (unsurprisingly) just a hair under the threshold, with S/CO ratio of 0.913, while the lowest positive was just a hair over at 1.058. The other ends of the extremes were 0.140 and 120.209 respectively.

              • Anoneuoid says:

                undetectable; these people had antibody levels below the seropositive threshold set by the manufacturer (which, as we know, represents a tradeoff between specificity and sensitivity).

                Yep, that is the definition of undetectable.

  21. Girish says:

    Could you comment on the accuracy of sampled sero-surveys for Covid19, in general? In regions with low prevalence (which is still most of the world), the geographical distribution of cases is highly clustered and non-uniform. Would even the best designed (small) samples be able to gauge prevalence accurately? Or is the geographical density distribution fat tailed and so any estimate will be inaccurate and converge slowly?

    • Mendel says:

      I wonder if anyone understands the accuracy of these surveys.

      a) https://finddx.shinyapps.io/COVID19DxData/ is a database of available tests. The difference between the sensitivity and 100% is the rate of false positives, and in a low prevalence situation, they drown out the data and can hide the true geographical distribution. It’s not well understood what drives the false error rate in these surveys, so you don’t know what the false error rate actually is in a specific situation because it varies. The Bendavid Santa Clara study compiled about a dozen evaluations for their test, with varying results. (There was a post about that on this blog.)

      b) The LA County study had a sample size of ~1000 and showed less seroprevalence in May than in April (though I can’t find the LA County Department of Public Health press release for that right now).

      My takeaway was not to trust sero-prevalence surveys in areas of low prevalence.

      • Girish says:

        The false positives/specificity issue in low prevalence situations is well studied, Bendavid et al, notwithstanding.

        I’m saying even if you used a test with practically no false positives (Wuhan RT-PCR survey of 11 million had only 300 positives – true Or false, for example), the clustering will still make your survey inaccurate for reasonably small sample sizes. Would like to know if there’s analysis addressing this issue.

  22. Berend de Boer says:

    This claim: “but it turned out that what Fund was doing was taking the upper bounds of Ferguson’s forecasts of past public health crises and pointing out that they were overestimates of the actual number of deaths in each case.”

    is not very strong if you know that in reality Ferguson, in each of these cases, has strongly advocated for the upper bound as the one reality will follow. For the mad cow diseases Ferguson predicted 136,000 (even 500,000 earlier!).

    A rival team at London’s School of Hygiene and Tropical Medicine developed their own model which predicted there would be up to 10,000 cases, with a “few thousand” being the best case scenario. Ferguson pooh-poohed the work of this rival team, saying it was “unjustifiably optimistic”.

    Instances abound: Ferguson always went with the upperbound. And honestly, a range of 10 to 500,000 will die from mad cow disease is not very helpful is it?

Leave a Reply