Doubting the IHME claims about excess deaths by country

The Institute for Health Metrics and Evaluation at the University of Washington (IHME) was recently claiming 900,000 excess deaths, but that doesn’t appear to be consistent with the above data.

These graphs are from Ariel Karlinsky, who writes:

The main point of the IHME report, that total COVID deaths, estimated by excess deaths, are much larger than reported COVID deaths, is most likely true and the fact that they have drawn attention to this issue is welcome. In a study of 94 countries and territories by Dmitry Kobak and myself – we estimate this ratio (based on actual all-cause mortality data) at 1.6. We believe this to be a lower bound since we lack data for much of the world, where more localized reports and studies demonstrate larger excess.

The issue with the IHME report is that it uses extremely partial data when much more encompassing (such as World Mortality) exists, the issue is that the country-level estimates they showed publicly are incredibly different than known ones (mostly higher) and that they purport to accurately estimate excess deaths where data simply does not exist – this undermines a tremendous effort currently underway to improve and collect vital data in many countries.

Karlinsky also quotes. Stéphane Helleringer:

I [Helleringer] do worry a lot though about false impression of knowledge and confidence that is conveyed by their estimates; especially the detailed global maps like the ones they just produced for COVID death toll and MANY other health indicators for which few or no data are available. The risk is that IHME figures, with their apparent precision, will distract some funders & governments from goal of universal death registration in low to middle Incomes countries. From their standpoint, if IHME readily estimates mortality, why invest in complex systems to register each death?

This is an interesting moral-hazard issue that comes up from time to time when considering statistical adjustments. I remember years ago that some people opposed adjustments for census undercount based on the reasoning that, once the census was allowed to adjust, that would remove their motivation for counting everyone. In practice I think we have to push hard in both data collection and modeling: work to gather the cleanest and fullest possible datasets and then work to adjust for problems with the data. If the apparently very seriously flawed IHME estimates are taken as a reason not to gather good data, that’s a problem not so much with statistics as with governments and the news media who have the habit of running with authoritative-sounding numbers from respected institutions and not checking. We saw that a few years ago in a different setting with that silly Democracy Index. The claims lacked face validity and were based on crappy data, but, hey, it was from Harvard! The University of Washington isn’t quite Harvard, but I guess the IHME had a good enough public relations department that they could get that air of authority. Also, they sent a message that (some) people wanted to hear. Also, the coronavirus authorities, for all their flaws, were lucky in their enemies. Say what you want about the IHME, they weren’t as dumb as last year’s White House Council of Economic Advisors or the Stanford-advised Pandata team or the Hoover Institution’s Richard Epstein, who, when he’s not busy jamming his fingers down people’s throats, made a coronavirus death prediction that was off by a factor of 1000.

P.S. See Karlinsky’s page for more details on data and estimates.

P.P.S. Instead of using legends in his graphs, Karlinsky should’ve placed labels on each line directly. For some reason, many people don’t seem to know about this trick, which allows people to read your graph without having to go back and forth and decode the colors.

14 thoughts on “Doubting the IHME claims about excess deaths by country

  1. Regarding the IHME, their authority, and their international influence, I thought this article was interesting: https://www.thenation.com/article/society/gates-covid-data-ihme/ It reminds me of the tensions and debates between centralization and de-centralization as well as Andrew’s previous posts on the cathedral vs. the bazaar.

    Re: the placing labels on each line directly instead of a legend, I gotta be honest, I agree with you 100 percent but was always too lazy to actually do it. So I just googled how to do it and I guess that was a terrible excuse since there are good packages that make this really easy to do in ggplot. Link for other lazy people: https://stackoverflow.com/questions/29357612/plot-labels-at-ends-of-lines

  2. Maybe you could explain what you mean saying their claims are “inconsitent with the data above.” For the US, they are claiming ~600K excess deaths, which is ~900K covid deaths minus ~300K fewer deaths from other factors.

    It feels like a rather extreme claim–it seems to suggest that a fake pandemic would “save” 300K lives, as well as claiming we missed 1 of 3 deaths from COVID. But it doesn’t really contradict anything.

    • Dzhaughn:

      From the quoted news article:

      Researchers at UW ultimately concluded that the extra deaths not directly caused by COVID-19 were effectively offset by the other reductions in death rates, leaving them to attribute all of the net excess deaths to the coronavirus.

      “When you put all that together, we conclude that the best way, the closest estimate, for the true COVID death is still excess mortality, because some of those things are on the positive side, other factors are on the negative side,” Murray said.

      So, yes, they’re estimating the number of covid deaths as equal to the number of excess deaths. An estimate of 900,000 contradicts the above graphs which show 600,000 excess deaths.

      • Well, that’s not my reading of:

        http://www.healthdata.org/special-analysis/estimation-excess-mortality-due-covid-19-and-scalars-reported-covid-19-deaths

        Not to waste your valuable time, but:

        Table 1 line 1 (also in Karlinski) says 905K Covid deaths vs. 570K reported. Eyeballing Figure 5 for the USA, I get the USA as undercounting COVID deaths by 1.5 – 1.75. Seems like the same datum.

        They list “drivers” a-f; the last 3 are things like reducing motor vehicle accidents, flattening the flu curve, and all the not-so-lucky people who died of COVID but would have died anyway. (The latter would not have been saved by a pretend pandemic, so I should take that back.)

        If the question is “how bad was it, really?” 600K is still the best body count. But to understand the rate the disease spreads, the actual cause of death matters.

        I interpret Murray’s quotation as saying as much: that roughly in the USA, the undercount of direct COVID deaths roughly matches the reductions in other-cause mortality. I suppose I think the NPR reporter is using “extra deaths” to mean “uncounted covid deaths” rather than “excess mortality.” But who knows? It was radio after all.

        • IHME goes through a list of several factors which can increase or decrease mortality “regardless” of COVID, and in the end they say it averages out, so that total excess = total covid. This is a reasonable assumption to make IMO. The issue is that their excess estimate for the US is way too high, as is the situation in many other countries where we have reliable excess death estimates – Germany, UK, Japan, Spain, France, Belgium…

          On the other side, their excess estimates for developing countries, where they have NO DATA are too small – so the whole thing is a mess.

  3. “Instead of using legends in his graphs, Karlinsky should’ve placed labels on each line directly. ”

    I like their lower graph where the line colors are in the title. Labeling lines is good too but all charts and maps should have a legend regardless of whatever other bling is provided. Sure, it’s hardly a need for two lines, but as charts get more complex a legend becomes a necessity and IMO the best habit is to have it then add whatever else you want.

    • I do like direct labels! there’s even a great R package for it that I use quite often:
      http://directlabels.r-forge.r-project.org/

      The issue is that these charts come from an automated script running on all the countries we have in WMD, and directlabels sometime require manual adjustment so they don’t overlap or show up in strange places on the chart. But I will take this challenge to improve the charts even in this process :)

  4. In 2017, the IHME got a ten-year, $279 million grant from the Gates Foundation to maintain the IHME’s ‘population health’ initiative. That’s $27.9 million per year, for an institute with over 300 staff. I can’t help wondering if that level of funding predisposes an institution towards interpreting their analyses in favor of ‘big’, provocative stories…….. like 900,000 ‘excess’ deaths. To justify big grants, maybe you need big stories.

    ( To give perspective to the Gates Foundation bestowal, the FY 2020 appropriation for the National Institute for Allergies and Infectious Diseases earmarks, among other things, $511 million for research related to combating antimicrobial resistance, about $200 million to advance basic, translational, and clinical research to develop a universal influenza vaccine, and $102 million to combat neglected tropical diseases. https://www.niaid.nih.gov/grants-contracts/budget-appropriation-fiscal-year-2020 )

  5. I thought US is overcounting deaths overall, and some undercounting in some places in the US.

    the CDC people should stand up and call for some news conference to stop this kind of claims. this is an example of misinformation.

    Some people in the US died of other causes but somehow had a positive test, so they were classified as covid related death. at least I know one locally reported in the news.

  6. So the IHME model suggests an underreporting factor >10x for Japan??? That can’t be right. Their model is just puzzled with the fact that Japan has a very old population but a very low (apparent) IFR.

Leave a Reply

Your email address will not be published. Required fields are marked *