More on the epidemiologists who other epidemiologists don’t trust

You know how they’ll describe someone as a musician’s musician? Or a chef’s chef? There’s also the opposite: those people who to outsiders represent a certain profession, but who are not respected by their fellow insiders. Guy Fieri, perhaps? I’m not really an expert on food celebs so I’m just guessing on that one.

An example is the team at the University of Washington’s Institute for Health Metrics and Evaluation (IHME).

Paul Alper points us to this article by Tim Schwab, who reports:

Are Bill Gates’s Billions Distorting Public Health Data?

Thanks to the Microsoft founder’s support, the IHME can make its own rules about how to track global health. That’s a problem.

In the early days of the pandemic, the IHME projected a far less severe outbreak than other models, which drew the attention of Donald Trump, who was eager to downplay the danger. At a March 31 press briefing, the White House’s coronavirus response coordinator, Debbie Birx, with the president at her side, used IHME charts to show that the pandemic was rapidly winding down. . . .

The IHME brushed aside the widespread criticism that emerged—“Many people do not understand how modeling works,” its director, Chris Murray, explained in a Los Angeles Times op-ed—and continued to push headline-grabbing projections that drew alarm from its peers.

No kidding, many people do not understand how modeling works!

Schwab continues:

Fueled by more than $600 million in funding from the Bill & Melinda Gates Foundation—a virtually unheard-of sum for an academic research institute—the IHME has outgrown and overwhelmed its peers . . . “It’s quite impossible to criticize or indeed comment on their methods, since they are completely opaque,” says Max Parkin, from the International Network for Cancer Treatment and Research. . . .

Despite such criticisms, the IHME’s dominion keeps expanding—thanks in large part to Richard Horton, the editor in chief of The Lancet, who has put the credibility of the famed medical journal behind it . . . The relationship between The Lancet and the institute was further underscored last year when Murray nominated Horton to receive the $100,000 Roux Prize from the IHME. . . .

Uh oh, Lancet . . . maybe not such a good sign . . .

Also:

Some experts are also reluctant to criticize the IHME for fear of upsetting the Gates Foundation, one of the most important funders in global health and academic research more generally. . . . Yet during the pandemic, the IHME’s early projections proved dramatically wrong—and damaging to public health, some say. But this reckoning has come only because the high stakes of Covid have brought a new level of scrutiny and competition from other researchers and because the institute has had to contend with the emergence of actual data on infections and deaths.

In much of the IHME’s other work in health metrics, these feedback mechanisms do not come to bear in the same way, even though its estimates may be just as influential and, in some cases, just as wrong. This includes its efforts to track hundreds of diseases in the most remote corners of the planet.

That’s an interesting point. You can go far if you have media power and your numbers can’t be checked.

In any case, actual University of Washington epidemiologists don’t like the IHME. See also here.

39 thoughts on “More on the epidemiologists who other epidemiologists don’t trust

  1. Yes, the IHME forecast was obviously wrong even to some of us amateurs. I was informally advising some local hospitalists on what to expect, and had to point out that the time-symmetrical peak IHME was predicting didn’t look anything like the peaks already seen in some countries, which had much longer tails than up-ramps. They didn’t even do the simplest validation procedure- does the theory more or less fit the available data?

  2. I’d be the first person to criticize IHME for not understanding modeling, but the suggestion that they’re somehow responsible for worsening the pandemic – at least any more than the “insiders” who got many things wrong – is laughable. And in WA the Democrat governor relied on them as well, so the implied claim that this was Bill Gates using is wallet to feed Trump with “Republican” information is also ridiculous. Not that I’m defending Gates either. As far as I can see the Gates foundation is mostly frittering money away on overpaid modelers, data analysts, statisticians and all variety of other useless-NGO-hangers-on. The NGO industry as a whole is useless at best and destructive at worst.

    But if we’re to blame some group for the severity of the pandemic, the blame should be squarely laid on the epidemiological **INSIDERS** who stubbornly refused to recognize the aerosol transmission mechanism for almost a year, sending people into paroxysms of surface sanitizing and other knuckleheaded preventative behaviors **even though there is strong evidence that aerosol was a major contributor to the 1918 pandemic and substantial early evidence that it was driving the current pandemic!**

    Whatever the case for epidemiological “insiders” the data analysis and modeling community outside of IHME has proven at best useless, so IHME-bashing serves other purposes as well.

    • Jim:

      I guess Bill Gates is a Democrat so, yeah, I wouldn’t say that he’s funding “Republican” information. I don’t see any implied claim that this was partisan, one way or another. I think the point is that the IHME people are insiders too, they’re just insiders who didn’t know what they were doing and, because of their funding, they were insulated from criticism from others in their profession.

    • “But if we’re to blame some group for the severity of the pandemic, the blame should be squarely laid on the epidemiological **INSIDERS** who stubbornly refused to recognize the aerosol transmission mechanism for almost a year, sending people into paroxysms of surface sanitizing and other knuckleheaded preventative behaviors **even though there is strong evidence that aerosol was a major contributor to the 1918 pandemic and substantial early evidence that it was driving the current pandemic!** ”

      100%.
      If this wasn’t a stats blog I’d write 110%.

      I’m still amazed by the abject failure of most governments to highlight aerosols. I checked the official Australian government guidelines a couple of weeks ago (I live in Brisbane). No mention of aerosols. Social distance. Wash hands. Clean surfaces. I talked to a few friends about it, they had some vague idea that government people might have mentioned it a while ago, but really had no idea of its importance, or the implications.

  3. Am not in any position to judge the validity of this criticism. My only interest in the article, and the criticism, is that, as a part-time job, I build predictive models. My criticism of the article is that it doesn’t provide a guide, or insight, on what fixes IHME should have.

    Should it be more transparent? If so, how are they opaque and how would being transparent help everyone in their field? This is important question because it points to the degree of difficulty of the problems. For instance, if IHME largely agrees with the criticism, and believes it can easily implement the fixes, than the problems are essentially trivial. If, however, IHME disagrees with the criticism and fixes, than the problems are indeed significant.

    One final note. In January of 2020, I built one of the first models of Covid. At the time, the only available data was number of deaths. From this, using, say, nonlinear mixed effects modeling, independent variables were derived to, say, use in SEIR or agent-based modeling. I thought this approach was too fragile. That is, there wasn’t a verification method for the reproductive number. The other option was to use the citizen-journalists of Hong Kong, who were collecting data in mainland China. (There are many problems with this, but I used this data.)

    I do know that IHME only used, in its early models, only data from deaths. How do I know? Because Murray gave an interview with 538, around March or April, 2020, and talked about the difficulty of building a model only from deaths. The subsequent models, of Covid, that IHME made were more accurate. It’s simply inaccurate to judge their Covid modeling from this early model.

    • I may be misremembering, but what I vaguely remember was that IHME assumed that the shape of the pandemic would be a gaussian, they fit the gaussian to the ramp-up of deaths, and then of course the ramp down had the same shape, and went to zero and never increased again.

      That was so laughable that it should have gotten the entire organization fired and replaced within a couple hours.

    • Here’s one of the earliest discussions from this blog: https://statmodeling.stat.columbia.edu/2020/05/05/university-of-washington-biostatistician-unhappy-with-ever-changing-university-of-washington-coronavirus-projections/

      early models were just curve fitting with the second derivative of the log of deaths. To fit a second derivative you require at least a quadratic, so yeah it’s kinda a gaussian shape. They eventually did a lot more, but their early models were so laughable it was like a clown show.

      Carlos Ungil showed how they kept changing their model projections dramatically https://statmodeling.stat.columbia.edu/2020/05/05/calibration-and-recalibration-and-more-recalibration-ihme-forecasts-by-publication-date/

      Curve fitting the second derivative gives us *zero* scientific information. It might be reasonable as a 1 week projection methodology if you somehow maybe needed to move resources around the country towards the places where you expected the most issues… but then it should have been billed as a “1 week projection” or something. it wasn’t.

      IMHO what they did was they front ran the modeling with crap to get publicity so that their real modeling effort which came out months later would have hype and conflict around it. It was a marketing effort, on the basis that “there’s no such thing as bad publicity”.

    • They made the source to their first model available, along with a high-level description, which is why it was known from the beginning that they were just doing a curve fit based on deaths data.

      They combined the weakness of this approach with an insistence on running model projections out for something like six months (or was it eight?). They then backed this up with media appearances and insistence that their results were useful for planning hospital utilization and the like.

      Yes, the Trump administration picked up on this model as it fit their prejudices and yes the IHME team basked in the publicity. But they basked in all publicity. Their quickly established prominence certainly didn’t hurt their position with the Gates Foundation.

      As their model results were shown to be a joke – they’d put out a prediction for deaths that would be surpassed after a few days – they did a total rewrite. They incorporated a SEIR model to drive the spread of infection part, and used their curve fit to generates deaths apparently based on estimates of the CFR/IFR or the like.

      They touted this as an improvement to their early model rather than simply acknowledge the fact that their curve-fitting original approach was sheer garbage.

      No one knows for sure because at this point, though they did publish a high-level description of the model, they closed the source and AFAIK it is still not available. Model performance did improve over the short term, I believe (at least the numbers being generated weren’t laughably low), but they continued with their insistence on projecting forward for months and still are. They’re still pulling numbers out of their rear for things like mask compliance and effectiveness AFAIK when they generate masking vs. no masking projections.

      Meanwhile, most modelers were at least smart enough to limit their projections to 3-4 weeks rather than months … and had a much lower PR budget.

  4. Here’s what I’m wondering about. Maybe someone can help me where I see this wrong.

    I think we should all agree that all models are wrong.

    Models are tools to help us evaluate probabilities.

    They will be wrong.

    Some people will have too much trust in them, thinking that models reduce uncertainty. They don’t. They help to evaluate probabilities. Not quite the same thing.

    On the flip side when models are wrong, as they will always be, people line up to be critical for whatever reason. Often the reason is that the critics want to show they’re smarter than the modelers. Or they want to complain about the political power of the modelers to influence polcy (happens a lot with climate change).

    Sometimes they’re right about that. So then models being wrong is a learning experience about how to, hopefully, make future models less wrong (although they will still be wrong).

    To really progress here we need to move being the right/wrong assessment. It’s not appropriate.

    • Joshua,

      All models are wrong to some extent but some models are wrong to the extent that they are ill-conceived, stupid and lacking in even the most rudimentary face validity. Treating a freshman stats class level exercise in curve fitting as though it is a flawed “model” that somehow will contribute to moving the science forward is some sort of weird false equivalence. The IMHE stuff was junk, not just science that happened to have a few flaws. And many people recognized it as such from the off.

    • All models are wrong; some are useful. Yes, we have to move onto the “useful” assessment.

      So what is the utility of the IHME model? What is the useful application of it?

      Generating media churn, perhaps?

      • Dzhaughn –

        > So what is the utility of the IHME model?

        My guess is that the utility is that their models have been useful, primarily, in demonstrating mistakes to avoid when modeling this pandemic. I’m guessing they’ve made different categories of errors over the course of the pandemic, although this article suggests their basic error was in the sams category as (what it seems to me to be) Levitt’s:

        The IHME model is based “on a statistical model with no epidemiologic basis,” the Annals of Internal Medicine critique argues.

        Although as with Levitt they also seem to have way underestimated uncertainty (CIs too narrow).

        https://www.google.com/amp/s/www.vox.com/platform/amp/future-perfect/2020/5/2/21241261/coronavirus-modeling-us-deaths-ihme-pandemic

        • I don’t know where IHME’s modeling went after the few few months of the pandemic (and don’t care). But initially they basically took a bunch of very likely fiddled data from China at face value and simply “modeled” the pandemic world-wide by overlaying that Chinese time-vs-cases curve onto various countries and ultimately even USA states with just the population numbers and starting point changing for each location.

          It’s the sort of thing you could do in Excel and because of its utter simple-mindedness it is much more appealing to innumerate “leaders” and politicians than messy mechanistic Epidemiologic models. It cuts right to the chase, asserts that it can predict the future with very reassuringly narrow confidence intervals and can be applied right down to the state level on a week-by-week basis. What’s not to love if you’re the sort of person who wants to spout off about “The data shows…” before embarking on whatever politically expedient policies you already intended to do in the first place.

          The only conclusion I can make, if I assume not everyone is a complete doofus, is that they set out to do exactly what they accomplished…become the go-to model for politicians by dumbing it down and getting “actionable insights” as directly and quickly as possible. It’s not science, it’s political hackery.

        • Brent –

          > It’s the sort of thing you could do in Excel and because of its utter simple-mindedness it is much more appealing to innumerate “leaders” and politicians than messy mechanistic Epidemiologic models.

          I get what you’re saying.

          But my question would be whether that approach wasn’t reasonable at the very beginning of the pandemic – as one choice of a way to go for dealing with the uncertainty. That could still be a reasonable choice that within the public domain got viewed improperly. It’s kind of like the Imperial College modeling of a projection contingent on various degrees of mitigation being misused by people who say their modeling was dangerously wrong by ignore those contingencies.

          So for me the timing aspect does become relevant. Maybe at this point it’s clear that no one should model pandemics just based on mathematical algorithms that are essentially uninformed by biomedicine or virology or immunology or epidemiology. Look at all the highly certain nonsense that was promoted about “herd immunity” based on such a limited methodology.

          But one way to learn those lessons was to do it, to model in a particular way, and then find out how inaccurate it was. Sure, even for someone as unsophisticated statistically as I am, the idea of modeling the pandemic without considering how specifics vary across circumstances seemed kind of nuts to me. I was shocked to see so many people, including expert epidemiologists like Ioannidis, effectively ignore confounding variables across context and rely on non-representative sampling to make broadly-applied predictions. But for all I knew at the time, maybe I was focusing on factors that in the end weren’t going to make a big difference.

          I”m trying to focus on what’s the lesson to learn here.

        • Joshua,

          The only lesson be learned (and I don’t think that’s really the right term here) is that the world is full of people ready to promulgate so-called “models” or “estimates” or “predictions” claiming certainty where none exists and that there are plenty of people in positions of power ready to buy that snake oil if it serves to promote their own agendas.

          A statistican can’t learn anything about modeling from the IMHE debacle because it was Fake Modeling. That’s my entire point. It wasn’t good faith effort to make some early progress toward models that might inform policy-making with a genuinely scientific attention to the uncertainties involved. It’s like asking what can Bernie Madoff teach us about econometrics.

  5. Brent –

    Thanks for the response. But here’s my problem.

    During this pandemic I have seen some people (e.g., Michael Levitt) who really understand nothing about epidemiological modeling, and who stare at curves on their screen and confidently predict the pandemic’s future trajectory without really even specifying confidence intervals. Ok. So I don’t put much weight on that. When someone like that turns out to be remarkably wrong (as Levitt has), I have no problem just dismissing their work as bad science.

    But it seems implausible to me that the modelers at IHME would simply fail to comply with statistics practices learned in entry level college courses. I can’t evaluate that myself (never having taken a statistics class). I could take your word for it. But that doesn’t seem very prudent. I have to assume there are statistics experts at IMHE that would disagree with you. Maybe if I saw evidence that isn’t the case… Or maybe if I could see that any statistics expert at the IMHE that disagrees with you is clearly an outlier in the statistics expert community…

    So I’m stuck with a basic view that their modeling was flawed. Saying that their science was junk doesn’t really move the bar for me. Of course, the importance of my view is vanishingly small… In fact, because it seems implausible to me that the statisticians involved would fail a freshman level class because of their modeling, the only thing it does is leave me questioning your viewpoint.

    • To provide some context for your comments on Michael Levitt. He’s a Nobel Prize winner in medicine and a professor at Stanford Medicine. On Twitter, about three months ago, after talking about for a year of the models he’s used to predict Covid, he asked how to find the reproductive number. At first, I thought he wasn’t serious. He was.

      Also, infamously, he predicted, during the spring of 2020, that Covid would be completely gone by the summer.

      If you want to get a flavor for his craziness, simply follow him on Twitter.

      • https://www.statnews.com/2021/05/24/stanford-professor-and-nobel-laureate-critics-say-he-was-dangerously-misleading-on-covid/
        – “There is little as powerful at conferring near-mythical status as the Nobel Prize. MacArthurs, Pulitzers, and Fields Medals all have pull, but for the general public, they don’t have the dazzling name recognition of the Nobel. That sort of reverence is laden with risk. Leave aside the war-mongering Peace Prize winners and the fascist, colonialist littérateurs, focus only on those objectivity-loving scientists, and you’ll still find a number of laureates who’ve gone off the rails.” … do tell.

      • My understanding is that IMHE builds models that results in projections complete with CIs.

        Levitt has made (Twitter) predictions about COVID outcomes (deaths and infections) all over the world, throughout the pandemic, many (most? all?) have been wrong, many very wrong, some by multiple orders of magnitude – and always in the same direction.

        I’ve asked him on Twitter to explain if he’s made an analytical error. My thinking is that if you make many predictions that are very far off, over an extended period of time, pertaining to many different localities, and they’re always off in the same direction, you’ve probably made a fundamental analytical error.

        I have yet to see him describe any fundamental analytical errors.

        • Eccentricity — at some universities — is common, in my experience. Levitt, however, is disturbing. He seems completely unaware of how his actions and approaches are perceived. I actually watched several of his (medicine-related) presentations on YouTube, to see what he’s actually like. There wasn’t any indication that he’s a total nutter.

        • Sam –

          > He seems completely unaware of how his actions and approaches are perceived.

          I don’t think that’s quite right. An aspect of Levitt’s whole schtick is that he makes a big deal of sometimes saying he was wrong, and then repeating that he said he was wrong, as if that should give him a get out of jail free card. His followers on Twitter often say something like “How refreshing it is to see a scientist admit that he was wrong. So inspirational. I wish that they [the ‘lockdowners] would do that like you do.”

          The problem, however, is that as far as I can tell he doesn’t bother to say in a meaningful way what it is he got wrong, or why he was wrong. I have asked him this question (on Twitter) multiple times in many ways and sometimes he’s responded directly or indirectly to my tweets but never actually answered the question. Of course, someone shouldn’t be judged on how they respond to nuts on Twitter – but as far as I can tell he has NEVER actually accounted for any kind of analytical error despite being wrong, very wrong sometimes, all throughout the pandemic on predictions all over the world, ALWAYS in the same direction.

          What does it mean for someone to say that they were wrong but never make explicit any analysis about why or how they were wrong – and then make a big deal of saying what a great person they are for admitting they were wrong?

          That suggests to me a volitional and deliberate rhetorical/PR tactic. Not just cluelessnes or unawareness. He’s tugging on a political string, within the fabric of the “big pharma/public health officals/etc. are hiding stuff from us. We know this because they won’t admit errors, but WE do because were the real scientists.”

          To be fair, I’m not saying it’s an evil or intentionally harmful schtick. I’m sure he’s doing what’s he sees to be the best for saving lives. But it’s still a schtick.

        • It’s an act, my friend. Every actor who amounts to anything knows he’s best when he plays himself. And it’s a hard act to keep up; playing himself. Imagine how much more difficult it would be — say — to play (and succeed) at being a concert-violinist; than to merely be one! But the confidence trickster who’s talented enough to get concert billing at the Kungliga Operan in the Mendelssohn concerto is capable of tricks beyond the run-of-the-mill. It doesn’t come easily; and the real (as opposed to the fraudulent) concert violinist is much, much more comfortable in his shoes. If you ask the fraudster to play a few bars of the Ysaye Humoreske in G flat he’ll play ’em, but break a string (on purpose) or spill a glass of water. Or he’ll play as much of the Mendelssohn concerto as there’s time for: because you see, he knows the bits and pieces of the repertoire and can fake his way through the opening bars of the Ysaye, the Vieuxtemp concerto, or almost anything; but the only thing he can *really* play is the Mendelssohn! It’s hard work; and he’s up with night-terrors because of it. The ordinary workaday-violinist may struggle with motivation and even stage-fright if you put him out there as a substitute; but he’s under no illusion, that to put bread on the table he’s got no alternative but to continue the act; that the great, indifferent and undiscriminating public should continue to judge him for being what his is; that for this public he subjects himself to the most vulgar of all affectations: that of playing the genius!

    • Eric:

      I have no idea about Schwab’s reasoning. I think the point is not that big money is necessarily bad but rather that big money insulates people from criticism. The IHME people got on the inside track with the big money, and this allowed them to build a big operation even though they didn’t know what they were doing, and it allowed them to power forward while ignoring reasonable criticism even from within their own university. Bill Gates can be a good guy; that doesn’t stop him from making mistakes.

      • We have seen researchers/institutions with far less money than Gates-funded IHME doing bad work being insulated from criticism. Focusing on the ‘big money’ part of it seems a bit misguided (if effective at engaging people) and does not get to the heart of the real issue.

        • Fred:

          But why think there’s a single “real issue”? Some researchers are insulated from criticism because of big money, others because of media connections/reputation (for example, Malcolm Gladwell, David Brooks, and that Ted talk sleep guy), others because of scientific prestige such as Nobel prizes, etc. None of these people are fully insulated from criticism—consider for example the criticism of IHME by other epidemiologists at the University of Washington—but they’re insulated in the sense that they can get away with batting away or ignoring criticism and not taking it seriously. I think money is part of this although I agree it’ not the whole story.

    • I think that Gates’ funding of the IHME was driven by sincerity.

      And if covid had not happened who knows? Maybe IHMEs efforts would’ve plodded along a somewhat more conventional path towards the understanding of disease spread, epidemics, costs and the like around the world.

      Covid was an enticement, and a chance for IHME to leap to prominence, and I think Murray et al leapt on it opportunistically. Gates’ role would’ve been hands off, it is his foundation that provided funding but his foundation (much less him personally) doesn’t run IHME. Murray does. Indirectly, large-scale funding does come with expectations and surely the Foundation was looking at IHME with the thought that hey, we’ve given you tons of money, belly up to the bar and show us what you’ve got.

  6. Joshua, the problem is that a simple curve-fitting approach as originally used by IHME is useless and was known to be useless at the time the released their model and began to promote its usefulness. States were using the model to plan for hospital capacities and the like.

    If we could really dig under the covers I think we’d find that the IHME didn’t have an model in place, saw an opportunity with covid to prove their capabilities to the world, and shoved something out ASAP that by necessity would have to be simple, and curve-fitting models are simple.

    At this point, no harm no foul, and if they’d just quietly moved forward with building a reasonable model I don’t think they’d be having to deal with the scorn that resulted from what they actually did. Promote it, encourage its use for planning, media appearances (shades of Ioannidis et al though with better intentions, IMO at least), etc.

    • dhgoza –

      I went back and looked at the other threads here…thought about Brent’s comments, etc.

      I think I may have been convinced.

      I’m reflexively resistant against the “bad faith scientists” narratives because they often make assumptions that seem implausible to me.

      But this may well be an exception.

    • I like how the graph in that article already shows the model prediction up as improbable.

      And the betting post shows that combining several models isn’t necessarily an improvement.

    • Are they seriously going to keep doing this every 6 months? Cases are going to rise in the winter, especially in the northern states. The default assumption is that Sars-2 behaves just like every other coronavirus, which is how we saw it behave last year.

      There must be some reason there is such resistance to formalizing the seasonality by putting it into the actual models, meanwhile almost everyone they interviewed expects to see it.

Leave a Reply

Your email address will not be published. Required fields are marked *