Calibration and recalibration. And more recalibration. IHME forecasts by publication date

Carlos Ungil writes:

The IHME released an update to their model yesterday.

Using now a better model and taking into account the relaxation of mitigation measures their forecast for US deaths has almost doubled to 134k (95% uncertainty range 95k-243k).

My [Ungil’s] charts of the evolution of forecasts across time can be found here.

I haven’t checked Ungil’s graphs—I haven’t looked at the data or code—but, conditional on them being accurate summaries of the IHME’s reports, they tell an interesting story.

Above are a few of the graphs. Lots more at the link.

67 thoughts on “Calibration and recalibration. And more recalibration. IHME forecasts by publication date

  1. What happened to the predicted spike in Wisconsin cases after the April 7th elections? Oh, it only exists for people who ignore the number of tests performed.

    • And further, all that we need to do is treat the illness correctly:

      Patient #1 is a middle-aged Hispanic male from the Tyson packing plant. He has no co-morbidities. Fever and test (positive) a week ago. Admitted 24 hours prior on 6L [liters of oxygen, CF] and quickly progressed to 8L, still hovering low 90% on pulse ox (oximetry – oxygen saturation, CF).
      […]
      By 9:15pm we had him in the HBO unit and ready to start. When he started the treatment he was visibly uncomfortable, and short of breath. We pretty quickly took him to 2 ATA (Atmospheres Absolute – two times sea level pressure which is the equivalent of being 33 feet under the ocean, CF). At 2.0 ATA he was visibly more comfortable and said he was much better.
      […]
      Treated for 60+ minutes of bottom time (bottom time is the total time that the chamber is compressed, CF). He started fidgeting and I didn’t want him to feel trapped, so we started bringing him up sooner than I would have wanted but he was improving and giving a thumbs up. Marathon, not a sprint after all.

      He had to have an oxygen mask on before exiting the chamber. When we pulled him out, he was visibly uncomfortable — through a translator he said he was nauseated. Why? His sense of smell had returned after HBOT and the mask odor made him sick to his stomach!! Alcohol swab did wonders and he was able to put his mask back on for transport back to the ward. He acknowledged he was feeling better, but his daughter left FaceTime as his wife was ill and she was going to bring her to the ER.

      didn’t send this email until I could see him this morning. He was lying with the bed flat. That’s a good sign since you can’t lie flat when you are unable to breathe. His oxygen saturation was 98%. He was still on 8 liters of oxygen, which is what he had been on the day before. (Reminds me that another detail to address is weaning down the supplemental oxygen after HBOT. However, when a ward has 7 COVID-19 admissions overnight, I suspect I will be doing the weaning of the oxygen). Best of all, he was sound asleep and visibly breathing much easier.

      https://carolinefifemd.com/2020/05/04/keeping-it-real-hbot-in-a-tyson-covid-19-outbreak/

      You don’t need to wait for an RCT when you see immediate improvement in the patients condition.

        • About 1375 hyperbaric chambers in the US

          No idea where you got that number but assuming the lowest possible one patient per chamber (doubtful) and 2 hrs per treatment that means without adding any capacity each can treat about 12 people/day. That is enough for 17k patients with severe illness at any given time.

        • Almost all are single patient, used for wound treatment. 2 hrs per treatment 12 people/day? Apparently you see no need to disinfect between uses?

        • Well, once again I have no idea where you are getting your information or how accurate it is. But at least this place has a chamber that fits 12:

          Dr. Harch began talking to doctors in China using hyperbaric oxygen too. Three dozen patients were treated with coronavirus, but it started with two patients who were seriously ill.

          “And they were able to save both of them. They called it ‘cure,’” Dr. Harch said, quoting Chinese doctors.

          So Dr. Harch is now hoping to do a study at UMC, with Louisiana patients who are very sick in the hospital but not yet on a ventilator, because he says their conditions are very similar to patients who suffered from the Spanish flu.

          “There’s a period of time where they begin to deteriorate and they’ve got about 12 hours before they end up on a ventilator. What we’re trying to do, is see if we can intervene and make a difference in that outcome, because once they’re on the ventilator, they have a very bad prognosis,” he explained.

          If his study is approved and funded, patients would be treated five times for an hour and a half each. And as many as 12 could be treated at one time in the large chamber.

          https://www.msn.com/en-us/health/medical/new-orleans-doctors-hope-hyperbaric-chambers-could-save-covid-19-patients/ar-BB12zMOw

          I have no idea how long disinfecting takes (or whether that is really necessary after most treatments) so added 30 minutes to the 1.5 hr protocol.

        • Group chambers are rare, I’m guessing they have one available because there’s a lot of oil production in the Gulf of Mexico and that industry employs a lot of divers.

          But good for them. I hope they have success with their study.

          If they do, and if treatment protocols can be widely applied, then indeed reducing CFR might make sense.

          But I’m not sure why modelers should do so before the Chinese study is replicated. I mean, hydroxycholorquine turned out to be a bust, surely modelers were prudent to not to base scenarios on the belief that it was going to be the magic bullet some claimed it would be.

        • By no means should anyone create a prediction that is entirely based on a hoped for magic bullet. That’s more or less Trumps every response…

          But we should absolutely show the *range* of possibilities given that fatality rates could either increase or decrease. New treatments could come online, and also the virus could mutate to a more severe form, particularly if there is selective pressure for more severe forms, for example if the main route of spreading is people isolating at home who then get very sick, and go to the hospital and spread it at the hospital.

          So, since this is a potentially varying variable, we should include it in the uncertainty, thereby widening our uncertainty intervals. The dominant message in all of this pandemic for me is: people who don’t acknowledge uncertainty properly made bad predictions.

        • Daniel:

          I think there’s value in your point but I’d argue against naively pooling too many sources of uncertainty that are qualitatively different. The uncertainty of “these facts about the virus are unknown” is different from the uncertainty of “well maybe we will come up with a new treatment”. I mean, as an absurd extreme we might end up arguing against developing those new treatments if our model outputs, inclusive of new treatment uncertainty, winds up claiming we don’t know if the virus will be serious in the future or not.

          See for example the people who use conditional model adjustments based on including the impact of social distancing to claim that this means the model shouldn’t have implied social distancing was worthwhile.

        • Daniel:

          “But we should absolutely show the *range* of possibilities given that fatality rates could either increase or decrease … the virus could mutate to a more severe form”

          How does one show the range of possibilities regarding the virus mutating into a more virulent, or less virulent, form? I don’t think there’s any relevant data available that can be used for this purpose (I know there’s some data on mutation rate per se).

          What are the odds of it mutating into something combining the deadliness of SARS-CoV-1 with the asymptomatic infectiousness of SARS-CoV-2? Or into something that antibodies to current strains aren’t effective against? Or, of course, to a milder form?

        • Zhou, I think your points are valid about the *presentation* of uncertainty.

          If we create a model which incorporates all known kinds of uncertainty, there is no reason we need to display its predictions marginalized against all forms. For example, we can condition on “assuming the virus stays as severe as it does” and then show those results, and then we can instead say “assuming the virus reduces in severity to the lower half of our plausible range”… and show those results… and same for upper half… or whatever

          @dhogaza

          Apparently there’s a 100% chance the virus will mutate to a more infectious form: https://www.latimes.com/california/story/2020-05-05/mutant-coronavirus-has-emerged-more-contagious-than-original

          The general process of mutation has been studied in many viruses, I assume if you convened a panel of virologists you could certainly get informed priors on mutation towards more infectious and less severe forms. There wouldn’t be agreement but there would be a spread of possibilities you could incorporate into models, and there are known mutation rates from extensive genetic sequencing studies of this virus… so it’s certainly possible to incorporate.

        • Also see this report of another mutation detected which mirrors a similar process seen in late stages of SARS epidemic: https://nypost.com/2020/05/05/new-mutation-indicates-that-coronavirus-might-be-weakening-study/

          I think the point is that it’s essentially 100% guaranteed the virus is mutating, the question is how likely is it for a mutation to both decrease the severity and increase the infectivity simultaneously… which would alter the pandemic in such a way that everyone could get the thing and have relatively few serious cases (like a cold or the flu).

        • It seems very reasonable to assert that there would be more transmission if you have large public gatherings of people, sure. The extent of this transmission would depend on a variety of factors, and a surge in infections might indeed be lost if the election-related infection is centred around young people who don’t tend to go get tested. Where is this claim that you have to ignore testing rate to make this judgement?

        • It seems very reasonable to assert that there would be more transmission if you have large public gatherings of people, sure.

          Yet, that didn’t happen. So looks like at least one of the assumptions did not hold. Public policy is being determined by the number of tests performed:

          Gov. Tony Evers’ plan to reopen the state calls for a downward trend in positive cases for 14 days.

          https://www.wisn.com/article/wisconsin-sees-spike-in-new-coronavirus-cases/32361185
          https://i.ibb.co/LpSq3Wz/wisconsin.png

          It is time to stop listening to the people who have been wrong every single step of the way because they listen to models that ignore the two most important factors.

        • Maybe people were able to appropriately social distance while casting their vote. Maybe voting sites with high volumes of people corresponded to communities with low disease prevalence. Maybe the fact that there’s a four week gap between infection and death (plus delays in reporting) means that we shouldn’t expect a spike in deaths until around about now, and the recent uptick in the Wisconsin deaths is the signal we were expecting.

          Abele’s statement was not a prediction about effect size. There are obviously big uncertainties – I don’t see how this has really anything to do with the usefulness of testing as a covariate, and the implication that considering testing as a covariate would necessarily make you think doing in person elections aren’t risky.

        • I mean, okay, do you want to make some predictions for what the daily death numbers will looks like in Wisconsin in the next 2 weeks or so?

      • I’m tired of seeing these models that ignore the role of testing in determining number of reported cases/deaths and the role of treatment in determining the death rate. No one should be making any decision based on these models that ignore the two most important factors that determine the data they are fit to.

        Just that you failed to make the connection shows how bad it is.

        • I think everyone here likely agrees that the IHME model sucks.

          And this newly released model of theirs isn’t just an iteration, it’s a total rewrite. A new model, effectively. Maybe the new model won’t suck as bad as the old model. Maybe it will suck more. We shall see.

          So you think that ignoring “the role of treatment in determining the death rate” is wrong.

          What do you think should be done, then? Modelers should ignore current CFR dates, but rather lower them drastically because you think there’s a hypothetical magic bullet in the form of hyperbaric chamber treatments?

        • What do you think should be done, then? Modelers should ignore current CFR dates, but rather lower them drastically because you think there’s a hypothetical magic bullet in the form of hyperbaric chamber treatments?

          You run different scenarios that include various lower future CFRs (I don’t think we need to do higher)… this is all common sense.

        • The IHME model adjusts CFR rates based on testing data that’s available, i.e.

          “To date, we have found as testing rates double, cases increase by an average of 22%. We then use this relationship to adjust case trends which then inform our death models, a vital step toward ensuring a more accurate representation of COVID-19 epidemic trends.”

          Now whether or not their methodology is sufficient, when you say

          “I’m tired of seeing these models that ignore the role of testing in determining number of reported cases/deaths”

          It appears that you’re tired of something that’s not true for the model under discussion, at least. The IHME model, for all its flaws, is reasonably well documented on their site, so you could’ve easily found out that they don’t ignore this on your own, if it is really a source of concern for you.

          Thus my focus on the second part of your stataement:

          “the role of treatment in determining the death rate”

          Current treatments of course are reflected in current CFRs, though there’s a time delay as to when the effect of new treatments shows up in the data since cases don’t die or recover immediately.

          I suppose that they could run projections by varying the CFR based on hypothetical future treatments that will lower it, but I don’t see how that would be particularly useful for planning purposes.

        • This has to be Carlos Ungil. No one else on this site posts so much data and quotes without providing sources.

        • The source I used for pointing out that the IHME model does adjust case rate (and therefore CFR) based on testing data, thus refuting your claim that they ignore it, is implied by this part of my comment:

          “The IHME model, for all its flaws, is reasonably well documented on their site”

          Do I need to provide you with a URL, or do you think you can find their documentation yourself? If you can’t, I’ll be more than happy to give you the URL. That was a direct cut-and-paste from their documentation.

        • Since you’re complaining about my not providing a source for my data and quotes, I missed the part where you provided a source supporting your false assertion that the IHME model – the one under discussion – ignores the impact of testing on CFR.

        • I see, it is here: http://www.healthdata.org/covid/updates

          But (afaict) not here: https://www.medrxiv.org/content/10.1101/2020.04.21.20074732v1
          Or here: https://github.com/ihmeuw-msca/CurveFit

          Correcting reported cases to account for scaling up testing. As more locations scale up testing for COVID-19, many places may report increases in cases; however, such increases usually reflect an increased detection of existing cases rather than a true rise in COVID-19 infections. Where data are available, we aim to adjust trends in reported cases based on the relationships between testing per capita and test positivity rates. To date, we have found as testing rates double, cases increase by an average of 22%. We then use this relationship to adjust case trends which then inform our death models, a vital step toward ensuring a more accurate representation of COVID-19 epidemic trends. Other COVID-19 estimation updates do not appear to account for this relationship between reported cases and expanded testing efforts; this could lead to very different conclusions about future epidemic trends.

          So I can’t figure out what exactly this means, but it does not sound like what I’m talking about. They should not be “adjusting case trends” for the rate of testing. They should be modeling the actual cases and then modelling testing of those cases to get what we see in the data and expect to see in future data.

          Also, “as testing rates double, cases increase by an average of 22%”. This is a confusing way of putting it. Doubling is an increase of 100%. So it means they saw that on average 22% of tests are positive regardless of the number of tests. I don’t know what data they based that on but I do agree that cases are a near constant function of the number of tests performed across time and space.

        • Thanks for looking for their documentation. The latest information is the updates page.

          The MedRXiv paper’s date is 4/21 … it doesn’t describe the current model rewrite (and in many ways it is really a new model, you can’t switch a key component from curve fitting to a dynamic SEIR approach and just call that an iteration, IMO!). In fact it predates at least one previous model iteration and I think at least two, not to mention this recent near-rewrite. So it is ancient.

          The source you found on github is also for an old version of the model. Note the last commit timestamps for the various modules. So none of the code for the SEIR component, or how they use their curve-fitting “death model” to inform their beta and I estimates for that component, or even the updated parameterized function they use to fit to the death data, is there. Again I

          It is totally fair for you to argue that their methodology for incorporating the effects of increased testing on CFR sucks. I’ve not said it doesn’t (and I’m not really the person to make that judgement). I only complained that you were wrong to state that they ignore that issue, when they clearly don’t.

        • I only complained that you were wrong to state that they ignore that issue, when they clearly don’t.

          Part of your post got cut off it looks like, but I the only evidence they don’t is a vague statement on a webpage that appeared recently.

          And using the average number of cases per test to “adjust case trends which then inform our death models” is not an appropriate way to deal with the one of the most important determinant of the numbers you are fitting to and predicting.

          But it is good to see this very important thing is moving from ignored to afterthought.

        • Anoneuoid –

          > I’m tired of seeing these models that ignore the role of testing in determining number of reported cases/deaths and the role of treatment in determining the death rate. No one should be making any decision based on these models that ignore the two most important factors that determine the data they are fit to.

          Accounting for testing in some fashion seems pretty basic to me. Dhogaza indicates below that they do.

          As for accounting for treatment efficacy, it seems a lot less straight forward. For example, I’m reading a whole lotta rightwingers complaining loudly that treatment with hydroxychloroquine is being ignored in the development of public policy to address the virus. Well maybe it is to some extent, but where do you draw the lines as to what to include in your modeling? Seems to me that waiting until you have solid evidence based on RCTs and studies conducted with control groups of some sort is not all together unreasonable.

        • I have a pending post about the testing thing. For the treatments just assume for different scenarios the death rate drops by 2x, 4x, etc and show the results. Assuming that death rates will not drop is the bad assumption. They already are. Also you can find nurses shipped into NYC from elsewhere claiming a lot of the deaths there are from medical errors.

          https://m.youtube.com/watch?v=CvhTQV5FNUE&feature=youtu.be

        • That is the number of cases divided by number of deaths at a given timepoint. I tried to explain to some people here that is what CFR referred to in the context of covid a week or so ago and ended up in a dumb argument.

          But yes, when deaths lag cases and number of new reported cases flatlines then CFR rises over time. That does not mean that the later patients are surviving less often.

        • Anoneuoid: correct, when the cases flattens, then as time goes on, the CFR approaches a stable and correct value. When the cases change through time especially rapidly, then the CFR at a point in time underestimates the CFR since most cases are in the early stages, pre-death.

        • the CFR approaches a stable and correct value.

          That isn’t what happening in the US. It is fluctuating more wildly and seems to be decreasing: https://i.ibb.co/HF9nNzx/covid55.png

          But there is no stable and correct value if the treatment changes to be more effective and this is slowly adopted across the country, etc.

        • Yes, not to mention that different populations can be infected at different times, and ascertainment can change through time in different places (for example states where almost no testing was available suddenly get tests etc).

        • ” For the treatments just assume for different scenarios the death rate drops by 2x, 4x, etc and show the results.”

          What treatments are available that aren’t in widespread use that have a real chance of lowering the death rate by 2x to 4x in the next several weeks?

          I assume you mean an additional 2x to 4x on top of the reduction in CFR due to increased testing.

        • What treatments are available that aren’t in widespread use that have a real chance of lowering the death rate by 2x to 4x in the next several weeks?

          HFNC, NIV, HBOT, HDIV. The latter two (hyperbaric and vitamin c) can be done in addition to mechanical ventilation but the first two are replacements for it. The only problem with hyperbaric treatments is we need more chambers, and the problem with the vitamin C treatment is ideally we could measure the blood levels but from what I hear no one wants to contaminate their HPLC. Someone with the money should donate one that blood samples from around the country can be sent to for start.

          I would also say that for the high flow and bipap/etc treatments they should probably start being more careful not to blast them with oxygen at the beginning because by the time the patients get there their tissues are probably adapted to low blood oxygen. Just like you don’t want to ascend/descend a mountain too fast:

          https://dolmatours.com/reverse-altitude-sickness/

        • Anoneioid –

          > Also you can find nurses shipped into NYC from elsewhere claiming a lot of the deaths there are from medical errors.

          Sure. But go to any rightwing website and you’ll find claims that hydroxychloroquine has had a miraculous impact. The problem is that there’s no way to know how individual perspectives or anecdotal observations inform at a more generalized level.

          I wouldn’t hurt to have some generic projections based on different scenarios – why not project from models that incorporate uncertainty about CFR into the CIs of the models…except there’s no particular reason to think that they’re more informative than the CFR currently being used for modeling.

        • Sure. But go to any rightwing website and you’ll find claims that hydroxychloroquine has had a miraculous impact

          If you read the clinical papers like 80% of the patients in the hospital are already getting it. Whether it is a good dose or timing I don’t know. Obviously, given the presumed mechanism, earlier would be better. But if you end up in the hospital, you will probably get HCQ. Eg, here is one of 91%: https://www.atsjournals.org/doi/pdf/10.1164/rccm.202004-1163LE

          Usually it is not even worth mentioning in the text apparently.

        • 91% of 66 patients in that paper, which was actually studying the efficacy of putting ventilated patients in the prone position. Nothing there about dosage nor is there any claim of effectiveness for the drug.

          Nothing there to support claims that “hydroxychloroquine has had a miraculous impact”, which is what Joshua was talking about.

        • 91% of 66 patients in that paper, which was actually studying the efficacy of putting ventilated patients in the prone position.

          Yes, like i said you will see the use in the tables of random clinical papers but not discussed in the text.

          And you totally missed the significance of what is reported in that paper. 34% with smoking history? 83.3% survival of patients on ventilators for a median of 16 days? Only 2% received supplemental oxygen before being put on the ventilator…

  2. Carlos: I think these charts would be clearer if there’s a better display of which parts of the line is predictive. For example you could use different colours or shade the background differently.

    • The red line showing actual values since the date of publication gives an indication of where the prediction started. But it’s true that as the number of updates has increased the charts are becoming narrower and it’s not so clear. I’ll look into adding some additional hints next time, thanks for the suggestion.

  3. Here is their model for New York: https://covid19.healthdata.org/united-states-of-america/new-york. I think this is based on the JHU data here https://github.com/CSSEGISandData/COVID-19/blob/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_deaths_US.csv.

    On 04/29 the graph on their model show: 23,319 deaths cumulative
    On 04/29 the JHU CSV says: 23477

    So I’m not sure its exactly the same data? but it’s a pretty small disagreement.

    If I take their “forecast” for 05/04 (yesterday).

    On 05/04 they forecast between 25,400-26,100 deaths cumulative
    On 05/04 the JHU CSV says: 25000

    Am I comparing the wrong datasets, or is this a time lag thing? Otherwise it seems like a 3 month forecast which falls out of its confidence interval 5 days in.

    • Hi Quantum: your trouble with the comment box notwithstanding, I think you’re right on that this model fails to predict more than a few days in advance, that’s what these graphs seem to show. It also seems to show that if you just wanted 3-4 days in advance you could extend a line from the last 5 days and it’d predict the next 5 days just fine. The issue seems to be that this model *insists* on a flattening of the curve almost immediately in time, and that just didn’t happen.

      • Daniel:

        “The issue seems to be that this model *insists* on a flattening of the curve almost immediately in time, and that just didn’t happen”

        Have you seen this in regard to the previous model iterations and the current model? It’s from their update documentation …

        “…we have increased the number of multi-Gaussian distribution weights that inform our death model’s predictions for epidemic peaks and downward trends.As of today’s release, we are including 29 elements, a substantial increase from our original seven and then 13 (which was introduced for our April 17 update). This expansion now allows for longer epidemic peaks and tails, such that daily COVID-19 deaths are not predicted to fall as steeply as in previous releases.”

        “Overall, these modeling improvements have resulted in considerably higher projections of cumulative COVID-19 deaths through August, primarily due to longer peaks and slower declines for locations that have passed their peaks.”

        So the parameterized function they previously used in their curve fitting “death model” couldn’t generate longish flat peaks or longish, slowly declining tails for their daily deaths projections regardless of the data. At least that’s how I’m reading it.

        • Also, two of them have mentioned to the press that premature re-opening by states is an important factor in the near-doubling of their death projection.

          Murray did mention the longer tails as a factor in the response I saw. The other team member didn’t, just the relaxation of distancing having driven the rise.

          Compare that with “primarily due to longer peaks and slower declines for locations that have passed their peaks”

          I think they’re trying to deflect attention away from just how badly the previous version(s) of the model sucked. Though with every previous iteration they promised us that it had improved and was useful for planning for hospital bed capacity etc.

        • Thanks for this. I’ve been more or less ignoring their models because my impression was they had zero chance of doing a good job. Carlos’s graphs indicate that I was more or less correct. The fact that they’re iterating and improving I guess is good.

        • Well the “improving” bit is yet to be seen :)

          Seriously, though, this is more or less a new model, with a SEIR component now bolted onto the side of the curve fitting “death model”. Closer to a rewrite than an iteration.

          They’re definitely in the camp that believes that increased complexity predicts improved performance :)

          I think we can say that it will either suck more, less, or the same as the early model architecture …

  4. Here is their model for New York: https://covid19.healthdata.org/united-states-of-america/new-york. I think this is based on the JHU data here https://github.com/CSSEGISandData/COVID-19/blob/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_deaths_US.csv.

    On 04/29 the graph on their model show: 23,319 deaths cumulative
    On 04/29 the JHU CSV says: 23477

    So I’m not sure its exactly the same data? but it’s a pretty small disagreement.

    If I take their “forecast” for 05/04 (yesterday).

    • (Continued)

      I see a prediction for between 25,400 – 26,100 deaths cumulative
      Whereas the JHU CSV reports 25,000 deaths

      Am I comparing the wrong dataset, or making some arithmetic error? Because otherwise it seem like a 3 month forecast which fell out of its prediction range 3 months in.

  5. Does this model take into account lags in death reporting? Some states are finding nursing home deaths, etc. that were missed early on, and there are lags in reporting more generally.

    I see it smooths to account for day-of-week effects, but it seems to me (not an expert) that lags in reporting could easily also affect the shape of the curve; if deaths early on are moved later, it could make a steeper curve (sharp peak and rapid decline) look like a flatter one, as some of the deaths from the “peak” days are moved into the “decline” days.

  6. Note that the main purpose of the IHME model was to predict hospital resource usage, not death count. It’s been even more of a trainwreck for that purpose than it has been for predicting deaths. When I first checked it on March 28 right after release, it was predicting NY State ICU usage that day of 5,400 (with lower bound of and upper bound of 3,900 and upper bound of 7,900) — and actual usage that same day was 2,037. So I immediately wrote it off. (Well, I actually sent them an email asking about the discrepancy and sharing the actual data from Cuomo’s briefing.)

    All that said, I don’t think it’s necessarily fair to cast the latest update reflecting eased distancing as a recalibration. The assumption that distancing would be put in place and remain in place was simply the policy that they modeled — they stated this all up front. Their predictions have been garbage, but it’s fair for them to change their policy assumption as the actual policy changes.

  7. Does anyone know of similar plots being made for projections of other models over similar time (non IHME ones). I’m interested in knowing how other models based on other methods are doing compared to the IHME one.

  8. I’ve been watching this one from U Texas:

    https://covid-19.tacc.utexas.edu/projections/

    They are also curve-fitting, and they assume that the mobility data they’ve gathered for the previous seven days will persist in the future. They only project 2-3 weeks out, because they recognize the severe limitations of what they’re doing and that long-term projections using their approach would be bogus.

    Why do I like them? Because they very openly state the limitations of their model and make no effort whatsoever to oversell it. I think they deserve some appreciation for this.

    Here is what they say about their own model:

    “Can your model tell us what would happen if social-distancing measures were relaxed?

    No. Our model explicitly assumes that social distancing behavior remains at the levels we’ve observed over the last seven days of data. If that doesn’t happen, you can throw our model’s projections out the window beyond about ten days in the future. Why ten days? Because that’s when we’d expect to see the very first deaths occurring if a renewed wave of transmisssion started today. The vast majority of all deaths that will happen over the next ten days will happen to people who’ve already been infected, implying that all the relevant social-distancing behavior over that time frame has already been observed.

    A bit more detail on this point: our model is a purely statistical “curve-fitting” approach, not an epidemiological model that tries to describe the process of disease transmission itself. As a result, our model somewhat restricted in the kinds of death-rate curves it can describe. Empirically, it seems to be effective at describing a single peak in the death rate that has been mitigated, to some degree or another, by social distancing. But it cannot account for multiple peaks in the death rate driven by distinct waves of COVID-19 transmission. To predict what would happen across multiple waves of transmission — the kind of pattern that epidemiologists would expect to see if/when social-distancing measures are relaxed — you really need a model with an underlying “epidemiological engine,” such as the SEIR models you might have read about.”

    Now if the IHME people had been this honest and straightforward about their model limitations, they might not be catching quite so much flak. All of the above applies equally to their pre-May 4th models.

Leave a Reply to Anoneuoid Cancel reply

Your email address will not be published. Required fields are marked *