“Fake Facts in Covid-19 Science: Kentucky vs. Tennessee.”

I’m writing this on 24 Apr 2020. I’ve been posting coronavirus items immediately and pushing previously scheduled material to the end of the queue (currently Oct and Nov). But this one is already forgotten so I might as well put it in lag. When it appears, you can read it and put yourself in the mindset of panic, misinformation, and politicization that was early 2020.

David Austin points us to this post by Ulrich Schimmack entitled “Fake Facts in Covid-19 Science: Kentucky vs. Tennessee.” Schimmack writes:

An idealistic picture of science views science as an orderly process where disinterested, objective experts look at crystal clear evidence to find the truth. In reality, science is a lot messier….

One important question for all countries is how effective social distancing is and how important it is to shut schools, close non-essential businesses, and to enforce quarantine and social distancing with laws. In the United States this question is filtered through a political lens with implications for the upcoming 2020 election….
As explained in a newspaper article, this graph was made by Stephanie Jolly from Kentucky and shared with friends on social media (Vergano, 2020). The problem is not Jolly’s graph, but the uncritical sharing of the graph as if it presented hard evidence that imposing stricter social-distancing measures earlier has a strong influence on the spread of Covid-19.

Although Tennessee reported more positive tests of Covid-19 there is no evidence that the virus is more prevalent in Tennessee than in Kentucky. The differences are mostly due to differences in testing. Thus, there is no evidence that differences in government rules influenced Covid-19 spread. As a result, the comparison of these two states cannot be used as a fact to support claims that states that issued stay-at-home orders later or not at all have a higher prevalence of Covid-19….

Science is a lot harder than collecting some data, making a graph, and telling a story about it. A big part of science is to check facts and to correct claims that turn out to be wrong. A key characteristic of science is that it is self-correcting, but often the process of self-correction is painfully slow. With Covid-19, we don’t have time. So, we need to speed up the self-correction process just as much as we need to speed up the process of hypothesis generation and testing. The best evidence that social-distancing is working – if anybody is doubting this – is that the spread of Covid-19 has slowed down all over the world, including the United States and disaster areas like New York, New Jersey, Michigan, and Louisiana. The reason Tennessee didn’t see more cases is that individuals enacted social distancing earlier than it was mandated by the state government. More data on individuals’ responses to Covid-19 will help to see how important governments’ response to Covid-19. Unfortunately, this will take time and we may have to make decisions without conclusive scientific facts, but at least, we can avoid making decisions on the basis of false facts.

I haven’t looked into this myself, but Schimmack is usually pretty reliable. And if he got this right, it’s a good reminder of the way that false facts can spread in social science. (If he’s wrong, it’s a good reminder of how difficult social science can be, especially in real time.)

30 thoughts on ““Fake Facts in Covid-19 Science: Kentucky vs. Tennessee.”

  1. This reminds me of the still very open question regarding whether elementary school kids are “super spreaders”, or not?

    Understanding this one is going to be very complicated.

    For example, many teachers have quit rather than go back to work in schools; what does this say about the behavior of those who remain?

  2. While this doesn’t tell us what the numbers really were back towards the end of April, as of today Tennessee has a cumulative per capita death rate that’s 1.48x higher than that of Kentucky.

    I don’t have the time series data from John Hopkins on the little laptop I’m using here, but if I were home I’d go look at the per capita cumulative death rates towards the end of May (to account for the roughly four week lag between case identification and death). That would give a reasonable indication IMO as to whether or not the level of infection was really higher in Tennessee at that time or not.

    • You can not simply compare per capita fatality. This is the simpletons version of “research.” We know 2 things… 1) This virus has a much greater mortality in individuals whom are.. aged, with co-morbidities and residing in long term care. 2) We know that no 2 areas have the same demographics. Having these 2 “knowns” should tell any person willing to do genuine research that you can NOT compare simple per capita mortality statistics. Always compare stats BY AGE. If one city happens to have a greater population that are in long term care, naturally that city will have a higher per capita fatality rate. Same is true when you compare nation to nation. You simply cant say x per capita and think you have found something.

  3. > When it appears, you can read it and put yourself in the mindset of panic, misinformation, and politicization that was early 2020.

    Oof this line really makes me realize how little progress we’ve gotten on the misinformation and politicization front since the early days. Or at least that’s what my memory says – someone correct me if I’m wrong. There does seem to be a lot less panic, though I’m not sure if that’s a good thing or not.

    A few unstructured thoughts on this article:

    – There’s a generalizability issue here. Whatever conclusion you want to make about the effect of stay-at-home orders on virus prevalence is conditional on these two states and this time. For example, if you believe there was no effect because people were already physically distancing prior to the stay-at-home orders, that doesn’t say anything about situations where people are currently not physically distancing. It’s also conditional on this time period where most people were aware of the virus and many were fearful of it. In other time periods e.g. now, people may be less fearful or aware of transmission rates and so a government order can put the virus back on these people’s radars.

    – Schimmack makes a conclusion himself off of some additional data related to mobility, testing, and deaths. However, similarly to how he criticized people for concluding too much from one graph I think he also may be concluding too much based on the data he looked at. It looks like either some of his graphs were removed or they’re otherwise not appearing on my screen. But it didn’t seem from his text that he was taking lags in death data into account. He also did not account for heterogeneity in infection rates, especially by age. Maybe stay-at-home orders influence mainly young people (where the death rate is lower) so the policies did indeed lower transmission while not lowering deaths.

    – I’m trying to compare data between [Tennessee](https://www.nytimes.com/interactive/2020/us/tennessee-coronavirus-cases.html) and [Kentucky](https://www.nytimes.com/interactive/2020/us/kentucky-coronavirus-cases.html) in the months following the time of the blog post. It’s a bit difficult because the graphs are on different scales and not population adjusted. It does seem though that the two states performed roughly similarly early on but Kentucky went on to do much better. On one hand, with an exponential disease, small differences early on can lead to sizeable differences later on. On the other hand, a lot happened besides the timing of stay-at-home orders in April onwards so there’s a ton of confounding.

    • >>Oof this line really makes me realize how little progress we’ve gotten on the misinformation and politicization front since the early days.

      I had exactly the same thought.

      >> There’s a generalizability issue here

      Yes. One thing I wonder is if mandates might work poorly in some parts of the US (especially if not enforced) due to the strong individualistic aspect of the culture in these places. IE do we know that more people will wear masks if masks are mandated (but not enforced/weakly enforced) than if masks are recommended but not mandated?

      I have seen people here in TX “wearing masks” on chins/ears so they technically ‘follow the rule’ but there is obviously no actual benefit…

  4. Are people still arguing about whether or not there is a relationship between the stay-at-home and the fall in case rates?
    The “debate” is like the preposterous anecdote in Kant, who describes two who “milk a he-goat: one does the milking and the other holds the bucket”.

    • There are still people arguing that if someone has low vitamun c levels and symptoms of scurvy they should not get vitamin c until there is an RCT.

      For masks and ventilators and lockdowns they dont require that though.

      • I think the English navy figured that one out nearly 200 years ago.
        But I don’t see the connection here.
        Of course that’s probably because I don’t have three letters after my name.

        • Everything is very low when your body is under stress. It could be viral/bacterial attack, extreme exhaustion (endurance sports) etc. and in those cases it makes perfect sense to supplement higher doses.

          That however doesn’t mean it will magically cure all our ills, bake us a pizza and tuck us in bed while singing a lullaby.

          I wish it was that easy, but if wishes were horses…

        • I believe the argument is that for some things the effect is obvious enough that a rigorous study is not really required to show an effect… but that public health agencies are inconsistent in applying that principle (being readier to approve things like mask mandates than things like vitamin D supplementation).

          I don’t know that I agree though.

          I mean the general principle is true, the RCT “gold standard” is I think overemphasized, and similarly statements that e.g. ‘we don’t know if COVID infection provides any immunity’ seem to totally ignore prior biological knowledge – but there is I think decent evidence that masks help at least somewhat.

        • I think I see the point you make and to that extent, I am in complete agreement. The obsession with the “gold standard” experiment, especially in this time of emergency — to my eyes — looks like a variety of superstitious behavior. It is a side-effect of the absurd orthodoxy according to which “evidence” has become a term of art, redefined so that only a certain class of preposterously circumscribed experimental setups merit consideration. The result: the inane churn in the supposed expert guidance, with respect to — say — masks. Much of it was a symptom of anxiety among those forced by events to lead during a crisis; but who’d never signed up for such exposure. The habit of hedging one’s assertions runs deep.

          Now it was also the case, that these heavily qualified “advises” also took on that exaggerated tone of high-church agnosticism, when they found themselves in an even more unpracticed position: having to rebut an ocean of crackpot certitudes on one or another related subject. There is an old chestnut: which cautions against responding too literally to lies — the act of doing so merely give the lie additional circulation.

        • >>having to rebut an ocean of crackpot certitudes on one or another related subject.

          Yes, definitely… I think in some cases (certainly the “we don’t know if prior infection provides immunity”) an attempt to argue against an overconfident statement was read as an overconfident/false statement of the opposite extreme.

          IE WHO’s statement on ‘immunity passports’ ended up sounding more like “we don’t know if antibodies do anything” rather than “we don’t know the degree and duration of immunity”.

          Similarly for some of the claims about serology/IFR… the data we have now seems to rule out both the highest and the lowest claims.

        • The WHO recommended putting anyone whose spO2 dropped below 94% on a ventilator based on nothing at all. As a result massive efforts were made to build ventilators and tens of thousands of patients were intubated that unnecessarily suffered and died in the most expensive and drawn out way possible. Many were in comas for weeks taking up space in the hospital doomed to death.

          To repeat, they recommended the most dangerous and expensive intervention based on no evidence at all and this was parroted by various health authorities like the DoD.

          Yet, there is “no evidence” for the safest and cheapest intervention of correcting a severe vitamin deficiency with the vitamin. And now it’s been 11 months and still almost no resources have been devoted to checking if maybe patients would be better off when not deficient.

          Also, vitamin c and vitamin d are both important. Both are commonly depleted in sick people (both are antioxidants that get depleted in injured/diseased tissue undergoing oxidative stress). But as vitamin C is the terminal extracellular antioxidant that is also a coenzyme for collagen synthesis (most common protein in the body) it’s best to focus on that first. In reality all these deficiencies should be corrected though.

        • I think the major thing against vitamins is that vitamins, especially vitamin C, have been claimed as a miracle cure for all sorts of random things for 50+ years. So doctors/public health types are coming into it with lots of skepticism. (I am not a doctor or otherwise expert at this, but that’s certainly how I feel.)

          I’ve also heard that the vitamin deficiency may be a result of COVID rather than a pre-existing risk factor, but only at an “internet rumor” level.

          The excessive focus on ventilators early on though was *definitely* problematic.

  5. “…as if it presented hard evidence…” I’m not sure what the technical definition of “hard” is in this context, but the graph is definitely evidence.

    “…there is no evidence that differences in government rules influenced Covid-19 spread.” Of course there is. There’s also evidence against it.

    “…the comparison of these two states cannot be used as a fact to support claims that states that issued stay-at-home orders later or not at all have a higher prevalence of Covid-19.” Of course it can. It can’t be used as *proof* but it can be among the factual observations with which one begins to build a case.

    I agree it would’ve been more responsible if she’d attached a caveat, something like: “*This is only one variable among many that are relevant and does not adjust for the others.” But it’s pretty hyperbolic to call the graph “fake facts,” especially when there are genuinely fake facts being actively spread by many (even back then). These are actual facts that could be misleading, depending on context. Suppose, for example, I showed you this graph and said, “This is only a correlation, and we hope results from scientific studies will come soon, but you can see the risk of not social distancing may be large.” That’s not much different from the public arguments doctors circa 1950 made for quitting smoking.

  6. FYI/FWIW

    I wrote the blog post a long time ago when a simple graph that wasn’t even meant to be serious science was passed around to back up substantial claims.

    Showing that the evidence at that point in time for these two states did not show what it was supposed to show has no implications for any serious investigation of the influence of policies or cultural factors on the spread of Covid-19.

    Clearly we have much better data today. I stopped following Covid-19 somewhere in July when it seemed pretty obvious which countries and what policies are working and which are not. The US as a whole and red states in particular are a disaster. A reasonable standard of comparison is Canada, your friendly neighbor North of the boarder. We had serious problems in May, but were able to get things under control. We are also having a fall rise in cases, but nothing like the US.

    I don’t know why my blog post is being discussed now, but it should not be used as evidence in a debate about Covid-19.

    Cheers,
    Uli

    • “The US as a whole and red states in particular are a disaster.”
      Do you have hard evidence for this claim? Do you regard NY as a red state or do you conclude (based on hard evidence) it has done much better than red states?

      By the way, not every positive test is a case. This careless language usage from statisticians have done incalculable damage.

      • I suppose it depends on how much “credit” you give the Northeastern state governments for doing the best they could with bad information. I doubt any of the red states surging now will catch up with NY or NJ in terms of per-capita deaths*, but that likely has as much to do with poor understanding of the disease in March as anything.

        Louisiana is also rather high in terms of per-capita deaths (though not so high as NJ/NY) probably because New Orleans was also hit very early.

        *I don’t know, maybe if the Dakotas go to 60%-70% infected or something. OTOH it might not be completely crazy to think the IFR in November could actually be 3x lower than in March, especially if the age distribution of those infected is significantly different.

        • I think it is reasonable to assume that the same information set was available to everyone. Whether political orientation played part in what part of the set people paid attention to is a different question. Though that would be an interesting question to consider. There is anecdotal evidence that information from even people with “correct credentials” was ignored.

        • >>I think it is reasonable to assume that the same information set was available to everyone.

          No, not really, because I’m talking about decisions made in NY/NJ in March, vs. decisions made in TX/AZ/FL in July or in the Dakotas now.

          Sure, political affiliation doesn’t determine access to information — but time does. The states hit in the first US surge were nearly all “blue” states (the only exception, I think, was New Orleans – and even there the rest of LA wasn’t much affected until the broader Southern surge in summer).

        • Or to put it slightly differently, sure, ND/SD had access to the same information back in March that NY/NJ did.

          But they didn’t have any cases …

          Now, ND/SD still have access to the same information as NY/NJ does today, they’re just not bothering to make use of it to control the number of cases.

        • Right.

          I think there are some things that the governors of NY/NJ *can* be blamed for, like sending COVID patients back to nursing homes due to an extreme panic over hospital capacity. But even that has to be weighed against the bad expert advice they were given — US hospital capacity has generally coped much better than expected*, though we haven’t seen a really unmitigated outbreak (until now in e.g. the Dakotas — even TX did a mask mandate near its peak).

          *I think this is because of too much reliance on comparisons to Europe: US has generally better ICU/million people and younger population, thus fewer hospitalization/ICU for the same number of infections.

        • Although, I think “not bothering to make use of it” is not exactly the situation. As far as I can tell it’s not that ‘they have better things to do’ or even ‘they don’t care’, I think it’s a political/ideological choice to not issue mandates.

          (And apparently South Dakota’s governor, Kristi Noem, has decent approval ratings on her COVID handling, so this may be simply a matter of doing what the voters want…)

        • By cases do you mean medical cases or reported positive tests?
          There is no evidence of increase in flu cases (medical cases, covid inclusive).

          If it is positive tests, why is controlling that an objective function worth considering?

          Mind you, the measures proposed by the governments so far are not costless. They are quite costly both in terms of $ and lives (suicides, mental illness, abuse victims, cancer deaths, compromised immune system, etc).

Leave a Reply to dhogaza Cancel reply

Your email address will not be published. Required fields are marked *