I was looking through the wrong end of the telescope.

Someone pointed me yesterday to a controversy around a recent New York Times article headlined, “Covid and Race: The death rate for white Americans has recently exceeded the rates for Black, Latino and Asian Americans.” Critics pointed out that, yes, the covid death rates for whites were indeed higher than those other groups, but not after age adjustment—and we should pretty much always age adjust for death rates, especially for a disease such as covid that is so much more likely to kill people at one end of the age distribution, in the same way that we should pretty much always use inflation-adjusted estimates when comparing dollars from different eras.

This led to a push to get the Times to correct the article, and I wrote a post, “Considering the recent NYT age-adjustment error as a particular example: When an error is published, when is there enough pressure so that it gets corrected?”, which appeared this morning. In the post I referred to past events when the Times refused to correct errors, most memorably a few years ago when they endorsed an unsupported claim from a Mars One press release. My question was, what is it about some errors that motivate a strong push for corrections, and what does it take for the newspaper to actually issue a correction.

But . . . as several commenters pointed out, the recent NYT article in question was not actually in error! Yes, it was imperfect—it showed some age comparisons but still keeping the 65-and-up category as a single group, which won’t quite work for covid—and, yes, the headline was a bit misleading, but claims that the author “disgraced himself” or was “dangerously wrong” . . . that’s too strong. Age adjustment is a tricky topic. A better description of the episode would be to say that the news article was very good, and it just missed one point, which is well explained by Katelyn Jetelina here (if you just take the technical parts and forget her use of the term “misinformation”). I do think that the news article would be improved by a link to Jetelina’s post. I don’t know that this would necessarily be called a “correction” of the original article.

So what happened?

When I saw this story, I was too willing to fit it into the “newspaper makes an error and refuses to correct it” framework. I’ve been annoyed at the Times in the past for not correcting obvious reporting errors—there was that Mars One story and also several examples in op-ed columns. Also, I know that age adjustment is confusing (indeed, the Times messed up last year in a covid story by doing too much age adjustment), so it was easy for me to put all these pieces together into a plausible story: NYT article fails to do age adjustment, people point out the error, NYT fails to issue a correction, how annoying!

But I got it wrong. As is sometimes the case, the truth is more ambiguous. The news article kinda had an error—I think it is a mistake to present non-age-adjusted comparisons without being clear on what you’re doing—but was only kinda an error, because the article kinda did age adjustment and it kinda fit the age comparison into the rest of the story. Meanwhile, the criticism of the article had some technical value but were over-the-top with descriptions such as “dangerously wrong.” Rather than asking, “when is there enough pressure so that it gets corrected?”, I should’ve asked, “when does a slightly misleading article get people so annoyed?”

Just to step back, I think the news communication story here is largely positive. It started with David Leonhardt writing a thoughtful article on covid death trends, followed by a thoughtful post by Katelyn Jetelina explaining challenges of age adjustment. Overall, things went as they should, and could be improved even further with a followup in the Times on age adjustment, not an admission of error but a clarification of the subtleties.

The thing I want to focus on in the present post is that I got things wrong. I took a routine case of subtleties in a news story and shoehorning it into a people-refuse-to-correct-their-errors narrative that I’m so familiar with. My bad.

The other lesson from all this is that blog commenters can be very helpful. Thanks, commenters, for pointing out my mistake!

13 thoughts on “I was looking through the wrong end of the telescope.

  1. It’s people getting upset all the way down.

    People get upset about an error that’s not exactly an error. Then people get upset about people getting upset about that.

    Recursively ad infinitum.

    Is this worse than it used to be? I think maybe so. I doubt that people are more polarized at some basic human level, but people have more information and it’s more immediately available and that feeds the outrage and othering. Throw in a healthy amount of confirmation bias and a dollop of motivated reasoning. Filter it all through a screen of fundamental attribution theory and pour the mixture into a binary-thinking, zero-sum scorched earth vessel.

    Shit’s hard. People rarely get everything right. With so many people looking for hot takes, and then so many feeling outraged when the hot takes are inevitably less than perfect, it can get pretty unpleasant.

    I dunno. I’m skeptical of theories about putative massive societal shifts, but I do wonder where we will be in 10 years?

  2. I greatly appreciate your willingness to reconsider first impression and giving Leonhardt fair treatment.

    That said, you still left out a crucial part of the NY Times article:
    ===
    Certainly, there are important caveats to the Covid story. For one thing, the total death rate remains higher for Black and Latino Americans, because the early disparities were so huge. For another, the unequal nature of underlying health conditions — and access to good care — means that a Black person remains more vulnerable on average to severe Covid than a white person of the same age, sex and vaccination status.
    ===

    Besides the importance of controlling for sex (the male female ratio is far lower for Black Americans), controlling for vaccination is crucial.

    The accusation against Leonhardt amounts to asserting that some people could reach mistaken conclusion without age-adjusted rates info. In reality, Leonhardt addressed this by supplying two graphs (not old and old) that made it perfectly clear that age-adjusted rates did not flip in 2021. In fact his method made it clearer for general readership than “age-adjusted” rates would (since many do not know what it means).

    What the accusers do not realize is that the very same logic could be used against them — and this time correctly. When you merely state that age-adjusted rates remained higher for Black Americans, you fail to address the possibility that this is because their vaccination rates are much lower.

    Leonhardt closed this loophole, which makes it clear that he cares about the plight of Black Americans during the pandemic.

    Why not then accuse his critics, who yell only about age-adjusted rates, of potentially misleading the public and NOT caring enough about Black Americans to clarify the age-adjusted death rates remain higher even after controlling for vaccination?

    Of course I am just pointing out the irony of these politically motivated attacks, not arguing we actually should scream murder anytime some potentially relevant info is missing.

    – David

    P. S. BTW, few people realize there is no single correct method for age adjustment — yet another good reason for simply splitting into two age groups, just as Leonhardt did, where the method is explicit.

    P. P. S. I did not find age+sex+vaccination adjusted rates but Black Americans have reached vaccination rates similar to those of Whites by the end of 2022, so Leonhardt’s assertion seems almost certainly true.

  3. Having actually analyzed the weekly data on an age-adjusted basis, which I don’t believe anyone else has done, I’d go a lot further than this.

    David Leonhardt was substantively correct on all of the major points he advanced in his article:

    (1) There has been strong convergence in racial disparities.

    (2) These death disparities recently “flipped” on age-adjusted terms (April 2022).

    (3) The probable flip in age-adjusted vaccination rates can plausibly explain this.

    He’s only “wrong” if you uncharitably interpret his pointing out that white crude rates were higher than blacks and latinos for most of 2022 implying that this had “flipped” at the time on a more conditional basis (especially age). Said other things he shared made it quite clear he didn’t intend or believe that, especially those two plots where crude rates are broken out by age groups.

    https://rpubs.com/random_critical_analysis/covid_racial_disparities_flipped

    https://twitter.com/RCAFDM/status/1537525365652037632

    • Random:

      Thanks. That’s super helpful. I added a long P.P.P.S. to my original post. As you say, the whole thing is tricky because the flipping of death rates occurred during a period when the death rates are falling fast. Lots of interesting things to chew on here.

      • The Slate article is a hatchet job that also manages to severely misinform about Dana Mackenzie.

        In fact Slate REVERSED the meaning of what Mackenzie said — by OMITTING the next sentence:

        Instead of cold stats, the issue is one of framing. Leonhardt framed the CDC data so that it contained some shock value; epidemiologists stepped in to say, in so many more words, that this is not how they would view the data in a million years (as Mackenzie puts in her post: “The general answer [to Simpson’s paradox] espoused by introductory statistics textbooks is: control for everything. If you have age data, stratify by age.”). It is fair criticism. And while the newsletter might not be wrong, per se, it probably left many of its millions of readers with the wrong impression.

        First, it makes it look as if Mackenzie is one of the critics of Leonhardt, when in fact that text came two years earlier and had nothing to do with Leonhardt.

        Furthermore, Slate OMITTED the next CRUCIAL sentence when quoting Mackenzie:

        This “one-size-fits-all” approach is misguided because it ignores the causal story behind the data.

        So Mackenzie was warning AGAINST the textbooks advice of “control everything”, but Slate REVERSED his intent by omitting his next sentence.

        – David

        P.S. BTW, Mackenzie is male per his Wikipedia page — not female, as per Slate.

    • There is link there to http://causality.cs.ucla.edu/blog/index.php/2020/07/06/race-covid-mortality-and-simpsons-paradox-by-dana-mackenzie/

      > If we want to know whether our health-care system discriminates against a certain ethnic group, then we want to hold all other variables constant that might account for the outcome, and see what is the effect of changing Race alone. In this case, that means stratifying the data by Age, and the result is that we do see evidence of discrimination.

      That’s it, all other variables that might account for the outcome means age. After controlling for age we get the result we intended to find so there is no point in looking further.

      • To be fair, he also says that [emphasis mine]: “Undoubtedly the diagram above is too simple; unfortunately, if we make it more realistic by including more variables, we may not have any data available to interrogate. […] while we can learn an excellent lesson about Simpson’s paradox and some __probable__ lessons about racial inequities, we have to present the results with some caution.”

        On the other hand, even if the data was perfect the question remains of how many more variables to include. It’s not clear what does it mean to change race (or sex or whatever characteristic you want to chek if your health-care system discriminates again) and keep _everything_ else constant.

      • The Mackenzie text applies only to the trivial diagram, not reality. Mackenzie would never say something as silly as that age matters but sex does not when dealing with race and covid.

        Furthermore, Mackenzie is discussing the effects OF race, NOT the impact ON race.

        Leonhardt was first discussing the impact ON race (overall death rate) and later the approximate effects OF race (controlling for race+sex+vaccination).

Leave a Reply

Your email address will not be published. Required fields are marked *