Palko points us to a post by journalism professor Jay Rosen sharing this criticism of a New York Times story:
The news media get things wrong all the time—that’s just the way it goes, they’re on deadline and run zillions of articles every day!—so we shouldn’t be surprised when this happens. In this case, it’s a subtle statistical error, comparable to that baby boom age-adjustment thing that tripped up a couple of economists a few years ago.
Epidemiologist Katelyn Jetelina explains the new story:
[The New York Times] reported that death rate for White Americans has recently exceeded the rates for Black, Latino and Asian Americans. . . . This is misinformation.
Misinformation always has a kernel of truth. The kernel of truth in this reporting is that if we plot COVID19 deaths over time by race/ethnicity, White Americans are dying more than Blacks and Hispanics. This started in mid-to-late January 2022. . . .
However, when we take into account confounders—or other variables that could also explain this phenomenon—it tells another story.
In this case, we need to take into account age. White Americans are far more likely to outlive Black Americans. They live longer for a whole myriad of reasons, like differences in access to care, trauma, stress, etc. This is important because age is the strongest risk factor, by far, for dying of COVID.
If we account for age, we see a very different story. . . . Before adjusting for age, White Americans account for 43 per 100,000 in 2022 compared to, for example, Black Americans, who account for 37 per 100,000. After we adjust for age, the story changes: Whites account for 31 per 100,000 while Blacks account for 40 per 100,000. A complete switch.
If we look at this phenomenon over time, the gap is narrowing. . . . But the association has not flipped like the NYT article concluded.
OK, so no great shame on the news reporter for getting this wrong [it would be more accurate to say, not that the reporter got this wrong, but that the article’s headline was misleading; see P.S. below. — AG], given that a couple of Princeton economists made a similar error a few years ago and not too many people really cared, and more recently the Times used a bad (in my opinion) age adjustment to claim wrongly (in my opinion) that “the U.S. death rate in 2020 was the highest above normal since the early 1900s—even surpassing the calamity of the 1918 flu pandemic.”
As we never tire of saying in this space, statistics is hard.
When is there pressure to make a correction?
OK, I have nothing further to say about the technical aspects of age adjustment. Maybe with three high-profile examples of age adjustment getting screwed up in academic publications and news reports, people will be slightly more aware in the future that these comparisons are not so easy.
The question I want to ask is, how does it work that some errors in the newspaper get pressure for correction and some don’t? A few years ago I complained that the Times refused to correct errors from op-ed writers Nicholas Kristof and David Brooks.
After this one I actually sent an email to two reporters I knew at the Times, but nothing came of it.
Another time I contacted the Times because they’d run a story falling for an evidence-free number from Mars One (remember them??), but again was rebuffed on the ground that the unsourced number “comes from Mars One’s news release. We would have no way of disproving it without doing our own investigation.” Which raises the question of when do they report numbers from press releases without question—i.e., when do they consider a press-release to be telling the truth unless it can be “disproved”? The NYT article was entitled, “A One-Way Trip to Mars? Many Would Sign Up,” and it included the sentence, “More than 200,000 people from dozens of countries applied.” “200,000” was the unsourced number.
OK, enough about the items that have ticked me off. My point is just that, yeah, it seems that to get a correction you either need a very special kind of reporter or columnist who is ok with correcting errors, or else lots of people screaming from the outside. It’s not clear to me why this recent age-adjustment mistake should bother people so much, but, hey, every correction is worth making.
I don’t want reporters, columnists, and editors to be afraid of making errors. I just want them to quickly, clearly, and prominently correct errors when they learn about them.
P.S. The commenters have convinced me that it would be more accurate to describe the statement from the NYT article that covid death rates have flipped to be “misleading” rather than an “error.” So I’ve changed the title accordingly. Again, my main interest here is not on the particular statistical issue but rather on what is it that gets people to push for a correction.
I guess that I was too wrapped up in the “newspaper makes an error and refuses to correct it” story—something we’ve seen in the past with the Mars One story and those op-ed columnists—that I wasn’t able to step back and see the full story!
P.P.S. A more elaborate “my bad” here.
P.P.P.S. A commenter on our other post points to a post, where the commenter (“Random Critical Analysis”) downloaded CDC data and plotted covid death risks by ethnicity and age:
As you can see, the covid deaths for whites really were higher than for these other ethnic groups. (The above graphs are only for 65+ but that’s where almost all the deaths are happening.)
Here are the age-adjusted death rates, giving weights by age groups that are the same for all ethnic groups:
As you can see, the rates for whites were about the same as for blacks and hispanics, and then since May the rates for whites have been slightly higher.
But then I was baffled: the original discussion referred to a post by Katelyn Jetelina, who’d downloaded CDC age-adjusted rates and found them to be lower for whites:
Whassup with that? How is it that the graph shows higher age-adjusted rates for whites and the table shows lower age-adjusted rates for whites??
I looked carefully and realized that the table is for all of 2022 and the graph is just since March.
The linked blog post also has a graph going back earlier:
If you just look at 2002, you’ll see that whites had a lower age-adjusted covid death rate than blacks and hispanics from January through March—and that was the period when the death rate was high. So it makes sense that if you average all of 2022, whites have a lower age-adjusted death rate. The blog commenter points this out at the end of the post, that covid deaths everywhere have been declining fast in the past few months.
Random Critical Analysis is correct that the age-adjusted death rates really did flip in recent months, and Jetelina is correct that this is not a flip if you count all of 2022, i.e., the flip is happening only in the recent period when death rates have been very low. It’s interesting how you can come to different conclusions from the same data. In any case, I think we can all agree that the criticisms of the original NYT article were off base, which brings us back to our original question of why did people get so worked up about it.
I agree that the age issue is useful context, but I wouldn’t call it an error and it’s certainly not misinformation. For instance, in another article, Leonhardt said the following:
“Incarceration rates for black men are about twice as high as those of Hispanic men, five times higher than those of white men and at least 25 times higher than those of black women, Hispanic women or white women.”
“Doing crime”, particularly “doing violent crime” is a major predictor of incarceration which varies across groups, which you could certainly attempt to capture the effects of and control for, and which this leaves out. Does not doing that make this an error? Of course not: the incarceration rate data is meaningful and interesting on its own, as are the relative death rates.
Or on the topic of COVID deaths, another important predictor of death is vaccination status. Was some of the prior differences across race in deaths being driven by differences vaccination rates? If so, do you have to mention that every time you mention those differences?
There is always something to take into account. There is always more context. We can disagree about what context is important, we can have that discussion, but when someone doesn’t provide context you think is important, that’s not misinformation.
E:
Good point. I agree that the term “misinformation” is too strong. The best would be for the article to say something like, “Proportionally more whites than minorities are dying of covid, which is no surprise given that covid kills old people, and whites are older on average than other groups.” As you say, it’s all about putting these numbers in context.
As noted in my post above, one of the things that puzzled me is why people were so upset about this particular omission (maybe too strong to call it an “error”), when there are so many things to be bothered by.
Yea, there’s just not an “error” here. People present uncontrolled comparisons all the time. If I read a headline saying “Death rate among group X is higher,” I would take that to mean that the raw death rate in that group is higher. It would be extremely misleading to say “Death rate among group X is *lower*” if what you mean is that it’s actually higher but then goes down after adjusting for something else.
For my money, the lady calling this “misinformation” is the one engaged in spreading misinformation unless she wants to call out every table of summary statistics ever published.
Did you end up reading the original Leonhardt story? I did and from what I saw in the post was surprised to find the line:
“For another, the unequal nature of underlying health conditions — and access to good care — means that a Black person remains more vulnerable on average to severe Covid than a white person of the same age, sex and vaccination status.”
This is followed by a chart explicitly showing that age-adjusted rates have still been better among white people recently. I also can’t find any indication that the article was updated with this at any point. So I’m not sure exactly what falsehood it’s being claimed that Leonhardt was spreading.
As far as I can tell the thesis of his article is that the flipping in non-age-adjusted death rates between racial groups was caused by increasing relative vaccination status among black people. This, of course, can still be true even if the age-adjusted rates remain somewhat lower among white people.
E wrote:
“There is always something to take into account. There is always more context.”
I had the same thought. If the unadjusted death rate went one way and now it goes the other way, that is news. It might need an explainer, but it does not need a correction. And it might be somewhat misleading about relative risk factors, but it is not misinformation.
“The question I want to ask is, how does it work that some errors in the newspaper get pressure for correction and some don’t?”
The NYT has a liberal slant and the story that whites were more affected than people of color didn’t fit with that leaning. The himmicanes, loneliness, and Mars One stories didn’t clash with the liberal slant.
I’m guessing that the Wall Street Journal would be less likely to issue a correction for a story like this. News outlets are as prone to motivated reasoning as the rest of us.
I completely agree with E. The real world is unadjusted, and it is not surprising that a journalist would provide numbers from the real world, not a hypothetical world that would pertain if all groups had the same age distributions when they do not. Of course Jetelina is also correct to point out that age-adjusted contrasts can be more informative for some interpretations of interest, but this does not make the crude data “misinformation”. Adjustment can clarify the contrast and bring a story into better focus, but this is not a matter of “right” versus “wrong” as the the Twitter critics claimed. And the age-adjusted contrast is not necessarily “correct” because there is no reason to assume that age is the only relevant covariate that could be important in this story. Why stop there? Jetelina helped refine this story and that was a positive contribution, which is exactly what Leonhardt wrote in his Twitter reply.
Jay:
As noted in my P.S., I am convinced by the commenters that it was too strong to label the statement in the article as an “error.” But I think you’re coming on too strong in the other direction! Age adjustment when comparing death rates across groups is standard in the same way that inflation adjustment is standard when comparing prices over time. CPI-adjusted figures (e.g., taking a salary from 1983 and converting to “2022 dollars”) and age-adjusted death rates are indeed, as you say, “hypothetical,” but nowadays they are the default, to the extent that it would be considered appropriate to specifically label raw dollars or death rates as not adjusted if that is the case.
That said, I agree with you that the best framing here is that Jetelina and others are adding context and the original news article could’ve been better, so that this is a move toward clarity rather than correction of an error.
Thank you. I agree that age-adjustment is normative in this context, but that does not mean that it should be applied reflexively. I think it depends on which question is being posed and what underlying structure is assumed between the variables. For example, Jetelina notes that age-adjustment is important in racial contrasts because black NH life-expectancy (LE) lags behind white LE. But this raises two important points. The first is that this disparity is highly variable by state. NY and MA now have almost no racial LE disparity, whereas the gap remains substantial in mid-western states. So state becomes another important covariate. Secondly, the LE disparity itself arises from systemic racism. So if you draw your DAG with LE and COVID as both affected by race as an exposure, then age-adjustment could be criticized as adjustment for a factor affected by the exposure (as in obesity paradox and Knox et al critique of Fryer). Discussion of these points helps us understand the complex scenario. A Twitter-mob demanding Leonhardt’s cancellation is not helping anyone.
Jay:
Regarding your last sentence: I’m sorry to have contributed to this!
I just went and read the original newsletter from Leonhardt, and not only are the basic claims stated accurately, later in the post he shows charts for different age groups and discusses the fact that, within each age group, death rates are lower for white people than for black people. He doesn’t use the phrase “Simpson’s paradox”, but he actually presents the phenomenon more clearly than does Your Local Epidemiologist in arguing that Leonhardt is spreading misinformation.
Noah:
Yeah, my bad for not reading that article carefully before posting.
It seems that Leonhardt wrote a good piece, but everyone got out of sorts about the one line description under the title: “The death rate for white Americans has recently exceeded the rates for Black, Latino and Asian Americans.” This is not misinformation nor misleading. The piece is not rooted in a mistake nor a fundamental statistical error. The age distribution of whites relative to other races that Jetelina and others pointed to has been relatively steady the entire pandemic. What Leonhardt seems interested in is why white Americans died at a higher rate than other races in the second year of the pandemic. This seems a valid question to ask, and since whites were always older, the answer is not just “whites are older”. Just as adjusted numbers provide context, unadjusted numbers can also have meaning in and of themselves.
Here’s Leonhardt giving the context he is criticized for not giving: “the unequal nature of underlying health conditions — and access to good care — means that a Black person remains more vulnerable on average to severe Covid than a white person of the same age, sex and vaccination status.” Also, “As I mentioned above, the narrowing of Covid’s gaps does involve some bad news: The share of white Americans who have received a Covid vaccine shot has barely budged since last summer. The main culprit is politics. Only about 60 percent of Republican adults are vaccinated, compared with about 75 percent of independents and more than 90 percent of Democrats, according to Kaiser. And Republicans are both disproportionately white and older. Together, these facts help explain why the white death rate has recently been higher than the Asian, Black or Latino rate.”
I think the curious thing about this episode is understanding why people got all out of sorts about that line. This is from Leonhardt’s article: “Why haven’t you heard more about the narrowing of Covid’s racial gaps? I think part of the reason is that many experts and journalists feel uncomfortable highlighting shrinking racial gaps in almost any area. They worry that doing so will somehow minimize the problem of racism and the country’s enduring inequities.” That might be part of it.
“In this case, we need to take into account age,” more or less begins today’s discussion of Covid 19 deaths and Simpson’s Paradox. However, back in 2003, Steven Senn wrote that wonderful book, “Dicing with Death,” in which he discusses an example of Simpson’s Paradox having to do with diabetes. On an aggregated basis, those diabetics who are non-insulin dependent die at a greater rate than those diabetics who are insulin dependent. When age, above or below 40, is introduced, then insulin-dependent patients die at a greater rate, regardless of age.
Is there anything special about 40 which makes things turn around? Is it just a convenient number divisible by 10? Is there a comparable (specific) age turn-around with regard to Covid’s effect on Whites vs. Blacks?