Ambarish Chandra writes:
I’m writing regarding the CDC study on school mask mandates from last month, which shows that counties with school mask requirements had lower pediatric covid cases than those without them.
As any social scientist knows, the study cannot possibly reveal causal effects due to the endogenous decision to adopt mask mandates by particular communities, and at specific times. Yet, the results of this study are very influential: they are repeatedly cited by the CDC director, posted prominently on the CDC website as “evidence” for school mask requirements and even repeated verbatim in other countries (e.g. Canada, Australia) to justify school mask mandates in those places. The CDC is not including any caveats about the study being correlational and not causal, and the media cannot tell the difference.
I attempted to replicate that CDC study, using the restricted CDC case data and matching it to data on school district mask mandates. It turns out that the CDC study was restricted to schools that opened by August 14, 2021, which are disproportionately in southern US states that were in the middle of the Delta wave when schools opened. I extended the study to use more recent data (up to the Oct 25 release) which also allows an examination of schools across the country. In short, the study does not hold up when extended. There is absolutely no difference between counties with school mask mandates and those without, in terms of either pediatric or adult cases. I’ve written all this up in this document, with figures included.
The document also has a short explanation of the pitfalls of using correlational studies. In the context of the pandemic, we should expect that private behaviour will be positively correlated with public health measures (e.g. places that impose mask mandates, school closures, or vaccine mandates are also likely to have citizens voluntarily reducing their social contacts, eating out less, improving ventilation etc), both cross-sectionally and temporally.
So, a simple correlation will likely overstate the true effect of PH measures. That’s why it is not surprising that the CDC study found a negative association between masks and pediatric cases—we would expect that even if the true causal effect is zero.
Regardless of how anyone feels about masks (and I have been on both sides of this fight over the past 18 months), we should not allow bad research to influence public policy. This study has been very influential and continues to shape policy. I am keen to publicize the fact that the CDC result is fragile, and completely disappears when extended to a larger sample.
If you have any thoughts on how I might get the word out about this critique, I would be grateful. I took a long shot and emailed David Leonhardt at the NYT, but that hasn’t worked out.
1. It’s great to see people reacting to published studies and doing their own analyses. I haven’t looked carefully at this particular analysis so I can’t really comment on the details, but I’m supportive of the general idea of looking at the numbers. In particular, it’s hard to untangle the effects of policies that are done at the same time, especially when behaviors are changing too. We discussed this regarding covid responses here and here. I appreciate that the CDC report is unambiguously correlational (“this was an ecologic study, and causation cannot be inferred”), but then it’s funny that at the end they say, “School mask requirements, in combination with other prevention strategies, including COVID-19 vaccination, are critical to reduce the spread of COVID-19 in schools.” That seems a bit like what Palko calls a “cigarettes and cocaine” argument.
2. I don’t know if I buy Chandra’s argument that “a simple correlation will likely overstate the true effect.” I agree that mask mandates are likely to occur in concert with private masking behavior and general carefulness that could’ve happened anyway, but I can also picture some things going in the other direction: for example, a mask mandate can be applied at a time when there’s a concern or expectation of an increase in risk. I think it’s fine to say that the correlational estimate has issues; I’m not so comfortable with using it as a unidirectional bound.
3. At the policy level, I see the appeal of a mask requirement in part because it addresses many parents’ and teachers’ concerns about health risks. Kids are required to go to school, so it’s important to do what it takes to make them and their parents and their teachers feel safe. All within reason, of course, but given what we know about the spread of covid, mask requirements seem to me to be within reason.
4. Regarding the last paragraph in Chandra’s note above: Yeah, I too have found it difficult to get the attention of reporters! My experience is that journalists are really busy people: they’ll contact me when they have questions or if they’re on a deadline, but otherwise they typically don’t seem to have the time to respond to email. That’s just the way it is. I guess if they spent too much time responding to messages, they wouldn’t have time for anything else. That’s true for me too, but I’ll respond to most things anyway because I appreciate the distraction.