Not so easy to estimate the effects of school mask requirements . . . how to think about this?

Ambarish Chandra writes:

I’m writing regarding the CDC study on school mask mandates from last month, which shows that counties with school mask requirements had lower pediatric covid cases than those without them.

As any social scientist knows, the study cannot possibly reveal causal effects due to the endogenous decision to adopt mask mandates by particular communities, and at specific times. Yet, the results of this study are very influential: they are repeatedly cited by the CDC director, posted prominently on the CDC website as “evidence” for school mask requirements and even repeated verbatim in other countries (e.g. Canada, Australia) to justify school mask mandates in those places. The CDC is not including any caveats about the study being correlational and not causal, and the media cannot tell the difference.

I attempted to replicate that CDC study, using the restricted CDC case data and matching it to data on school district mask mandates. It turns out that the CDC study was restricted to schools that opened by August 14, 2021, which are disproportionately in southern US states that were in the middle of the Delta wave when schools opened. I extended the study to use more recent data (up to the Oct 25 release) which also allows an examination of schools across the country. In short, the study does not hold up when extended. There is absolutely no difference between counties with school mask mandates and those without, in terms of either pediatric or adult cases. I’ve written all this up in this document, with figures included.

The document also has a short explanation of the pitfalls of using correlational studies. In the context of the pandemic, we should expect that private behaviour will be positively correlated with public health measures (e.g. places that impose mask mandates, school closures, or vaccine mandates are also likely to have citizens voluntarily reducing their social contacts, eating out less, improving ventilation etc), both cross-sectionally and temporally.

So, a simple correlation will likely overstate the true effect of PH measures. That’s why it is not surprising that the CDC study found a negative association between masks and pediatric cases—we would expect that even if the true causal effect is zero.

Regardless of how anyone feels about masks (and I have been on both sides of this fight over the past 18 months), we should not allow bad research to influence public policy. This study has been very influential and continues to shape policy. I am keen to publicize the fact that the CDC result is fragile, and completely disappears when extended to a larger sample.

If you have any thoughts on how I might get the word out about this critique, I would be grateful. I took a long shot and emailed David Leonhardt at the NYT, but that hasn’t worked out.

My reply:

1. It’s great to see people reacting to published studies and doing their own analyses. I haven’t looked carefully at this particular analysis so I can’t really comment on the details, but I’m supportive of the general idea of looking at the numbers. In particular, it’s hard to untangle the effects of policies that are done at the same time, especially when behaviors are changing too. We discussed this regarding covid responses here and here. I appreciate that the CDC report is unambiguously correlational (“this was an ecologic study, and causation cannot be inferred”), but then it’s funny that at the end they say, “School mask requirements, in combination with other prevention strategies, including COVID-19 vaccination, are critical to reduce the spread of COVID-19 in schools.” That seems a bit like what Palko calls a “cigarettes and cocaine” argument.

2. I don’t know if I buy Chandra’s argument that “a simple correlation will likely overstate the true effect.” I agree that mask mandates are likely to occur in concert with private masking behavior and general carefulness that could’ve happened anyway, but I can also picture some things going in the other direction: for example, a mask mandate can be applied at a time when there’s a concern or expectation of an increase in risk. I think it’s fine to say that the correlational estimate has issues; I’m not so comfortable with using it as a unidirectional bound.

3. At the policy level, I see the appeal of a mask requirement in part because it addresses many parents’ and teachers’ concerns about health risks. Kids are required to go to school, so it’s important to do what it takes to make them and their parents and their teachers feel safe. All within reason, of course, but given what we know about the spread of covid, mask requirements seem to me to be within reason.

4. Regarding the last paragraph in Chandra’s note above: Yeah, I too have found it difficult to get the attention of reporters! My experience is that journalists are really busy people: they’ll contact me when they have questions or if they’re on a deadline, but otherwise they typically don’t seem to have the time to respond to email. That’s just the way it is. I guess if they spent too much time responding to messages, they wouldn’t have time for anything else. That’s true for me too, but I’ll respond to most things anyway because I appreciate the distraction.

30 thoughts on “Not so easy to estimate the effects of school mask requirements . . . how to think about this?

  1. My impression is that there have been a lot of observational studies of masking, so just because one doesn’t replicate doesn’t make a big news item. There are also relevant experimental studies of the effects of masks on droplets and aerosols. And, here is a caution about changes over time. There used to be a strong negative correlation between the percentage of fully vaccinated people in states and the number of cases per 100,000 per week (or whatever the NY Times reports). The same was true for counties with California. Then, after the number of cases dropped in the South, the correlation went away, which might make you think that the vaccines don’t work. However, if you look at cases among vaccinated and unvaccinated people, it is clear that they do. The only puzzle for me in this is why cases went down in the South.

  2. Thanks for posting this Andrew. The link you posted is broken, here is the correct one:
    https://github.com/ambarishchandra/CDC-critique/blob/main/CDC_critique.pdf

    Comments welcome. As for Andrew’s point #3, I see the logic, but I’m not sure I agree. This seems to be saying that, since people are worried about transmission in schools, we should take measures that create the appearance of safety. But of course, these measures can just as easily be counter-productive. Plexiglass shields are now thought to worsen airflow, for example.

    • Ambarish:

      It’s not just the “appearance” of safety. I’m talking about actual safety, as it seems pretty clear that masks help. Whether the increase in safety is worth the inconvenience is another story.

      • Why does it seem pretty clear that masks help? In the context of a post essentially debunking the CDC’s primary “evidence” for them, that seems like an odd thing to assume.

        I suspect headline fatigue is behind most people’s default assumption that masks work. Who cares that whenever we pick a specific study to dig into in the results are, to be charitable, underwhelming; if a 1000 headlines suggest they work then surely the other unread 999 are solid!

          • Thanks for the pointer.

            Without re-hashing the discussion here, this isn’t positive evidence in favor of mask’s efficacy. It’s not even “theory” exactly as the contact+droplet mental model discussed, as the article describes, is based on now known flawed mechanics. To my read, the basic conclusion is: it may be useful heuristically to still believe masks work despite broken theory and failed experiments.

            I do think sniping clearly subpar CDC churn is almost unfair. I’m legitimately curious as to what is the single best piece of actual evidence in favor of mask’s efficacy?

            • d –

              I’m legitimately curious why you’re looking for a single best piece of actual evidence.

              Reviewing the literature, you will no doubt find many different pieces of evidence that support a variety of different conclusions – and none of the evidence, imo, should be expected to jump over a bar of “single best actual.”

              The use of masks during a pandemic across all variety of contexts, where there are so many variables and mediators and moderators and interaction effects, is not a subject where’d I’d expect any single piece of evidence to be even remotely available.

              Seems to me the best approach is to try to look at the full range of available evidence and then from that consider a risk analysis that includes a cost/benefit evaluation along with an evaluation of factors such as exponential growth and low probability, high damage function risk as well as the interfering influence of politics-based polarization.

              • I agree with you that even if they masks work the cost/benefit of their use is still unclear. This calculus is downstream of knowing their effect though, which if its zero is unnecessary.

                If the CDC’s stated 80% reduction were true, surely after 2 years we could point at at least one medium-quality piece of evidence for this? If not then I suspect a meta-analysis on “the full range of available evidence” is garbage in / garbage out.

              • d –

                I agree with you that while we can’t be absolutely certain they work or how much they work, a cost/benefit analysis makes sense. The cost seems likely low to me, and the upside if they do work might be quite high, given the compounding effect of any benefit at the population level. So I say might as well go for it.

                Although there is indeed the cost of hearing the constant whining from some folks about how much their freedoms are suffering from a minor inconvenience of wearing a mask. But my sense is that they’d be whining about something else if it weren’t about wearing a mask. Some folks just really like being a freedom fighter and seeing them eves as a victim of tyranny. Not much you can do about that anyway. 😉

  3. I don’t know what arguments are there against masking. A large randomized trial in Bangladesh has shown that masks reduce COVID-19 infections in communities ( https://insights.som.yale.edu/insights/in-first-randomized-study-shows-that-masks-reduce-covid-19-infections ). Since COVID-19 spreads through aerosols ( https://www.science.org/doi/10.1126/science.abd9149 ), masks should help but may not be enough at schools.

    But maybe this post is more about CDC’s study and how they should not be making causal claims without caveats.

    • This headline is borderline misinformation:

      > especially among older people and when surgical masks are used

      The actual study itself shows that masks help ONLY for the elderly — i.e. not for children — only if they are surgical masks, and importantly only in conjunction with other social distancing practices. There is no evidence in this study that (cloth) masks in isolation are beneficial for children.

      Even ignoring all that, you’ll note the studies suggested all-in effect size is an order of magnitude smaller than the 80% the CDC claims.

      • I have not read the CDC study. IF the observational study claims causal without enough care, it deserves criticism and should not be advertised widely. Most of what I write in this comment is not going to be about the CDC study but mask usage generally. Heads up to the reader.

        The study I shared was done on a very different population from school children in the US. But that’s what we’ve got. That’s much more than what we had early in the pandemic.

        My question is, what is the harm in using masks? Maybe kids will not know how to use them well? Will it have some psychological effect on them? The WHO has really been going on about the false sense of security and the resulting behaviour substitution (when people start being careless just because they use masks). Possible? Sure. I went with it too initially. But then that’s as much guesswork as anything else. It’s very normal for people in many parts of the world to wear masks on a daily basis. Why are US schoolchildren so different?

        I personally would not need an RCT to be convinced of masks. I just need to think about the mechanics of aerosol transmission. Masks probably help a lot in keeping the virus from getting in the air from our breath. Surgical masks and masks with high filtration are probably better than cloth masks. But I would also want there to be good ventilation and centralized air filtration systems in place in schools.

        Sometimes, we don’t get evidence that’s all neat and tidy. It does not mean we should take data and pretend that it tells us something it can’t. But then we still have to make decisions.

  4. > At the policy level, I see the appeal of a mask requirement in part because it addresses many parents’ and teachers’ concerns about health risks. Kids are required to go to school, so it’s important to do what it takes to make them and their parents and their teachers feel safe. All within reason, of course, but given what we know about the spread of covid, mask requirements seem to me to be within reason.

    At the policy level, I see the appeal of no mask requirement in schools in part because it addresses many parents’ and teachers’ concerns about developmental risks by children not seeing faces. Kids are required to go to school, so it’s important to do what it takes to make them and their parents and their teachers learn how to be humans. All within reason, of course, but given what we know about the spread of covid in schools in Sweden and statistically similar effects on teachers as on other professions, mask requirements seem to me to be lacking justification.

    • Country:

      I agree. I see the appeal of a mask requirement and also the appeal of no mask requirement, as you say. That paragraph of mine was not intended to be an argument that there should be a mask requirement, just an argument that the requirement is within reason. “Within reason” can include many things.

      I think that your paragraph overstates things—“learn to how to be humans” is a bit over the top—but I agree there are arguments for this on both sides, and much will depend on context, including parents’ and teachers’ expectations. Personally, I’d prefer to teach maskless, but I’m cool with Columbia’s mask requirement because it seems that this is what it takes for everyone to be ok with in-person teaching. And I much much prefer in-person teaching, even if masked, to remote teaching of any sort. But I could see that in other places, attitudes could be different.

      • “”But I could see that in other places, attitudes could be different.”

        Yes. Here in Japan, no one goes unmasked, and I expect that to continue indefinitely. Japan’s life expectancy has _INCREASED_ during the pandemic, mainly* due to masking also reducing pneumonias and flu. (We just trucked (well, public transportationed (local commuter train)) up to Shinjuku station (one of the world’s busiest train stations), and everyone really was masked. No one without. Not even one. Even the apparently homeless bloke I noticed. Despite 75% of the population being vaccinated, things “opening up”, and new cases and deaths down to tiny fractions of those in other industrialized countries.)

        From watching Japan, it seems nuts to think that not masking is at all acceptable. Anywhere. Anyone.

        *: Also, of course, due to radically reduced international travel. I had a vicious “flu” in the spring of 2019 that my GP/PCP said was probably something a tourist brought, so that year’s flu shot didn’t work. Unfortunately, the Japanese government is in the pocket of the travel industry and plans to return to pushing its existence as a major tourist destination, despite global warming and the associated disease burden tourism imposes on the local population.

          • Yes. Even on Colorado rivers. That’s why the seven-day average number of new confirmed cases for Nov. 19 was 138 new cases. That was for all of Japan.

            Seriously, anytime there’s the possibility of interacting with someone outside of one’s immediate family, everyone wears a mask. Even Buddha.

            https://pbase.com/davidjl/image/171733506

            (Of course, a mask only work when you actually wear it, so restaurants have been the main source of Covid surges in Japan.)

            • This is where I think masking becomes ridiculous. A family in a boat on a Colorado river may well be several miles from the nearest person they could possibly breathe on. The limit for me is about 15-20 *feet* (5m). If you are outside and more than 5m from the nearest person who isn’t in your household a mask is pointless. Even 3m is probably sufficient outdoors.

              Of course if you are in a situation where the distance is constantly changing, sure wear a mask.

              • “A family in a boat on a Colorado river…”

                Your sense of humor seems to be on vacation.

                I suppose with lots of people saying over the top things here, my over the top expression of the sensibleness of masking may come off as over the top, but it’s actually not over the top. Really. It’s not.

            • I suppose that they just forgot to mask all that well during the case surges in January, May and August, then?
              Particularly August, when daily cases averaged 25,000?

              Anyway, this discussion of whether masks ‘work’ is beside the point. The discussion here is of whether it is legitimate to use correlational studies to make causal claims about the effectiveness of masks or other NPIs.

              • When the Japanese were told it was safe to party, they did, and without masks. (The shutdowns here were for restaurants, which were the main problem, and when the government declared things safe (most recently because they wanted to run the Olympics when only the elderly were vaccinated), people went out eating, drinking, and karyoki-ing (which happens in small, unventilated rooms here) and lots of them got sick.)

              • mark –

                > Anyway, this discussion of whether masks ‘work’ is beside the point. The discussion here is of whether it is legitimate to use correlational studies to make causal claims about the effectiveness of masks or other NPIs.

                What would you suggest as an alternative – given that in this context RCTs and lab-based modeling have limited generalizability?

                My impression is that the CDC and other public health officials try to make an assessment using all available evidence.

    • Country –

      > it addresses many parents’ and teachers’ concerns about developmental risks by children not seeing faces.

      I don’t doubt that some teachers and parents are concerned, but I also think those concerns are not substantiated by evidence and are likely a function of ideology and tribalism.

      > Kids are required to go to school, so it’s important to do what it takes to make them and their parents and their teachers learn how to be humans.

      They wear masks a lot in a lot of counties in Asia. They have for years. Do you think Asians are less human than Americans?

Leave a Reply

Your email address will not be published.