The point here is not the face masks; it’s the impossibility of assumption-free causal inference when the different treatments are entangled in this way.

Adam Pearce writers:

When I read your Another Regression Discontinuity Disaster post last year, I was curious how much shifting the breakpoint would change the fit lines.

A covid paper making the rounds this weekend used a similar technique so I hooked it up to an interactive widget that lets you tweak the start and end dates to change the recommendation on mask wearing.

The point here is not the face masks; it’s the impossibility of assumption-free causal inference when the different treatments are entangled in this way.

From Pearce’s post:

A paper [by Renyi Zhang et al.] published last week claiming mask usage prevented 66,000 infections in NYC has received widespread media coverage. It has also been heavily criticized [by Kate Grabowski] for, among other things, a tenuous analysis of a regression discontinuity.

Fitting lines to the number of new cases each day in New York City before and after face masks were mandatory, the authors attribute the steeper decline in cases to face masks: new cases were falling by 39 per day before the order and 106 after, so face masks probably sped up the rate of decrease by about 67 cases a day.

There are some bold claims along the way (“After April 3, the only difference in the regulatory measures between NYC and the United States lies in face covering on April 17 in NYC”) . . .

But:

With large day-of-week patterns, the regression is sensitive to the exact start and end dates . . . Add a seven day lag to account for incubation, testing and reporting to the paper’s dates and the regression makes it look like there’s a strong case for banning masks! . . . But it’s not really possible to do causal inference with case counts from just three regions like this paper tries to do, especially if you’re fitting straight lines to exponential infection curves.

Indeed, even if the date of the policy effect were unambiguous, I wouldn’t trust this regression, because mask use was only one of the many behaviors that were changing at that time.

This is similar to the difficulty discussed by Donohue and Wolfers in trying to estimate the deterrent effect of the death penalty. Too many other things are changing at the same time.

It’s not about regression discontinuity at all—except to the extent that regression discontinuity is considered to be some sort of “identification strategy” that is thought to produce rigorous causal inference without the need to wrestle with the challenges of an observational study.

7 thoughts on “The point here is not the face masks; it’s the impossibility of assumption-free causal inference when the different treatments are entangled in this way.

  1. On Donahue/Wolfers, I think that was the paper with the “Canada graph”, looking at homicide rates in the U.S. and Canada over a long period in which the two countries were out of sync on whether the death penalty was allowed. Despite this the homicide rates went up and down together. To me this was decisive. If the death penalty had a deterrent effect, it was pretty weak. No statistical tests were needed. See Figure 2 here:
    http://users.nber.org/~jwolfers/papers/DeathPenalty(SLR).pdf
    (but this was not the original).

  2. While the conclusion comports with my pro-mask bias, it seems a giant leap to infer the comparative magnitude of microscopic phenomena from macroscopic data. Not that it couldn’t be done, but it’s like using a sledgehammer for work that requires fine tweezers.

    It also struck me on reading that the wording of the study is more certain than typical for studies, especially where an indisputable giant leap is being taken. ~Ian

Leave a Reply

Your email address will not be published. Required fields are marked *