Following up on our recent discussion of synthetic control analysis for causal inference, Alberto Abadie points to this article from 2021, Using Synthetic Controls: Feasibility, Data Requirements, and Methodological Aspects.
Abadie’s paper is very helpful in that it lays out the key assumptions and decision points, which can help us have a better understanding of what went so wrong in the paper on Philadelphia crime rates that we discussed in my earlier post.
I think it’s a general concern in methods papers–mine included!—that we tend to focus more on examples where the method works well, than on examples where it doesn’t. Abadie’s paper has an advantage over mine in that he gives conditions under which a method will work, and it’s not his fault that researchers then use the methods and get bad answers.
Regarding the specific methods issue, of course there are limits to what can be learned from N=1 treated units, whether analyzed using synthetic control or any other approach. It seems that researchers sometimes lose track of that point in their desire to make strong statements. On a very technical level, I suspect that, if researchers are using a weighted average as a comparison, that they’d do better using some regularization rather than just averaging over a very small number of other cases. But I don’t think that would help much in that particular application that we were discussing on the blog.
The deeper problem
The question is, when scholars such as Abadie write such clear descriptions of a method, including all its assumptions, how is it that applied researchers such as the authors of that Philadelphia article make such a mess of things? The problem is not unique to synthetic control analysis; it also arises with other “identification strategies” such as regression discontinuity, instrumental variables, linear regression, and plain old randomized experimentation. In all these cases, researchers often seem to end up using the identification strategy not as a tool for learning from data but rather as a sort of springboard to overconfidence. Beyond causal inference, there are all the well-known misapplications of Bayesian inference and classical p-values. No method is safe.
So, again, nothing special about synthetic control analysis. But what did happen in the example that got this discussion started? To quote from the original article:
The research question here is whether the application of a de-prosecution policy has an effect on the number of homicides for large cities in the United States. Philadelphia presents a natural experiment to examine this question. During 2010–2014, the Philadelphia District Attorney’s Office maintained a consistent and robust number of prosecutions and sentencings. During 2015–2019, the office engaged in a systematic policy of de-prosecution for both felony and misdemeanor cases. . . . Philadelphia experienced a concurrent and historically large increase in homicides.
After looking at the time series, here’s my quick summary: Philadelphia’s homicide rate went up since 2014 during the same period that it decreased prosecutions, and this was part of a national trend of increased homicides—but there’s no easy way given the directly available information to compare to other cities with and without that policy.
I’ll refer you to my earlier post and its comment thread for more on the details.
At this point, the authors of the original article used a synthetic controls analysis, following the general approach described in the Abadie paper. the comparisons they make are to that weighted average of Detroit, New Orleans, and New York. The trouble is . . . that’s just 3 cities, and homicide rates can vary a lot from city to city. There’s no good reason to think that an average of three cities that give you numbers comparable to Philadelphia’s for the homicide rates or counts in the five previous years will give you a reasonable counterfactual for trends in the next five years. Beyond this, some outside researchers pointed out many forking paths in the published analysis. Forking paths are not in themselves a problem—my open applied work is full of un-preregistered data coding and analysis decisions—; the relevance here is that they help explain how it’s possible for researchers to get apparently “statistically significant” results from noisy data.
So what went wrong? Abadie’s paper discusses a mathematical problem: if you want to compare Philadelphia to some weighted average of the other 96 cities, and if you want these weights to be positive and sum to 1 and be estimated using an otherwise unregularized procedure, then there are certain statistical properties associated with using a procedure which, in this case, if various decisions are made, will lead to choosing a particular average of Detroit, New Orleans, and New York. There’s nothing wrong with doing this, but, ultimately, all you have is a comparison of 1 city to 3 cities, and it’s completely legit from an applied perspective to look at these cities and recognize how different they all are.
It’s not the fault of the synthetic control analysis if you have N=1 in the treatment group. It’s just the way things go. The error is to use that analysis to make strong claims, and the further error is to think that the use of this particular method—or any particular method—should insulate the analysis from concerns about reasonableness. If you want to compare one city to 96 others, then your analysis will rely on assumptions about comparability of the different cities, and not just on one particular summary such as the homicide counts during a five-year period.
You can say that this general concern arises with linear regression as well—you’re only adjusting for whatever pre-treatment variables that are included in the model. For example, when we estimated the incumbency advantage in congressional elections by comparing elections with incumbents running for reelection to elections in open seats, adjusting for previous vote share and party control, it would be a fair criticism to say that maybe the treatment and control cases differed in other important ways not included in the analysis. And we looked at that! I’m not saying our analysis was perfect; indeed, a decade and a half later we reanalyzed the data with a measurement-error model and got what we thing were improved results. It was a big help that we had replication: many years, and many open-seat and incumbent elections in each year. This Philadelphia analysis is different because it’s N=1. If we tried to do linear regression with N=1, we’d have all sorts of problems. Unfortunately, the synthetic control analysis did not resolve the N=1 problem—it’s not supposed to!—but it did seem to lead the authors into a some strong claims that did not make a lot of sense.
P.S. I sent the above to Abadie, who added:
I would like to share a couple of thoughts about N=1 and whether it is good or bad to have a small number of units in the comparison group.
Synthetic controls were originally proposed to address the N=1 (or low N) setting in cases with aggregate and relatively noiseless data and strong co-movement across units. I agree with you that they do not mechanically solve the N=1 problem in general (and that nothing does!). They have to be applied with care and there will be settings where they do not produce credible estimates (e.g., noisy series, short pre-intervention windows, poor pre-intervention fit, poor prediction in hold-out pre-intervention windows, etc). There are checks (e.g., predictive power in hold-out pre-intervention windows) that help assess the credibility of synthetic control estimates in applied settings.
Whether a few controls or many controls are better depends on the context of the investigation and on what one is trying to attain. Precision may call for using many comparisons. But there is a trade-off. The more units we use as comparisons, the less similar those may be relative to the treated unit. And the use of a small number of units allows us to evaluate / correct for potential biases created by idiosyncratic shocks and / or interference effects on the comparison units. If the aggregate series are “noiseless enough” like in the synthetic control setting, one might care more about reducing bias than about attaining additional precision.
Agree entirely that there is no reasonable way Abadie could have rewritten his paper to avert its conversion into an overconfidence springboard.
I think the issue is that graduate programs systematically train researchers in all the gymnastics required to stick these leaps to conclusions.
First-year grad students speaking about their own work are often the exemplars of scientific reserve and skepticism. As a dissertating grad student myself, it has been interesting and troubling to watch peers develop their ability to perform confidence and expertise—especially in respect to muddy topics. No one is born knowing how to say, “we conducted robustness checks and the results are unchanged”.
That said, some people obviously learn how to be candid, eloquent, and pragmatic about the limits of scientific knowledge (see all sorts of posts and comments on this very blog) so obviously the grad school socialization isn’t all bad.
Thank you, Andrew.
I’ll add some more context on synthetic controls for the readers of the blog.
Synthetic controls were designed to assess the effects of large-scale events, notably those stemming from policy decisions at the level of a city, state, etc. Many statistical methods focus on estimating average effects over a population of interest from which we have obtained sufficiently large samples of units, with many of them exposed to the intervention of interest. However, in reality, many of the interventions that economists care about don’t fit this mold. Indeed, many interventions that researchers and policymakers focus on aren’t deployed at the individual level or across many disaggregate units. Instead, we often aim to gauge the effect of policies that impact a limited number of aggregate units.
I agree with Andrew’s point that having just one treated unit (or a small number of treated units) presents more challenges compared to a larger sample. Yet, as mentioned above, these scenarios often emerge in real-world contexts. Synthetic controls are based on the idea that lower noise/signal ratios, combined with the strong co-movement in outcomes across units often observed in aggregate data, can help estimate policy effects.
The fact that a synthetic control is a convex combination of untreated units acts as a regularization device for the weights. I also agree with Andrew that in certain settings additional regularization could be beneficial. Guido Imbens and co-authors have produced excellent work on synthetic controls with L2 (and elastic net) regularization, which creates dense weights.
Originally, synthetic controls stemmed from comparative case studies where the outcomes of aggregate treated units are compared with those of a few carefully selected untreated units (e.g., David Card’s Mariel Boatlift paper). While the dense weights that come with L2 regularization can improve precision, the trade-off is that they might also muddle the interpretation of the counterfactual estimates and the evaluation of potential biases (created, for example, by interference effects).
Alberto Abadie