Ethan Steinberg sends in these delightful regression discontinuity graphs:
He reports that the graphs come from a study from the University of Pennsylvania trying to estimate the effect of a care management intervention for high risk patients.
Nothing special here, just another day at the sausage factory. People are taught this is what to do, it spits out publishable results, everybody’s happy.
P.S. Just to explain some more: Sometimes people frame this as a problem of trying to figure out the correct specification for the running variable: should it be linear, or polynomial, or locally linear, whatever? But I don’t think this is the right way to think of things. I’d say that the original sin of the “regression discontinuity” framing is the idea that there’s some sort of purity of the natural experiment so that the analysis should be performed only conditioning on the running variable. Actually, these are observational studies, and there can be all sorts of differences between exposed and unexposed cases. It’s poor statistical practice to take the existence of a discontinuity and use this to not adjust for other pre-treatment predictors. Once you frame the problem as an observational study, it should be clear that the running variable is just one of many potential adjustment factors. Yes, it can be an important factor because of the lack of overlap between exposed and control groups in that variable. But it’s not the only factor, and it’s a weird circumstance of the way that certain statistical methods have been developed and taught that researchers so often seem to act as if it is.
P.P.S. A student asked for more detail regarding my concerns with certain regression discontinuity analyses. If you want more of my thinking on the topic, you can google *statmodeling “regression discontinuity”* to see some posts, for example here and here, and you can look at my articles with Imbens and Zelizer. We also discuss the topic in a less critical and more constructive, how-to-do-it perspective in section 21.3 of Regression and Other Stories.
P.P.P.S. Commenter Sam looked at the above-linked article more carefully and reports that their main analysis adjusts for other pre-treatment predictors and also includes alternative specifications. I still think it’s nuts for them to use this quadratic model, and even more nuts to include different curves on the two sides of the boundary—this just seems like noise mining to me—but they did also do analyses that adjusted for other pre-treatment variables. I don’t find those analyses convincing either, but that’s another story. Maybe the general point here is that it takes a lot for this sort of statistical analysis to be convincing, especially when no pattern is apparent in the raw data.