After giving my talk last month, Better Than Difference in Differences, I had some thoughts about how diff-in-diff works—how the method operates in relation to its assumptions—and it struck me that there are two relevant ways to think about it.
From a methods standpoint the relevance here is that I will usually want to replace differencing with regression. Instead of taking (yT – yC) – (xT – xC), where T = Treatment and C = Control, I’d rather look at (yT – yC) – b*(xT – xC), where b is a coefficient estimated from the data, likely to be somewhere between 0 and 1. Difference-in-differences is the special case b=1, and in general you should be able to do better by estimating b. We discuss this with the Electric Company example in chapter 19 of Regression and Other Stories and with a medical trial in our paper in the American Heart Journal.
Given this, what’s the appeal of diff-in-diff? I think the appeal of the method comes from the following mathematical sequence:
Control units:
(a) Data at time 0 = Baseline + Error_a
(b) Data at time 1 = Baseline + Trend + Error_b
Treated units:
(c) Data at time 0 = Baseline + Error_c
(d) Data at time 1 = Baseline + Trend + Effect + Error_d
Now take a diff in diff:
((d) – (c)) – ((b) – (a)) = Effect + Error,
where that last Error is a difference in difference of errors, which is just fine under the reasonable-enough assumption that the four error terms are independent.
The above argument looks pretty compelling and can easily be elaborated to include nonlinear trends, multiple time points, interactions, and so forth. That’s the direction of the usual diff-in-diff discussions.
The message of my above-linked talk and our paper, though, was different. Our point was that, whatever differencing you take, it’s typically better to difference only some of the way. Or, to make the point more generally, it’s better to model the baseline and the trend as well as the effect.
Seductive equations
The above equations are seductive: with just some simple subtraction, you can cancel out Baseline and Trend, leaving just Effect and error. And the math is correct (conditional on the assumptions, which can be reasonable). The problem is that the resulting estimate can be super noisy; indeed, it’s basically never the right thing to do from a probabilistic (Bayesian) standpoint.
In our example it was pretty easy in retrospect to do the fully Bayesian analysis. It helped that we had 38 replications of similar experiments, so we could straightforwardly estimate all the hyperparameters in the model. If you only have one experiment, your inferences will depend on priors that can’t directly be estimated from local data. Still, I think the Bayesian approach is the way to go, in the sense of yielding effect-size estimates that are more reasonable and closer to the truth.
Next step is to work this out on some classic diff-in-diff examples.
At least for me, I’d want to sharply distinguish between a randomized experiment where we already have valid inference for a treatment effect (but we want to make things more precise) and an observational study where this adjustment is ostensibly critical to identification and valid inference.
I think in andrew’s example, even though it was an experiment, there was concern that the experiment might be broken somehow and the DiD was to correct potential bias.
Z:
In that example, I am concerned that the experiment had problems—and the researchers flat-out refused to share their data, which was particularly annoying given that they were working at a U.S. government lab, i.e. those data were obtained using my tax dollars—, but even without any concern about the experiment being broken, it was still a terrible mistake for them to have used that diff-in-diff estimator, as it doubled their variances (i.e., effectively halved their sample size) by adding what was essentially pure noise to their estimates.
To put it another way: yes, the DID was there to correct potential bias. It’s just that (a) neither data nor theory give evidence for any such bias, and (b) if there were any bias, it could indeed still appear in a completely clean, unbroken experiment. The lack of bias is an empirical thing, not a design issue.
On the same page, didn’t mean that you personally had concern of bias, just that bias concerns were the original motivation for DiD in this case (even though it was an experiment, to Dean’s point)
> I’d rather look at (yT – yC) – b*(xT – xC), where b is a coefficient estimated from the data, likely to be somewhere between 0 and 1.
1) Why is b likely to be between 0 and 1. Do you just choose what to call ‘y’ such that that holds?
2) For the ‘one study’ setting, sensitivity/bias analysis for DiD where you put a distribution on the magnitude of deviation from parallel trends already exists. How does this improve on that since the data don’t tell you about b?
Regarding your point 1:
I think setting b between zero and one is a quite intriguing idea. I say that as someone who did ot study diff-in-diff in detail.
We can rewrite Andrew’s first equation (yT – yC) – b*(xT – xC). Ordered as in his abcd example, we get
(yT-b*xT)-(yC-b*xC)
=[(1-b)*baseline_T +Trend_T +effect +error_yT -b*error_xT] – [(1-b)*baseline_C +Trend_C +error_yC -b*error_xC].
Assuming Trend_T=Trend_C, we get
(1-b)*(baseline_T -baseline_C) +(error_yT -error_yC) +b*(error_xC -error_xT) +effect
If b is between zero and one, the proposed modification allows for some bias by not completely cancelling out the baseline effects. But it also shrinks the error terms from the pretreatment period, thus reducing the variance of the estimated treatment effect. As I understand it, Andrew is proposing to trade off bias against variance.
Please correct me if I am wrong!
Z:
1. I think b is likely to be between 0 and 1, partly from experience, and partly from general regression-to-the-mean principles if the before and after measurements are on the same scale.
2. My colleague will need to read up on the literature more carefully to answer this sort of question. For now, let me just say that I think there are major statistical efficiency gains possible by using a reasonable value of b. As with Bayesian inference in general, this will depend on the prior, and as with Bayesian inference in general, I think there will be many settings where there is no good prior reason to think that b=1 is even close to the best adjustment.
In DiD, people commonly look at ‘pre-trends’, i.e. outcome data for many timepoints prior to the treatment time in both the treatment and control units. Even in the single study setting, pre-trends could be a source of information about a good value of b. For example, if pre-trends are very flat, then this could mean b should be shrunk toward 0.
Z:
Given your position at the end of the alphabet, I can see why you’re a fan of long time series.
Correct me if I’m wrong, but should not
“((d) – (c)) – ((a) – (b)) = Effect + Error”
be
“((d) – (c)) – ((b) – (a)) = Effect + Error”,
i.e. b and a switched?
Typo fixed; thanks.
I think the specific appeal of difference-in-differences is that it is a consistent estimator of the average treatment effect on the treated under the parallel trends assumption. Since the parallel trends assumption is about unobserved counterfactuals, we cannot empirically evaluate it, but we can empirically evaluate whether pre-trends were parallel. This provides an empirical way to motivate the identifying assumption that underpins a causal interpretation of the estimate. I do not believe the regression estimator you have proposed is consistent with respect to the average treatment effect on the treated under the parallel trends assumption, and so does not enjoy this property. That said, the estimator is noisy relative to more model-heavy ways of analyzing the data, but with large sample sizes this may not matter so much.
Ram:
Yes, the issue is that we don’t have infinite data, so doubling the noise of estimates has serious applied consequences. We discuss this in the context of our example at the bottom of page 6, top of page 7, and section 3.5 of our article.
Understood, but my question is whether your proposed alternative is a consistent estimator of the ATET, and under which assumptions. If it isn’t, then I suspect users of difference-in-differences will not be interested in your alternative since it is not estimating the same quantity. Or if it is, but only under assumptions that data cannot be used to evaluate at all, then this will also be less appealing since at least with parallel trends there is the possibility of evaluating whether pre-trends were parallel.
Ram:
I don’t think any of these procedures are consistent, in that they are always applied to some average of data, in which case they are estimating some sort of local average effect. Consistency is a property that arises with infinite sample size, and the way that researchers increase their sample size is by averaging over more conditions, which has the effect of changing the underlying quantity being estimated. Consistency here is an essentially meaningless condition, as it refers to tracking a moving target.
^replying to prior thread…
Thanks for this response. Given your views on consistency, I wonder how you determine what it is than an estimator is estimating. For example, you suggested an alternative estimator to difference-in-differences, namely a certain regression specification. How do you determine what it is that this procedure is estimating? The classical view would be that you’re assuming the regression model you’re proposing is correctly specified, and that you’re estimating the parameters of that model. But I’m not sure in what sense that’s true other than that the estimator is consistent wrt those parameters if the model is true. So I wonder if you can spell out your view here, or if I’ve misunderstood something.
Ram:
In the linked paper, I’m doing Bayesian inference for the parameters theta_j, which correspond to the treatment effects in experiments j = 1, …, J.
> If the sham estimates are indeed nothing but noise, then it makes sense not to include them in the estimated treatment effects. […] We could almost describe this as “analysis ignoring the sham data” but that would not quite be correct. We did not ignore the sham data: we only decided to exclude the sham data from our inferences after first analyzing the sham results and finding no evidence distinguishing them from pure noise.
Does it really look like pure noise? There are four streaks of five positive or negative sham estimates in a row. If my calculations are correct, the probability of such a thing with pure noise is less than 0.7%. (I don’t have access to the original paper so I don’t know if the ordering has any physical meaning but I assume it may correspond to the sequence of experiments.)
In any case, is it wise to take the absence of evidence as evidence of absence? When is it justified to ignore data – or decide to exclude data – because it could be nothing but noise?
> Perhaps most importantly, the overall impression of the data would have changed. […] the upper-left plot of Figure 1 shows a mix of positive and negative results, and most are not statistically significant. In contrast, in the lower-left plot all the point estimates are positive, making it clear that the results are consistent with a general pattern of positive effects with uncertainty at individual frequencies.
The contrast is not so strong, in the lower-left plot it’s also the case that most of the results are not statistically significant. It’s true that two have changed sign but I’d say that both charts are consistent “with a general pattern of positive effects with uncertainty at individual frequencies” just like both charts are consistent with the idea “that certain frequencies are effective in causing enhance calcium-ion efflux while others are not”.
It’s not obvious why “making clear” the former would be preferable anyway. In the subsequent analysis you also seem to find that it’s an improvement that the estimates from the hierarchical model have been partially pooled toward the common mean. But there is no reason why the physical effect measured for different frequencies should be normally distributed around some value!
The distribution doesn’t look particularly close to normal: https://imgur.com/a/S7DU7AA Could it be that you’re forcing the low estimates up – and farther from zero – for no good reason?
By the way in Gelman and Stern [2006] you present a chart with standard errors as an alternative to the barplot where colors are used to mark statistical significance and while I agree that the alternative chart is more informative I don’t quite agree with the following:
> Some of the most dramatic features of the original data as plotted in Figure 2(a)—for example, the negative estimate at 480 Hz and the pair of statistically-significant estimates at 405 Hz— do not stand out so much in Figure 2(b), indicating that these features could be explained by sampling variability and do not necessarily represent real features of the underlying parameters.
I’d say that the peak at 405Hz stands just as much – or even more as it becomes clear why the corresponding bars where blak in the original chart. In fact it would make sense to pool the two measurements at the same frequency and it would become even clearer. (Or maybe you mean that it’s not an isolated signal but theres is a build-up as frequency gets closer to 405Hz but I’d say that was also visible in the original chart.)