John Bullock sends along this from Joshua Angrist and Jorn-Steffen Pischke’s Mostly Harmless Econometrics—page 223, note 2:
They don’t seem to know about the idea of adjusting for the group-level mean of pre-treatment predictors (as in this 2006 paper with Joe Bafumi).
I like Angrist and Pischke’s book a lot so am happy to be able to help out by patching this little hole.
I’d also like to do some further analysis updating that paper with Bafumi using Bayesian analysis.
I’m sure they do; this is known as the Mundlak estimator [https://pages.stern.nyu.edu/~wgreene/Econometrics/Mundlak-1978.pdf] in econometrics. This approach has some appealing properties when treatment effects are heterogeneous [https://arxiv.org/abs/1807.02099]
lal:
OK, then this just shifts the question back one step. If the authors were indeed aware of the idea of adjusting for the group-level mean of pre-treatment predictors, then why did they write that passage which makes it seem that they are unaware of it?
How much of this is just the difference in terminology between what fixed/random effects mean in economics vs statistics vs biostatistics?
https://stats.stackexchange.com/questions/27662/what-are-the-major-philosophical-methodological-and-terminological-differences
Including group-level means of your X in the regression is algebraically the same as running fixed effects (i.e., including a dummy variable for each group). Not only do the authors know about it, it’s precisely what they’re recommending!
Absentminded:
Yes if you’re fitting the model using least squares, no if you’re fitting a hierarchical model or more generally if you have priors on the coefficients. See section 5 of my 2004 paper on parameterization for some discussion of how you can have two models that are equivalent to each other if you’re fitting least squares or maximum likelihood, but become different when you consider Bayesian models.
One reason this is important in practice is that if you adjust for lots of predictors using least squares, you will overfit—overadjust for noise.
Andrew, would you mind expanding on that note regarding the adjustment for many predictors using least squares? I have noticed this in practice and greatly prefer sampling from the posterior to generate reasonable parameter estimates, it seems like using least squares optimization algorithms often get caught in local minima, converging to nonphysical parameter values.
Also, do you find it makes a difference if we explicitly specify the noise model and fit the noise parameter during the least squares optimization as well?
Kiran:
For a linear model you don’t need to worry about local minima, but you can get unreasonable estimates. “Unreasonable” = “contradicts prior information,” hence the benefit of including priors.
To put it another way: in a causal inference setting we might be regressing the outcome on the treatment variable, adjusting for various pre-treatment variables. In the regression, this “adjusting” is done using estimated coefficients for the pre-treatment variables. If you have a lot of pre-treatment variables, these estimates can be noisy—they can even go in the wrong direction, which is not so good for your adjustment!
Agreed. But if they don’t discuss Bafumi and Gelman, it’s not because “they don’t seem to know about the idea of adjusting for the group-level mean of pre-treatment predictors,” but because they’re not into priors and Bayesian modeling.
Absentminded:
Just to be clear, I’m not expecting anyone to cite Bafumi and Gelman, which is an unpublished paper! My problem was not that they didn’t cite our paper, it’s that they wrote that bit about the varying coefficients being uncorrelated with the regressors, without pointing out that this assumption can be easily satisfied by including group-level predictors in the model.
I guess another way to put it is that it’s not correct to say “GLS requires stronger assumptions than OLS.” OLS is just a procedure; it doesn’t require any assumptions at all! Or, if you want to talk about assumptions, the so-called fixed effects model can be considered a special case of the so-called random-effects model with the group-level variance set to infinity.
I get what they’re trying to do: they’re trying to give useful advice and convey the idea that fixed effects or OLS is the safe, reasonable choice. But . . . that’s bad advice! Or, I should say, it often can be bad advice. It’s not generally good advice.
Fixed effects (dummy variable regression estimated using OLS) is consistent under weaker assumptions than random effects (GLS), because fixed effects doesn’t require the individual intercepts to be uncorrelated with the regressor of interest. Including the group means of the regressor of interest doesn’t rehabilitate random effects–it simply turns it into fixed effects. There might be a terminology difference. In MHE, “random effects” refers to GLS estimation of the panel model where the individual intercepts are considered part of the error term. Bayesian methods which might go by the “random effects” moniker in other settings are not on the menu here.