Robert Neumann writes:
in the JEP 24(3), page18, Daron Acemoglu states:
Why Development Economics Needs Theory
There is no general agreement on how much we should rely on economic theory in motivating empirical work and whether we should try to formulate and estimate “structural parameters.” I (Acemoglu) argue that the answer is largely “yes” because otherwise econometric estimates would lack external validity, in which case they can neither inform us about whether a particular model or theory is a useful approximation to reality, nor would they be useful in providing us guidance on what the effects of similar shocks and policies would be in different circumstances or if implemented in different scales. I therefore define “structural parameters” as those that provide external validity and would thus be useful in testing theories or in policy analysis beyond the specific environment and sample from which they are derived. External validity becomes a particularly challenging task in the presence of general equilibrium and political economy considerations, and a major role of economic theory is in helping us overcome these problems or at the very least alerting us to their importance.
Leaving aside the equilibrium debate, what do you think of his remark that the external validity of estimates refers to an underlying model. Isn’t it the other way around?
My reply: This reminds me a lot of Heckman’s argument of why randomized experiments are not a gold standard. I see the point but, on the other hand, as Don Green and others have noted, observational studies have external validity problems too! Whether or not a model is motivated by economic theory, you’ll have to make assumptions to generalize your inferences beyond the population under study.
When Acemoglu writes, ” I therefore define ‘structural parameters’ as those that provide external validity,” I take him to be making the point that Bois, Jiang, and I did in our toxicology article from 1996: When a parameter has a generalizable meaning (in our context, a parameter that is “physiological” rather than merely “phenomenological,” you can more usefully incorporate it in a hierarchical model. We used statistical language and Acemoglu is using econometric language but it’s the same idea, I think, and a point worth making in as many languages as it takes.
I don’t know that I completely agree with Acemoglu about “theory,” however. Theory is great—and we had it in abundance in our toxicology analysis—but I’d think you could have generalizable parameters without formal theory, if you’re careful enough to define what you’re measuring.