I have always been taught that the randomized experiment is the gold standard for causal inference, and I always thought this was a universal view. Not among all econometricians, apparently. In a recent paper in Sociological Methodology, James Heckman refers to “the myth that causality can only be determined by randomization, and that glorifies randomization as the ‘‘gold standard’’ of causal inference.”
It’s an interesting article because he takes the opposite position from all the statisticians I’ve ever spoken with (Bayesian or non-Bayesian). Heckman is not particularly interested in randomized experiments and does not see them as any sort of baseline, but he very much likes structural models, which statisticians are typically wary of because of their strong and (from a statistical perspective) nearly untestable assumptions. I’m sure that some of this dispute reflects different questions that are being asked in different fields.
Heckman’s article is a response to this article [link fixed–thanks Alex] by Michael Sobel, who argues that Heckman’s methods are actually not so different from the methods commonly used in statistics. It’s all a bit baffling to me because I actually thought that economists were big fans of randomized experiments nowadays.
P.S. As noted by an anonymous commenter, some controversy arose from this issue of Sociological Methodology, but I’m not going into detail here since said controversy is not very relevant to the scientific issues that arise in these papers, which is what I wanted to post on.
Clicking link to Sobel's article only gave me the Heckman's article…
Fascinating discussion. Thanks for posting it.
This is a very interesting topic from my perspective, as an economics student taking a year off to study for a master's in statistics. You should have a look at a recent paper by Hal White at UCSD. He's doing some very interesting work on untangling causality from observational studies.
The Heckman and Smith (1993) Journal of Economic Perspectives paper is a good quick introduction to why economists do not think that experiments solve all problems everywhere. I think it is too strong to say that Heckman is not interested in randomized experiments – various randomized experiments play a key role in the case he makes for more policy emphasis on early childhood interventions.
What is true, I think, is that he, and many in the profession, worry about those who see experiments as a substitute for thinking. There are many questions they cannot answer (e.g. questions about general equilibrium effects) and many more that they cannot answer well. Furthermore, experiments can be implemented well or poorly. The latter is often the case even in highly publicized experiments (Perry Preschool comes to mind). Thus even experimental evidence requires thoughtful analysis and careful scrutiny.
I skimmed the paper, but I did not understand why he claims why he says that randomization is a "crutch." I appreciate the interest in specifying the selection model in econometric models. Yet, I don't see any argument why the magic of randomization (that on average over repeated experiments you have equality on all observed and unobserved variables) may be weaker for causal purposes than a selection model.
In my opinion, the main weakness of experiments is construct validity: you never really know whether you can ascribe the differences in treatment effects to any particular aspect of the treatments. (A secondary weakness is the "on average over repeated experiments" which means that many experiments lead to Type I and II errors.)
What am I missing: can you elaborate on Heckman's reasoning? What's your take on experiments as the gold standard? Thanks.
Andrew, economists disagree with each other (of course!) on these issues. Many are just as wary of structural models as statisticians are. See, e.g., Angrist and Krueger's articles on IV and empirical strategies, and Angus Deaton's book The Analysis of Household Surveys.
I don't agree with any of his politics or much of his economics, but on econometrics, Heckman is a mensch. He is fighting something of a proxy war here; I think his real opponent is Stephen Levitt and the unbearable cutesy "Freakonomics" gang (referred to as the "natural experiment movement".
Heckman's point as I see it is that economic behaviour takes place in an economy – an economy being an ongoing historical process of production, consumption and exchange. Because of that, economists (and therefore econometricians) have to respect the fact that they are dealing with events that take place in historical, unrepeatable time, rather than repeatable scientific events.
Obviously you have to believe that past and future events are in some way capable of being compared and generalised about (unless you are going to get into the really radical critiques of econometrics), but taking the real uncertainty and noise of the actual historical process and substituting a lab-created randomisation is a bad idea unless you are going to admit that you are now doing psychology rather than economics.
Lots of economists do now try to carry out randomised experiments, but if you look at what they've produced in terms of genuine, applicable propositions about economic paper (as opposed to reinventions of the experimental psychology literature and awards to themselves of prizes for doing so), they really have spent a lot of time and effort and achieved very little.
I think Heckman has a very valid critique here when he points out that the "strong assumptions" of structural models are at least explicit and made at the estimation stage. If you try to do econometrics without a structural model, what you end up doing is estimating a model that gives you a statistical association, and then putting all your naughty strong structural assumptions back in when you try to give it an economic interpretation, often (as is invariably the case with Levitt) without clearly telling your readers you are doing so.
Here's a link to <a>Deaton's intro.
Andrew at July 7, 2006 12:46 PM wrote:
"It's an interesting article because he takes the opposite position from all the statisticians I've ever spoken with (Bayesian or non-Bayesian)."
For what it is worth, the many (most?) philosophically inclined Bayesians (Bayesianism a big, wide tent) are actually not very sympathetic (or actually hostile) to randomization. Suppes (1982) says "it is hard to think of a more controversial subject than that or randomization" Of those Bayesians who think randomization is helpful, there is no agreement on rationale. Here are a few cites that hint and the broad
disagreement on the subject:
Suppes, Patrick, “Arguments for Randomizing,” Philosophy of Science Association Proceedings,
1982, 2, 464–475.
Swijtink, Zeno, “A Bayesian Justification of Experimental Randomization,” Philosophy of Science
Association Proceedings, 1982, 1, 159–168.
Harville, D. A., “Experimental Randomization: Who Needs It?,” American Statistician, 1975,
29, 27–31.
Rubin, Donald B., “Bayesian Inference for Causal Effects: The Role of Randomization,” Annals
of Statistics, 1978, 6, 34–58.
Berry, Scott M. and Joseph B. Kadane, “Optimal Bayesian Randomization,” Journal of the
Royal Statistical Society, Part B, 1997, 59 (4), 813–189.
Is the original Heckman article available for download? I only see his response to Sobel above. Or am I missing the link?
Also, the claimed link to Deaton's intro does not work for me.
David, here's a corrected link to Deaton. Readers of this blog may be interested in his thoughts on experiments, structural modeling, and graphical methods. Also, today Marginal Revolution links to an article by Abhijit Banerjee on randomized experiments and foreign aid, with comments by Deaton and others.