Chris Blattman writes,
Several aspiring graduate students have written me [Blattman] about becoming an impact evaluator. . . . I think the best advice is: don’t get a PhD to do evaluations. The randomized evaluation is just one tool in the knowledge toolbox. . . . Yes, the randomized evaluation remains the “gold standard” for important (albeit narrow) questions. Social science, however, has a much bigger toolbox for a much broader (and often more interesting) realm of inquiry. . . .
I pretty much agree with Chris on the substance of his remarks, but I think he’s missing something when he merges “impact evaluation” and “randomize evaluation” into a single concept. Policy analysis is a big area, and it certainly includes observational studies. We care about the impacts of all sorts of policies that can’t be directly studied using experimentation.
P.S. In a different direction, it’s interesting to me that policy evaluation is considered part of economics (a little bit) but not really part of political science–but maybe things are changing.
Policy evaluation is (was?) not considered part of Political Science, perhaps just because till recently this discipline was not (in general) quantitative enough for the job. But this is changing, I believe.
In medicine, sooooo many well conducted, gigantic observational studies have been over turned on randomized controlled trial that I don't understand how people dissmiss it's importance in other fields. I feel like economics especially dismisses their importance seemingly ignoring data to the contrary. Seeing Tyler cowen or robin hanson post on medicine after one of there rtcs are "only the gold standard" is extremely aggravating. I just don't get the sense that economics values rtcs enough to warrant these typenof backlash posts
When I took program evaluation, the professor frequently explained it this way: "People do not run programs because they want to solve problems. They run programs because they want to run programs. If they wanted to solve problems, they would care more about seeing whether the programs actually solved the problems." Very few people like evaluating their preferred policies and programs, in political science or otherwise, because it might prove you wrong. (Or, by your view, produce one misleading result that will do harm to something that you know works.) Political science students may not present great demands for evaluation courses.
Funny, I was just this a.m. reading on Volokh (a libert-right legal blog populated by lots of lawprofs with aspirations of sounding like economists) about how we can't know anything about anything *unless* we do a RCT (see this).
Sort of recalls this old thing…