There’s an article by Abhijit Vinayak Banerjee in the Boston Review recommending randomized experiments (or the next best thing, “natural experiments”) to evaluate stragies for foreign aid. Also, here’s a link to the Boston Review page which includes several discussions by others and a response by Banerjee.

On the specific topic of evaluating social interventions, I have little to add beyond my coments last year on Esther Duflo’s talk: randomized experimentation is great, but once you have the randomized (or “naturally randomized”) data, it still can be a good idea to improve your efficiency by gathering background inforomation and using sophisticated statistical methods to adust for imbalance. To quote myself on Dfulo’s talk:

There are a couple ways in which I think the analysis could be improved. First, I’d like to control for pre-treatment measurements at the village level. Various village-level information is available from the 1991 Indian Census, including for example some measures of water quality. I suspect that controlling for this information would reduce the standard errors of regression coefficients (which is an issue given that most of the estimates are less than 2 standard errors away from 0). Second, I’d consider a multilevel analysis to make use of information available at the village, GP, and state levels. Duflo et al. corrected the standard errors for clustering but I’d hope that a full multilevel analysis could make use of more information and thus, again, reduce uncertatinties in the regression coefficients.

Why don’t we practice what we preach?

Nonetheless, I am not sure myself that large-N studies are always a good idea. And, in practice, I rarely do any sort of formal experimentation when evaluating interventions in my own activities. Here I’m particularly thinking of teaching methods, where we try all sorts of different things but have difficulty evaluating what works. I certainly do make use of the findings of educational researchers (many of whom, I’m sure, use randomized experiments), but when I try things out myself, I don’t ever seem to have the discipline to take good measurements, let alone set up randomized trials. So in my own professional life, I’m just as bad as the aid workers who Banerjee criticizes for not filliong out forms.

This is not meant as a criticizm of Banerjee’s paper, just a note that it seems easier to give statistical advice to others than to follow it ourselves.