I just saw Beth Tipton speak at the Institute of Education Sciences meeting on The Generalizer, a tool that she and her colleagues developed for designing education studies with the goal of getting inferences for the population. It’s basically MRP, but what is innovative here is the application of these ideas at the design stage.
The online app looks pretty cool. Here’s the demo version. It’s set up specifically for experiments in schools, so it won’t be relevant for most of you, but the general principle should also apply to medical experiments, economic interventions, A/B tests, etc.
This seems like the right way to go. There is no substitute for careful measurement and design—and ultimately measurements and designs should be evaluated based on the goals of analysis.
The citation for the webpage on the last page seems to point to the wrong place (http://www.generalizer.org instead of http://www.thegeneralizer.org).
That is neat. I don’t have anything to add other than instrumenting things seems cool. It certainly seems more interesting than just grabbing as much of whatever info you can get.
How do these studies get carried out anyway? Do you just call the elementary schools you got randomized and hope that the principal is friendly to the cause?