Elin:

Yes, I wrote this post in preparation for part 2, which is about universal child care (or something like it). So the title of this post was a bit misleading.

]]>Chris,

Following up on your last point – if I am doing the same calculation you are, then I get a 95% uncertainty interval (Andrew’s blog, Andrew’s preferred term) of [0.07, 0.53] for the effect on log earnings. That’s pretty wide. I don’t think it’s exactly kosher to do it this way (p. 11 of the working paper version) but I’d guess that if you did it properly you’d get something similar.

So … effect imprecisely estimated, and the interval includes modest as well as very large effects. My complaint is similar to yours (Chris’) – had they reported intervals, they would have been doing readers a service.

]]>Chris:

Part of the issue is effects that are higher than my priors. Not just my priors, but lots of prior information. Recall this quote from Charles Murray: “To me, the experience of early childhood intervention programs follows the familiar, discouraging pattern …small-scale experimental efforts staffed by highly motivated people show effects. When they are subject to well-designed large-scale replications, those promising signs attenuate and often evaporate altogether.” Charles Murray isn’t always right but the point is that it’s not so unreasonable to be skeptical about large claims from small studies. You refer to the literature, but we know that just about any published literature will overestimate effect sizes, sometimes by extreme amounts.

And that brings me to my second concern, which is that by seeking and publishing statistically significant estimates, researchers are biasing their effect size estimates; see here. When people present results from biased estimates and don’t try to correct these biases, I’m concerned.

]]>Much of the issue seems to be that the estimates of the ATE are higher than your priors. 42% (for all workers) or 25% (for full-time jobs) do seem at first glance like implausibly large effects, but they’re not, to me at least, surprisingly large once we view them in context. The sample is not merely from a developing country (where we might expect such interventions to have larger effects), but critically the sample was of stunted children: the average study participant had a height three standard deviations below the average child’s height, conditional on age. The literature on potential “catch up” of children subject to early adversity is small but consistent with the idea that these children may respond particularly well to interventions such as studied here. The result is more or less that the intervention increased the earnings of stunted children such that those children eventually had earnings comparable to those of non-stunted children in this developing country, which doesn’t seem incredible. I’ve read extensively in the relevant literatures, and my prior on the effect of this particular intervention would be both very diffuse and place substantial weight on very high, like 25% or 42%, outcomes. I am not sure why yours places so little weight on large outcomes.

Can they reject a 10% effect? An annoyance with the paper is they only report p-values, so there is rounding error, but if we consider the highlighted estimate from the published version: the effect on log-earnings is 0.3, p~=0.01, implies we should should reject nulls of returns lower than about +7%.

]]>