Someone writes in:

After your recent Jamaican Me Crazy post, I dug into the new JECS paper a bit, and the problems are much deeper than what you mentioned. The main problems have to do with their block permutation approach to inference.

The article he’s referring to is “Effect of the Jamaica early childhood stimulation intervention on labor market outcomes at age 31”; it’s an NBER working paper and I blogged about it last month. I was surprised to hear that it had already been published in JECS.

I did some googling but couldn’t find the JECS version of the paper . . . maybe the title had been changed so I searched JECS and the author names, still couldn’t find it, then I realized I didn’t even know what JECS was: Journal of Economic . . . what, exactly? So I swallowed my pride and asked my correspondent what exactly was the new paper he was referring to, and he replied that it was the Jamaican Early Childhood Stimulation study, hence JECS. Of course! Jamaican me crazy, indeed.

Anyway, my correspondent followed up with specific concerns:

1. The first issue that I noticed is in their block 5. They lay out the blocks in Appendix A. The blocks as described in the body of the paper are stratified first by a mother’s education dummy and assignment to the nutritional supplement treatment arm (supposedly for being unbalanced at baseline), and then gender and an age dummy, which is how the study’s initial randomization was stratified. However, block 5 is on broken up by age, not gender. There’s no reason I can see for doing this – breaking it up by gender won’t create new blocks that are all treatment or control, nor will they be exceptionally small (current blocks 1, 3, and 6 are all smaller than what the resulting blocks would be). Regardless, this violates the exchangeability assumption of their permutation tests. Considering block 5 is 19% of their sample, splitting it could create a meaningful difference in their inferences.

2. Their block 1 is only mothers with higher education, it isn’t broken out by supplement, gender, or age. Again, this violates the exchangeability assumption, no reason is given as to why, and if you were only to read the body of the paper, you would have no idea that this is what they were doing. The actual design of the blocking I’ve attached here.

3. In the 2014 paper, the blocking uses mother’s education, mother’s employment status, a discretized weight-for-height variable, and then gender and age. No reason is given for why they dropped employment and weight-for-height and added supplement assignment – these are all baseline variables, if they were imbalanced in 2014, they’re still imbalanced now! Stranger still, the supplement assignment isn’t even imbalanced, since it was originally a treatment arm!

I [the anonymous correspondent] found the 2014 data on ICPSR, and ran a handful of analyses, looking at the p-values you get if you run their 2021 blocking as they ran it, 2021 with block 5 split properly, some asymptotic robust p’s, and their 2014 blocking as I think they did it. I say “I think” because their replication code is 125 MATLAB files with no documentation. If you do it as described in the 2014 paper, you have 105 kids divided into 48 blocks, you end up with lots of empty or single observation blocks, so I’m sure that isn’t what they did, but it’s my best guess. I attached the table from running those here as well:

There were other issues, like their treatment of emigration, but this email is already long. You might also be interested in something on the academic side. I showed this to my advisor, and was basically told “great work, I don’t think you should pursue this.” . . . He recommended at most that I create a dummy email account, scrub my PDFs of any metadata, and send it anonymously to you and Uri Simonsohn. So at least for now, like your original correspondent, I live in the Midwest and have to walk across parking lots from time to time, so if you do blog, please keep me anonymous.

“Their replication code is 125 MATLAB files with no documentation”: Hey, that sounds a bit like my replication code sometimes! And I’ve been known to have some forking paths in my analyses. That’s one reason why I think the appropriate solution to multiplicity in statistical analysis is to fit multilevel models rather than to try to figure out multiple comparisons corrections. It should be about learning from the data, not about rejecting null hypotheses that we know ahead of time are false. So . . . I’m not particularly interested in the details of the permutation tests as discussed above—except to the extent that the results from those tests are presented as evidence in favor of the researchers’ preferred theories, in which case it’s useful to see the flaws in their reasoning.

Also, yeah, laffs and all that but for reals it’s absolutely horrible that people are afraid to express criticism of published work. What a horrible thing it says about our academic establishment that this sort of thing is happening. I don’t like being called Stasi or a terrorist or all the other crap that they throw at us, and I didn’t like it when my University of California colleagues flat-out lied about my research in order to do their friends a solid and stop my promotion, but, at least that was all in the past. The idea that it’s still going on . . . jeez. Remember this story, where an econ professor literally wrote, “I cam also play this hostage game,” threatening the careers of the students of a journal editor? Or the cynical but perhaps accurate remarks by Steven Levitt regarding scientific journals? I have no reason to think that economists behave worse than researchers in other fields; maybe they’re just more open about it, overtly talking about retaliation and using terms such as “hostage,” whereas people in other fields might just do these things and keep quiet about it.

Just to be on the safe side, though, look both ways before you cross that parking lot.