Brian Nosek is a leader of the replication movement in science and a coauthor of an article on replicability that we discussed the other day.

They discussed the rigor-enhancing practices of “confirmatory tests, large sample sizes, preregistration, and methodological transparency, and in my post I wrote that those were not the first things I’d suggest to increase rigor in science. My recommendations were (1) Make it clear what you’re actually doing, (2) Increase your effect size, e.g., do a more effective treatment, (3) Focus your study on the people and scenarios where effects are likely to be largest, (4) Improve your outcome measurement: a more focused and less variable outcome measure, (5) Improve pre-treatment measurements, and finally (6) the methods listed in the above-linked article: “confirmatory tests, large sample sizes, preregistration, and methodological transparency.”

I sent this post to Nosek, and he replied:

For your list of practices:

#1: We did this for both methodological and statistical practices.

#2: I suspect that every lab was motivated to get the largest effect that they could given the research question that they were studying (ours certainly was). But, you’ll observe in the findings that we didn’t get very large effect sizes on average. Instead, they are what I believe are around what most “real” effect sizes are for the messy concepts that social scientists study.

#3: We didn’t do this. Each lab used a sampling firm and all studies were conducted through that firm. It is possible that a lab would have tried to tailor the design to the sample, but these were very heterogeneous samples, so that would not likely have been very effective.

#4: I suspect that every lab did this the best that they could. Simultaneously, most of the research in this is pretty on-the-edge discovery work, so not necessarily a lot of existing evidence to make use of (with variation across experiments and labs).

#5: I suspect that this was done for a couple of experiments from some labs, but not others. (None from mine did so.)

I like all of your suggestions for improving rigor. I would counterargue that some of them become more meaningfully impactful on the research process as the evidence base matures (e.g., where to get the largest effect size, what are effective pretreatment measurements). In the context of discovery research like the experiments in this paper, we could only speculate about these in trying to design the most rigorous studies. The practices that we highlight are “easily” applied no matter the maturity of the domain and evidence base.

On your other points: I think the paper provides proof-of-concept that even small effects are highly replicable. And, I am much more sanguine than you are about the benefits of preregistration. Maybe we can find some time to argue about that in the future!