Johannes Haushofer asks:
Later in the thread, the Millennium Villages Project (MVP) evaluations came up:
I thought:
1. Our MVP evaluation says:
The design was completed before endline data collection, and a peer-reviewed evaluation protocol was registered with The Lancet.
Could the protocol have been more detailed and constrained more researcher degrees of freedom ? Yes.
This may not be a solution though, as Michael Betancourt writes:
Preserving a preregistered model only perpetuates its optimistic assumptions about the realization of an experiment. In practice we almost always have to adapt our model to the peculiarities of the observed data.
The Bayesian workflow authors say:
Our claim is that often a model whose assumptions withstood such severe tests is, despite being the result of data-dependent iterative workflow, more trustworthy than a preregistered model that has not been tested at all.
2. Eliciting domain expertise should probably involve project leadership. But how ?
3. I’m flattered to be considered by Johannes to be a trustworthy researcher. I also prefer publication procedures that rely less on authors’ reputation and more on methods clarity.
Andrew thought:
It’s funny that they characterize us as being “independent,” given that both of us had Earth Institute associations at the time we did the project.
In the sentence right before the one that you quote, Betancourt writes, “An immediate corollary is that while ideas like preregistration can be helpful for documenting the initial inferential goals of an experiment they are nowhere near sufficient for a successful analysis in practice.”
I thought the main idea of preregistration was preregistration of inferential goals. It seems like this is what the twitter comment is more toward, and not preregistration of the model.
It seems to me that preregistering the inferential goals would help with the scenario where a large study is conducted, the original inferential goals turn out to be ‘not significant’, and so the publication focuses on a bunch of other stuff that was ‘found’ instead, for which the original study was neither designed for nor powered for.
Preregistering every detail of the model wouldn’t seem to make much sense anyway.
And to the extent you need to diverge from your pre-registration, that’s fine. Do all the post hoc analyses you want or need to, no one’s stopping you! You just have to say that’s what they are. I don’t understand why opponents of pre-registration all seem to balk at this same point and all apparently find adding 2 words like `# Unregistered Analyses` to be such an incredible burden – it is among the very easiest of all methodological reforms ever proposed.
Gwern:
I’m sure I’m missing something, but my impression is that the criticisms of preregistration come from two directions:
1. People who do unreplicable or unreplicated research, or who suspect their research is unreplicable, and who don’t like the idea of a system such as preregistration that would allow other people’s research to be reliably replicated.
2. People who think preregistration has been oversold, to the extent that they feel that, on net, the “preregistration” package has done more harm than good by focusing on less important aspects of science reform and replicability.
Position #1 annoys me (and of course I guess that just about nobody would admit to holding that position). Position #2 I disagree with—or, to put it another way, even if “the preregistration package” has done more harm than good, that’s no reason why we can’t do plain vanilla preregistration and make our research better—but I can kinda see where people who hold that position are coming from, as it’s related to the point I’ve often made that all the preregistration and clean science in the world won’t help you if your data are noisy and your theories are crap.
Anyway, I agree with your position and I’m pro-preregistration myself. I just think it’s good for us to recognize point #2, even when people stating that point mess it up by making large unsupported claims.