Brendan Nyhan writes:
Wanted to share this open letter. I know preregistration isn’t useful for the style of research you do, but even for consumers of preregistered research like you it’s essential to know if the preregistration was actually disclosed to and reviewed by reviewers, which in turn helps make sure that exploratory and confirmatory analyses are adequately distinguished, deviations and omissions labeled, etc. (The things I’ve seen as a reviewer… are not good – which is what motivated me to organize this.)
The letter, signed by Nyhan and many others, says:
It is essential that preregistrations be considered as part of the scientific review process.
We have observed a lack of shared understanding among authors, editors, and reviewers about the role of preregistration in peer review. Too often, preregistrations are omitted from the materials submitted for review entirely. In other cases, manuscripts do not identify important deviations from the preregistered analysis plan, fail to provide the results of preregistered analyses, or do not indicate which analyses were not preregistered.
We therefore make the following commitments and ask others to join us in doing so:
As authors: When we submit an article for review that has been preregistered, we will always include a working link to a (possibly anonymized) preregistration and/or attach it as an appendix. We will identify analyses that were not preregistered as well as notable deviations and omissions from the preregistration.
As editors: When we receive a preregistered manuscript for review, we will verify that it includes a working link to the preregistration and/or that it is included in the materials provided to reviewers. We will not count the preregistration against appendix page limits.
As reviewers: We will (a) ask for the preregistration link or appendix when reviewing preregistered articles and (b) examine the preregistration to understand the registered intention of the study and consider important deviations, omissions, and analyses that were not preregistered in assessing the work.
I’ve actually been moving toward more preregistration in my work. Two recent studies we’ve done that have been preregistered are:
– Our project on generic language and political polarization
– Our evaluation of the Millennium Villages project
And just today I met with two colleagues on a medical experiment that’s in the pre-design stage—that is, we’re trying to figure out the design parameters. To do this, we need to simulate the entire process, including latent and observed data, then perform analyses on the simulated data, then replicate the entire process to ensure that the experiment will be precise enough to be useful, at least under the assumptions we’re making. This is already 90% of preregistration, and we had to do it anyway. (See recommendation 3 here.)
So, yeah, given that I’m trying now to simulate every study ahead of time before gathering any data, preregistration pretty much comes for free.
Preregistration is not magic—it won’t turn a hopelessly biased, noisy study into something useful—but it does seem like a useful part of the scientific process, especially if we remember that preregistering an analysis should not stop us from performing later, non-preregistered analyses.
Preregistration should be an addition to the research project, not a limitation!
I guess that Nyhan et al.’s suggestions are good, if narrow in that they’re focused on the very traditional journal-reviewer system. I’m a little concerned with the promise that they as reviewers will “examine the preregistration to understand the registered intention of the study and consider important deviations, omissions, and analyses that were not preregistered in assessing the work.” I mean, sure, fine in theory, but I would not expect or demand that every reviewer do this for every paper that comes in. If I had to do all that work every time I reviewed a paper, I’d have to review many fewer papers a year, and I think my total contribution to science as a reviewer would be much less. If I’m gonna go through and try to replicate an analysis, I don’t want to waste that on a review that only 4 people will see. I’d rather blog it and maybe write it up on some other form (as for example here), as that has the potential to help more people.
Anyway, here’s the letter, so go sign it—or perhaps sign some counter-letter—if you wish!
I’d love to check pre-registrations as a reviewer. But the typical pre-reg form, I’m thinking of OSF’s, is terrible for that purpose. It could easily double the review process for a lot of papers. They really need to just write up a guide / checklist that people can use for a narrative pre-registration statement. It would make life so much easier all around.
“If I’m gonna go through and try to replicate an analysis, I don’t want to waste that on a review that only 4 people will see. I’d rather blog it and maybe write it up on some other form (as for example here), as that has the potential to help more people.”
Ideally, a journal that publishes a paper with a pre-registration should allow and encourage replications for the paper in the same journal, maybe with a lighter peer review. As it stands now, the discussions are mostly elsewhere than in the journals where they should be.
Quote from above: “I’m a little concerned with the promise that they as reviewers will “examine the preregistration to understand the registered intention of the study and consider important deviations, omissions, and analyses that were not preregistered in assessing the work.””
I would feel better when this open letter would talk about how and why the readers of the actual paper should be able to verify things, and for the pre-registration information to be available to them first and foremost. This automatically would also result in peer-reviewers being able to have access to pre-registration information I reason.
Subtleties like that matter a lot to me, especially since I am worried about the possible abuse, commercialization, and control concerning many things in present-day science. In that light, I’m a little concerned as well, but perhaps for slightly different reasons.
I’m mainly concerned that this emphasis on the peer-review role of checking the pre-registration might be one step towards the commercialization of peer-review in a similar way that perhaps the publishing of papers has been commercialized. This could possibly go like follows:
Step 1: make pre-registration a “thing”
Step 2: incorporate pre-registration into the peer-review process (e.g. see “Registered Reports” or whatever they are called) or emphasize that peer-reviewers should be able to “verify” pre-registration (and not for instance the reader of the paper)
Step 3: have some third party “verify” pre-registration (and perhaps subsequently not even make the pre-registration information available to the reader of the paper), because of “reasons” (e.g. peer-reviewers don’t have time to do this all)
Step 4: profit
Step 5: extend to other things (e.g. study design) for more bureaucracy, profit, and control
Qote from above: “Anyway, here’s the letter, so go sign it—or perhaps sign some counter-letter—if you wish!”
I wish I could sign an open letter on the reasons why open letters should not be used or given attention.
Quote from above: “I mean, sure, fine in theory, but I would not expect or demand that every reviewer do this for every paper that comes in. If I had to do all that work every time I reviewed a paper, I’d have to review many fewer papers a year, and I think my total contribution to science as a reviewer would be much less.
Thank goodness reviewers DO have time to critically check and verify and assess statistical analysis, choices concerning measures used, soundness of reasoning and argumentation, etc. Imagine what the scientific literature in certain fields might look like if reviewers didn’t have time to check and examine and assess those things…
I sometimes have titles and ideas for manuscripts that involve wordplay. It seems that it’s one of the things I like the most in scientific writing. I recently had the idea of a manuscript with the title “Psych101 or Psych1984?” that would talk about certain things in Psychological Science that I find weird and/or absurd.
I dislike reading books, and most other things substantially longer than a quote or a blog-comment, but I think the book “1984” includes certain things that I could connect to certain things that I find weird and/or absurd in Psychological Science. This blogpost and some remarks reminded me of this all.
For example, to me it’s weird and/or absurd:
– that some people who use pre-registration apparently do not add a link in the paper (?) so peer-reviewers and readers can read the information when I think that’s the most important function of it all
– that some people think peer-reviewers don’t have time to check the pre-registration information when I think that that’s just as time consuming as checking other things in a paper that could/should be checked. Also: what are peer-reviewers even for when they would not have time to check these kinds of things?!
– that some people seem to value scientific credit and ownership (or whatever the appropriate term is) by using and possibly even emphasizing names and authorship, but to me seem to also value and possibly even emphasize the role of anonymous peer-review
I may have more examples like this, and could possibly start beginning to write a manuscript about this all, but I would have to start reading certain manuscripts from certain people on certain topics again and I don’t want to do that anymore. Thank goodness for blogs like this one where I can sort of still enjoy wordplay in a way so to say.
Those who find pre-registration onerous may cite instead Philip Cole’s “Hypothesis generating machine” (Epidemiology, 1993). Any and all causal associations pre-specified!
More seriously, I don’t see how pre-registration improves science unless all pre-registered analyses are made public, interesting or not, no exceptions. Is that part of the deal? An alternative might be for researchers to explain candidly how a certain observation was made, planned or serendipity. But if the latter is seen as moral weakness or misconduct, that is unlikely to happen.
Shouldn’t they have conducted an RCT to see if signing at the top or the bottom promotes better research practice?
Dale:
Haven’t you been keeping up with the literature?? These effects depend on the weather. It’s November already, so that signing-at-the-top thing will no longer work, and they don’t want to wait for the spring.
Am I doing this meme-thing correctly when linking to the following youtube clip?
https://www.youtube.com/watch?v=ebYA4cj6MiE
Anon:
If you caught that reference to “The Impact of Weather on Women’s Tendency to Wear Red or Pink when at High Risk for Conception,” then . . . you, like me, spend too much time on the internet!
When reading “These effects depend on the weather” in your reply I was thinking you were likely refering to some priming-effect like research from a few years back. So yes, I think I got the gist of the reference. Although I did not exactly have that specific paper in mind.
Bit surprised to see the following statement:
“I guess that Nyhan et al.’s suggestions are good, if narrow in that they’re focused on the very traditional journal-reviewer system. I’m a little concerned with the promise that they as reviewers will “examine the preregistration to understand the registered intention of the study and consider important deviations, omissions, and analyses that were not preregistered in assessing the work.” I mean, sure, fine in theory, but I would not expect or demand that every reviewer do this for every paper that comes in. If I had to do all that work every time I reviewed a paper, I’d have to review many fewer papers a year, and I think my total contribution to science as a reviewer would be much less.”
How can one assess the quality and validity of an experiment, without knowing if it is actually deductive research vs. potentially p-hacked? Also, it usually only takes me a few minutes to read a pre-registration (that one usually, not always, can find quickly, at least in my field).
Carsten:
In reviewing a paper, I just want to do my part. I don’t imagine that my review will be a comprehensive assessment of the paper. Correctness of the claims is ultimately the responsibility of the author. My goals as reviewer are: (a) to help the author improve the paper, and (b) to help the editor decide whether to publish it. For reasons of efficiency and diminishing returns, I don’t spend a lot of time on the review. In my review, I just need to move the ball forward; I don’t need to score a touchdown.
Simulated data ought to be an expected component of any data analysis. I really don’t understand how this still isn’t part of the standard for a minimal experiment design.
Dan:
I dunno, but I think that something like 99.9% of designs don’t use simulated data.