There are probably lots of clever answers to this one, but I’ll go with: One of them was hyped in the media as a clean-cut fresh face that would restore fan confidence in a tired, scandal-plagued entertainment cartel—and the other is a retired baseball player.
Let me put it another way. Derek Jeter had three salient attributes:
1. He was an excellent baseball player, rated by one source at the time of his retirement as the 58th best position player of all time.
2. He was famously overrated.
3. He was a symbol of integrity.
The challenge is to hold 1 and 2 together in your mind.
I was thinking about this after Palko pointed me to a recent article by Rose McDermott that begins:
Pre-registration has become an increasingly popular proposal to address concerns regarding questionable research practices. Yet preregistration does not necessarily solve these problems. It also causes additional problems, including raising costs for more junior and less resourced scholars. In addition, pre-registration restricts creativity and diminishes the broader scientific enterprise. In this way, pre-registration neither solves the problems it is intended to address, nor does it come without costs. Pre-registration is neither necessary nor sufficient for producing novel or ethical work. In short, pre-registration represents a form of virtue signaling that is more performative than actual.
I think this is like saying, “Derek Jeter is no Cal Ripken, he’s overrated, gets too much credit for being in the right place at the right time, he made the Yankees worse, his fans don’t understand how the game of baseball really works, and it was a bad idea to promote him as the ethical savior of the sport.”
Here’s what I think of preregistration: It’s a great idea. It’s also not the solution to problems of science. I have found preregistration to be useful in my own work. I’ve seen lots of great work that is not preregistered.
I disagree with the claim in the above-linked paper that “Under the guidelines of preregistration, scholars are expected to know what they will find before they run the study; if they get findings they do not expect, they cannot publish them because the study will not be considered legitimate if it was not preregistered.” I disagree with that statement in part for the straight-up empirical reason that it’s false; there are counterexamples; indeed a couple years ago we discussed a political science study that was preregistered and yielded unexpected findings which were published and were considered legitimate by the journal and the political science profession.
More generally, I think of preregistration as a floor, not a ceiling. The preregistered data collection and analysis is what you need to do. In addition, you can do whatever else you want.
Preregistration remains overrated if you think it’s gonna fix science. Preregistration facilitates the conditions for better science, but if you preregister a bad design, it’s still a bad design. Suppose you could go back in time and preregister the collected work of the beauty-and-sex-ratio guy, the ESP guy, and the Cornell Food and Brand Lab guy, and then do all those studies. The result wouldn’t be a spate of scientific discoveries; it would just be a bunch of inconclusive results, pretty much no different than the inconclusive results we actually got from that crowd but with the improvement that the inconclusiveness would have been more apparent. As we’ve discussed before, the benefits of procedural reforms such as preregistration are indirect—making it harder for scientists to fool themselves and others with bad designs—but not direct. Are these indirect benefits greater than the costs? I don’t know; maybe McDermott is correct that they’re not. I guess it depends on the context.
I think preregistration can be valuable, and I say that while recognizing that it’s been overrated and inappropriately sold as a miracle cure for scientific corruption. As I wrote a few years ago:
In the long term, I believe we as social scientists need to move beyond the paradigm in which a single study can establish a definitive result. In addition to the procedural innovations [of preregistration and mock reports], I think we have to more seriously consider the integration of new studies with the existing literature, going beyond the simple (and wrong) dichotomy in which statistically significant findings are considered as true and nonsignificant results are taken to be zero. But registration of studies seems like a useful step in any case.
Derek Jeter was overrated. He was a times a drag on the Yankees’ performance. He was still an excellent player and overall was very much a net positive.
As said elsewhere “Pre-flight checklists and pre-operative checklists are also unnecessary if you are thinking prudently but they’re still good policy”
https://twitter.com/dggoldst/status/1567362698282782720
Yup; well put.
Thanks for writing this commentary Andrew. It saves me lots of time answering questions and writing a similar one myself.
One group of individuals that need to get on board with what pre-registration is really about is reviewers. You do run into some from time to time who don’t get that it’s just about openness and not about having to follow the original plan to the letter. But they’re perhaps less common than the ones who don’t look at pre-registrations at all.
Psyoskeptic:
No problem. Youall pay enough for this blog; I should give you value for the money.
I believe just about everything outside of social science is “pre-registered”, isn’t it? If you expect to spend three seasons in the field on an ecological study, you’re expected to present the question(s) you expect to answer and show how the data you plan to obtain and the analysis you plan to perform will answer the question(s). If you’re lobbying for a mission to Mars, you’re expected to explain what you’re trying to achieve and specifically how you plan to achieve it. None of that stops researchers from further exploring the existing data, acquiring additional data or performing additional analysis and often it’s justified to obtain additional data that doesn’t directly bear on the stated questions for a small expense, given that logistics are already paid for.
Chipmunk:
No, lots of things outside of social science are not preregistered. I can speak from experience: of the various non-social-science projects here, exactly zero were preregistered.
Your confusion might arise in part from terminology. If you want funds for an experiment, whether it be a mission from Mars or a medical experiment, you’ll need to write some sort of proposal and give some sort of plan of what you will do, but that plan will not typically be a “pre-registration” in the sense that this term is used. A statement along the lines of, “We plan to study X, Y, and Z, and we have the demonstrated capacity to do the experiment and analyze the data,” is not a pregregistration in the sense of being a full protocol for design, data collection, and analysis.
>full protocol for design, data collection, and analysis.
Well, the proposals for funding that I have worked on involve all of those things…they’re basically like a pre-registration in the sense that you seem to use.
So how much registration detail constitutes ‘pre-registration’ in the sense that you use it on this blog? I tried to Google a definition, and found this (among others):
“Preregistration, in its simplest form, is a one-page document answering basic questions such as: What question will be studied? What is the hypothesis? What data will be collected, and how will they be analyzed? In its most rigorous form, a “registered report,” researchers write an entire paper, minus the results and discussion, and submit it for peer review at a journal, which decides whether to accept it in principle. After the work is completed, reviewers simply check whether the researchers stuck to their own recipe; if so, the paper is published, regardless of what the data show.”
https://www.science.org/content/article/more-and-more-scientists-are-preregistering-their-studies-should-you
I thought the main purpose was simply to document in advance the inferential goals.
https://statmodeling.stat.columbia.edu/2022/08/18/preregistered-vs-adaptable-bayesian-workflow-and-who-should-do-the-work/#comment-2072391
For example, pre-registering the exact model specification is problematic as pointed out by Betancourt in the quote in the above linked post. However, I think it’s useful to state the primary outcomes at least.
What amount of detail in preregistration is useful?
Jd:
There are different goals. In no sense should preregistration be thought of as restricting. You don’t preregister “the” exact model specification; you preregister “an” exact model specification, and you’re always free to fit other models once you’ve seen the data.
Statistics as usually taught, tries to “shed probability”. In effect, it takes instances where there’s slight opportunity for something and de-facto declares there’s no possibility of it happening. A sort of probability laundering.
As long as you’re equating “slight opportunity” with “no possibility”, it creates endless problems. And that’s true even if the calculation of “slight opportunity” was done correctly and based on true assumptions, which typically it is not.
You can. for example, string a series of such “slight opportunity = no possibility” assertions together and get results that are hopelessly out of wack with reality as the errors build up. Indeed, you can reach any conclusion you want this way.
Or you can fish around for an instance of “slight opportunity = no possibility” which leads to an error that suits your purposes.
Preregistration, like most solutions to these problems, tries to fix this by artificially limiting the number of cases where a researcher can declare “slight opportunity = no possibility”. And in a sense it works. But a better solution would be to go all the way and never make the claim that “slight opportunity =no possibility”. In effect, eliminate every opportunity for doing so.
Now if only there was a way of doing statistics that propagated all uncertainties throughout, so that if the data suggests something is possible, it always remains a possibility, with no probabilities being “shed” ….
Jbayes:
Yes, what you’re referring to at the beginning of your comment is what I refer to as “premature collapsing of the wave function.”
So in short, some pre-registered designs may be bad designs, and some non-registered designs may be good designs.
I want to advocate pre-registering interpretations and actions! I find it a useful defence against people who have pre-conceived conclusions from which they won’t budge regardless of evidence. For example, the marketing team believes that spending $$$ on ads will definitely increase sales. Working with statisticians, they pre-register a test to measure the effect of ads on sales. What I push to pre-register in addition to test design is what we would do once the analysis is finished. Of course, if the ads are shown to increase sales, we both agree that they should receive the budget to run more such ads. What if the ads are shown to decrease sales? In pre-registration, we agree that they would stop running these ads. WIthout the preregistration, statisticians are frequently told to keep running analyses until a story can be created to justify continuing the ads despite a negative outcome.
Just like pre-registered designs, pre-registering these intentions does not mean that the analyst cannot investigate other interpretations or actions. It’s a floor, not a ceiling, as you said.
Many decades ago, in a previous life, I worked in the field we used to call “computers” but would now be called “IT”. I was contacted by a company who wanted to hire me under contract for a few years with a salary far above what I was earning at the time. So I did the interview.
They wanted me to prepare an “analysis” of their computing needs in order to justify spending a huge amount of money switching from one mainframe vendor to another. The vendor they wanted to switch to was the company with which I was at the time employed. That’s why they wanted to recruit me to do the changeover.
Based on what they showed me, there were certainly advantages to making the switch. At least from my point of view as somewhat of an expert in the system they wanted to switch to. But it was an open question whether the advantages were worth the enormous cost of revamping their entire computing infrastructure company-wide.
So I had to ask “elephant in the room” question. What if my analysis showed that the benefits of switching were not sufficient to justify the change? Wouldn’t I be out of a job after just a year or so?
The person I was interviewing with patiently explained that by hiring me to do the analysis they were confident my own self-interest would guarantee I’d come up with the right answer. So OK, bonus points for honesty but it just all smelled fishy to me.
Today I routinely encounter research that has been peer reviewed and published where the same sort of incentives strongly apply to both the researchers and the peer reviewers. Guess I missed the boat all those years ago and should have just taken the money and done a very well paid 3-4 year gig that I could leverage into another similar position at the next company.
Have you seen “The Laws of Science” by Niko McCarty (https://newscience.substack.com/p/laws-of-science)?
He discusses preregistration in some detail, but in particular, his very first figure looks (results of trials with and without mandatory registration) pretty alarming. (It was reproduced from Kaplan R.M. & Irvin V.L. PLOS ONE (2015))
Dace:
I like some of that post by McCarty but I think he’s a bit too idealistic about the legal system, which he is using as his comparison point.
McDermott is building an army of strawmen.
Mark:
My (uninformed) guess is that McDermott is annoyed at the grand claims that have been made for preregistration and is reacting by disparaging the idea, in the same way that people were annoyed at the Derek Jeter hype and followed up by underrating him.