This is Jessica. Null results are hard to take. This may seem especially true when you preregistered your analysis, since technically you’re on the hook to own up to your bad expectations or study design! How embarrassing. No wonder some authors can’t seem to give up hope that the original hypotheses were true, even as they admit that the analysis they preregistered didn’t produce the expected effects. Other authors take an alternative route, one that deviates more dramatically from the stated goals of preregistration: they bury aspects of that pesky original plan and instead proceed under the guise that they preregistered whatever post-hoc analyses allowed them to spin a good story. I’ve been seeing this a lot lately.
On that note, I want to follow up on the previous blog discussion on the 2023 Nature Human Behavior article “High replicability of newly discovered social-behavioural findings is achievable” by Protzko, Krosnick, Nelson (of Data Colada), Nosek (of OSF), Axt, Berent, Buttrick, DeBell, Ebersole, Lundmark, MacInnis, O’Donnell, Perfecto, Pustejovsky, Roeder, Walleczek, and Schooler. It’s been about four months since Bak-Coleman and Devezer posted a critique that raised a number of questions about the validity of the claims the paper makes.
This was a study that asked four labs to identify (through pilot studies) four effects for possible replication. The same lab then did a larger (n=1500) preregistered confirmation study for each of their four effects, documenting their process and sharing it with three other labs, who attempted to replicate it. The originating lab also attempted a self-replication for each effect.
The paper presents analyses of the estimated effects and replicability across these studies as evidence that four rigor-enhancing practices used in the post-pilot studies–confirmatory tests, large sample sizes, preregistration, and methodological transparency–lead to high replicability of social psychology findings. The observed replicability is said to be higher than expected based on observed effect sizes and power estimates, and notably higher than prior estimates of replicability in the psych literature. All tests and analyses are described as preregistered, and, according to the abstract, the high replication rate they observe “justifies confidence of rigour-enhancing methods to increase the replicability of new discoveries.”
On the surface it appears to be an all-around win for open science. The paper has already been cited over fifty times. From a quick glance, many of these citing papers refer to it as if it provides evidence of a causal effect that open practices lead to high replicability.
But one of the questions raised by Bak-Coleman and Devezer about the published version was about their claim that all of the confirmatory analyses they present were preregistered. There was no such preregistration in sight if you checked the provided OSF link. I remarked back in November that even in the best case scenario where the missing preregistration was found, it was still depressing and ironic that a paper whose message is about the value of preregistration could make claims about its own preregistration that it couldn’t back up at publication time.
Around that time, Nosek said on social media that the authors were looking for the preregistration for the main results. Shortly after Nature Human Behavior added a warning label indicating an investigation of the work:
And if I’m trying to tell the truth, it’s all bad
It’s been some months, and the published version hasn’t changed (beyond the added warning), nor do the authors appear to have made any subsequent attempts to respond to the critiques. Given the “open science works” message of the paper, the high profile author list, and the positive attention it’s received, it’s worth discussing here in slightly more detail how some of these claims seem to have come about.
The original linked project repository has been updated with historical files since the Bak-Coleman and Devezer critique. By clicking through the various versions of the analysis plan, analysis scripts, and versions of the manuscript, we can basically watch the narrative about the work (and what was preregistered) change over time.
The first analysis plan is dated October 2018 by OSF, and outlines a set of analyses of a decline effect, where effects decrease after an initial study, that differ substantially from the story presented in the published paper. This document first describes a data collection process for each of the confirmation studies and replications in two halves that splits the collection of observations into two parts, with 750 observations collected first, and the other 750 second. Each confirmation study and replication study are also assigned to either a) analyze the first half-sample and then the second half-sample or b) analyze the second half-sample and then the first half-sample.
There were three planned tests:
- Whether the effects statistically significantly increase or decrease depending on whether the effects belonged to the first or the second 750 half samples;
- Whether the effect sizes of the originating lab’s self-replication study is statistically larger or smaller than the originating lab’s confirmation study.
- Whether effects statistically significantly decrease or increase across all four waves of data collection (all 16 studies with all 5 confirmations and replications).
If you haven’t already guessed it, the goal of all this is to evaluate whether a supernatural-like effect resulted in a decreased effect size in whatever wave was analyzed second. It appears all this is motivated by hypotheses that some of the authors (okay, maybe just Schooler) felt were within the realm of possibility. There is no mention of comparing replicability in the original analysis plan nor the the preregistered analysis code uploaded last December in a dump of historical files by James Pustejovsky, who appears to have played the role of a consulting statistician. This is despite the blanket claim that all analyses in the main text were preregistered and further description of these analyses in the paper’s supplement as confirmatory.
The original intent did not go unnoticed by one of the reviewers (Tal Yarkoni) for Nature Human Behavior, who remarks:
The only hint I can find as to what’s going on here comes from the following sentence in the supplementary methods: “If observer effects cause the decline effect, then whichever 750 was analyzed first should yield larger effect sizes than the 750 that was analyzed second”. This would seem to imply that the actual motivation for the blinding was to test for some apparently supernatural effect of human observation on the results of their analyses. On its face, this would seem to constitute a blatant violation of the laws of physics, so I am honestly not sure what more to say about this.
It’s also clear that the results were analyzed in 2019. The first public presentation of results from the individual confirmation studies and replications can be traced to a talk Schooler gave at the Metascience 2019 conference in September, where he presents the results as evidence of an incline effect. The definition of replicability he uses is not the one used in the paper.
Cause if you’re looking for the proof, it’s all there
There are lots of clues in the available files that suggest the main message about rigor-enhancing practices emerged as the decline effects analysis above failed to show the hypothesized effect. For example, there’s a comment on an early version of the manuscript (March 2020) where the multi-level meta-analysis model used to analyze heterogeneity across replications is suggested by James. This is suggested after data collection has been done and initial results analyzed, but the analysis is presented as confirmatory in the paper with p-values and discussion of significance. As further evidence that it wasn’t preplanned, in a historical file added more recently by James, it is described as exploratory. It shows up later in the main analysis code with some additional deviations, no longer designated as exploratory. By the next version of the manuscript, it has been labeled a confirmatory analysis, as it is in the final published version.
This is pretty clear evidence that the paper is not accurately portraying itself.
Similarly, various definitions of replicability show up in earlier versions of the manuscript: the rate at which the replication is significant, the rate at which the replication effect size falls within the confirmatory study CI, and the rate at which replications produce significant results for significant confirmatory studies. Those which produce higher rates of replicability relative to statistical power are retained and those with lower rates are either moved to the supplement, dismissed, or not explored further because they produced low values. For example, defining replicability using overlapping confidence intervals was moved to the supplement and not discussed in the main text, with the earliest version of the manuscript (Deciphering the Decline Effect P6_JEP.docx) justifying its dismissal because it “produced the ‘worst’ replicability rates” and “performs poorly when original studies and replications are pre-registered.” Statistical power is also recalculated across revisions to align with the new narrative.
In a revision letter submitted prior to publication (Decline Effect Appeal Letterfinal.docx), the authors tell the reviewers they’re burying the supernatural motivation for study:
Reviewer 1’s fourth point raised a number of issues that were confusing in our description of the study and analyses, including the distinction between a confirmation study and a self-replication, the purpose and use of splitting samples of 1500 into two subsamples of 750, the blinding procedures, and the references to the decline effect. We revised the main text and SOM to address these concerns and improve clarity. The short answer to the purpose of many of these features was to design the study a priori to address exotic possibilities for the decline effect that are at the fringes of scientific discourse.
There’s more, like a file where they appeared to try a whole bunch of different models in 2020 after the earliest provided draft of the paper, got some varying results, and never disclose it in the published version or supplement (at least I didn’t see any mention). But I’ll stop there for now.
C’mon baby I’m gonna tell the truth and nothing but the truth
It seems clear that the dishonesty here was in service of telling a compelling story about something. I’ve seen things like this transpire plenty of times: the goal of getting published leads to attempts to find a good story in whatever results you got. Combined with the appearance of rigor and a good reputation, a researcher can be rewarded for work that on closer inspection involves so much post-hoc interpretation that the preregistration seems mostly irrelevant. It’s not surprising that the story here ends up being one that we would expect some of the authors to have faith in a priori.
Could it be that the authors were pressured by reviewers or editors to change their story? I see no evidence of that. In fact, the same reviewer who noted the disparity between the original analysis plan and the published results encouraged the authors to tell the real story:
I won’t go so far as to say that there can be no utility whatsoever in subjecting such a hypothesis to scientific test, but at the very least if this is indeed what the authors are doing, I think they should be clear about that in the main text, otherwise readers are likely to misunderstand what the blinding manipulation is supposed to accomplish, and are at risk of drawing incorrect conclusions
You can’t handle the truth, you can’t handle it
It’s funny to me how little attention the warning label or the multiple points raised by Bak-Coleman and Devezer (which I’m simply concretizing here) have drawn, given the zeal with which some members of open science crowd strike to expose questionable practices in other work. My guess is this is because of the feel-good message of the paper and the reputation of the authors. The lack of attention seems selective, which is part of why I’m bringing up some details here. It bugs me (though doesn’t not surprise me) to think that whether questionable practices get called out depends on who exactly is in the author list.
What do I care? Why should you?
On some level, the findings the paper presents – that if you use large studies and attempt to eliminate QRPs, you can get a high rate of statistical significance – are very unsurprising. So why care if the analyses weren’t exactly decided in advance? Can’t we just call it sloppy labeling and move on?
I care because if deception is occurring openly in papers published in a respected journal for behavioral research by authors who are perceived as champions of rigor, then we still have a very long way to go. Interpreting this paper as a win for open science, as if it cleanly estimated the causal effect of rigor-enhancing practices is not, in my view, a win for open science. The authors’ lack of concern for labeling exploratory analysis as confirmatory, their attempt to spin the null findings from the intended study into a result about effects on replicability even though the definition they use is unconventional and appears to have been chosen because it led to a higher value, and the seemingly selective summary of prior replication rates from the literature should be acknowledged as the paper accumulates citations. At this point months have passed and there have not been any amendments to the paper, nor admission by the authors that the published manuscript makes false claims about the preregistration status. Why not just own up to it?
It’s frustrating because my own methodological stance has been positively impacted by some of these authors. I value what the authors call rigor-enhancing practices. In our experimental work, my students and I routinely use preregistration, we do design calculations via simulations to choose sample sizes, we attempt to be transparent about how we arrive at conclusions. I want to believe that these practices do work, and that the open science movement is dedicated to honesty and transparency. But if papers like the Nature Human Behavior article are what people have in mind when they laud open science researchers for their attempts to rigorously evaluate their proposals, then we have problems.
There are many lessons to be drawn here. When someone says all the analyses are preregistered, don’t just accept them at their word, regardless of their reputation. Another lesson that I think Andrew previously highlighted is that researchers sometimes form alliances with others that may have different views for the sake of impact but this can lead to compromised standards. Big collaborative papers where you can’t be sure what your co-authors are up to should make all of us nervous. Dishonestly is not worth the citations.
Writing inspiration from J.I.D. and Mereba.
Data Colada should look into this.
That’s like that movie, No Way Out, where Kevin Costner is tasked to solve a crime, and all the evidence points to him!
And he’s also guilty.
Well, he was a Russian spy. But he didn’t kill Sean Young.
Sorry for the pedantry, but it may be of interest that the plot structure of No Way Out, which has been used several times in movies, traces back to the 1946 novel The Big Clock, by Kenneth Fearing, which is good!
I’d like to pre-register a prediction that JID will become the greatest rapper of his generation.
I’ll define the criteria for winning after he’s won.
I see your preregistration and I’ll vouch for it no matter what you do to that data.
“Interpreting this paper as a win for open science, as if it cleanly estimated the causal effect of rigor-enhancing practices is not, in my view, a win for open science”
even if the study were properly pre-specified, isn’t this a misinterpretation of its estimand, since there is no control group who didn’t use the “rigor-enhancing practices”?
“The observed replicability is said to be higher than expected based on observed effect sizes and power estimates, and notably higher than prior estimates of replicability in the psych literature.”
^ this seems to be the only benchmark they provide. doesn’t seem like a great control group–maybe they just chose highly likely to replicate estimates?
Jessica:
I remember that 2019 Open Science meeting! It was a strange mix of talks, some by science-reformers such as Nosek and myself and some by old-school types. I remember one talk that started going on about how brilliant Albert Einstein or Elon Musk was at the age of 11, another talk all about Nobel prize winners . . . that stuff got me so annoyed I just quietly slipped out of the auditorium and walked outside the building into the warm California sun . . . and there I met some other conference participants who were equally disgusted by that bogus hero-worship thing . . . I’d found my science-reform soulmates! I also remember Schooler’s talk, not in detail but I do remember being stunned that he was actually talking about ESP. Really going there, huh? It gave off a real 70s vibe, kinda like when I took a psychology class in college and the professor recommended that we try mind-altering drugs.
The funder for the event and this study is quite strange. It lists only two funded projects (this one, and one more), and really does evoke a 70s infatuation with the mystical and spiritual. They seem to be particularly interested in figuring out if our physical understanding of the world is incomplete. This would explain funding work on the supernatural, but doesn’t explain how these authors got (quantum?) entangled. Bem should get in touch.
“Is scientific physicalism incomplete? What metaphysical assumptions underpin modern science? Is the reality that science describes absolute or model-dependent? How to differentiate science from pseudoscience? Might a future science include free will and consciousness in its frame? Is a relational science truly possible?”
Jessica:
Stellar analysis!
My hypothesis (exploratory, not preregistered) is that even if you’re a science reformer extraordinaire, a five-year, highly intensive study must be balanced with favorable results to justify the work. The authors’ other lower-resource (shorter) studies may be worth a look to see if they have fewer deviations.
It seems that another bias-mitigation tool/method beyond preregistration is needed. Maybe some kind of psychological intervention to help authors split their identity from the work invested in a study.
“a five-year, highly intensive study must be balanced with favorable results to justify the work. ”
An alternate way to justify all that work is to do some *actual science* that *actually tests* a sensible hypothesis using *sound methods* that have previously proven successful! whoa, where’s the “mind is blown” emoji on my keyboard?
If you have a real hypothesis – e.g., something based on actual knowledge and grounded in theory rather than the typical social science hypothesis like “I hypothesize that when men flex their muscles in the mirror before asking a woman on a date they will be more successful”; and a real method – rather than recruiting six 12 year olds from the local playground to test the hypothesis among themselves, analyzing the data with NHST methods, generalizing the results to all human beings and demanding the UN spend $12B on corrective action to help men be more succesful in mating – then the negative is just as important as the positive and there is no such thing as a “failed” experiment. That way you don’t have to worry about null results or p-hacking or forking paths or false assumptions any of the other attacks quack science unjustly suffers at the hand of the Science Stasi.
I agree that the longitudinal aspect probably made it feel like they deserved a paper in some prestigious journal, even though in the end the paper is not really contributing any clear, important insight. Some things are better published quietly (I feel this way about my own work a lot actually!) It’s like some people instead attempt to oversell in the hopes of clouding others’ judgments.
There are 17 authors. Isn’t the first question to ask, “who did what?” — in part so we can search for research assistants to blame.
Statistically speaking it was likely John (Jon/Jan) or Matt(Matthew).
Thank you for saving many of us a great deal of time Jessica… oops, starting writing that before your closing where you’re telling us to do all the work too…
rats…
Great post, Jessica.
This sentence (below) from your post highlights an underappreciated point in science: almost all scientists out there have basically no idea how to keep track of the files they generate while working on a paper.
” Nosek said on social media that the authors were looking for the preregistration for the main results.”
It doesn’t matter whether one has a PhD in computer science, statistics, whatever. All we can do is create one docx file after another, labeling it Final, Really_Final, This_One_is_REALLY_Final, and then lose them all
because you are using a Windows machine and/or because the files are stored in different directories all over your computer. Until quite recently, I was no different. File management was just pathetic. I’ve gotten better over the last 10 years.
After reading your post, my main question is: why hasn’t Nosek arranged for a retraction of this paper? Doesn’t he have a rep to protect? Or is he now also too big to fail?
I’m again struck by the fact that that Gino lady loses her job for what is probably total incompetence in data interpretation and analysis (hardly atypical), but her co-author Ariely, and these people’s shenanigans have zero impact on their papers. At the very least this paper needs an extensive correction; the best way is that it be retracted.
Are there any examples of an author willingly retracting a paper central to the claims they’ve built their career on?
LOL (I assume this was a joke)
https://pubmed.ncbi.nlm.nih.gov/20947655/
The pattern still showed up in their other data though…
I predict that any correction made will be minor, and the authors will proceed as if the paper is big news for open science.
Reminds me of the newspaper headline that ran “Dewey beats Truman ” when in fact Truman won the election in 1948
Hey, at least “Dewey beats Truman” was preregistered.
This post is misclassified. It should be under Zombies.
Schooler lists under honors and awards on his CV:
Clarivate Analytics Web of Science™ Highly Cited Researcher 2022
he has several more of these on his cv.
I was wondering how this is an honor; it’s certainly not an award. I assume that if one has a certain count of citations, one gets listed automatically on this list; in which case, is the honor the honor of being cited some 40k times? If so, one could just list one’s google scholar citation count itself as an honor on one’s cv, right? I wonder if I should try this.
Jessica:
I saw some criticism online of this post, not with any evidence that you got anything wrong, but rather that there was somehow something improper for you to comment on this replication issue because, apparently, some processes are going on beyond closed doors that will provide future relevant information.
With that in mind, I want to offer my opinion that it’s always acceptable to criticize published work, and it’s acceptable to offer that criticism in an article, a book, a newspaper, a blog, instagram, twitter . . . any form. I do prefer some forms to others. In contrast to journal articles and books, blogs allow replies and discussions at any length.
My basic argument is simple: If work has been published, it is public, and it’s open to criticism.
To add a step here: If an article is published, anyone (or, possibly, anyone who’s willing to pay) can read it. It’s out there. To argue that you shouldn’t criticize because there’s some secret process happening, and a future report is coming . . . that’s just ridiculous! One might as well argue that nobody should criticize BDA3, because at some time in the future BDA4 will come out.
To say it one more time: the people who tell us No no you can’t criticize, You should wait until some new information comes out, etc. . . . those people never seem to have any problem when there is positive publicity about this work. They’re not out there telling people, No no you can’t cite this paper, No no you can’t praise it or use it to get publicity or fundraising, etc.
If these shushers were to start on their side by refusing to publicizing their work, refusing all media requests, refusing to use their work in grant applications, etc etc., then I’d have more respect for their position saying that we should shut up and wait.
I want to know who the heck said such a thing. You should call them out publicly; they shouldn’t be able to get away with it any more than the people who called their critics statistical terrorists or whatever it was.
Shravan:
Some of these people are psychology researchers, and I think one thing that annoys them about Jessica and me and others like us is that we’re not ourselves psychologists. I’ve published psychology papers and collaborate with psychologists, Jessica works in the psychology-adjacent field of human-computer communication, Blake McShane works in the psychology-adjacent field of marketing, as does Berna Devezer, etc., but we’re not in the guild. So I think that’s part of it. Again, lots of psychology researchers are in agreement with us, and I get tons of supportive emails from psychology researchers—but from the standpoint of some insiders, we’re just ignorant interlopers. So I think that’s the background for some of this. But, maybe that’s just me psychologizing!
Also, psychology has a long tradition of doing statistics within the field: there’s psychometrics, psychophysics, etc. As I never tire of saying, many of the most important methods in statistics were first introduced in psychology. Some methods researchers within psychology end up with a go-it-alone, we-don’t-need-any-statisticians-telling-us-what-to-do sort of attitude, and they can get annoyed at people like us who are out there giving advice to psychologists and criticizing psychology papers.
This reminds me of the days when political scientist types used to criticize Noam Chomsky for having no formal credentials that could qualify him to comment on US and other countries’ politics.
I’m very psychology-adjacent but with no formal psych qualification (only have a linguistics, statistics, computer science, and languages–Japanese and French–background). I have a lot to do with psychology research as a reviewer and it is pretty rare for me to see a paper in which an egregious error, either technical or just p-hacking or a misinterpretation of the p-value-based result, does not occur. I think probably the type of stats-savvy psych people you are thinking of are maybe 5% of the population.
Shravan:
I agree that the vast majority of psychology researchers are not statistics experts and do not consider themselves to be statistics experts. Which is fine—that’s not their job!
What I was saying in my above comment is that I think that some of the hostility toward the writings of Jessica and me and others on science reform is coming from some subset of the people in psychology who are statistics experts within that field. Then here we are coming from outside the field of psychology and acting as if we’re the real experts. I can see how this could annoy some of the local experts within psychology.
The promise and peril of inter-disciplinary research.
Psycho-linguists know it well.
I prefer the term psycho linguist.
Thank you, I completely agree!
Also, as Steve Haroz pointed out, there’s something very ironic about defending preregistration through appeal to private documents.
That’s indeed one oddity about that comment: The expectation that this should be handled behind closed doors. For a paper that’s already published and is being cited extensively.
The other oddity, I find, is what boils down to “How dare you talk about this before hearing my thoughts on it?” As if any scientist owes anyone else their deference and attention. As if there is a final authority in science and we should all respect that. As if someone is entitled to have the last word.
Everything is out there for anyone to read and make up their mind. We only owe it to ourselves and to the profession to evaluate all the evidence properly before arriving at a judgment and/or commenting on it. But this ‘waiting for the final authority to have their say’ thing is complete BS.
I guess they are pro pre-registration but anti-post-publication-peer-review :) I would have bet that the intersection of those two groups would be the empty set.
I know most of this article about pre-registration. But one thing is often missed is that most of these studies are just simply showing “association” so the title and the conclusion should use “association” and less of causal language. The same was for the study that shows registered reports lowered positive findings. But this is again is association and could be just due to selection bias.
I don’t see many people bothered by it. but this is an important issue since we might be spending money and time on things which may not have a causal impact.
I’ve been thinking about how even when reviewers require that authors change how they talk about results to tone down causal statements, or add some acknowledgement near the end like `technically our study didn’t implement conditions necessary to establish causality…’, authors are often still very adept at slipping in language that implies causality. And of course these kinds of implications will shape how some readers recall the paper, or how they cite it.
In this paper, the statement in the abstract that “This high replication rate justifies confidence in rigour-enhancing methods to increase the replicability of new discoveries” tempts causal interpretation. So does setting up the idea that the rigor-enhancing practices “might promote higher replicability than previously reported in the literature” then putting a lot of effort into establishing that “oh, by the way, replicability is unexpectedly high no matter how we look it,” without appearing to put much effort into the reminding the reader that there are many alternative ways to define replicability that might change the story.
I could see someone arguing that once you remove the misleading statement about the preregistration in this paper, the evidence is all right there as to what the paper did and what was confirmatory, and so any good reader should be able to see that the study wasn’t designed to estimate the causal effect of these practices and that the replicability measure was chosen post-hoc. But subtext does matter, and that’s a place where authors need to take more responsibility.
What the title initially made me think of was: High replicability has always been achievable. Just set your alpha to 1.
This was the focus of Joe and my commentary Jessica linked to. Preregistration was only tangentially relevant to our core critique. The design did not support the causal conclusion; there was no treatment of preregistration or other open science practices. No control condition. The “rigor-enhancing methods” were the baseline for all the studies in this meta-study. Hence the conclusion should not even use “association” label. Without variance, there is no association either. Then there’s the selection bias in the effects chosen and arbitrariness of the replicability measure. I was bothered by all of this before even we knew about the missing preregistration.
Thanks Berna. I read your critique and now the article and you are spot on!
And adding to your points. I also think picking 4 labs who are some of the best psych researchers in the country is a problem too. These are folks who are doing everything methodologically right and asking the right questions. So is it the open practices or just the good researchers tend to replicate their work more often? Also, even if they pick random researchers, there is some very serious hawthrone bias at play when they know their work will be scrutinized like no other. And as you say the hypthesis to do a confirmation study has issues. The procedure to pick the study to run confirmatory test should have been pre-registered and studies selected by an external committee.
Cosnidering these are some of the major advocates of transparency and open science, they should have added a line to the conlusion saying saying something along the lines of Considering the lack of a control group and randomization, the results or causality should be interpreted with caution “
“Big collaborative papers where you can’t be sure what your co-authors are up to should make all of us nervous.”
If you can’t explain what each of your co-authors did, then you’re not a CO-author and you shouldn’t be getting credit for the published work if you’re not also willing to suffer the consequences if the paper is full of misconduct.
Maybe there should be separate sections for authors and contributors. I mean this already exists for anyone in the know. The first and last authors are responsible and everyone in between contributed something.
I don’t see a need to change this but it is somewhat opaque.
Anyway, “enumerating badness” is a strategy doomed to fail. Essentially you need a whitelist of good practices rather than trying to detect the infinite number of bad ones.
Independent replication, and derive predictions that can distinguish between your theory vs others, then check it on new data.
This was solved hundreds of years ago.
There’s a commonplace selection effect that large fluctuations are more likely to be published, hence the effect decline in replications. From the description (I haven’t read the paper), it sounds like at least some of the authors were looking to test whether there could be some other factor at work that biased against replications?
Dan:
There are many many reasons to expect decline. I think I’ve posted on the topic but I’m not sure where, so maybe a new post on the topic is in order. In any case, I think it’s pretty ridiculous for Schooler et al. to posit a
supernatural“quantum” explanation for it. But, then again, I’m a skeptical guy. I don’t believe in ghosts either.This appears straightforward. I can’t help but wonder whether the co-authors have read it and how they’re feeling about the paper as a result.
Nosek has defended the paper on social media, though without addressing the inconsistencies between what is claimed and what is in the repo brought up in this post. I guess those cannot be discussed due to the investigation NHB is doing.
There’s a long conversation between Nosek and Tal Yarkoni (one of the former reviewers of the paper) on bluesky where Yarkoni summarizes very bluntly why critics like me and Bak-Coleman and Devezer find the claims this paper is presenting so hard to take and Nosek responds to those critiques: https://bsky.app/profile/talyarkoni.com/post/3kpbu5povbw2u
First, Pre-registration is not the solution for “cheating”. The very term (PR) is not well defined, nor is there universal agreement on what a “good” study is, how power should be determined, and what statistical procedures are the “correct” ones given the data. Researcher often do not even know what is meant by “effect size”. I can’t help thinking about Skinner’s discovery of partial reinforcement: SERENDIPITY ( no research plan, no concern for sample size, no statistical procedure to test a null hypothesis) . I suppose that if Skinner had tried to get his work published in one of our current APA journals , it would be rejected because it wasn’t Pre-registered.
Second, there is a way to increase the probability that a would-be article is legitimate ( that is, makes are important contribution to the development of knowledge). Assuming that the action editor is known, all reviewers (referees) need to sign their reviews and need to sign a statement that they were the sole author of their reviews. All data need to be archived (available to anyone, as is the current practice) along with the signed reviews. The action editors response to the authors of an article also needs to be included in the data archive. These procedures would distribute the responsibility for the publication of a poor quality article across those who have contributed to the publication or to a decision not to publish.
As psychologist let us be a little more humble and a little more transparent. Humble, because the fact is, we don’t have the answers, either statistical, methodological or applied, to be certain about the value of our work. Transparent because transparency is the only way to avoid an authoritarian discipline.