Kelsey Piper writes:
I’m writing about the replication crisis for Vox and I was wondering if you saw this blog post from one of the DARPA replication project participants, particularly the section that argues:
I frequently encounter the notion that after the replication crisis hit there was some sort of great improvement in the social sciences, that people wouldn’t even dream of publishing studies based on 23 undergraduates any more (I actually saw plenty of those), etc. Stuart Ritchie’s new book praises psychologists for developing “systematic ways to address” the flaws in their discipline. In reality there has been no discernible improvement.
Your blog post yesterday about scientists who don’t care about doing science struck a similar tone, and I was curious: do you think we’re in a better place w/r/t the replication crisis than we were ten years ago? Or has the replication crisis been a long series of conversations that haven’t influenced publishing and research practices much if at all?
My discussion of that above-quoted blog post appeared a couple years ago. I agreed with some of that post and disagreed with other parts.
Regarding Piper’s question, “has the replication crisis been a long series of conversations that haven’t influenced publishing and research practices much if at all?,” I don’t think the influence has been zero! For one thing, this crisis has influenced my own research practices, and I assume it’s influenced many others as well. And it’s my general impression that journals such as Psychological Science and PNAS don’t publish as much junk as they used to. I haven’t done any formal study of this, though.
P.S. For some other relevant recent discussions, see More on possibly rigor-enhancing practices in quantitative psychology research and (back to basics:) How is statistics relevant to scientific discovery?.
Quote from above: “Regarding Piper’s question, “has the replication crisis been a long series of conversations that haven’t influenced publishing and research practices much if at all?,” ”
Maybe the replication crisis has influenced publishing and research practices, but whether that’s a “good thing” concerning science and the scientific process is another question perhaps…
It reminds me of Edwards and Roy (2017) who have written a paper titled “Academic research in the 21st century: maintaining scientific integrity in a climate of perverse incentives and hypercompetition” which, if I remember correctly, has a Table 1 which notes several implemented proposed “improvements” of the past which may have (partly) caused some of the problematic issues that academia is dealing with currently. With this possible process in mind, and thinking about how things might develop over time, I am very skeptical concerning several proposed and/or implemented “improvements” of the last 10 years
Perhaps it all also depends on how you want to look at things, and which time frame you take doing that. Maybe certain changes are and/or seem positive for the first few years, but later on it becomes clear that they may have led to several problematic issues down the road. Take this “open practices badges”-stuff for example. It could easily lead to a whole commercialization of “verifying” the badges, which could in turn perhaps become just as filled with problems as the scientific publishing industry. Or, this whole “Registered Reports”-stuff could perhaps easily lead to a massive influence of “peer-reviewers” who now might even influence the actual design of studies.
Let’s just say I am not necessarily a big fan of many of the proposals for improvements of the past 10 years or so…
Have financial frauds (Ponzi, Enron, Madoff, Theranos, Bankman-Fried, etc.) influenced financial ethics at all?
Has political misbehavior (Nixon, Stevens, Trump, Santos, well too many to name) influenced political ethics at all?
Perhaps academia is a bit better – at least we have a name (replication crisis) whereas the two above are just normal accepted behavior.
Quote from above: “Perhaps academia is a bit better – at least we have a name (replication crisis) whereas the two above are just normal accepted behavior.”
I even have questions regarding the term, and declaration of, the “replication crisis”.
I was following things in parts of Psychological Science around the time a case of certain fraudulent social psychologist got attention and the “replication crisis” was declared around 2012. I always have had issues with that name, because it sort of implicitly implies that there was perhaps nothing wrong with the “first” study, but more so with the “replication” aspect of it.
Or put differently, why wasn’t there a “validity crisis” declared back in 2012 instead of a “replication crisis”? Especially perhaps given all the criticism of Psychological Science, and whether it is even a “science”, in the decades before that.
Bob: Harry, we’ve got a problem. Sh!t is about to hit the fan. We’ve got that Stapel guy who was a member of our society, and made up data and published his made up nonsense research in our journals for years and years. The public knows this now Harry. The public knows!
Harry: I know Bob, but what can we do? There will soon also be a major report made public by Levelt et al. about the investigation concerning the fraud. There are major detrimental statements and conclusions in that report Bob. Major!
Bob: I know Harry, I know. We’ve got to come up with something, or else the money will dry up.
Harry: Wait, we can perhaps try and blame it on some vague thing and make it seem like a certain other thing. Like, we could blame “the incentives” for just about everything, and not really talk about how it all might really be a bunch of BS but make it about “replication”. That’s where the problems lie: “the incentives” and “replication”.
Bob: Brilliant Harry! Just brilliant. I’ll spread the word, make some calls. This is good stuff Harry, really good stuff. I’m sure you’ll get that promotion soon.
Bob: Harry, Bob here. You know that whole “it’s all because of the incentives” stuff you thought of? That’s working like a charm. People are eating it up!
Harry: Yeah, I know. It’s been pretty interesting to see actually. We are trying to see what more we can do with it. People aren’t really asking questions about this “the incentives” stuff, like what that even implies exactly, or what it entails specifically. Or even whether it makes sense given things like “tenure” and the touted role of “independent peer-review” and all that stuff.
Bob: Do you think this might come in handy for some other stuff as well?
Harry: Well, it seems that people aren’t really going to really think about what is even good for science or why, or whether certain changes can possibly lead to “bad” things for science. And for the first few years they will likely also don’t even think about whether the changes we propose are even recognized and “rewarded” by universities and other institutions so they even can be considered to be useful new “incentives”. This leaves us with lots of options to propose things we want in the name of “aligning” or “changing” the incentives.
Bob: Wait, what? Do you think that could work?!
Harry: Have you looked at the Levelt et al. report!? Anyway, just to give an example. I think people in the coming years will be focused more on whether publishing and research practices will change in a certain way than they will think about whether that is even good for science, or what is even good for science.
Bob: Well, perhaps we can also try to “reward” the people who listen to us and support us. Maybe we could hand out “badges” to provide some “incentives”, like when you were 5 years old and got a sticker from the teacher when you did a good job. Like, we could use an image that depicts working together as a group. That’s super important in science, to work together as a one big group!
Harry: Yeah, good idea. Do you have any image in mind that represents this all which we could use on these badges that we could hand out to people who do what we say is “good” for science without asking questions or thinking about things for too much?
Bob: Sheep! A picture of a sheep!!
Harry: Brilliant Bob, just brilliant. I’ll call the badges-folks right away!
Dale:
I don’t think it’s quite right to identify the replication crisis with fraud. I say this for two reasons:
1. Lots of non-fraudulent studies are not replicating. If a study is super-noisy, there’s no reason to think it will replicate. No fraud is required for this to happen. For example, I don’t think there are any accusations of fraud regarding the beauty-and-sex-ratio paper, the ovulation-and-voting paper, the elderly-slow-walking paper, etc etc etc, but all of these studies are dead on arrival, and any replication that would occur would be entirely accidental.
2. When researchers are told that their published finding will not replicate, they will often indignantly reply that, no, they did not cheat, how could you accuse them of cheating, etc etc. In this twist, the association of replication failure with fraud is used as a sort of defense, which I think is inappropriate.
You are right that the replication crisis is different than fraud. I was commenting on a more general level – that once poor practices have been identified, does anything change? It is conceivable that things become worse – people may get the impression that everyone does it, so why not? The lack of ethics in politics, business, and academic research is appalling, although I still think the latter is not quite as bad as the others (though that belief is increasingly shaken).
Dale: fraud in financial markets is niether accepted nor legal and in fact society spends ample resources prosecuting people who commit fraud.
That’s more than I can say for academia, where plagiarism – a form of fraud – by university presidents is brushed off as though it were a mistaken comma. So spare us the lecture on fraud in finance – academia commits ZERO resources to fraud and usually simply ignores it.
I have also been thinking about this issue. My impression, but I have no data, is that the scientists who wanted to do good work, but perhaps misunderstood some things, have made some changes and are now doing better work; while the scientists who were “cutting corners” before continue unimpeded. Some corner cutters now do some things to make it appear that they are doing things properly (like preregister their studies) but they are actually just pretending to do that. I suspect that some of them are going to get in trouble because the extra openness they now practice makes it easier to discover that they are not actually doing good work.
Quote from above: “I suspect that some of them are going to get in trouble because the extra openness they now practice makes it easier to discover that they are not actually doing good work.”
That’s perhaps where this “it’s pre-registered but it also kind of really isn’t” stuff might come in handy!
Just ask people behind the “Registered Reports” stuff where Ioaniddis and Hardwicke (2018) found that certain pre-registratoin information was not available if I remember correclty. Or ask some other “open and transparent” people who apparently are not really that open and transparent themselves when it comes to certain things as we have recently learned on this blog…
It’s all super complicated apparently this pre-registration stuff!
Quote from above: “That’s perhaps where this “it’s pre-registered but it also kind of really isn’t” stuff might come in handy!”
Coming soon:
“Redefine pre-registration” from the authors that previously brought you “Redfine statistical significance”.
The solutions (pre-registration, etc) don’t solve the problem. Solving it is simple:
1) Describe what you did in enough detail so someone else can repeat it and get similar results. If there are all sorts of other things that matter missing from your description, that is a problem. That is solved by iteratively repeating this process.
2) Derive actual predictions from your theory, and compare those to the data. Don’t use some default null hypothesis of no correlation/difference.
It is very simple, *use science*. If you don’t like the above practices, then you do not actually like science. So go do something else with your life.