Luke Ford writes:
[Regarding] the replication crisis, researcher degrees of freedom, and the gap between what statistical methods claim to establish and what they can actually support . . . Looking at the current state of the social sciences, do you think the culture of research has genuinely changed since the replication crisis became widely discussed, or has it mostly generated new compliance rituals around pre-registration and open data while leaving the underlying incentive structure intact?
And a question about Columbia specifically since you are there: the university has had a difficult two years in ways that have played out publicly. From your position as someone who thinks carefully about institutional incentives and measurement, what do you think the administration consistently misread, and is there a statistical or social scientific way of understanding how institutions lose the ability to accurately perceive their own situation?
My reply:
1. I’m loath to give an answer about the changes in the culture of research because I have not studied this systematically. My impression is that, yes, there’s more skepticism and less acceptance of noisy N=38 papers in psychology, etc., and less toleration for unfalsifiable evolutionary psychology and that sort of thing. On the other hand, perhaps this has just shifted from the science establishment to social media. Ten or fifteen years ago, there was a pipeline (partly abetted by Jeffrey Epstein) from researchers at top universities to publication in top journals to books, NPR, Ted, Gladwell, Freakonomics, etc., and lucrative speaking and consulting gigs. So you get people like Marc Hauser or Albert-Laszlo Barabasi or Brian Wansink or Dan Ariely doing the basic research (such as it is), academic middlemen such as Steven Levitt and Cass Sunstein as promulgators, and the universities, journals, and prestige news media as part of this system (as for example here: https://statmodeling.stat.columbia.edu/2023/08/31/the-variation-ignoring-junk-science-thats-promoted-by-association-for-psychological-science-and-related-academic-celebrities-its-like-a-poker-player-thinking-okay-if-push-all/).
Nowadays, though, social media runs on its own steam, and the models for academic junk science are researchers such as Andrew Huberman and Dr. Oz, who cut out the middleman and promote junk science directly, sell supplements, etc. And social media is full of fake news and AI slop. They don’t really need NPR, Ted, Gladwell, Freakonomics, etc., anymore; they can do it on their own. So, in short, yes, I do have the impression that science has reformed from the bad old days of 2010-2015 (about which, see this article with Simine Vazire: https://sites.stat.columbia.edu/gelman/research/published/jmmss-3062-gelman.pdf), but maybe the public intellectuals don’t need academic science anymore; they can just make up whatever they want on their own.
2. My take on Columbia is similar to my take on many institutions, which is that they have an executive function but minimal legislative or judicial functions; I discussed this here: https://statmodeling.stat.columbia.edu/2018/01/19/lesson-charles-armstrong-plagiarism-scandal-separation-judicial-executive-functions/ and here: https://statmodeling.stat.columbia.edu/2025/11/11/from-the-three-branches-of-government-to-the-bidirectional-nature-of-legal-reasoning-in-a-way-that-is-similar-to-how-statistics-works-and-should-work-in-the-real-world/. As a result, their decisions are made on consequentialist rather than proceduralist gounds, and over and over again the administration takes the seemingly reasonable decision to cover up misdeeds.
Ford posted this discussion on his blog. It was kinda weird seeing myself discussed as a sociological object, but, fair enough, I’m a public figure, and people can say what they want as long as they don’t misrepresent my writings or claim that I said something I never said.
Ford’s assessment is accurate that I’m not very good at strategic behavior so often I don’t even try. It’s similar to how I’m a bad negotiator so usually I’ll just try to make my goals clear and not try to optimize, following the “Getting to Yes” principle that the main thing getting in the way of smooth negotiation is ignorance of other people’s goals. I think back to various successful and botched negotiations I’ve been involved with in the past, and almost always the problems come with struggles over details without there being clarity on the goals of the different parties.
So much to say about this all, as it just fits very well with my attempts and experiences of the last 14 years or so. Let me start with something very interesting to me I noted in the question sent: “(…) or has it mostly generated new compliance rituals around pre-registration and open data while leaving the underlying incentive structure intact?”.
One thing about “the incentives” that has annoyed me from the start back around 2012, is a seemingly flawed or simplified explanation of the processes supposedly involved here: a scientist “has to” publish as many papers as possible, preferably in top journals, to get a job or position and that’s why they might use questionable research practices because that might give them an advantage. If that’s a fair description of the proposed general “incentives” and “publish of perish” explanation, I keep wondering how this then applies to tenured professors who can’t be fired (easily). Why would these tenured professors use questionable practices to try and publish as many papers as possible (anymore)?
Now, perhaps other factors might play a role with the tenured scientists for why they want to try and publish many papers in top journals: Raking in lots of grant money of which a substantial part goes to the university? Attempting to provide sponsored evidence for X, Y, or Z?). Whatever it is, I reason “publish or perish” seems not applicable anymore. Or, at least there might be different versions of “perishing” (not being hired or promoted VS not receiving lots of grant money to you can hire a new batch of PhD students and help pay the electricity bill of the university), which might be useful to make more explicit.
What’s also extremely strange to me, is that all this “incentive” talk seems to mostly involve just a very particular level, one that might not even be the crucial one. Researchers are now supposed to provide open data, or pre-regsiter their studies, etc. But, I wonder, where are all the data and papers and proposals that mention gathered information concerning these issues at the level of actually hiring or promoting scientists? That’s, presumably, a very crucial level, is it not? You can have people writing lots of paper about “the incentives” or how to be a better scientists by providing open data and those kinds of things, but if scientists who follow such recommendations and practices are not hired, what’s the point? You can have a journal adhering to certain possibly better standards for publishing, but when publishing in this journal might be harder, one might be at a disadvantage compared to another scientist who had an easier time publishing more papers in journals without these possibly better standards when it comes to unversities hiring “productive” candidates. Again, what’s the crucial level and point in this all?
“Ford posted this discussion on his blog.”
This link goes to your blog. The links in the linked post are all either to your blog again, or don’t work.
Link fixed; thanks.
Quote from above: “(1) “Do you think the culture of research has genuinely changed since the replication crisis became widely discussed (…)”
The following two papers might be interesting for the person asking the questions, and possibly others, should they not be familiar yet:
Hardwicke et al. (2024) “Prevalence of Transparent Research Practices in Psychology: A Cross-Sectional Study of Empirical Articles Published in 2022”:
“In conjunction with prior research, our results suggest transparency increased moderately from 2017 to 2022. Overall, despite considerable infrastructure improvements, bottom-up advocacy, and top-down policy initiatives, research transparency continues to be widely neglected in psychology.”
Klonsky (2024) “Campbell’s Law Explains the Replication Crisis: Pre-Registration Badges Are History Repeating”:
“Pre-registration mandates are positioned as an antidote. However, I argue that such efforts, perhaps best exemplified by pre-registration badges (PRBs), are history repeating: Another useful tool has been converted into an indicator of strong science and a goal in and of itself. This, too, will distort its use and harm psychological science in unanticipated ways.”
And, here are two of my own manuscripts that might make some things clear(er) concerning the question quoted above: “Psychological Science Replicates Just Fine, Thanks”, and “Pre-registration, Grocercy Lists, and Particular Pre-registration Issues” which can both be found on SSRN.
I started reading the discussions about you posted by Luke Ford. First, I was really confused by the chronology of what was being responded to or discussed. In a post ostensibly made in late March, he’s discussing things you (Andrew) did today (late August)! Here today, you wrote that Ford’s assessment that you are not good at “strategic behavior”. I kept reading Ford to find an assessment of you mentioning that, and could not find. His analysis of you (Andrew) seemed really oddScrolling to the bottom to see how much longer the post was, I came across a disclaimer that “Most of my posts since January 2025 are written with AI.” That’s ten minutes of blgo-reading I won’t get back.
Needless to say, I’ve lost interest in what appears on Luke Ford’s blog.
+1. It’s written in a very strange and choppy style too.
This choppy style is apparently Claude AI’s (default?) style, as seen here:
https://www.theatlantic.com/technology/2026/08/paul-claude-gigot-why-i-am-right-about-ai/688434/
(archived version here: https://archive.is/tjrTJ)
Gregory:
I know nothing about Luke Ford—he’s just someone who emailed me with interesting questions. He posted the responses on his blog after I sent them to him, and then I wrote this post. We have a queue: I wrote the post several months ago.
Andrew:
I figured your post had been in the queue, It was Ford’s anachronistic inclusion of things from (approximately) now that made the timeline confusing; after seeing Ford’s admission that he doesn’t write his blog, I gave up trying to figure out the chronology. (Also, apologies for the several typos in my original comment!)
This was the first time I learned of your article with Simine Vazire. I have a small nitpick: In that article you claim that Simmons et al. (2011) introduced the terms “researcher degrees of freedom” and “p-hacking”. The first is true but the second is not; the 2011 article does not contain that word, and to the best of my knowledge, Simmons and colleagues first started using it in presentations of the ideas behind “False-Positive Psychology” in the year or two after publication of that paper. The earliest written record I can find is January 2013 (papers dot ssrn dot com slash sol3 slash papers.cfm question_mark abstract_id equals 2205186). An interesting sighting of the Mandela Effect, perhaps.
I did some searching online and found a 2019 WIRED article titled “We’re All ‘P-Hacking’ Now” in which the following can be read:
“A year later, the team went public with its new and better name for this phenomenon. At a psychology conference in 2012, Simonsohn gave a talk in which he used the term p-hacking for the first time.
“We needed a shorter word to describe [this set of behaviors], and p-dash-something seemed to make sense,” Simmons says. “P-hacking was definitely a better term than ‘researcher degrees of freedom’ because you could use it as a noun or an adjective.”
The phrase made its formal debut in a paper the team published in 2014, where they wrote “p-hacking can allow researchers to get most studies to reveal significant relationships between truly unrelated variables.”
They weren’t the first to identify what can go wrong when scientists exploit researcher degrees of freedom, but by coining the term p-hacking, Simonsohn, Simmons, and Nelson had given researchers a language to talk about it. “Our primary goal was to make it easier for us to present our work. The ambitious goal was that it would make it easier for other people to talk to each other about the topic,” Nelson says. “The popular acceptance of the term has outstripped our original ambitions.”
Nick:
In their 2014 paper, “P-Curve: A Key to the File-Drawer,” Simonsohn, Nelson, and Simmons us the term “p-hacking” and write:
Maybe that’s the same paper that was a preprint in 2013.
I prefer the term “garden of forking paths” because it does not imply the intentionality that I see in the term “hack” in “p-hacking.”
Also, I like that paper with Vazire, but I’m frustrated as it seems that the overlap between the set of people who care about meta-science and the set of people who’ve heard of that Virginia Woolf quote is approximately zero.
Replying for the sole purpose of letting you know that maybe it’s close to it but it’s definitelty not an empty set!
OK, we’re up to N=2 now.
Whoa his blog is a bit creepy. Did you know he was psting so much about you?