John “not Indiana Jones” Williams writes, “I thought you would be interested in and a bit annoyed by this review from Science of a book on one aspect of the replication crisis. There is no mention of significance tests.”
Williams is pointing to a review by historian of science Elizabeth Lunbeck of a book by Ruth Leys about the discredited social-priming work (see here for some background) of John Bargh and others.
We’re already several levels deep in abstraction here:
1. The actual psychological phenomenon of social priming, such as it is.
2. The experimental techniques used by Bargh et al. to study that phenomenon.
3. The journal articles used to promote the idea. These articles cannot simply be considered as instantiations of the experiments: it takes a lot of effort and intellectual “technology,” as it were, to transform ambiguous or equivocal data into convincing and confident claims of empirical evidence and theoretical coherence.
4. The structures of the science and news media establishments that led to the claims in the journal articles being widely disseminated and believed.
5. The backlash within psychology and elsewhere against social priming, including failed replication experiments, statistical theory exploring how it all could’ve gone so wrong (that’s the “significance tests” thing to which Williams was referring), and news media reports of the debunking. I was involved in some of this myself.
6. Leys’s book, which tells some aspects of the above stories.
7. Lunbeck’s review of Leys’s book, which is what I’ll be writing about.
8. The reaction by Williams and me to Lunbeck’s review.
9. This post, which injects this all into social media discussion, so that you and others can react to my reaction to Lunbeck’s review of Leys’s book on the backlash to the promotion within science and the news media to journal articles that provided (unintentionally) misleading summaries of experiments designed to investigate a dubious psychological theory.
It’s an inverted pyramid balanced on a pinpoint.
I haven’t read Leys’s book, so my comments here are on Lunbeck’s review. She tells the story well, focusing on the psychology theories, not on the statistics or the academic politics.
I appreciate that. There’s lots to say about the statistical problems of unreplicated or unreplicable research–indeed, I’ve had a lot to say about the topic myself!–but that’s all just a means to an end. And, as I and others have argued, the fundamental problem with a lot of this bad research is not the bad statistics but rather the bad substantive theory, along with bad connections between theory and data. The bad statistics enables the bad science to appear successful; it does not in itself make the science bad.
To put it another way: bad science analyzed using good statistics does not produce good science; at best, it just makes it harder to be fooled by noise. Conversely, good science can proceed just fine using bad statistics. Good statistics should make good science more efficient and effective, but it’s usually not necessary.
So I think it’s appropriate that Lunbeck (and, I assume, Leys in the book being reviewed) does not dwell on the statistical errors that led to decades of overconfidence; rather, she focuses on the problems with the social priming theory and experiments:
Bargh marshaled this striking finding to support his claim that “automaticity,” not free will or intentionality, powerfully governs behavior. . . . Automatic responses–quick, efficient, intuitive–were just as powerful in shaping behavior as were more cognitively complex and considered ones, the theory went. . . . Critics charged that the original experimenters had ignored a long tradition of research and theorizing focused on so-called demand characteristics, the motivations at work in both subjects and researchers in the setting of the psychological experiment. . . . [Leys] is particularly focused on psychology’s long history of downplaying intentionality in human behavior. . . . priming researchers were repeatedly snared by conceptual and theoretical traps of their own devising. For instance, they eventually posited that “moderators,” such as desires to affiliate or gender, influence individuals’ responses to primes, but this undermined the generalizability of their experimental results.
Well put.
Just one more thing regarding “the generalizability of their experimental results.” Those published results are consistent with null effects. This was the point of the classic Simmons, Nelson, and Simonsohn (2011) methods paper. I think Lunbeck is aware of this–she refers to “confirmation bias and a ubiquity of tautological statements dressed up as theory.” I just wouldn’t want readers of the review to be left with the impression that those experimental results could be interpreted as claimed, even locally.
To put it another way: Mind the gap between the direct experimental results (the entire raw data along with precise descriptions of the conditions of the experiment) and how these results are presented in the published journal articles.
And, as we discussed yesterday, mind the gap between the extravagant claims in the title and abstract of a published paper, and the actual measurements that were conducted.
Generalizability is an important but separate concept.
Replication answers the question: Do we understand the phenomenon well enough to control all the important factors and communicate how so to others?
The theory could be completely wrong, so there is no generalization. But you can still say if someone does A, B, C then x, y, z will happen.
The early days of electricity (eg, the Leyden jar) provide a great example: https://books.google.nl/books?id=UlTLRUn1sy8C&pg=PA309&redir_esc=y#v=onepage&q&f=true
Then, once there is a theory (eg about why the jar needs to be held in the hand and events must occur in a seemingly arbitrary order), you check if the theory can generalize to new circumstances.
I’m amused that just before reading this post, I read today’s XKCD and ruefully noted the alt text: “If you think curiosity without rigor is bad, you should see rigor without curiosity.”
https://xkcd.com/3101/
“the fundamental problem with a lot of this bad research is not the bad statistics but rather the bad substantive theory… The bad statistics enables the bad science to appear successful; it does not in itself make the science bad.
To put it another way: bad science analyzed using good statistics does not produce good science; at best, it just makes it harder to be fooled by noise.”
Science isn’t just a body of theory; it’s a process for reality-testing theories and killing off those that don’t match reality. So in that sense, “bad science [bad theories] analyzed using good statistics” performs the function of helping to kill off erroneous theories… and that IS good science [as a process].
But the problem is that the bad statistics DON’T kill off the bad theories. The folks with the bad theories are so convinced of their bad theories that they do bad statistics to support said theories, and then when folks point out that they did the statistics wrong, they claim that the critics are quibbling. (Although, come to think of it, it’s not so much that the statistics is wrong, but that the experiements are (often subtly) bad and don’t provide data for which doing the statistics will give the right result.)
That is, the bad* theories in cog. sci. are really really bad. Trivial, stupid, inane. And every other pejorative term you can think of. Power posing. Signing at the top. This stuff really is stupid and inane. Meanwhile, we humans figured out the standard model and the Langlands program.
The problem with the book is that the author isn’t quite ready to say “the nudge idea is stupid”, so she ends up trying to hard to argue on philosophical grounds. Going through John Bargh’s various arguments across multiple decades of his carreer in excrutiating detail.
*: Note the heavy lifting “bad” is doing here. It excuses me from claiming my actual belief, which is that the whole field is hopeless.
I said that bad science analyzed using GOOD statistics helps kill off erroneous theories… so how does the failure of BAD statistics to do so a refutation?
Aren’t you just arguing that good statistics is important? The same as I was saying?
Sure. If the people doing bad science would do us the favor of doing good statistics, we’d be home free.
(Also, Andrew’s point/claim was that noisy data can mean that even statistics done right might not get the right answer. I’m adding in fraud. But, whatever, I’m really really happy that Prof. G. is on the same page as I am about the underlying science being bad. My ranting on the reasons for the badness often gets more than a bit afield.)
But it looks to me that (much of the time) you don’t get good statistics with bad science. So the good statistics doesn’t get to fix the problem until some third party does the work. But in the meantime, the bad papers are out there. And it’s real hard to get papers withdrawn, even really bad ones. (We’ve been fortunate in that a couple of the bad papers had fudged data that was so amateurishly fudged that it was obvious. But how many papers are there out there in which the fudged data was fudged more carefully???)
(Long rant deleted. But I included the following reference, the reason therefor being left as an excercise for the reader. Folks familiar with my schtick will have no trouble guessing. (Hint: I’m betting that this is a good paper with the statistics done right.))
**: Proc. Natl. Acad. Sci. U.S.A. (2025) 10.1073/2406684122 (mentioned on page 373 of the 25 April 2025 edition of Science: “Feedback from higher-level visual processing centers in the brain influences the early stages of object recognition”)
“Conversely, good science can proceed just fine using bad statistics.”
How can you possibly know that it’s good science if the data analysis is flawed?
Kevin:
First, some experiments give results that are so clear that no statistical analysis is necessary, and a bad statistical analysis can still point to what’s going on. For example, if I compare two treatments in a clean experiment and the estimated effect is 0.5 +/- 0.1, for some problem where 0.5 is a large effect, then if someone does something silly like compute the p-value is 0.000…–or even if they compute the p-value wrongly–it won’t really matter. Along the same lines, sometimes bad statistics is bad because it is inefficient, not making full use of available information. But the analysis that’s actually done can yield a strong enough result that it’s fine.
Second, statistics is often used to suggest directions of future work. Good science can proceed just fine using bad statistics, if these statistics are used in a sensible way by scientists, as a motivation for where to look next.
Third, “bad statistics” is a continuum. No statistical model is perfect. For example, a few years ago we wrote about problematic tail behavior in the Economist and Fivethirtyeight election forecasts. I think these forecasting models were good science–not ideal science, but good science–despite the flaws in their statistical models. At some point a model can be so bad that it turns the forecast into bad science–here, I’m thinking about the 2016 Princeton election consortium model which assumed statistical independence of state-level uncertainties, leading to an extremely overconfident national-level forecast.
Andrew,
I agree there are clear cut experiments where no statistics is required. But I don’t think there can be a good social science without good statistics. At some point you want to generalize your experimental findings to more realistic, observational, settings. And then you have to adjust for various things that you could easily control for in your experiment via randomization. Also in the real world effects will vary depending on period, culture, location and so on and you want to model that.
So I think there can be good individual scientific studies without good statistics, but there can be no social science (as the whole endeavour) without good statistics, because: How do you generalize without good statistics?
In a Danish study involving nudging with well over 20,000 (!!) patients:
https://www.sensible-med.com/p/good-ideas-require-testing-the-lessons?utm_campaign=email-post&r=396huk&utm_source=substack&utm_medium=email
“In the group of patients who received a letter, 65.1% filled prescriptions vs 65.9% in the control arm. The less than 0.8% difference did not reach statistical significance.”
“For patients with providers who received letters, 63.9% vs 64.4% in the control arm received prescriptions. This tiny difference did not reach statistical significance.”
Was it necessary to throw in “did not reach statistical significance”? Wouldn’t the famous “inter-ocular test” [it hits you between the eyes] suffice? Or, relegate the buzz words to a footnote.
On the other hand, perhaps not because “statistical significance” is now part of the dance.
I looked at the paper. They basically sent a marketing spam email to people who had entrusted their doctors with their personal information, which was then added to some database. The email then implied the senders of the email (not anonymous, but may as well be since they are unknown to the patients) knew better than their current doctor about the best treatment.
They would have better luck with a personable tiktok influencer doctor, imo.