Olaf Zimmermann points to this article by Gerald Haeffel, which begins:
Nearly 100% of the published studies in psychology confirm the initial hypothesis. This is an amazing accomplishment given the complexity of the human mind and human behaviour. Somehow, as psychological scientists, we always find the expected result; we always win! Recently, however, the legitimacy of psychology’s winning streak has been called into question. Major replication projects show that only about half of psychological findings replicate. Further, there is evidence that psychology’s winning streak may be due to cheating. Similarly to how baseball’s homerun chase in the United States was fuelled by steroids (e.g. Bonds, McGuire, Sosa) and Lance Armstrong’s Tour de France streak was aided by doping, psychology’s winning streak may be the result of questionable research practices . . .
Haeffel summarizes:
This is a problem because science progresses from being wrong. For decades, there have been calls for better theories and the adoption of a strong inference approach to science. However, there is little reason to believe that psychological science is ready to change. Although recent developments like the open science movement have improved transparency and replicability, they have not addressed psychological science’s method-oriented (rather than problem-oriented) mindset. Psychological science still does not embrace the scientific method of developing theories, conducting critical tests of those theories, detecting contradictory results, and revising (or disposing of) the theories accordingly.
Well put. And it’s not just psychology. I’ve seen a lot of this in political science and economics as well. In medical research the problems are more complicated, as there’s a mix of declaring victory from noise and declaring null effects just cos a comparison is not statistically significant.
Regarding psychology in particular, I’ll point to my article with Simine Vazire, Why Did It Take So Many Decades for the Behavioral Sciences to Develop a Sense of Crisis Around Methodology and Replication?, which considers some possible reasons why, on or about December 2010, the behavioral sciences changed.
The link to Gerald’s post is wrong.
Link fixed; thanks.
I’m old enough to remember an earlier change. When I was young, in the 1950s, Freudian ideas were generally accepted in the liberal circles. My physician father was persuaded, and had me do a couple of years of psychoanalysis when I got depressed in the early 60s. I wasn’t paying attention, but over the next couple of decades Freudian ideas seemed to go out of fashion, and I’m wondering whether there was some kind of over-reaction to Freudian methods in psychology. (I’m not defending psychoanalysis — sometime in the 70s, when I was well along to becoming a scientist, I came across a psychoanalytical journal in the library, and boy, was it pathetic.)
There is a new magazine called Parapraxis that aims to revive Freudianism for a general, educated audience.
I have started to wonder more and more whether lots of things are cyclical, including processes and features of (social) science.
If you only look behind you and in front of you for a short while and looking at a short distance you might think that your steps lead you to some place different and even lead to making progress, but you may just be walking in a circle.
John:
See our discussion, Evaluating Sigmund Freud: Should we compare him to biologists or economists?
Thanks. From your earlier post: “The problem is, one could say, that Freud is presenting social-science-style theorizing as if it’s biological science.” But, if you accept that Freud was trying to base his ideas in biology, then you need to think about the state of the biology science that was available to him, which was not great. For example, testosterone wasn’t discovered until the 20s, so theorizing about libido was about all you could do.
I’ve been thinking about and collecting insights for just this question.
There are relatively few opportunities for junior researchers to learn about critical skills like forming ontologies, theorizing, considering practical implementation of interventions, and conducting a deep synthesis of many diverse and often contradictory results. In my grad program, all of this is under the table and at the discretion of one’s lab affiliation. We need some resources here (especially well-considered interdisciplinary ones).
Given that much of the metascientific literature is beginning to repeat itself and often rehashes things found in introductory textbooks, it’s time to shift towards developing curricula and technologies/systems for scaffolding credible research practices (as an alternative to QRPs).
The lowest-hanging fruit here would be to run workshops for students in a few different disciplines and catch them early. It’s interesting to note the opportunities for interdisciplinary theorizing in my department (Information Studies) that are neglected. By the time conference season arrives, we’re dealing with the issues of poor problem-framing and lack of initial conceptual work. Peer review needs two stages (prior to IRB submission and after write-up).
Hi. This sort of issue was once (for a while famously) raised by Allen Newell, who lamented the loss of theory in psychology. There was a Golden Age of theory once when competing hypotheses derived from theories were compared experimentally, but it died out in the 1950s, replaced by the AI-fueled eclecticism that Newell lamented in Newell, Allen (2018). You can’t play 20 questions with nature and win : projective comments on the papers of this symposium. Carnegie Mellon University. Journal contribution. https://doi.org/10.1184/R1/6612977.v1
“Thus,far from providing the rungs of a ladder by which
psychology gradually climbs to clarity, this form of
conceptual structure leads rather to an ever increasing
pile of issues, which we weary of or become diverted
from, but never really settle.”
That’s nicely put by Newell and calls to mind Meehl’s line about academics making careers of exploring the ever-growing branches of a research program without ever touching the trunk.
Also good! (last one)
“So loose jointed are our edifices that a divide and conquer strategy can be used. A part of the totality can be pulled out and attacked in isolation with seeming impunity.
What should be the case? A challenge to one part of a pattern of experimental results should not be permitted unless it can successfully challenge (or be shown to be consistent with) a substantial part of the total existing pattern.”
“Psychological science still does not embrace the scientific method of developing theories, conducting critical tests of those theories, detecting contradictory results, and revising (or disposing of) the theories accordingly.”
This sounds like an overly naïeve description of science (too strictly Popperian?) / at odds with what history and philosophy of science have learned us about how science works in the last half century or so.
Kris:
It’s interesting to know how science works, but I’d also like to know how science should work. The history and philosophy of science will tell us that a dominant mode of many subfields of science, for example, social psychology, is a fallacious but popular reject-the-null-hypothesis-and-consider-that-as-a-confirmation-of-one’s-preferred-story form of reasoning. It’s what people do, and it’s led to debacles such as the papers on himmicanes etc.—where the debacle is, in a Kinsley-gaffe sort of way, not so much that such ridiculously bad papers were published in prestigious journals but that this was standard practice. So I think we have to be careful about taking too seriously “how science works,” given that much of how science works is to misapply statistical methods in order to make strong claims that are not supported by data or theory.
I should probably have said “how good science works” (physics, astronomy, genetics, …). Shoving aside things that just superficially resemble good science (the things you describe) makes sense, but seems no less justifiable on grounds of “how (good) science works” as on naïeve Popperianism. That being so, much of the reform movement in psychology seems to get inspiration from Popper, so perhaps the next centrury of (social) science may later be described as more Popperian than what was before – interesting related paper: https://scholar.google.be/scholar?hl=nl&as_sdt=0%2C5&as_vis=1&q=putting+popper+to+work&btnG=#d=gs_qabs&t=1695852134142&u=%23p%3DJ0HNkUPe2G4J