The problem is not individual research papers, or even the field of psychology. It’s the way that academic culture filters papers, and the way that the larger society gets their results. . . .
Journalists . . . easily fall into the habit (and I’m sure an enterprising reader can come up with at least one example on my part), of treating studies not as a potentially interesting result from a single and usually small group of subjects, but as a True Fact About the World. Many bad articles get written using the words “studies show,” in which some speculative finding is blown up into an incontrovertible certainty.
I’d just replace “Journalists” by “Journalists and researchers” in the above paragraph. And then there are the P.R. excesses coming from scientific journals and universities. Researchers are, unfortunately, active participants in the exaggeration process.
Psychology studies also suffer from a certain limitation of the study population. Journalists who find themselves tempted to write “studies show that people …” should try replacing that phrase with “studies show that small groups of affluent psychology majors …” and see if they still want to write the article.
Indeed. Instead of saying “men’s upper-body strength,” try saying “college students with fat arms,” and see how that sounds!
More from McArdle:
We reward people not for digging into something interesting and emerging with great questions and fresh uncertainty, but for coming away from their investigation with an outlier — something really extraordinary and unusual. When we do that, we’re selecting for stories that are too frequently, well, incredible. This is true of academics, who get rewarded with plum jobs not for building well-designed studies that offer messy and hard-to-interpret results, but for generating interesting findings.
Likewise, journalists are not rewarded for writing stories that say “Gee, everything’s complicated, it’s hard to tell what’s true, and I don’t really have a clear narrative with heroes and villains.” Readers like a neat package with a clear villain and a hero, or at least clear science that can tell them what to do. How do you get that story? That’s right, by picking out the outliers. Effectively, academia selects for outliers, and then we select for the outliers among the outliers, and then everyone’s surprised that so many “facts” about diet and human psychology turn out to be overstated, or just plain wrong. . . .
Because a big part of learning is the null results, the “maybe but maybe not,” and the “Yeah, I’m not sure either, but this doesn’t look quite right.”
Yup. None of this will be new to regular readers of this blog, but it’s good to see it explained so clearly from a journalist’s perspective.