E. J. Wagenmakers pointed me to this recent article by Roy Baumeister, who writes:
Patience and diligence may be rewarded, but competence may matter less than in the past. Getting a significant result with n = 10 often required having an intuitive flair for how to set up the most conducive situation and produce a highly impactful procedure. Flair, intuition, and related skills matter much less with n = 50.
In fact, one effect of the replication crisis can even be seen as rewarding incompetence. These days, many journals make a point of publishing replication studies, especially failures to replicate. The intent is no doubt a valuable corrective, so as to expose conclusions that were published but have not held up.
But in that process, we have created a career niche for bad experimenters. This is an underappreciated fact about the current push for publishing failed replications. I submit that some experimenters are incompetent. In the past their careers would have stalled and failed. But today, a broadly incompetent experimenter can amass a series of impressive publications simply by failing to replicate other work and thereby publishing a series of papers that will achieve little beyond undermining our field’s ability to claim that it has accomplished anything.
I [Baumeister] mentioned the rise in rigor corresponding to the decline in interest value and influence of personality psychology. Crudely put, shifting the dominant conceptual paradigm from Freudian psychoanalytic theory to Big Five research has reduced the chances of being wrong but palpably increased the fact of being boring. In making that transition, personality psychology became more accurate but less broadly interesting.
Poe’s Law, as I’m sure you’re aware, “is an Internet adage which states that, without a clear indicator of the author’s intent, parodies of extreme views will be mistaken by some readers or viewers for sincere expressions of the parodied views.”
Baumeister’s article is what might be called a reverse-Poe, in that it’s evidently sincere, yet its contents are parodic.
Just to explain briefly:
1. The goal of science is not to reward “flair, intuition, and related skills”; it is to learn about reality.
2. I’m skeptical of the claim that “today, a broadly incompetent experimenter can amass a series of impressive publications simply by failing to replicate other work.” I’d be interested in who this experimenter is who had this impressive career.
In fact, the incentives go in the other direction. Let’s take an example. Carney et al. do a little experiment on power pose and, with the help of some sloppy data analysis, get “p less than .05,” statistical significance, publication, NPR, and a Ted Talk. Ranehill et al. do a larger, careful replication study and find the claims of Carney et al. to be unsupported by the data. A look back at the original paper of Carney et al. reveals serious problems with the original study, to the extent that, as Uri Simonsohn put it, that study never had a chance.
So, who’s the “broadly incompetent experimenter”? The people who did things wrong and claimed success by finding patterns in noise? Or the people who did things carefully and found nothing? I say the former. And they’re the ones who were “amassing a series of impressive publications.”
Baumeister’s problem, I think, is the same one as the problem with the “statistical power” literature, which is that he sees “p less than .05,” statistical significance, publication, NPR, Ted Talk, Gladwell, Freakonomics, etc., as a win. Whereas, to me, all of that is a win if there’s really a discovery there, but it’s a loss if it’s just a tale spun out of noise.
Here’s another example: When Weakliem and I showed why Kanazawa’s conclusions regarding beauty and sex ratio were essentially unsupported by data, this indeed “undermined psychology’s ability to claim that it has accomplished anything” in that particular area—but it was a scientific plus to undermine this, just as it was a scientific plus when chemists abandoned alchemy, when geographers abandoned the search for Atlantis, when biologists abandoned the search for the Loch Ness monster, and when mathematicians abandoned the search for solutions to the equation x^n + y^n = z^n for positive integers x, y, z and integers n greater than 2.
3. And then there’s that delicious phrase, “more accurate but less broadly interesting.”
I guess the question is, interesting to whom? Daryl Bem claimed that Cornell students had ESP abilities. If true, this would indeed be interesting, given that it would cause us to overturn so much of what we thought we understood about the world. On the other hand, if false, it’s pretty damn boring, just one more case of a foolish person believing something he wants to believe.
Same with himmicanes, power pose, ovulation and voting, alchemy, Atlantis, and all the rest.
The unimaginative hack might find it “less broadly interesting” to have to abandon beliefs in ghosts, unicorns, ESP, and the correlation between beauty and sex ratio. For the scientists among us, on the other hand, reality is what’s interesting and the bullshit breakthroughs-of-the-week are what’s boring.
Anyway, I read through that article when E. J. sent it to me, and I started to blog it, but then I thought, why give any attention to the ignorant ramblings of some obscure professor in some journal I’d never heard of.
But then someone else pointed me to this post by John Sakaluk who described Baumeister as “a HUGE Somebody.” It’s funny how someone can be HUGE in one field and unheard-of outside of it. Anyway, now I’ve heard of him!
P.S. In comments, Ulrich Schimmack points to this discussion. One thing I find particularly irritating about Baumeister, as well as with some other people in the anti-replication camp, is their superficially humanistic stance, the idea that they care about creativity! and discovery!, not like those heartless bean-counting statisticians.
As I wrote above, to the extent these phenomena such as power pose, embodied cognition, ego depletion, ESP, ovulation and clothing, beauty and sex ratio, Bigfoot, Atlantis, unicorns, etc., are real, then sure, they’re exciting discoveries! A horse-like creature with a big horn coming out of its head—cool, right? But, to the extent that these are errors, nothing more than the spurious discovery of patterns from random noise . . . then they’re just stories that are really “boring” (in the words of Baumeister) stories, low-grade fiction. The true humanist, I think, would want to learn truths about humanity. That’s a lot more interesting than playing games with random numbers.