When does a misunderstanding reach the point where it is recognized to be flat-out ridiculous?

James Lasdun reviews a book by Ariel Sabar telling the story of a conman who sold a fake Bible-related document to a Harvard professor, leading to academic publications and media publicity before the whole thing fell apart.

The most amusing of many amusing bits:

An Egyptologist at Brown University, Leo Depuydt, found a ‘colossal double blunder’ in the Coptic grammar. As a younger man, Depuydt had once warned the Oxford journal Discussions in Egyptology not to publish a grammatically flawed version of a Gnostic text. They went ahead anyway, only to learn that they had been pranked when a reviewer’s daughter pointed out that the name of the man who had ‘discovered’ the text, Batson D. Sealing, sounded awfully like ‘bats on the ceiling’. The entire issue had to be recalled (it’s now a collector’s item).

This is a story we’ve heard many times before. Some low-grade scholarly work appears (perhaps an actual faked document such as this Jesus text or the Hitler diaries from the 1980s, or garbled or misrepresented data such in the climate economics papers of the gremlins guy or the monkey experiments of the Evilicious guy, or pizzagate-style data that might be fake or might be just garbage, or sleepy misrepresentation of others’ data, politically or religiously-tinged noise mining as in that study of beauty and sex ratio or that Bible code paper, or just run-of-the-mill cargo-cult science such as we’ve seen for ESP, himmicanes, air rage, ages ending in 9, ovulation and voting, and lots of other things you can google), it hits the media heights of NPR or Ted or Freakonomics or Gladwell or Time Magazine or whatever, and then, eventually, the truth comes out.

Another set of examples are speculative bubbles such as Theranos or MIT’s “personal food computer” which never had anything to offer to their wealthy backers except for the presence of other wealthy backers on the inside. Given that the insiders get rich off this sort of thing and don’t seem to suffer any professional or financial consequences, we can expect more of these.

The common feature of all these stories is that at first seems there’s some possibility that the claim is legit, then there’s some criticism by insiders and outsiders that’s brushed aside, then at some point it’s not even debatable and in retrospect it’s all a big joke.

The other common feature is that even when the original results are conclusively found to be fraudulent, or not supported by the data, there’s still some underlying hypothesis that could be true—and it’s also possible that the opposite is true. The data supply no valuable evidence, which is something that people seem to have a hard time understanding. (Evidence vs. truth.) It’s possible that prettier parents are more likely to have girls—or that they are less likely to have girls. It’s possible, as claimed by those faked Bible document, that Jesus was married. It’s possible that Cornell students have ESP, even though it does not really appear in the data that have been shown to us, etc.

But what interests me is the transition—the “tipping point,” if you will—when the lack of evidence becomes overwhelming, when you start to see arguments of this caliber, where there’s not even an attempt at linking to the real world:

(More here.)

As they say at Harvard, the replication rate is “statistically indistinguishable from 100%.”

And, of course, some of the people involved in these efforts never give up. I think the beauty-and-sex-ratio and ovulation-and-clothing researchers and still stand by their claims. On the plus side, they’re not trying to overturn a democratic election; they’re just pushing scientific arguments that are not supported by their data in the way that they claim.

P.S. Oddly enough, my later post on the Jesus’s Wife book already appeared last month. I guess I screwed up somewhere with my lagged posts.

18 thoughts on “When does a misunderstanding reach the point where it is recognized to be flat-out ridiculous?

  1. It’s possible that Cornell students have ESP […] But what interests me is the transition—the “tipping point,” if you will—when the lack of evidence becomes overwhelming, when you start to see arguments of this caliber, where there’s not even an attempt at linking to the real world:

    Trump’s disregard for reality is no greater than Bem’s (or those who sought but failed to reproduce his results) and there’s no such “tipping point” in that case. The evidence against (retrocausal) ESP was overwhelming before he decided to ‘investigate’ it with experiments incapable of ruling out “deception”.

    If parapsychologists followed the methodology of scientific inquiry, they would look what we know about the laws of physics, realize that their purported subject of study had already been ruled out, and within thirty seconds would declare themselves finished.

  2. Ms. King played her cards poorly. Before defending her bold finding to skeptical scholars, she should have built a coalition of like-minded scholars interested in the rehabilitation of women as active agents in biblical texts. Then these other folks could have referred to her critics as methodological Stasi. Many outsiders would have nodded in agreement, and the tipping point might have come and gone without the world noticing.

    That’s how the pros do it:

    https://www.forbes.com/sites/kimelsesser/2018/04/03/power-posing-is-back-amy-cuddy-successfully-refutes-criticism/?sh=1ac5442a3b8e

    “Now, [Amy] Cuddy can legitimately claim that power posing is science. Cuddy’s new academic paper published in March in Psychological Science, offers ample evidence (that even passes the p-curve test) that adopting an expansive posture makes people feel more powerful. Cuddy now refers to the effect as postural feedback rather power posing (perhaps to appease to those who claimed her research was more pop than science). Cuddy’s analysis examined over 55 studies and clearly demonstrates a link between expansive postures and feelings of power. Even the replication study that set off the original controversy found that those in high power poses felt more powerful than their low pose counterparts.”

    Never let them see you sweat.

    • Matt:

      Yes, the power pose people did a good job at working the refs. Regarding the specifics, Carol Nickerson looked into the claim of 17 replication of power pose and found that this claim lacked empirical support; see here.

    • In fairness, I doubt that many of the coalition of defenders of power pose cared at all about power pose, or even about Cuddy personally. If she had just outright fabricated data, they would have thrown her and her work under the bus and claimed the moral high ground, e.g., Stapel.

      I think they were more interested in protecting the dubious methods used to “discover” power pose, upon which their own careers rested. These methods essentially boil down to combining small convenience samples with a statistical significance threshold to make broad causal claims only tangentially related to the actual experimental manipulations and measurements.

      The analogous coalition King would have needed to build to defend her methods would have needed folks willing to defend obvious grammatical errors in historical languages, failure to look into provenance of artifacts, and hiring your in-laws with no relevant experience to conduct chemical analyses of ink and papyrus. So far, looks like only Harvard is defending those methods.

      • “tangentially related to the actual experimental manipulations and measurements.”

        I question the “related” part.

        My recollection of the power pose paper – it’s been a while since I read it but I did read the entire methods – was that the “experiment” was analogous to a group of children playing “job hunt”. To even call it an “experiment” is laughable. It’s shocking that a person with a PhD (or in the processes of getting one) would propose to call such a thing a viable “experiment” – and even more shocking that it seems to have been generally accepted by the establishment.

        Here’s a similar experiment:

        Put some kids in the sandbox and give some of them Tonka excavators and have them building a pipeline. Then we’ll have some other kids be the environmentalists and try to stop them! Which methods of protest are most effective at stopping the pipeline? Oh! yeah and the pipeline contractors get a piece of bubble gum if they finish the pipeline in two hours – now they have realistic incentives to oppose the environmentalists! Ha, we could even test the power pose – if the environmentalists use the power pose, will the excavators be more likely to stop?

        Here’s variation B1: what if the pipeline contractors (e.g., kids playing with excavators) encounter a native shell midden (buried modern shells) while they’re digging? If we prepare the contractors before hand with two different papers with statements on the value of Native American sites, which one will be most effective at encouraging the contractors to handle the unexpected encounter according to regulations?

        Wow! The sandbox sure is a fertile ground for experiment!! Removing reality really allows for some great experiments!

        • Hah, fair enough! I guess when I said “tangentially related”, I didn’t think about being tangent to reality…

      • Gec:

        I think you’re on to something here. Consider the linguist Steven Pinker, who’s been tangentially involved in the Jesus’s Wife and Power Pose stories. Sabar’s book has a story of how Pinker opposed a Divinity School course on Reason and Faith to be offered in Harvard’s general education curriculum on the grounds that “universities are about reason pure and simple . . . Faith—believing something without good reasons to do so—has no place in anything but a religious institution, and out society has no shortage of these. Imagine if we had a requirement for ‘Astronomy and Astrology’ or ‘Psychology and Parapsychology.’” Unrelatedly, Pinker came out in favor of power pose, I suspect out of a mix of personal connections and a general support for the hidden-influences school of psychology research. From that perspective, the low quality of the power pose paper wasn’t as important as the fact that it made claims that he supported. I don’t think it’s about Pinker’s career; it’s more about who he considers to be scientific allies.

    • Paul:

      I clicked through and read the review. I disagree with its statement that Sabar’s book is “34 percent too long” with “too much detail.” I liked the detail!

  3. || It’s possible that Cornell students have ESP, even though it does not really appear in the data that have been shown to us, etc.

    Its also possible, and far more likely, that Bem was simply practicing the dark-arts data analytic strategy he advised aspiring psychologists to embrace over 30 years ago in his cynical 1987 “Writing the empirical article” book chapter:

    “Examine (the data) from every angle. Analyze the sexes separately. Make up new composite indexes. If a datum suggests a new hypothesis, try to find additional evidence for it elsewhere in the data. If you see dim traces of interesting patterns, try to reorganize the data to bring them into bolder relief. If there are participants you don’t like, or trials, observers, or interviewers who gave you anomalous results, drop them (temporarily). Go on a fishing expedition for something—anything —interesting. ” (Bem, 1987, p. 171)

    Bem, D. J. (1987). Writing the empirical journal article. In M. P. Zanna & J. M. Darley (Eds.),
    The compleat academic: A practical guide for the beginning social scientist (pp. 171-201). New York: Random House.

  4. > It’s possible that Cornell students have ESP, even though it does not really appear in the data that have been shown to us, etc.

    That’s only because the people with ESP are able to use their abilities to fool the researchers.

  5. Scott Alexander’s concept of a “respectability cascade” seems related.

    He exresses the tipping point is a social phenomenon, rather than an evidentiary one. Once enough legit people hold a point of view for legit reasons, suddenly lots more will admit to it.

  6. Interesting post! We’ve seen this a few times – I was a physician using hydroxychloroquine early in the pandemic because the jury was still out on its effectiveness. Then the evidence started to converge and we stopped using it. Not suddenly – there never one singular point or study where we all stopped, but as the studies accumulated there were fewer and fewer colleagues still prescribing. But, still, the only people who stopped using it were those who were willing to re-evaluate their priors when new evidence was released. If that’s not a process you go through, then you keep beating the drum as a matter of faith and recapitulation of identity: HCQ works, the election was stolen, and now, “the lab-leak hypothesis is true”. From that perspective, anything to the contrary must be biased or politically motivated.

    Essentially it seems what you’re talking about here on a small scale is when consensus flips. But on a larger scale, this seems like when a Kuhnian revolution can take place. Different individuals have different thresholds, and these, collectively, relate to cultural/societal thresholds (which sociologists have tried to capture with terms like ‘uncertainty avoidance’) but we’d be better off talking explicitly about those thresholds and the values/biases that reinforce them than eliding them.

Leave a Reply to Martha (Smith) Cancel reply

Your email address will not be published. Required fields are marked *