I happened to be looking at this post from 2007 on the fallacy of the one-sided bet and came across this comment from Koray:
I’ve been recommending the book How We Know What Isn’t So by Gilovich for this sort of stuff. He starts out with the misconception that is the hot hand in basketball (apparently players don’t score in streaks, but everybody think so) and moves on to all kinds of wrong beliefs we form.
That’s funny. The most famous thing Gilovich ever did was to get the hot hand wrong, and he wrote a book called How We Know What Isn’t So!
Which reminds me of when my friend Seth wrote back in 2006 about the now-discredited and quite-possibly-never-performed “bottomless soup bowl” experiment:
An experiment in which people eat soup from a bottomless bowl? Classic! Or mythological: American Sisyphus. It really happened.
It’s hard to fault Seth for getting conned by this guy, given that other dupes included media-savvy folks such as Richard Thaler, Cass Sunstein, and basically the entire science staff at NPR.
I wonder if Gilovich is going to issue a new edition of his book, called, “How We Know What Isn’t So (and by We, I mean me).”
Could be worse. At least, when describing Wansink’s studies he didn’t follow up with, “The results are not made up, nor are they statistical flukes. You have no choice but to accept that the major conclusions of these studies are true. More important, you must accept that they are true about you” (Kahneman, on social priming studies, for anyone who hasn’t read his book).
A truly remarkable claim, especially coming from a Nobel-prizewinning scientist. Even if the studies on social priming had been perfectly valid, it would hardly follow that they had to be true for every individual human being.
Kevin,
You’ve picked up on a weird thing about social science. Sometimes it’s about universal properties of people or of social interactions; other times it’s about individual differences.
Ah, the benefits of being an academic fraudster and ripping off the entire public!! Wansink et al are in an ever growing club!
EW has some new girlfriends in the “Ooops I’m not native!!” club:
“An ethnic studies professor who was accused of claiming to be Native American has agreed to part ways next year with the University of California Riverside following a complaint filed by colleagues, documents show. After her resignation date, Smith will be eligible to retire as emerita professor and receive UC benefits, according to the agreement. ”
Such horrible consequences, a life languishing in the prison of guilt and valuable retirement benefits. :( I’m sure a lot of honest people are pretty happy they don’t have to deal with the guilt of getting the benefits of full professor fraudulently.
And this:
“Earlier this year, another California professor issued an apology for claiming to be Native American…Elizabeth Hoover previously stated she grew up believing she was Mohawk on her mother’s side and Mi’kmaq on her father’s, but that questions raised about her true identity had prompted her to do genealogical research, which resulted in her finding no records of tribal citizenship for any family members. Hoover remains part of UC Berkeley’s faculty. ”
Oh, sorry, I lied to get my job. I’ll feel bad for a few more seconds about ripping off the public. It’s great that she even has the specific fantasy tribes. Oh, but it was just a little secretarial error.
https://www.cnn.com/2023/08/28/us/andrea-smith-resigns-native-ancestry-questioned-reaj/index.html
Maybe one of these two conscientious ladies will run for Feinstein’s senate seat in 2024!
Hey, Andrew, you and I were fooled by the Hot Hand stuff too! Many people were. The results seemed really convincing — a whole season of the Philadelphia 76ers, back when simply collecting a whole year of box scores was a labor-intensive thing to do — and the analysis method seemed so simple and intuitive …indeed any time I think about it, I have to spend five minutes re-figuring-out why it doesn’t work. And the authors of the study didn’t just show (supposedly) that there was no Hot Hand in the data, they showed why people think there was one, illustrating that in order to NOT see a hot hand in random coin flips, the coin flips would have to be anti-correlated. (And that latter part is still true).
I try to be careful not to be patronizing when other people make mistakes like this because they’re so easy to make. There, but for the grace of God, etc.
Phil:
Indeed, I was fooled by the hot hand paper, to the extent that I used the hot hand in lectures and articles as an example of a fallacy. I have a post coming up on how the hot hand paper seemed so perfect when it came out. That’s actually how I originally came to be contacted by Josh “hot hand” Miller: he contacted me to explain why I was wrong on that.
My problem with Gilovich is not that he got the hot hand wrong; it’s that he continues to cling to the error. I could’ve continued to cling to the error too, but then I would’ve lost an opportunity to learn and to better understand the world.
Sure, I agree that it’s a mistake not to learn from mistakes. But Gilovich wrote his book back in 1991, when he and you and I and everyone else thought the Hot Hand had been pretty much disproved, or rather shown to be a small effect. If he wrote that book _now_ it would be inexcusable, but it’s unfair to expect him to have learned in 1991 from a mistake that wasn’t recognized until 20 years later!
And in general I think this post takes a patronizing attitude towards people who are fooled or conned by published work, and I think that’s unjustified. Or, rather, it’s unjustified as a general principle. In specific instances there can be good reason to be suspicious: a study is obviously underpowered, or the authors are known to be clowns, or the reported experiment has obvious methodological problems, or whatever. We should all have a healthy skepticism, especially for surprising results that aren’t backed up by a substantial body of evidence.
But, for instance, consider the soup bowl experiment-that-never-happened. OK, _now_ we know Wansink is one of the aforementioned clowns, but I don’t think that was known (or at least not widely known) in 2006. A respected-at-the-time researcher publishes work that claims that people eat more from a soup bowl that secretly replenishes the soup than from one that doesn’t…it seems like a pretty goofy experiment, and with a very predictable outcome — surely, if I had to bet, that’s the way I would bet, which might have been Wansink’s rationale for not bothering to actually do the experiment — but there doesn’t seem to have been anything suspicious about it, even in retrospect. Why wouldn’t one believe it? You don’t have to be a famously credulous person like whoever chooses the the NPR science stories to believe that the experiment was conducted and that the result was what is claimed.
Phil:
Regarding the Nudge authors: my problem was not that they got fooled by Wansink; my problem was that after they learned about the problems with Wansink’s work, they just memory-holed him from the second edition of their book. It would’ve been much better for them to have acknowledged the mistake, even in some general way (for example, “We and many others were fooled by . . .”).
Regarding Gilovich: my problem is not that he and his colleagues made mistakes with their hot hand study, fooling themselves into thinking that the hot hand was an illusion. You and I made those mistakes too! My problem is that, in recent years, well after being aware of the Miller and Sanjurjo results, as well as other issues with the hot hand study, Gilovich still finds it difficult to let go. The title of the 1991 book is just amusing in retrospect; the problem right now is that Gilovich persists in not fully acknowledging the problems with his famous paper.
Two comments. When I look at the book on amazon the date is given as 2008 as if it were a new edition… but it looks like it’s an unmodified reprint of the 1991 edition. So perhaps Andrew thought the book was revised, but in fact it was just reprinted?
Second, I’m finding it to be the case that the stuff I’ve learned through now 15 years or so of interacting with this blog is still not even close to widely known. In particular it’s extremely easy to find people asking questions on Reddit r/AskStatistics or similar about essentially “how to p hack my way into a significant result for my masters thesis”. or “my such and such test has a large p value therefore can I assume I’ve met the requirements to run XYZ test of difference in proportions for a 4 way blablabla”
I’ve learned to just read those kinds of things and not comment because if you come in and discuss maybe all of NHST is a bad idea and maybe you should learn some bayesian statistics and build a model that includes some insights about your specific problem and etc… the response can vary a lot, but in many cases it would be as if Zaphod Beeblebrox walked into a hat maker’s convention and asked for hats that simultaneously fit two heads…
It’s a sort of “There are more things in heaven and earth Horatio than are dreamt of in your philosophy” situation, and I feel like Hamlet, the one who everyone agrees “is crazy”.
Because of that, there’s some value to pushing back hard against about a century of really terrible stuff, even if the people who were doing that terrible stuff were just going along with the “standards for their times”.
How early *could* people have “known better”? Certainly by the 1960’s https://statmodeling.stat.columbia.edu/2016/05/06/needed-an-intellectual-history-of-research-criticism-in-psychology/
So if in 1991 people were still doing this same old crap, at least 30 years after as Andrew says “Paul Meehl was saying [it] 50 years ago. And it was no secret” then maybe it’s not terrible to call people out on it. Maybe we need to hold people to a standard where “just because it’s in all the textbooks doesn’t mean you are excused for believing it”…
hard to argue that though I think.