Seth Green writes:
Apropos of your post PNAS GIGO QRP WTF: This meta-analysis of nudge experiments is approaching the platonic ideal of junk science (which I reference all the darn time), a new umbrella review of nudge studies has a Wansink citation that made me chuckle.
The paper is The role of nudges in food choices: An umbrella review, and it’s relatively focused on two aspects of rigor:
* Quality of primary studies: “the overall trend indicated significant methodological limitations, with the majority of studies rated at high risk of bias or poor quality.”
* Quality of review: “only one review was rated as high quality, 16 as moderate, 9 as low and 10 as critically low, indicating insufficient quality.”
And then we get: “nudges are particularly well suited to the area of appetite, where food choices are often automatic, habitual and emotionally driven (Meier et al., 2022; Wansink & Sobal, 2007).”
I guess this is how Wansink got >2K citations last year. But also, if this is still the relevant citation for that claim, that raises my subjective assessment that there’s no there there.
Green follows up:
A nudge review from 2020 addresses the Wansink issue like so:
Authors’ Note
Six articles coauthored by Brian Wansink have been included in this study. Brian Wansink was relieved of his academic position on the grounds of academic misconduct. We do, however, include the six papers here, as they have not been subject to corrections or retraction, and are still available to be read in respectable journals.
The word “respectable” really gets under my skin.
And:
Sorry to harp on this but what is it with behavioral scientists still citing this Wansink & Sobal paper??
In JEP: Nudging Meat off the Plate in Foodservice? A Systematic Review and Meta-Analysis Identifying Moderators in Field-Based Intervention Studies: we get:
The default option requires less effort and is effective due to consumers’ passive decision-making and/or cognitive or attentional limitations, which is particularly the case for daily food choices (Wansink & Sobal 2007).
Is it a consensus among a research subculture that I’m not a part of that the idea of mindless eating is still valid even if the data were fudged? (But even if that were true, wouldn’t you cite something else?) I am wondering, why risk alienating those of us who care about stuff like this to pay tribute to someone who’s totally out of the game? It’s certainly not to curry favor with Jeremy Sobal, who passed away last year. I just don’t get it.
My reply: I think of this as a special case of a general phenomenon, which is that people–including credentialed scientists, including credentialed scientists who work in areas where there is high variation–have this idea that the evidence should all go in the same direction. It’s connected to what Tversky and Kahneman called in their 1971 paper, the belief in the law of small numbers, which they define as to “regard a sample randomly drawn from a population as highly representative, that is, similar to the population in all essential aspects.” In this case, it’s not sampling balls from an urn, it’s sampling experiments from the hypothetical population of all possible experiments in a research area.
With regard to the Nudgelords, and, more generally, many other researchers in the human sciences, there are three stages to this erroneous belief:
1. The belief that if a general phenomenon (e.g., “nudge,” or “evolutionary psychology,” or “social priming”) is real, that it will show up in the same direction no matter how it is studied (for nudging this could be investment decisions or food choice or risky behaviors; for evolutionary psychology this could be the sex of babies or religious behavior or voting behavior; for social priming this could be walking speed or hormone levels or political attitudes; etc.).
2. The belief that any specific effect being studied will appear in any data, ideally with statistical significance at the conventional 5% level, but otherwise with statistical significance at the 10% level, or a result in the right direction, or else explained away as the result of an interaction (as with that notorious claim that the effect of monthly cycle on clothing choice was moderated by outdoor temperature) or as unreliable data from a low-power study.
3. The belief that an effect found in the aggregate will apply to each individual or, if not, can be explained away with some elaborate theory. I shared a particularly ridiculous example a few years ago from Ian Ayres, a law professor notorious for copying others’ work without attribution (although in this case the error does not seem to have been copied from another source). My point here is not that elaborate theories are wrong but rather that it is an error to think that some explanation is necessary when a particular data point does not fit an average pattern. Unless your regression has an R-squared of 100% and a residual variance of zero, you shouldn’t expect your data to all go in the same direction.
Putting these together: if you’re coming into your research project or meta-analysis with the conviction that the effect you’re studying is real, then you’ll expect all data and all studies to conform to the pattern, with any failures attributable only to noise or interactions, never to the possibility that the effect sometimes does not appear or even goes in the wrong direction, and never to the possibility that a positive finding (statistically significant or not) in the desired direction is meaningless or wrong. So, the fact that these Wansink papers could well report experiments that were never performed, or data that were not conducted as reported, or data that were selected and tortured to yield their findings . . . that doesn’t even matter, because the authors’ (unexamined) belief in the law of small numbers tells them it doesn’t matter.
P.S. At this point we have to avoid falling into the trap we’ve just identified. No, I don’t think the belief in the law of small numbers is universal. What I think is that the concept of “the belief in the law of small numbers” is a useful way to understand many different sorts of cognitive errors, including what the Nudge people are doing. And, indeed, Tversky and Kahneman first wrote about many of these errors in the context of psychology researchers, in particular researchers who expected that small studies would routinely return positive results, despite the mathematics that would say otherwise.
With the Nudgelords there’s the additional factor that they’ve received fame and fortune so they have some motivation to not back down and the standing to pretty much ignore their critics (occasional swipes at “Stasi” notwithstanding). But ultimately I see this as more of an intellectual error than a moral failure or a mismatch of incentives: they start with the implicit belief in the law of small numbers, and that carries them through to a sort of universal belief in all of their evidence, and from that point it’s easy enough to ignore dissenters or consider them (us!) to be ignorant haters.
Maybe they meant “respective”.
Really interesting framing of the replication crisis! I would add the following. Beyond people sometimes believing the effect of a paper to be real, I think sometimes people believe that _if_ a study is well designed, the empirical estimates must be “real” and believed. In economics seminars (and in these comments!), I often see a lot of skepticism about empirical strategies (Did they adjust for all relevant variables? Is the instrumental plausibly exogenous?), but then once people believe the strategy, the estimates take on a life of their own. Then it becomes a free for all to hypothesize why the effect is different than in other studies, why the effect seems to differ between subgroups, etc. But of course, in many cases that variation comes from noise. It is somehow hard to internalize that even a well designed study will only give you the treatment effect in expectation.
There is no rigorous way to explain “the effect” besides without using it for hypothesis generation then checking the predictive skill of *your model* on *new data*.
All the problems stem from testing the bizarro null hypothesis of no difference rather than the research hypothesis, which inverts all the logic and incentives of science.
Of course, once realized, you will not design studies looking for “the effect” to begin with (essentially all scientific content in the data has already been removed/ignored during the design). Instead you will study the dynamics of the phenomenon and try to model that.
Couldn’t Tversky and Kahneman have come up with a better term for this… calling something a ‘“law” that’s more of a cognitive bias… and reusing the expression used for the Poisson limit of the binomial?
Jyd,
I’m not sure what would be a better term. “Belief in universal representativeness,” perhaps? “Belief that the part will replicate the whole?” “Belief in coherence of evidence”?
In defense of the term, they call it the “belief in the law of small numbers.” For other alternatives, Wikipedia has a few under “hasty generalization.” I like the “fallacy of the lonely fact,” maybe a variation along the lines of the “fallacy of the lowly sample” works.
Re: “Why wouldn’t you cite something else?” There’s an explanation other than the citers believing the results of the cited paper: they don’t actually read the paper and certainly don’t read any subsequent discussion in the field. I’m constantly amazed by the number of citations that, if one looks at the cited paper, point to junk or that garble the actual conclusions of the (good) research. For many people, “citing something” means entering some keywords into Google (or now, AI), glancing at the title / abstract, and stopping there. To be fair, there is such a deluge of papers that keeping up is impossible and the lazinesss is understandable, but it’s not good for science.
Quote from above: “For many people, “citing something” means entering some keywords into Google (or now, AI), glancing at the title / abstract, and stopping there.”
There is a recent paper by Schneider et al. (2024, PLoS ONE 19(8): e0304342) concerning questionable research practices involving a Danish and an international sample if I understood things correctly. This is a quote from the paper:
“Interestingly, the three most admitted uses of QRPs in both surveys are all potentially questionable citing practices: ‘Selective over-citing’ (#16) (DK 64%, INT 65%), ‘cite without reading’ (#7) (DK 59%, INT 60%), and ‘irrelevant citing to please’ (#15) (DK 59%, INT 62%).” (p. 15)
LLMs actually make finding good citations easier. You have to get it to give you exact quotes from the sources it decides are relevant. It won’t work out of the box, it requires multiple agents checking each other and doing QC. But once you can trust it is working at ~80% without hallucinating, it can literature search (in parallel) orders of magnitude faster than a person ever could. Then have it write a report to follow up on.
In fact, current legacy citation practices are very inefficient because an entire article is referenced at once. Even manually, I give exact quotes wherever possible so the reader does not have to search through the article to find it. Such quotes should be standard in the references.
Quote from above: “Even manually, I give exact quotes wherever possible so the reader does not have to search through the article to find it.”
I have noticed in my own writing that I have quoted more and more in more recent manuscripts. Also because I reason it might minimize misinterpreting and/or misrepreseting the author(s) I cite.
I think I have read some “official” writing recommendations once by some “official” organization that stated something like that you should paraphrase “so it fits better with your own writing” or something like that. I think that might increase the chance of stating things that the original authors may not have intended and/or stated.
I also like using quotes because sometimes people just write things down very interestingly, or well, or whatever word is most appropriate. It’s a bit like a poem, or piece of art to me. I don’t want to touch the words, they should remain as they were written.
And I think it’s also fun to sort of combine quotes from different authors, perhaps even different topics, and sort of “mix” them to “create” something new that may be more than the combination of the quotes. It’s like picking and choosing and mixing ingredientes to cook something new. Or it’s like taking music samples and mixing them and creating something new that way. Something like that.
In case you are unaware, there is also the PMC API:
Quote away. It stopped updating during the shutdown so is a month or so out-of-date but looks like that should be back soon.
https://www.ncbi.nlm.nih.gov/research/bionlp/APIs/BioC-PMC/