Uncanny academic valley: Brian Wansink as proto-chatbot

Back in 2017 I posted this exchange that psychologist Tim Smits had had in 2012 with former food researcher and NPR star Brian “Pizzagate” Wansink.

I happened to come across this post yesterday pretty much at random–I was writing something else that linked to my post on storytelling as predictive model checking–and it linked to the previous and following posts on the blog:

Pizzagate update: Don’t try the same trick twice or people might notice

Measurement error and the replication crisis

Both these old posts looked and, indeed were, interesting. One of the advantages of having all this old material is that I can reread it and be reminded of some story or bit of reasoning that had, over the years, receded beneath my threshold of consciousness.

Rereading Smits’s decade-old exchange with Wansink from the perspective of modern chatbots was an uncanny experience.

I’ll share the exchange, but first some context.

You know how we keep talking about how chatbots are excellent bullshitters, they can emit a string of words that on the surface seem reasonable, and they can even associate those strings of words with what looks like logical reasoning, if you don’t probe too carefully? That’s just what Wansink did!

In his email, the now-disgraced food researcher did a kind of Turing-test performance, simulating scientific inquiry in the same way that so many published papers present words and sentences and images in a way to imitate real research in a way that is convincing enough to fool journal editors and reviewers–or, at least to exhaust them into submission.

I could tell at the time that Wansink was bullshitting (and Smits could tell this too; that’s why he emailed it to me); in hindsight this form of response just seems so familiar. Even to the point where, after Wansink is called on the absurdity of his response, he quickly pivots to a new line of b.s. As with the chatbot, once you let go of the idea that it understands or wants to understand what’s going on, we can step back and be amazed by the impressive fluidity of the text, in some ways more impressive given that it is pure association with no underlying meaning.

OK, here’s the story from Smits:

In 2011, the Cornell research published an article (Zampollo, Kiffin, Wansink & Shimizu, 2011) on how children’s preferences for food are differentially affected by the how the foods are presented on a plate compared to adults. . . . some of the findings were incomprehensible from the article . . . I [Smits] wrote a polite email. Asking for some specific information about the statistics. This was the response I got.

Dear Tim, Thank you for being interested in our paper.Actually there are several errors in the results section and Table 1. What we did was two step chi-square tests for each sample (children and adults), so we did not do chi-square tests to compare children and adults.As indicated in the section of statistical analysis, we believe doing so is more conclusive to argue, for example, that children significantly prefer six colors whereas adults significantly prefer three colors (rather than that children and adults significantly differ in their preferred number of color). Thus, for each sample, we first compared the actual number of choices versus the equal distribution across possible number of choices. For the first hypothesis, say #1=0, #2=0, #3=1, #4=0, #5=2, #6=20 (n=23), then we did a chi-square test (df=5) to compare those numbers with 3.83 — this verified the distribution is not equal. Then, we did second chi-square test (df=1) to compare 20 and 0.6 (the average of other choices), which should yield 18.3. However, as you might already notice, some of values in the text and the table are not correct — according to my summary notes, the first 3 results for children should be: 18.3 (rather than 40.4)16.1 (rather than 23.0)9.3 (rather than 26.88) Also, the p-value for .94 (for disorganized presentation) should not be significant apparently. I am sorry about this confusion — but I hope this clarify your question.

Well, that was interesting. Just one email, and immediately a bunch of corrections followed. Too bad the answer was nonsensical. So I [Smits] wrote back to them (bold added now):

When reading the paper, I did understand the first step of the chi-square tests. I was puzzled by the second step, and to be honest, I still am a bit. The test you performed in that second step boils down to a binomial test, examining the difference between the observed number of counts in the most preferred cell and the H0 expected number of counts. Though this is informative, it does not really tell you something about how significant the preferences were. For instance, if you would have the following hypothetical cell counts [0 ; 0 ; 11; 0; 0 ; 12], cell 6 would still be preferred the most, but a similar binomial test on cell 3 would also be strongly significant. In my opinion, I thus believe that the tests do not match their given interpretations in the article. From a mathematical point of view, your tests on how much preferred a certain type of plate is raise the alpha level to .5 instead of .05. What you do test on the .05 level is just the deviation in the observed cell count from the hypothesized count in that particular cell, but this is not really interesting

Then, this remarkable response came. . . . they agree with the “shoddy statistics” . . . Moreover, they immediately confess to having published this before.

I carefully read your comments and I think I have to agree with you regarding the problem in the second-step analysis.I employed this two-step approach because I employed similar analyses before (Shimizu & Pelham, 2008, BASP). But It is very clear that our approach is not appropriate test for several cases like the hypothetical case you suggested. Fortunately, such case did not happen so often (only case happened in for round position picture for adults). But more importantly, I have to acknowledge that raising the p-value to .5 in this analysis has to be taken seriously. Thus, like you suggested, I think comparing kids counts and adults counts (for preferred vs rest of cells) in 2×2 should be better idea. I will try to see if they are still significant as soon as I have time to do.

So, back in 2012, someone sent Wansink a devastating criticism, he or someone in his lab responded in a cordial and polite way, gave tons of thanks, and then for the next several years they did essentially nothing. As Smits put it, “Their own press releases and outreach about that study did not show a single effort of self-correction. You can still find some of that material on their website. Similarly, despite the recent turmoil, I have seen them just continue their online communication efforts.”

At the time, I saw this as a story about Wansink and his Food and Brand Lab, and about Cornell University’s continuing support of that organization, and, more broadly, the academia and news-media environment that incentivized such behavior.

And, yes, that’s all still a concern. But, looking back, what strikes me most is the chatbot-like nature of Wansink’s replies.

Just as the development of flying machines gave us insight into the ways that birds fly, so does the development of chatbots give us insight into human reasoning, including bullshitting, which is some associative form of reasoning, not quite the same as logical/rational/conscious/attentive reasoning but of interest for its own sake.

And this isn’t just about Wansink (who remains shameless), or even just about junk science; it’s also about how we have implicitly been trained to respond to criticism, which is to supply smooth words that imply a logical train of thought without necessarily being coherent.

27 thoughts on “Uncanny academic valley: Brian Wansink as proto-chatbot”

  1. This post, for me, once again highlights the possible importance of proper training and honing of reasoning principles and skills. Once again I note that, if I remember correctly, I wasn’t explicitly taught much, if anything, about this all during my time at university. Once again, I wonder how this might be for other people who went to university.

    It also highlights, once again, for me that I think certain features of present-day academia and social science might facilitate 1) (certain?) people that bullshit, and 2) that this bullshitting might partly, or even largely, goes unnoticed (at least for some time), and 3) that there might be few protective and corrective and preventive processes that remedy 1) and 2).

    I was very recently pondering again whether to try and see if I can make some sort of possible manuscript-outline regarding possible issues such as these three points. The focus however would be on psychopathy in academia at large and psychological science in particular, and would be a follow-up of something I have written already. It would focus on why exactly the environment of present-day academia and psychological science might particularly facilitate (people with) psychopathy, which I reason might have (at least some) overlap with bullshitters and bullshitting.

    • I think that depends on your discipline and national tradition: Americans have to hope they take a class or two of philosophy or communications and learn explicitly about logic and argumentation while in the Dutch tradition undergrads start with a class on “what is a historical explanation?”

      Outside formal education you have the good ways of learning how to know what isn’t so (something like Jona Lendering’s essays on historical methods or Skepdic), the bad (Eliezer Yudkowsky’s blogging), and the mixed (say Daniel Kahneman’s trade books). And because these are all in the marketplace of ideas, its up to the learner to separate the lemons from the useful tools.

    • In my Canadian undergraduate education I had advances seminars in computer science where we were given one of the instructor’s papers and told “find and correct the error in this proof” and in history with extended study of specific problems. A trouble is that generic effective thinking only gets you so far and what works in specific contexts is learned by experience and practice. Notoriously some archaeologists spend time early in their education having long theoretical debates which turn out to be mostly irrelevant if they ever do interesting field archaeology.

      • Thank you for the replies!

        I went to a Dutch University and enterered somewhere inbetween the 2nd and 3rd year of a clinical psychology bachelor via some course credits (or something like that). I don’t think that matters in light of my point, which I think is especially relevant for the master parts of the education, like my Research Master in Behavioural Science which I did follow from start to end in the more usual way.

        Here is something I wrote in one of my manuscripts, also with that Research Master education in mind, that further expresses my point of view and wonderment:

        The more I think about it, the more I reason that less might be more, and that simple might be better (cf. Cohen, 1990, p. 1304-1306). I think a course based on fundamental topics and concepts like logic, reasoning, and argumentation (also see Sayers, 1947) would have been more valuable than most of the courses that were part of the curriculum of my education. Courses which involved papers, information, and “knowledge” (nine-tenths of which I will never use again) of questionable quality and validity given all the problematic issues mentioned in the introduction. I think logic, reasoning, and argumentation are extremely important with regard to comprehending scientific writing, integrating scientific findings, critical thinking, hypothesizing, designing
        experiments, (re-)formulating theories, writing papers, etc., which in my view further justifies an entire specific course. In light of this all, I have wondered more and more why I wasn’t taught (much) about these things.

        After some searching I discovered an online version of a book by Dowden (2022) titled “Logical Reasoning”. The table of contents includes: “what is a statement?”, “what is an argument?”, “rewriting arguments in standard form”, “deductively valid and inductively strong”, “improving your writing style”, “distinguishing deduction from induction”, “what is logic?”, “logical forms of statements and arguments”, “reviewing the principles of scientific reasoning”, etc. I think learning about these fundamental things, in a certain order, might be of tremendous value. That a course on these topics might be useful is underscored by Mulnix (2010) who notes that far too many college undergraduates have yet to achieve proficiency in thinking critically, and that many students are
        unable to recognize even the simplest of evidential relationships between statements (p. 11). Mulnix (2010) further notes that “This appears to suggest that the fundamental concern of any critical thinking course ought to be teaching students to ‘grasp’ inferential or evidential connections.” (p. 11). A course that covers the topics and concepts discussed in the book by Dowden (2022), that involves learning names for argument patterns and fallacies (cf. Mulnix, 2010, p. 12), that involves rewriting arguments in “standard form” (cf. Dowden, 2022, p. 41) or using “argument maps” (cf. Mulnix, 2010, p. 12; van Gelder, 2005, p. 44-45), and that involves practicing by editing papers (Mulnix, 2010, p. 13) might be very appropriate and useful. I wish such a course had been part of the curriculum of my education. I wonder why it wasn’t.

        References:

        Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304-1312
        Dowden, B. H. (2022). Logical Reasoning
        Mulnix, J. W. (2010). Thinking critically about critical thinking. Educational Philosophy and Theory, 1-16
        Sayers, D. L. (1947). The lost tools of learning. Paper read at a Vacation Course in Education, Oxford 1947
        Van Gelder, T. (2005). Teaching critical thinking: some lessons from cognitive science. College Teaching, 53, 41-46

  2. Having used Claude a lot lately for coding assistance, this reminds me of what Claude does whenever I chide it for making an error: “Good catch!” it says. As valuable as Claude is in writing beautifully structured tedious stretches of code, the sycophancy programmed in following my discovery of (extremely common logical) errors palls pretty quickly.

    The difference, of course, is that unlike Wansink, Claude doesn’t submit its output for publication.

  3. Andrew wrote: “As with the chatbot, once you let go of the idea that it understands or wants to understand what’s going on, we can step back and be amazed by the impressive fluidity of the text, in some ways more impressive given that it is pure association with no underlying meaning.”

    Huh? “with no underlying meaning.”??? Huh???

    Andrew, you sound like me. That’s not a good look.

  4. Not sure why you say Brian Wansink is still being shameless. I clicked your link and it looks like a blog post about his travel and music interests? He resigned his position at Cornell a decade ago. Was he supposed to swallow a cyanide capsule too?

    • Dewayne:

      What a horrible thing to say! Of course I don’t want Wansink to hurt himself in any way. What I would like is for him to acknowledge that he was doing junk science, or at least move away from his junk-science claims. But if you go to his website, it says:

      The tabs are filled with useful tips and tricks on how people can eat to be healthier and happier. They’re based on 35+ years of our published research.

      And, from his self-description:

      People call me a “pracademic” — an academic who has a practical mission to help change behavior and improve people’s lives. I was a professor for 30 years, and I worked at discovering new ways to help people be happier, healthier, and more meaningfully connected. . . .

      I resigned from Cornell in 2019, and I’m very grateful and proud of my practical discoveries and publications related to eating behavior. I hope this website can provide initial direction for people who are interested in changing eating behavior either in their own life (see the For You or For Families tabs above, or the book Mindless Eating) or changing eating behavior on a wider scale (see the For School Lunches, For Workplace Wellness, Grocery Shoppers, or Restaurant Lovers tabs above, or the see the books Slim by Design or Marketing Nutrition).

      ​The ideas and tools we discovered have been published in 300 or so peer-reviewed journal articles . . .

      No mention of the retractions, the many papers based on nonexistent or garbled data, etc.

      But, sure, I have no problem with the blog posts about his travel and music interests. If that were all that was on the webpage, that would be cool. He seems like a sincere and nice guy who perhaps just doesn’t realize that it’s not so cool to purport to present empirical evidence if it isn’t there. There was never anything stopping Wansink from just writing papers with cool ideas for experiments without pretending to have data and statistical significance. But he chose to go that extra step, and I do find it a bit otherworldly that he continues to present his claims as if they represent real empirical research.

      In my ideal world, Wansink would remain a happy, upbeat person who can also come to terms with the fatal flaws in his research. He already took a lot of government and private money based on what seem to me to be essentially false pretenses (that he had strong evidence for his claims about eating behavior etc.), and it bothers me that he still is pushing this stuff. So, yeah, that seems shameless to me. Again, I wish him no unhappiness; indeed, I’d like to believe that he could be even happier if he could unburden himself. But it’s a free country; if he wants to remain shameless, that’s his choice.

      • I think people who behave shamefully deserve some unhappiness. There are many, many worse people than Wansink, and I’m not saying he deserves to spend twenty years in prison or anything. But he did defraud funding agencies and employers out of hundreds of thousands of dollars, so I think something like five years in prison would be good. Or at least two, for crying out loud. And he should be appropriately unhappy.

        • But he did defraud funding agencies and employers out of hundreds of thousands of dollars, so I think something like five years in prison would be good. Or at least two, for crying out loud.

          Do you realize that this will soon be used for people with ideas, methodology, and conclusions you like? It is pure argument from authority. Bad actors will just strive to be that authority.

        • > I think something like five years in prison would be good. Or at least two

          That’s an astonishing thing to say. As I see it, Wansink and others (e.g., Cuddy and many others mentioned on this blog in the past) were all simply unaware of the extent of the statistical abuse they were engaging in. I doubt very much that they actively set out to deceive. There are many active academics out there who continue to abuse statistics; I doubt that any of them is actively trying to defraud their audience. They just don’t get the problem, and the reason is probably that their advisors didn’t get it either. People just keep on passing on misinterpretations from one generation to the next (“it deepens like the ocean shelf”; Larkin).

          Of course, as Andrew says, it would be good if people like Wansink would just at least acknowledge the retractions. I suspect that the inability to do that is the relentless (to my mind, quinessentially American) focus on the positive, and heavily downweighting to zero all the negative stuff.

          Putting these people in jail seems absurd to me. The push should be to try to provide the right kind of education in these fields.

        • In my experience, a prevalent view out there is that statistics is just a convenient tool for quantitatively bolstering an idea that you already believe in. That is why almost no scientist out there ever finds evidence against their favorite theory. It would be just silly to put them all in jail for abusing statistics to make a point that they already believe in. I think that the best one can do is try to educate the next generation better.

        • Shravan:

          I too would not want Wansink to go to prison, and you may be correct that he was “simply unaware of the extent of the statistical abuse they were engaging in,” but I don’t think he was completely blameless. From what I’ve seen, it seems like he knowingly lied about his experiments, claiming he had done data collection that he never had, taking data from one experiment and claiming it was from another, etc. At the very least, his inconsistent and continuing changing stories imply something like a complete indifference to the facts, to the extent that just making things up was part of his method.

          I expect he sincerely believes in his ideas, but that’s another story. Being a true believer is not at all a bar to lying. You write that people like Wansink didn’t “actively set out to deceive” and “just don’t get the problem”–but, whatever he originally “set out” to do, it does seem that he established a pattern of deception, and one of the “problems” he didn’t “get” is the idea that it’s a problem to lie about your data. He may have well thought that everybody does it.

          Consider an analogy to an embezzler who thinks that the company money is rightfully his, because that’s just how everyone succeeds in business. The embezzler might not originally set out to steal and then, once he steals, he might just not get the problem–but he can still be aware that he’s breaking the rules, lying about it, making up alternative explanations, trying to lead investigators off the scent with affable lies, etc.

          But, yeah, I would not recommend prison. I’d recommend restitution: that he pay back to the government and his private funders all the money he got for the experiments that he didn’t do or that he misrepresented, and in addition some sort of fine, and some community service such as repairing trails in the woods or whatever, something that he might feel good about doing. If I were in charge of the world, which I’m not (and which is a good thing).

        • @Shravan: Compare Table 5 of 10.2139/ssrn.2715021 with Table 2 of 10.1017/S0021849903030290, and tell me that Wansink didn’t realise what he was doing.

        • Shravan. I’m not sure of what’s been proven sufficiently, but my impression about Wansink is that he claimed that he did things that they never even came close to doing. It wasn’t just about using some p values in a stupid way, it was about claiming to have used money to perform physical experiments that never were performed, or to pay RAs to analyze data that didn’t ever exist or similar.

          Stuff I remember is that there’s no real evidence that the “bottomless soup bowl” was ever constructed, set up, and used, that there were datasets about something like maybe veterans administration records that were analyzed but he never had that data, and the reported statistics were mathematically impossible, or extraordinarily unlikely… wansink was reported to have said things that lead one to believe he was a fraud in ways that would be very similar to Bernie Madoff (though obviously much less money involved), just make up stuff, say you’re doing one thing when you’re doing another, use “fake data” about what you’re up to to convince people to invest more money in your projects, etc.

          If it’s true he was defrauding the govt and private investors by claiming to have performed experiments that he never performed, and by analyzing made up data, then what he did was substantially different and more actively harmful than merely “doing what he was taught and not knowing better”.

          Again, this has not been proven in court so I can’t speak to the actual facts, only the allegations, but those allegations were pretty strong.

      • As a result of the new comments here about the website and such things I decided to share something I thought about after also taking a look at the website hours agao. It also seems in line with something I expressed earlier here about the possible similarities between bullshitting and psychopathy.

        Anyway, I recently wrote something about psychopathy and encountered a well-known book by Cleckley (1941) titled “The Mask Of Sanity – An Attempt To Clarify Some Issues About The So-Called Psychopathic Personality”. It is my understanding that psychopathy can be seen as being dimensional, and consisting of a collection of certain personality characteristics. In light of the blog post and comments here, I read some parts of the book by Cleckley again, more specifically a section about lack of remorse or shame. Here are two quotes from the book:

        “(…) but often he will go through an idle ritual of saying that much of his trouble is his own fault. When the latter course is adopted, subsequent events indicate that it is empty of sincerity-a hollow and casual form as little felt as the literal implications of “your humble and obedient servant” are actually felt by a person who closes a letter with such a phrase.” (p. 343)

        “Whether judged in the light of his conduct, of his attitude, or of material elicited in psychiatric examination, he shows almost no sense of shame. His career is always full of exploits, any one of which would wither even the more callous representatives of the ordinary man. Yet he does not, despite his able protestations, show the slightest evidence of major humiliation or regret.” (p. 343)

        I also thought about this all because I have wondered about shame more specifically in light of the above comment I made about possibly writing abother manuscript about why psychopathy might particularly be present and facilitated in present-day academia and psychological science. I wonder whether several processes involved in optimally performing science or being a good scientist involve personality characteristics that people possess to different degrees, such as shame.

        I think, for instance, that shame might be related to certain scientists to be extra careful in their next experiments or data analyses or writing papers after they made a mistake and were pointed to that previously by others. That in turn might result in better work, or more careful considerations and reasonings, etc. All in all, that can perhaps be considered to be part of some sort of growth (or whatever word is most appropriate here), and in turn might be good for science overall, but it might involve the abililty or instance to feel shame by the scientist who made the mistake in the first place…

    • You forgot to read the “About” page.

      “I resigned from Cornell in 2019, and I’m very grateful and proud of my practical discoveries and publications related to eating behavior.
      …
      ​The ideas and tools we discovered have been published in 300 or so peer-reviewed journal articles, but many of them can also be found in much more readable “How-to” books:”

      Looks like seriously shameless to me.

  5. FWIW, you are talking about the same phenomenon some blokes at the University of Tokyo do in a recent paper.

    Here’s the press release. I don’t buy the tech they’re using (“energy landscape analysis”), but the obvious point that people with Wernike’s Aphasia (a condition in which it appears that normal thought processes have become disconnected from language generation processes) generate text that sounds real similar to the chatbot overconfident BS phenomenon.

    https://www.u-tokyo.ac.jp/focus/en/press/z0508_00410.html

    Oh, yes. I think you’ve nailed where chatgpt gets this from.

    “And this isn’t just about Wansink (who remains shameless), or even just about junk science; it’s also about how we have implicitly been trained to respond to criticism, which is to supply smooth words that imply a logical train of thought without necessarily being coherent.”

  6. Interesting, Wansink used to have a website where he freely admitted to his past deeds while still expressing belief in his views about healthy eating. That website is still linked to his homepage (https://www.tipsforphds.com/) but the link is broken. Perhaps he decided the statute of limitations had run out and he had serve his sentence? In any case, he now appears shameless but it was not quite how he appeared a year ago.

      • Dale:

        I followed the link, where he writes: “I had some disasters like having 7% of my articles retracted and having to retire earlier than I wanted.”

        I don’t see where on that page he admitted, readily or otherwise, to his misdeeds. He mentions the retractions but, at least as it’s written there, it just seems like something that happened to them, not the result of his misconduct. Also I get the impression that the “7%” there is supposed to minimize the problem, as if the other 93% of his articles are just fine.

        So I’d say that this now-disappeared page was a baby step in the right direction, but then even that was too much openness than he was prepared to handle. We could imagine an alternative world in which he’d followed up that mention of “disasters” with a frank admission of the places where he lied, why he felt that it was ok to do so, etc. And maybe he’ll go that route in the future. But for now he seems to decided to play pretend.

        And, who knows, he might get rehabilitated without ever acknowledging the problem. Maybe the next step is some softball interviews from Steven Levitt, Sean Carroll, and the rest of the science-boosting community. If Al Sharpton can be treated with respect in the political community, all things are possible.

        • Well, when those pages were still active, I remember him discussing each of the retracted papers and how he would recommend doing the studies now (then). Still not saying he messed up, but I’d still not classify it as “shameless.” I’ve seen far worse.

        • Dale:

          I fear that the main problem he saw with those studies was that he got caught. If not, he’d be interested in how he should’ve done all those non-retracted studies that had similar problems.

  7. *”it’s also about how we have implicitly been trained to respond to criticism, which is to supply smooth words that imply a logical train of thought without necessarily being coherent”*

    I recently posted some criticism of a DuoLingo post on LinkedIn, which discussed a research article that, according to DuoLingo, provided “independent evidence that the DET [DuoLingo English Test] is a valid, accessible option for evaluating international applicants”. (DET is a relatively new English proficiency test that they have marketed very wel and that strongly competes with traditional tests such as Educational Testing Service’s Test of English as a Foreign Language, TOEFL).

    The problem is that the article that they linked to provides no evidence whatsoever that DET is a valid test. I’m sharing my comment below, and part of their response. It’s fairly classic in that it doesn’t address my main point, doesn’t acknowledge their mistake, explains something to me that I just explained to them, follows it up with a nonsensical statements that is presented as logical, moves the goalposts, etc. A nice example of “smooth words that imply a logical train of thought without necessarily being coherent”.

    My comment:
    “I’m not sure I understand how the results of this paper provide evidence that the DET is “a valid option for evaluating international applications”.

    If you read the paper carefully, you will see that the correlation between DET test score and first-year GPA score was not statistically significant. In other words, DET test score did not predict first-year GPA score.

    Most of the correlational tests reported for IELTS and TOEFL also yielded results that were not statistically significant. What I think this suggests is that the study did not have enough statistical power (not enough data points) to detect small correlations between test scores and first-year GPA scores, let alone to detect differences between DET / IELTS / TOEFL in predicting first-year GPA scores. (Assuming that such differences exist.)

    This study does not provide strong evidence that “DET scores predict academic performance just as reliably as TOEFL and IELTS”.”

    The start of their response:
    “Thanks for your thoughtful comment! It’s important to keep in mind that statistical significance is very sensitive to sample size: with a large enough sample, even very small correlations will usually be statistically significant. That’s why researchers look at the size of the correlation itself and not just whether it reaches significance. (…)”

    It reads a little LLM-y, now that I read it again.

    Full exchange [here](https://www.linkedin.com/posts/duolingo-english-test-_how-does-the-det-compare-to-toefl-and-activity-7373709994141204480-kWFX?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAkSHQ4BO6Ohv-4DXky6x8iqKezUeeJVNEE).

    • Try just ignoring all the p-value/significance stuff when reading a paper. Think of it like some kind of rendering/printing glitch. I have for 10 years, its entirely irrelevant. Too bad it also determines what gets published to begin with, so we also have to assume over half of what we read would not replicate.

  8. My favourite Wansink-as-BS-merchant moment is in an interview that he gave to Retraction Watch quite early in the saga at
    https :// retractionwatch.com /2017/02/16/social-science-isnt-definitive-like-chemistry-embattled-food-researcher-defends-work/ (link sliced up to avoid the need for moderation). We had questioned why, in a study of children eating in a school cafeteria, “carrots served by the staff” minus “carrots returned on the plate at the end of the meal” did not equal “carrots eaten”. In response to the interviewer asking this, Wansink replied:

    “… there were times that the math of what food kids were given and what they ate vs. left behind didn’t add up to a perfectly round number because this study was done in elementary schools (and based on the well-cited quarter plate data collection method referenced in that paper). For anybody who can remember school lunches as an 8 year old – some amount of it ends up on the floor or in pockets.”

    Now, this sounds quite reasonable, until you realise that the authors only measured “carrots served” and “carrots returned”. That is, “carrots eaten” was *defined* as the difference between those two measurements. The children were not equipped with oesophagus-mounted sensors that counted how many carrots went down their gullet. Carrots were counted as “eaten” even if they did end up on the floor.

    I still don’t know whether Wansink realised this (we might call it “conscious bullshit”) or not (“clueless bullshit”).

Leave a Reply

Your email address will not be published. Required fields are marked *