Concerns about Brian Wansink’s claims and research methods have been known for years

1. The king and his memory

There’s this stunning passage near the end of Josephine Tey’s classic The Daughter of Time. Most of the book is taken up with the main characters laboriously discovering the evidence that Richard III was not really a bad guy, he didn’t really kill those little princes, etc. Having made this discovery, the characters are planning to write a book and break the big news to the world.

But, then, some hours later:

And Brent came in.
But it was not the Brent who had last gone out.
Gone was the jubilation. Gone was his newly acquired breadth.
He was no longer Carradine the pioneer, the blazer of trails.
He was just a thin boy in a very long, very large overcoat. He looked young, and shocked, and bereaved.
Grant watched him in dismay as he crossed the room with his listless unco-ordinated walk. There was no bundle of paper sticking out of his mail-sack of a pocket today. . . .
‘What is it? Have you discovered that there was a general rumour about the boys before Richard’s death, after all?’
‘Oh, much worse than that.’
‘Oh. Something in print? A letter?’
‘No, it isn’t that sort of thing at all. It’s something much worse. Something quite—quite fundamental. I don’t know how to tell you.’ He glowered at the quarrelling sparrows. ‘These damned birds. I’ll never write that book now, Mr Grant.’
‘Why not, Brent?’
‘Because it isn’t news to anyone. Everyone has known all about those things all along.’
‘Known? About what?’
‘About Richard not having killed the boys at all, and all that.’
‘They’ve known? Since when!’
‘Oh, hundreds and hundreds of years.’
‘Pull yourself together, chum. It’s only four hundred years altogether since the thing happened.’
‘I know. But it doesn’t make any difference. People have known about Richard’s not doing it for hundreds and hundreds—’
‘Will you stop that keening and talk sense. When did this—this rehabilitation first begin?’
‘Begin? Oh, at the first available moment.’
‘When was that?’
‘As soon as the Tudors were gone and it was safe to talk.’
‘In Stuart times, you mean?’
‘Yes, I suppose—yes. A man Buck wrote a vindication in the seventeenth century. And Horace Walpole in the eighteenth. And someone called Markham in the nineteenth.’ . . .

Yup, it was known all along. But nobody wanted to hear it. Nobody wanted to spoil the pretty story.

OK, I don’t know anything about Richard III. And The Daughter of Time is fiction. So I’m expressing no opinion on what happened to the princes in the tower. Rather, I’m drawing attention to the compelling idea that an error in historical memory can remain in plain sight, that certain mistakes seem to need to be rediscovered to be believed.

I’ve remembered the above scene (not the exact words, but the general idea) ever since I read that book, nearly thirty years ago.

2. The discredited researcher

During the past year we’ve been hearing a lot about the misdeeds of Brian Wansink, the Cornell University business school professor who’s received contracts with major corporations, millions in government research grants, and tons of largely uncritical media coverage over the past decade or so based on his “nudge”-style research on eating behavior. It appears that that Wansink repeatedly was manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record. (I use that particular phrasing because it aligns with the NIH’s definition of research misconduct, from a webpage that was linked to by the Cornell Media Relations Office.)

The main discoveries of misconduct occurred in 2016 and 2017, but Wansink, for people in his lab, have a track record of ducking criticism, acting like they take it seriously but then continuing with sloppy scientific procedures; here’s a story that goes back to 2012.

The ratty sweater that is Wansink’s research record continues to unravel. Here for example is a post by Bettina Elias Siegel, “New Study Casts More Doubt on ‘Smarter Lunchrooms’ Data,” reporting on a debunking by Eric Robinson on some of the most influential work from that lab.

What I want to point to here, though, is something different. Linked from Siegel’s blog is this post of hers from 2014, “Moms, ‘Food Fears’ and the Power of the Internet.” Key quotes:

Dr. Brian Wansink, a professor of consumer behavior at Cornell University and director of the Cornell Food and Brand Lab, has published a new study in the journal Food Quality and Preference entitled “Ingredient-Based Food Fears and Avoidance: Antecedents and Antidotes.” This study, co-authored by Aner Tal and Adam Brumberg, seeks to determine why people – mothers in particular — develop so-called “food fears” about certain ingredients (such as sodium, fat, sugar, high fructose corn syrup, MSG and lean finely textured beef) and what the food industry and government can do about it.

The study’s ultimate conclusion, that “food fears” can be addressed by “providing information regarding an ingredient’s history or the other products in which it is used,” is hardly controversial. But some other things about this study raise red flags . . . in Wansink’s own YouTube video created to promote the study, he tells us that people with “really bad ingredient food fears have three things in common:”

First of all, they tend to hate the foods the product’s in, almost more than the [unintelligible] ingredient itself, meaning they tend to hate potato chips or candy or soft drinks almost more than the ingredients themselves.

Second of all, they get most of their information . . . from the Internet, they look at their favorite websites, they don’t get it from mainstream media and they certainly don’t get it from health care professionals.

The third thing they have in common is that they are much more likely to need social approval.

The problem is, Wansink’s study simply does not support these characterizations of individuals who get their food information from the Internet, and Wansink’s own recap of his study is in some ways as grossly inaccurate as the media reports I cite. Here’s why. . . .

The Study Did Not Address Social Media At All

In a survey of 1,008 women who had two or more children, the question posed to respondents about where they obtained information about food ingredients did not include the words “Facebook,” “Twitter,” “newsfeed” or even the more general term “social media.” Instead, respondents were simply asked if they obtained such information from “Internet/Online,” an incredibly broad descriptor which could include anything from the sketchiest of blogs to the website of the Institute of Medicine.

So, in fact, the study has nothing at all to say about the role of Facebook, Twitter or other social media per se in stoking “food fears.”

The Study Failed to Distinguish Between Types of Online Media

Wansink also contrasts what he sees as the largely biased Internet with more trustworthy “mainstream media,” but without acknowledging that almost every local and national news outlet operating in traditional media now also has its own website. (In fact, where did I find Wansink’s own study, the news coverage about it, and every other citation in this post? Online, of course.) . . .

So when Wansink says in his video that people with “food fears” “look at their favorite websites, they don’t get [food news] from mainstream media,” he has no basis at all on which to make this key distinction. . . .

Conclusions About Sharing “Food Fears” on Social Media Are Entirely Unsupported

As noted, Today reports that moms with “food fears” “feel strongly about sharing these opinions on social media or their own blogs,” and Wansink notes that “they have a higher need to tell other people about their opinion.” In his video, he says such people “are much more likely to need social approval.” (Emphasis mine.)

But while the study did find that “some individuals who avoid ingredients may have a greater need for social approval among their reference group than those with a more moderate view,” the study’s authors were forced to admit that “such effects were small in our sample.” So Wansink’s “much more likely” characterization is patently false.

And even if this finding were significant, the supposed need for social approval was not measured by respondents’ use of social media or blogs. Rather, it was measured using a standard “social desirability” assessment tool that has nothing to do with social media, and also by asking respondents if they agreed with two statements (“It is important to me that my friends know that I buy Organic Foods and Beverages” and “It is important to me that my friends know that I buy Natural Foods and Beverages”), neither of which have anything to do with “food fears” or social media. . . .

Those With “Food Fears” Are Not “Haters” of Junk Food

Wansink tells us in his YouTube video that those with “food fears” actually “hate” the product in which the feared ingredient is found more than the ingredient itself. Specifically, he tells us, the study found that they “tend to hate potato chips or candy or soft drinks almost more than the ingredients themselves.”

Now here’s what Wansink’s study actually found. Participants were asked to rate the healthfulness of four foods (yogurt, granola, pre-sweetened cereal and cookies). Some participants were then told that these four products contained HFCS and among that subset, the “healthy” rating went down for yogurt, granola and pre-sweetened cereal, but not for cookies (presumably because cookies are not thought to be healthful in the first place.)

And that’s it. Not a word in the study about “potato chips, candy or soft drinks.” Not a word about “hating.” But Wansink apparently likes this fictional finding so much he mentions it in his video not once, but twice.

People keep rediscovering that Wansink does not tell the truth about his experimental procedures, or his data, or his results. But, even all these years later, his claims are driving policy. Here’s Bettina Elias Siegel from 2017:

To date, the U.S. Department of Agriculture has spent $8.4 million to directly fund the Smarter Lunchrooms research and implementation, and another $10 million in grants to help schools put the Smarter Lunchroom principles into practice. The agency also now requires schools to implement some Smarter Lunchroom techniques in order to qualify as “HealtherUS Schools,” and the Obama-era federal rule on local wellness policies specifically informs school districts that

at a minimum, FNS [the USDA’s Food and Nutrition Service] expects [districts] to review “Smarter Lunchroom” tools and strategies, which are evidence-based, simple, low-cost or no-cost changes that are shown to improve student participation in the school meals program while encouraging consumption of more whole grains, fruits, vegetables, and legumes, and decreasing plate waste.

At the very least, perhaps the president of Cornell University could call up the USDA and apologize for all these wasted tax dollars, not to mention all the kids who had to put up with all this b.s. in their lunchrooms for so many years.

What’s striking about this story is that the relevant information has been in the public record all along. I guess that’s one reason Wansink thought he could keep doing it: he’d got away with it for years and nobody cared, so why expect anything to change. You think they’ll ever take him off the Ted webpage? I wonder what it takes to get de-listed by Ted.

P.S. Siegel did not start out as a Wansink skeptic; here’s an interview from 2011 where she fell for his whole shtick (and where Wansink, ironically in retrospect, is labeled as a “Master of Lunchroom Trickery”). The interview’s good: Wansink’s a good talker. I guess it’s easy to be persuasive, at least in the short term, if you’re not constrained by the truth.

P.P.S. In the months since I wrote the above post, lots more has appeared on Wansink; see for example here, here, here, and here.

That last link is particularly amazing—it’s the story of a $10,000 Kickstarter campaign “guaranteed to help users lose 2+ lbs each month,” which never got started:

Those who gave at least $30 were supposed to receive three months’ worth of the service, which would start right after the campaign ended in September 2014, according to the Kickstarter page. . . .

A few weeks after the campaign met its goal, Wansink emailed three dozen contributors . . . “We anticipate this will go live in January 2015 . . . Really exciting results so far, now we’re smoothing out the interface.” . . .

In February 2015, Star Li, who was listed as the campaign’s manager, wrote in response to Ruth, the donor in Israel who had kicked in $30: “This monthly method plan should be available within the next month or two. Our apologies for the delay.”

When Ruth later tried to follow up again, Wansink wrote in January 2017, “We’re still working on this.” . . .

In his [February 2018] email to donors, Wansink . . . said his team had worked with programmers for a year to create the service, but it didn’t materialize by the time he left for sabbatical in 2015. It was then that he decided to put it “on the back-burner” and develop a substitute service on a website and app. These were launched in beta-testing form, but have since been taken down for adjustments, he wrote.

It really is a lot easier to say you’re gonna do something, or to even say you’ve done something, than to actually do it.

Again, what’s stunning about this whole story is that so many of the problems were hiding in plain sight, just sitting out there for all to see, who chose to look. The strategy of affable denial worked so well for so long.

32 thoughts on “Concerns about Brian Wansink’s claims and research methods have been known for years

  1. I’d prefer to leave this after a less intemperate comment, but I take what’s on offer. Andrew, sometimes your thought processes get in the way of accepting the statistical meaning of the situation. Your entire objection in these experiments are an archetype of a kind of inquiry that is not statistically valid, where validity is variable across the domain of methods – i.e., so it might not be designed, implemented, collected, analyzed, presented – and that expresses as a replication measure. You’ve actually been assigning a value to the variables within the graphable finding of replicable. You even organize the summations of the methods in a multi-level manner. The answer is that as statistics has developed in methods it has become a manner by which you can rank replicability. Instead of p-statistics and other variations that say ‘the results of this test are potentially true’, you shift it to ‘the results of this test are potentially replicable’. This particular answer best fits the reality of research: there is work which is replicable and work which is not, but some non-replicable work is considered investigatory; it’s in a direction the researchers posit has potential, much like digging for a new vein of ore. It answers the question by shifting the measure to an abstracted level which takes into consideration the methods by which research fails to be statistical.

    You have to admit, that kind of backwards phrasing lies at the heart of the original statistical approach, that is by abstracting the existing methods of farming you can ascribe a number to the various combinations of methods, and that number relatively evaluates the success or failure of each combination. What’s generally not realized is the extent to which data points represent the shearing away of other possible outcomes. To me, the best way to visualize that is to plot them across a stacked matrix set up as a lattice with each regressable point assigned a depth within the lattice. It’s easy to idealize this to a cube or sphere, but it’s also useful to think of this in terms of surfaces, so the points can be seen as away or toward. If you rotate the lattice, you’ll see the band of rotations where the image is relatively stable. That gives you a depth stability factor: you can treat it as a literal visibility test for when this or that issue becomes visible in your model. Can you imagine how hard this would have been to do a few years ago? It’s like when you pull up the paper on how to bootstrap: reads like something from the dark ages.

    Isn’t it time we started treating this stuff like the imagery it is? Think nuclear medicine. I remember when my dad would inject contrast dye into the femoral artery – and it clearly hurt because in those days you didn’t put people under nearly as often because the risk of anaesthesia was material. I remember when a liver scan was the equivalent of a Geiger counter moving back and forth, mechanically tied to a marker that made a dot with every click, making a harder dot that kind of matched amplitude so a regular scan would have some spacing of light and dark areas. A big enough tumor would show as a blob. Now it’s full color, fully dimensional, rotatable imagery. Regression actually draws a picture in a lattice, same as a liver. We have all the tools necessary to assign values that render these ‘points’ relatively across a lattice.

    Think of the applications: wavefront analysis that gives you estimations of course, the Star Trek warp drive realized. Think about that: how would they know where they were going by plotting in a course and then engaging the ‘warp drive’? The science fiction idea behind that is reading the warp of space-time so they fit to the shortest allowable course. I used to wonder how far the warp drive could read. That was implied by the fact they kept coming up on unexplored space and because there was a sense of distance – something the movies have completely blown in their disrespect for science fiction as principled action. In actuality, of course, it’s far more useful in the sense of radar now, though eventually it becomes a way of unlocking pathways.

    I have to go to work now.

    • Sorry, but if someone can translate that into English and a concise comment, I’d appreciate it. I’m getting too (old, impatient, something….) to make sense out of it. Andrew’s post was a bit rambling (in my eyes), but did not get in the way of “accepting the statistical meaning of the situation.” The face that poor practices have been in view for a long time and that denial has been effective for so long has little “statistical meaning” in my eyes. It is part and parcel of poor academic and scientific practice and says more about the sociology and politics of these practices than anything statistical, per se. But perhaps I have just totally misunderstood the post. I have certainly misunderstood the comment.

      • Dale:

        That’s right, my post is not about statistics at all, any more than a post on Lance Armstrong’s legal struggles is about bicycle racing, or, to take much more extreme examples, posts on Michigan State or Penn States would be about gymnastics or football. In all these cases, the topic is the appalling willingness of people to cover up wrongdoing, along with the appalling willingness of many many others to look away.

        As a social scientist, the pizzagate story interests me in large part because research such as Wansink’s represents much of the public face of social science; see for example here. And as a statistician, the story interests me in large part because it is through statistics that Wansink’s claims get credibility as science, rather than just folk wisdom or one man’s opinion.

      • +1

        Jonathan’s credibility: -1.

        OP was pretty easy to understand: a lie gets half-way around the world by the time the truth gets its boots on.

        Jonathan’s comment may have had some sort of insight about non-replicability not being identical to refutation (or something) buried deep, deep inside the verbiage, but it’s not my job to make someone else’s post clear.

  2. What about all the tax payers money going to “climate change” propaganda. Their methods/behavior are worse than this, I don’t see any of you complaining.

    • Jack:

      We post things on this blog that demonstrate various issues of statistics and social science, and we also post on things that happen to pique our interest. We post on literature, art, sports, and other topics of little worldly importance. There are tons of horrible things going on in the world that we don’t post on at all. All over the world, there’s rape and murder, there’s corruption of all sorts. So please don’t think that, when we post on topic X, that we mean to imply that there aren’t lots of things worse than X going on right now.

      We have occasionally posted on climate change—I’ve even done some research on the topic. I’m not quite sure what is the propaganda that you’re referring to, but I suppose that one person’s information is another’s propaganda. But, again, the enumeration in this blog, of certain issues, shall not be construed to deny or disparage others retained by the people.

      • i’m just pointing out because most people that are eager to criticize Wansink acts in a completely opposite way about other bogus research. Climate change papers are the best ones to point this out because of all the media attention, and how everyone who does not understand the ridiculousness of the papers acts as if just because it was published it is “science”.

        On a different note, is Susan David the new Amy Cuddy: http://www.susandavid.com ?

        It seems our society, specially Harvard, is tuned to produce BS? Most of “science” seems to serve to provide an “evidence based” discourse.

        • Jack:

          It does seem there are positive incentives to produce B.S. in many fields of research. But I don’t know what you’re talking about regarding climate change papers. I’ve written on the topic, and I don’t think my papers are ridiculous. I’ve never tried to make a systematic study of the quality of research in any given field, and on this blog we tend to consider examples one and a time. A couple years ago, I posted on some low-quality research on the economics of climate change.

          Regarding Susan David, I’ve never heard of her before. Just speaking in general terms, I have no objection to inspirational books and lectures, nor am I bothered by recommendations that are based on personal experiences and anecdotes. Every day we have important decisions to make in our lives; rigorous research covers only a small part of the things we care about, so in the meantime we need to make decisions based on whatever information is available. If someone has some ideas of how we can live our lives better, and can present these ideas persuasively, that’s great. One of my favorite books ever is How to Talk so Kids will Listen and Listen so Kids will Talk, by Adele Faber and Elaine Mazlish. I have no idea if their recommendations are backed by careful research; if so, great, but if not, I still feel that I’ve still got a lot from reading that book. My objections, as expressed on this blog, and elsewhere, to some promoters of some psychology research, arose not from their popular Ted talks but from their claims of rigorous scientific support for claims for which there was no good evidence, or in some cases no evidence at all.

        • If you watch her TED talks she is always saying: “my research found…”.

          I agree not every book has to be scientific. Actually, I despise vulgar scientism and I belief the bulk of human knowledge does not come from scientists, but engineers. My red flag here is that it seems she’s doing the same thing Amy did, travestying self-help as science. Some people should look into those papers that back what she is saying.

      • Interesting paper about tree-ring data (A Model-Based Approach to Climate Reconstruction
        Using Tree-Ring Data). It looks like a significant addition to the literature.

        A big complaint about climate science has been that it needs more rigorous statistical analysis, and I have long wondered why competent statisticians weren’t drawn to the field — it looks like such low-hanging fruit for them.

        More specifically, tree ring reconstructions have long been criticized for underestimating uncertainties, and this paper addresses one of the ignored sources of error that I had always wondered about: why are potential errors in earlier stages in the analysis dropped in later stages?

        • I haven’t read all the analyses by the statisticians involved — I am just saying that I believe these are competent statisticians. If you believe they are not competent, why do you think they are not?

    • Climate change is a difficult area to express skepticism about because there seems to be a core of scientific truth to it that goes back to Arrhenius in the 1890’s and the work (by I forget who) in the 1930’s.

      So the shenanigan’s appears to be exaggeration and tenuously extending the ideas to related areas. That is a trickier case to make. Still, the field could use more gimlet-eyed assessment by outsiders.

  3. The stench from Cornell corresponds to the recent fuss between the NCAA and schools whose athletes got paid. In both cases the administration buries its head(s) in the sand and nothing short of making a federal case out of it gets anybody’s attention. The common denominator is, of course, the usual suspect: money.

    • Torq:

      I don’t think it’s just money; I suspect it’s more about wanting to avoid bad publicity, or even just not wanting to think bad things about one’s friends and colleagues, and then just natural taking of sides. I’m guessing that people in the Cornell administration have been absolutely furious about Wansink for over a year—they’d probably love to push a button that could just make him disappear—but they might even be more furious at those of us who’ve pointed out Wansink’s errors and misrepresentations. The whole shoot-the-messenger thing.

      • > they’d probably love to push a button that could just make him disappear
        My guess (with some discussion with senior university administrators from another university) the fear is about collateral damage. What impact whatever they do might have on other faculty. In the sense of “if they can step on a Wansink they might step on me/my colleagues next”.

        • Most of the research in that area is shoddy as well. If Cornell fires him, this could trigger the lack of faith in the whole field. For society, this would be good. For the universities that manage to get money from others by fooling the public into believing this is science, this is bad. That’s why we won’t see people suffering consequences that easily, unless we relentlessly call BS science out.

      • I wouldn’t rule out administrative inattention. Cornell has been through 4 presidents and 3 provosts in the past 5-6 years. One of the presidents passed away after just a few months in office, but not before she merged the Dyson School/Ag Econ (Wansink’s tenure home) with two other units to create a new business college. This may or may not be a good decision in the long term, but in the short term it created upheaval and a leadership vacuum. There’s also been natural turnover (retirement) in the Vice Provost for Research, who oversees the offices that manage external grants and police research ethics.

        I’m not making excuses for Cornell’s inaction, by any means. My point is just that when no one is minding the store, a dishonest actor can get away with shoplifting even if the store manager isn’t on the take, too.

    • Jack,

      The problems with p-values are not just with p-values.

      And I disagree with your statement, “bad science is just due to bad scientists.” Lots of scientists are trying to do their best, and one problem is many scientists have been fooled bad statistical methods into thinking that it’s possible to learn enduring truths about human nature from sloppy experimentation and noisy data. Call it statistical significance, call it a p-value, call it a Bayes factor, call it whatever you want: if it’s a number that’s been extracted from noisy data, watch out!

      • I can’t see how this is fault of the statistical methods. It seems the authors have no clue of what they are doing and want to outsource thinking to a procedure. Unfortunately, most researchers are like that.

        • Jack:

          It’s complicated. These methods have been promoted for close to a hundred years by authority figures, have been put in textbooks from intro through graduate level, have apparently been used to great success in many fields. It doesn’t make sense to expect psychology researchers to be statistics experts—hell, even a lot of statisticians aren’t true experts in the statistical methods that they use. I do believe that by developing better methods, and by making clear the problems with various existing methods, I can make a difference in the research that people do, in part by motivating the best researchers go gather better data, and in part by reducing the incentives for the Wansinks of the world to promote weak claims that are based on bad data.

        • > have no clue of what they are doing and want to outsource thinking to a procedure
          What would better for that than statistical methods as taught to non-statisticians?

          Uncertain if effects are different by gender – no problem my statistical consultant can do a test to sort that out ;-)

  4. I also found the Kickstarter story interesting.

    First of all, what does Wansink need 10K for if he gets millions in grants? As a case in point, Wansink is listed as a super donor on his own campaign, suggesting to me that he just funded the rest of the money himself to guarantee he hit his goal.

    His emails to his donors telling them that the site will be ready in another month or so is consistent with emails I’ve seen. In fact, I’d say it might be his M.O. He seems to keep promising he’ll do something in a month, then you get back to him a month later, and he says he needs another month. It’s almost as if he’s hoping you’ll just forget about it.

    Dan Ariely gave 5K to the campaign? That’s not strange at all. Ironically, Ariely has authored a couple papers very applicable to Wansink:

    “Dishonesty in scientific research.”: https://www.ncbi.nlm.nih.gov/pubmed/26524587

    “The dark side of creativity: original thinkers can be more dishonest.”: https://www.ncbi.nlm.nih.gov/pubmed/22121888

    “Temporal view of the costs and benefits of self-deception.” https://www.ncbi.nlm.nih.gov/pubmed/21383150

    • The “Dishonesty in scientific research.” is very interesting e.g. “Thus, dishonesty is easy to fix. All one needs to do is to amp up the costs (or reduce the benefits) and make everyone aware of them.”

      However, one worries the background studies feeding into that, on the surface seem like Wansink’s.

      Have you checked out those studies at all?

Leave a Reply to Andrew Cancel reply

Your email address will not be published. Required fields are marked *