Did that “bottomless soup bowl” experiment ever happen?

I’m trying to figure out if Brian “Pizzagate” Wansink’s famous “bottomless soup bowl” experiment really happened.

Way back when, everybody thought the experiment was real. After all, it was described in a peer-reviewed journal article.

Here’s my friend Seth Roberts in 2006:

An experiment in which people eat soup from a bottomless bowl? Classic! Or mythological: American Sisyphus. It really happened.

And here’s econ professor Richard Thaler and law professor Cass Sunstein in 2008:

Given that they described this experiment as a “masterpiece,” I assume they thought it was real.

Evidence that the experiment never happened

We’ve known for awhile that some of the numbers in the Wansink et al. “Bottomless bowls” article were fabricated, or altered, or mis-typed, or mis-described, or something. Here’s James Heathers with lots of details.

But I’d just assumed there really had been such an experiment . . . until I encountered two recent blog comments by Jim and Mary expressing skepticism:

For me, for sure, if I got a 6oz soup bowl that refilled itself without me knowing I’d just go right on eating gallon after gallon of soup, never noticing. . . . There’s no way he even did that! That has to be a complete fabrication.

If you try to imagine designing the refilling soup bowl, it gets harder and harder the more you think about it. The soup has to be entering the bowl at exactly the right rate. . . . I don’t think they really did this experiment. They got as far as making the bowls and stuff, but then it was too hard to get it to work, and they gave up. This would explain why an experimental design with 2 bottomless and 2 non-bottomless subjects per table ended up with 23 controls and 31 manipulations . . .

I searched the internet and found a photo of the refilling soup bowl. Go to 2:36 at this video.

See also this video with actors (Cornell students, perhaps?) which purports to demonstrate how the bowl could be set up in a restaurant. The video is obviously fake so it doesn’t give me any sense of how they could’ve done it in real life.

I also found this video where Wansink demonstrates the refilling bowl. But this bowl, unlike the one in the previous demonstration, is attached to the table so I don’t see how it could ever be delivered to someone sitting at a restaurant.

So when you look at it that way: an absurdly complicated apparatus, videos that purport to be reconstructions but which lack plausibility, and no evidence of any real data . . . If seems that the whole thing could be a fake, that there was no experiment after all. Maybe they built the damn thing, tried it out on some real students, it didn’t work, and then they made up some summary statistics to put in the article. Or they did the experiment in some other way—for example, just giving some people more soup than others, with the experimentalists rationalizing it to themselves that this was essentially equivalent to that bottomless-bowl apparatus—and then fudged the data at the end to get statistically significant and publishable results.

Or maybe it all happened as described, and someone just mistyped a bunch of numbers which is why the values in the published paper didn’t add up.

To paraphrase Jordan Anaya: I dunno. If I’d just designed and carried out the most awesome experiment of my career—a design that some might call a “masterpiece”—I think I’d be pretty damn careful with the data that resulted. I’d’ve made something like 50 copies of the dataset to make sure it never got lost, and I’d triple-check all my analyses to make sure I didn’t make any mistakes. I might even bring in two trusted coauthors just to be 100% sure that there were no missteps. I wouldn’t want to ruin this masterpiece.

It’s as if Wansink had found some rare and expensive crystal goblet and then threw it in the back of a pickup truck to bring it home. A complete disconnect between the huge effort required to purportedly collect the data, and the zero or negative effort expended on making sure the data didn’t get garbled or destroyed.

Evidence that the experiment did happen

On the other hand . . .

Perhaps the strongest argument in favor of the experiment being real is that there were three authors on that published paper. So if the whole thing was mde up, it wouldn’t be just Brian Wansink doing the lying, it would also be James Painter and Jill North. That moves our speculation into the conspiracy category.

That said, we don’t know how the project was conducted. It might be that Wansink took responsibility for the data collection, and Painter and North were involved before and after and just took Wansink’s word for it that the experiment was actually done. Or maybe there is some other possibility.

Another piece of evidence in favor of the experiment being real is that Wansink and his colleagues put a lot of effort into explaining how the bowl worked. There are three paragraphs in Wansink et al. (2005) describing how they constructed the apparatus, how it worked, and how they operated it. Wansink also devotes a few pages of his book, Mindless Eating, to the soup experiment, providing further details; for example:

Our bottomless bowls failed to function during the first practice trial. The chicken noodle soup we were using either clogged the tubes or caused the soup to gurgle strangely. We bought 360 quarts of Campbells tomato soup, and started over.

I’m kinda surprised they ever thought the refilling bowl would work with chicken noodle soup—isn’t it obvious that it would clog the tube or clump in some way?—but, hey, dude’s a b-school professor, not a physicist, I guess we should cut him some slack.

Scrolling through the Mindless Eating on Amazon, I also came across this:

It seems that when estimating almost anything—such as weight, height, brightness, loudness, sweetness, and so on—we consistently underestimate things as they get larger. For instance, we’ll be fairly accurate at estimating the weight of a 2-pound rock but will grossly underestimate the weight of an 80-pound rock. . . .

They’re having people lift 80-pound rocks? That’s pretty heavy! I wonder what the experimental protocol for that is. (I guess they could ask people to estimate the weight of the rock by just looking at it, but that would be tough for lots of reasons.)

But I digress. To return to the soup experiment, Wansink also provides this story about one of the few people who had to be excluded from the data:

Cool story, huh? Not quite consistent with the published paper, which simply said that 54 participants were recruited for the study, but at least some recognition that moving the soup bowl could create a problem.

Summary

Did the experiment ever happen? I just don’t know! I see good arguments on both sides.

I can tell you one thing, though. Whether or not Wansink’s apparatus ever made its way out of the lab, it seems that the “bottomless soup bowl” has been used in at least one real experiment. I found this paper from 2012, Episodic Memory and Appetite Regulation in Humans, by Jeffrey Brunstrom et al., which explains:

Soup was added or removed from a transparent soup bowl using a peristaltic pump (see Figure 1). The soup bowl was presented in front of the volunteers and it was fixed to a table. A tall screen was positioned at the back of the table. This separated the participant from both the experimenter and a second table, supporting the pump and a soup reservoir. Throughout the experiment, the volunteers were unable to see beyond the screen.

The bottom of the soup bowl was connected to a length of temperature-insulated food-grade tubing. This connection was hidden from the participants using a tablecloth. The tubing fed through a hole in the table (immediately under the bowl) and connected to the pump and then to a reservoir of soup via a hole in the screen. The experimenter was able to manipulate the direction and rate of flow using an adjustable motor controller that was attached to the pump. The pre-heated soup was ‘creamed tomato soup’ (supplied by Sainsbury’s Supermarkets Ltd., London; 38 kcal/100 g).

And:

Participants were then taken to a testing booth where a bowl of soup was waiting. They were instructed to avoid touching the bowl and to eat until the volume of soup remaining matched a line on the side of the bowl. The line ensured that eating terminated with 100 ml of soup remaining, thereby obscuring the bottom of the bowl.

So it does seem like the bottomless soup bowl experiment is possible, if done carefully. The above-linked article by Brunstrum et al. seems completely real. If it’s a fake, it’s fooled me! If it’s real, and Wansink et al. (2005) was fake, then this is a fascinating case of a real-life replication of a nonexistent study. Kind of like if someone were to breed a unicorn.

42 thoughts on “Did that “bottomless soup bowl” experiment ever happen?

  1. I also imagine that to meet health and safety regulations (or just to satisfy your garden variety overzealous IRB), the tubes and pumps would need to be emptied and cleaned after each use. So (if the experiment took place) either Wansink had multiple pumps to move the experiment along at a reasonable pace (seems like a big hassle) or multiple participants’ spoons were in the same soup distribution network. And wouldn’t the participants hear the pump (or even if the pump was quiet, I’m sure whatever actuator that switched the pump on would be heard) or bang their knees against those hanging tubes. I would guess the experiment took place (just not the way it is described in the paper) but the data was (at best) massaged.

  2. An experimental procedure ought to be described in sufficient detail that a motivated expert could reproduce it with adequate resources. Does it matter whether it occured if what exactly was done cannot be reconstructed?

    • Dan:

      The whole thing’s a joke and I don’t believe any of Wansink’s conclusions. None of this matters. It’s just an object of curiosity. I’m curious whether Wansink and his colleagues just made everything up, or did they actually collect data as described and then make up the summary statistics, or did something else happen? The answer to this question does not matter for science—it’s just something I’m wondering.

      • I understand your curiosity – the question might be phrased as how much of a scoundrel is the guy? or, how sad is it that a famous university employs such a person? However, there is a general issue made apparent by your question that perhaps deserves more attention. One of the many things that makes difficult deciding whether statistics have been done properly in papers is that, even in serious papers by apparently reputable people, the procedures/experiments/data are often inadequately explained, so that one cannot judge what is being evaluated or how reliable it is. For the reader there is no effective operational difference between an experiment that was never performed and one that is not explained in any detail.

      • For the record, based on what is in the papers, it seems difficult to believe that the experiments were performed, at least as the headlines lead one to believe.

  3. My guess is that most probably (a) the apparatus was indeed built, (b) the experiment was indeed planned, (c) people indeed came into the lab (in groups of 4, with 2 being bottomless and 2 not), and then… the rubber met the road. Participants got bored of eating soup after 10 minutes. many will have guessed what was going on. Some probably discovered the mechanism (being told “Don’t touch the bowl” would have caused suspicion right off the bat) So the lab was left with a lot of investment of time and effort to test a theory that makes soooo much sense that it must be true… and you can fill in the rest.

    I suspect that something like this happens more often than we would like to imagine in other branches of experimental psychology too.

    • Nick:

      That all seems plausible, but then, given the design of the experiment it’s funny that they didn’t bother to give fake results where N_treatment = N_control. Maybe they thought that equal numbers would look suspicious so they made up unequal N’s which would seem more realistic.

      Also there’s the question, if there was lying, of how many of the three authors were in on the deception.

  4. I am surprised that none of these participants balked at the taste of Campbell’s tomato soup. I recently tried it after three kids basically refused to eat more than a spoon of it and found it just as unpalatable as they did.

        • If there are millions of people doing something and reproducing it is hard to call them “not in their right mind”. More likely they are working based on different information or just have different priorities than you.

          I guess the modern equivalent would be using these various tech sites. I don’t understand why anyone would use facebook, twitter, or google search at this point. That doesn’t mean they are “out of their minds” though. I know people who still somehow do not use an adblocker online! Does making such a poor decision mean they are crazy or mentally unfit?

        • It sounds like he loved Campbell’s soup:

          When the art critic G. R. Swenson asked Warhol in 1963 why he painted soup cans, the artist replied, “I used to drink it, I used to have the same lunch every day, for twenty years.”[30][38]

          […]

          Several stories mention that Warhol’s choice of soup cans reflected his own avid devotion to Campbell’s soup as a consumer. Robert Indiana once said: “I knew Andy very well. The reason he painted soup cans is that he liked soup.”[40] He was thought to have focused on them because they composed a daily dietary staple.[41] Others observed that Warhol merely painted things he held close at heart. He enjoyed eating Campbell’s soup, had a taste for Coca-Cola, loved money, and admired movie stars. Thus, they all became subjects of his work. Yet another account says that his daily lunches in his studio consisted of Campbell’s Soup and Coca-Cola, and thus, his inspiration came from seeing the empty cans and bottles accumulate on his desk.[42]

          https://en.m.wikipedia.org/wiki/Campbell%27s_Soup_Cans

        • Back in the day, I used to sneer along with Pete Seeger at the “boxes on the hillside all made of ticky-tacky”. Then I saw pictures of how people lived in the Depression and now I cringe at my sneering. It is one of the reasons I am no longer a leftist. Now I think of Pete Seeger as an evil Stalinist.

          Here is a Zillow page showing the tacky boxes on the hillside today. Rather nice for a family of limited means compared to the tar-paper shacks of the Depression.

          https://www.zillow.com/homes/Levittown-NY_rb/

    • That could very well be the same Jill North, given the Illinois connection (the BB study was run at UIUC and submitted to the journal in December 2003, some time before Wansink moved to Cornell).

  5. Quick thoughts:

    1) I stir my soup. It helps keep heat and consistency even. Some soups are liquidy, others have things in the soup. I tend to think my spoon would notice an anomaly at the bottom of the bowl.

    2) Is original video available? I’ve seen what I believe to be original video of experiments like the Milgram experiment and Stanford Prison experiment and others.

    3) Free food at events. I probably do eat more of whatever (pizza, fruit, etc.) when it is free, there is plenty of it, and it is a one-off event.

  6. My wife went to Cornell. The community is not that big. If this really happened,
    we should be able to find some of those who participated. Let’s get David Brooks on it!

  7. Must be a slow day. Wansink has already been discredited, however beating a dead horse must not be enough. The whole post IS conspiracy thinking at best, inviting one commentator to muse about the possibility of such shams occurring in “experimental psychology”. Wansink was not a trained psychologist, and generalizing his behavior to “psychology” strikes me as poor thinking.

  8. I’m still going with total fake.

    1) There’s just no way you would not notice a six ounce soup bowl refilling
    2) In a real restaurant setting the bowl is brought to the table – not sitting full on the table when you arrive
    3) the mechanics of filling would be either noticeable or impossible for almost anything but tomato soup.
    4) No one ever got to the bottom of the bowl to see the refilling mechanism? Masterful refilling technique! Years of practice!
    5) Wansink’s talk has no technical information whatsoever
    6) his stories are totally contrived
    7) he offers no photographic evidence of anything, just the one shot of a really crappy set up that’s being built

  9. “In a real restaurant setting the bowl is brought to the table”

    But as far as I know, Wansink never said that the soup in his experiment was brought to the table. Indeed, the anecdote of one of the excluded participants drinking the soup like a Viking would say otherwise.

    Furthermore, it’s not as if the bottomless soup bowl is even remotely impossible. Tricky to engineer, perhaps, but it probably wouldn’t be that hard to find someone at Cornell who had the relevant expertise. Judging from one of the YouTube videos, it looks like the bowls are fed through the siphon effect, so it wouldn’t be *that* difficult.

    Furthermore, it requires far more of a guilty mind to fabricate an experiment than it does to p-hack or mess up an analysis. The latter two really only require sloppiness and self-deception, both of which are more plentiful than malice. Bear in mind that Wansink was caught because he bragged about a graduate student being willing to torture the data from a failed study until it confessed, as if he didn’t realize what a problematic statement he was making: https://web.archive.org/web/20170130155126/http://www.brianwansink.com/phd-advice/the-grad-student-who-never-said-no That suggests that he doesn’t have the guile to do an outright hoax.

    • “Wansink never said that the soup in his experiment was brought to the table”
      Exactly the point.

      “Tricky to engineer”
      Yes and the kind of thing that the people who engineer it like to get a nice little paper out of it with credit for their work.

      “Furthermore, it requires far more of a guilty mind to fabricate”
      I don’t know about his mind or anyone else’s. What I see is that there isn’t any evidence there ever was an experiment. Just data.

    • Jj:

      Just to be clear, Wansink did not merely “p-hack or mess up an analysis.” Based on the reanalyses by James Heathers, Tim van der Zee, Nick Brown, etc., of his published papers, it’s clear that, in many different papers Wansink included numbers that could not have correspond to any actual data. You can’t get that just by p-hacking or messing up. At the very least, there has to be have massive misreporting going on regarding data and experimental methods.

  10. About the Brunstrom apparatus, Figure 1 above.
    Why not set the reservoir above the level of the bowl? Then no pump would be needed, and soup flow could be regulated by a simple release thingamajig, like they have on iv lines in the ICU. Gravity is effective, and silent.

  11. Such an experiment would require an answer to IRB regarding the deception employed.

    The methodology would also be described in IRB paperwork.

    I, for one, seriously doubt these procedures were followed.

    I, too, think the ‘experiment’ was never conducted.

    • I don’t have access (free) to the paper, but I am not surprised if people eat more soup if more is made available to them – unless they have to pay according to how much they eat. I do wonder about the 73% that did not feel like they ate more soup – exactly how is that measured? Are they asked for a quantity of soup that they consumed? Measured in ounces, cups, or what? Replication is great, but I just have all sorts of questions about the design of this experiment (notwithstanding those old concerns about whether the self-filling soup bowl ever existed).

Leave a Reply

Your email address will not be published. Required fields are marked *