Is the effect they found too large to believe? (the effect of breakfast macronutrients on social decisions)

Someone who wishes to remain anonymous writes:

Have you seen this paper?

I [my correspondent] don’t see any obvious problems, but the results fall into the typical social psychology case “unbelievably large effects of small manipulations”. They even say so themselves:

We provided converging evidence from two studies showing that a relatively small variation in breakfast’s macronutrient composition has a striking impact on social decisions.

The article in question is “Impact of nutrition on social decision making,” by Sabrina Strang, Christina Hoeber, Olaf Uhl, Berthold Koletzko, Thomas F. Münte, Hendrik Lehnert, Raymond Dolang, Sebastian Schmid, and Soyoung Park. From the abstract:

Breakfasts with a high-carbohydrate/protein ratio increased social punishment behavior in response to norm violations compared with that in response to a low carbohydrate/ protein meal. We show that these macronutrient-induced behavioral changes in social decision making are causally related to a lowering of plasma tyrosine levels.

And here’s their evidence:

I’m concerned about implausible effect size estimates, which is what you can get from the combination of noisy data, small samples, and forking paths in analysis.

The graph on the left is from experiment 1 which is observational data (not assigning breakfasts but just asking people what they ate), but still:

Within the low-carb/ protein group, 24% of subjects decided to reject unfair offers. In contrast, 53% of the high-carb/protein group decided to reject unfair offers.

I don’t care if it is p=0.03, I don’t expect to see this in a replication.

There’s a lot more data, and there could be something going on—I have no idea. I think they should do a Nosek, Spies, and Motyl and replicate the whole thing from scratch. Or someone else can do the replication.

Until then, I’m skeptical of these claims:

The findings indicate that, in a limited sense, “we are what we eat” and provide a perspective on a nutrition-driven modulation of cognition. The findings have implications for education, economics, and public policy, and emphasize that the importance of a balanced diet may extend beyond the mere physical benefits of adequate nutrition.

Or this:

In this study, we demonstrated that the macronutrient composition of food acutely influences our social decisions, showing a modulation in the dopamine precursor as the underlying mechanism.

Exploratory experimentation and analysis are fine—that’s what science is all about. Let’s just not forget that finding some statistical significant comparisons in data is not the same thing as scientifically “demonstrating” a hypothesis. Their hypothesis could well be true, or maybe not, or maybe it depends on context. Nothing special about this particular study, we just need to give such studies a modern reading.

7 thoughts on “Is the effect they found too large to believe? (the effect of breakfast macronutrients on social decisions)

  1. If I’m guessing correctly, “UG” stands for “ultimatum game”, which Wikipedia describes as “a game that has become a popular instrument of economic experiments.” I’m very skeptical that an experiment based on this game could result in findings that “have implications for education, economics, and public policy”. This sounds like making a mountain out of a molehill. (Hey — this has a great acronym: MAMOOAMH.)

    • The Ultimatum Game is just a very simple experiment designed to more carefully isolate simple social preferences from reciprocity than historically earlier games did. The even simpler Dictator Game (https://en.wikipedia.org/wiki/Dictator_game) takes this still further.

      Whether you think methdologically sound estimates from experimental trials run on these kinds of games could have “have implications for education, economics, and public policy” should mostly depend on whether you think the notion of context-independent social preferences is meaningful, I think.

    • MAMOOAMH indeed! I’d be suspicious that behavior in an ultimatum game says much about behavior in education, economics, and public policy even if the subjects had been randomly assigned. But in this case, it is observational, which makes causation virtually impossible. I’m sure that my behavior is affected by the doughnut I eat every morning but I’m even more sure that my daily doughnut reveals things about me that make me different from people that have healthier breakfasts (makes me wonder, is it the doughnut or the hole, to raise a mixed metaphor).

  2. “the findings have implications for education, economics, and public policy”

    they mean the findings have implications for troughers who make a living extracting money from tax payers.

Leave a Reply to Terry Cancel reply

Your email address will not be published. Required fields are marked *