If this article portrays things accurately, the nutrition literature is in even worse shape than I thought

Forget Pizzagate. This is the stuff we really care about.

John Ioannidis writes:

Assuming the meta-analyzed evidence from cohort studies represents life span–long causal associations, for a baseline life expectancy of 80 years, eating 12 hazelnuts daily (1 oz) would prolong life by 12 years (ie, 1 year per hazelnut) [1], drinking 3 cups of coffee daily would achieve a similar gain of 12 extra years [2], and eating a single mandarin orange daily (80 g) would add 5 years of life [1]. Conversely, consuming 1 egg daily would reduce life expectancy by 6 years, and eating 2 slices of bacon (30 g) daily would shorten life by a decade, an effect worse than smoking [1].

Could these results possibly be true? Authors often use causal language when reporting the findings from these studies (eg, “optimal consumption of risk-decreasing foods results in a 56% reduction of all-cause mortality”).[1] Burden-of-disease studies and guidelines endorse these estimates. Even when authors add caveats, results are still often presented by the media as causal.

This is indeed ridiculous; it’s the piranha problem.

Indeed, it’s just as stupid as pizzagate and beauty-and-sex-ratio and ovulation-and-voting and himmicanes and all the other noise-mining social science hype we’ve been screaming about all these years—but worse, because it’s life and death we’re talking about here. More’s at stake than academic careers and who gets their research featured on NPR.

I just want to make sure of one thing, though, and that is that Ioannidis characterized this work correctly.

The above quote references two papers:

[1] Schwingshackl L, Schwedhelm C, Hoffmann G, et al. Food groups and risk of all-cause mortality: a systematic review and meta-analysis of prospective studies. Am J Clin Nutr. 2017;105(6): 1462-1473.

[2] Poole R, Kennedy OJ, Roderick P, et al. Coffee consumption and health: umbrella review of meta-analyses of multiple health outcomes. BMJ. 2017;359:j5024.

So let’s see. I’m kinda busy so I’ll just look at article [1], which is here, and I’ll focus on the hazelnuts. I don’t see anything in that paper about hazelnuts specifically, but there is this in the abstract:

With increasing intake (for each daily serving) of whole grains (RR: 0.92; 95% CI: 0.89, 0.95), vegetables (RR: 0.96; 95% CI: 0.95, 0.98), fruits (RR: 0.94; 95% CI: 0.92, 0.97), nuts (RR: 0.76; 95% CI: 0.69, 0.84), and fish (RR: 0.93; 95% CI: 0.88, 0.98), the risk of all-cause mortality decreased . . .

Later on in the paper a serving is defined as “28 g nuts/d,” and they provide some references to the original studies; I assume that hazelnuts count as nuts for this serving size.

The next question is how to map an estimated risk ratio of 0.76 to increased life expectancy. There’s gotta be some standard formula for this: take the instantaneous probability of death and integrate it over the distribution of ages and you get life expectancy; multiply these probabilities by 0.76 and you’ll get a different life expectancy; take the difference and, according to Ioannidis’s calculations, you get 12 years. Here’s an overview from David Spiegelhalter, where he goes through the calculations and finds that a 13% increase of risk, from age 40 onward, corresponds to a 1-year decrease in life expectancy. This would imply that the risk ratio of 0.76, from age 40 on, corresponds to roughly a 2-year increase in life expectancy. I guess that Ioannidis’s calculation assumes the risk ratio starting at birth, not age 40, so that will bump up the effect on life expectancy. It doesn’t seem to me like this would give you another factor of 6, but maybe I’m missing something.

25 thoughts on “If this article portrays things accurately, the nutrition literature is in even worse shape than I thought

  1. I’ve already had 3 cups of coffee this morning. My wife finished up the hazelnuts yesterday (they’re hard to find in a midwestern grocery store; TJ’s is about the only place that carries them regularly), and I’m preparing to eat an orange.

    Only problem: my retirement funds don’t look like they’re in shape to get me to 107.

    Very interesting question. I’ve posted this on CrossValidated to see what answers we might get there.

    https://stats.stackexchange.com/questions/389287/how-to-map-an-estimated-risk-ratio-of-0-76-to-increased-life-expectancy

  2. I wonder how much (if any) people’s tendency to assume a relationship is causal is influenced by the specific language that is used to present the results? In the quoted example, “With increasing intake (for each daily serving) … the risk of all-cause mortality decreased,” I could see how to some people causality is implied. If you increase this, you decrease that. Unfortunately I can’t think of a simple phrasing that avoids this problem, but maybe simplicity is less important than avoidance in this case and it would be better to say something like “People who consumed an additional daily serving…had an substantially lower risk of dying in the next year than those who didn’t (RR = 0.76,…).” Would this help emphasize that this is a comparison between different groups of people, not a prediction of what would happen to the risk of one group if they changed their behavior?

    • You’d probably need to include a phrase like, “But people who eat hazelnuts might also be more likely than those who don’t to have habits that contribute to longevity.”

  3. This post reminds of a new report that China overtakes US for healthy lifespan this year. Current life expectancy for a man is 76.9 in US vs. 74.6 in China. I was really surprised by this considering the vast gaps in QoL between the two countries in terms of air quality, food safety, quality of healthcare, etc. Not to mention that over 50% of Chinese men smoke vs around 15% of US men and according to CDC smoking reduces life expectancy by at least 10 year (not sure how they got that number). It made me wonder how much return US is getting from the huge healthcare investment.

    • I don’t see China here, but following the Spiegelhalter link Andrew gave above through U.K. official statistics provides this table, which doesn’t show much bang for US health care buck:

      Table 1: Life expectancy in selected countries, males, 2015 to 2017
      Latest year At birth At age 65
      Switzerland 2016 81.5 19.8
      Norway 2017 80.9 19.2
      Japan 2015 80.8 19.4
      Singapore 2017 80.7 19.1
      Sweden 2017 80.7 19.1
      Iceland 2017 80.6 19
      Italy 2016 80.6 19.1
      Spain 2016 80.3 19.1
      The Netherlands 2017 80.1 19
      New Zealand 2015 to 2017 80 19.4
      France 2017 79.5 19.4
      UK 2015 to 2017 79.2 18.6
      Belgium 2017 79 18.3
      Denmark 2016 to 2017 79 18.1
      USA 2015 76.3 18
      Poland 2017 74 15.9
      Source: Office for National Statistics
      Notes:
      1. The latest year of available data for each country has been used therefore the reference years shown vary between countries.

      Sorry about the poor formatting. The table (and table 2 for women, showing the US behind even Poland) can be found much more royally glorious at https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/lifeexpectancies/bulletins/nationallifetablesunitedkingdom/2015to2017

      • It is simplistic to compare health care systems by comparing life expectancies.

        The US has two obvious factors that decrease life expectancy in a meaningful way relative to Europe.
        1. The much higher rates of death by firearms (make sure to include suicides! there are more suicides by firearms than there are homicides by firearms!)
        2. The greater distances driven per capita (with the corresponding higher risk of death in automobile accidents).
        Both factors result in decreased life expectancies, particularly for males, that are not directly attributable to the health care system per se (they are of course public health issues, but that is a different statement).

        • It is interesting how risks are socially constructed. I was struck by this after a recent visit to a new physician. As a man in my early 50s, I was warned about things that, as far as I can tell, will have a negligible impact on my life expectancy over the next couple of decades—like whether I wear sunscreen or eat “too much” “fat”—but I was not asked whether I own a firearm (no) or take frequent car trips (nope! Walk to work and for groceries), both of which would increase my risk of early death.

        • Interesting – might not be just perception of risk but also a sense of having meaningful advice (a hammer to give a good bang) – wear sunscreen or don’t eat “too much” “fat” seem like obvious good advice (they may not be) but give up your firearm and get a job that you can walk to?

        • Another way to look at it is, this is an HMO. Reducing my time spent on the road and, yes, disposing of a firearm would measurably reduce the risk that I will need urgent medical care that the HMO will need to pay for; whereas, at this age, basically, either I am primed by my past behavior for skin cancer or whatever trouble people think eating fats causes today, or I am not. Other things equal [which they never are], a few more or fewer suntans or Egg McMuffins won’t put me in the hospital.

        • Let’s do a thought experiment here. Suppose he asks the next 20 patients whether there is a firearm in the house.
          That’s about 40%, say 8 of the 20. This is going to be a highly variable percentage, of course.
          https://www.statista.com/statistics/249740/percentage-of-households-in-the-united-states-owning-a-firearm/

          Of those 8, I would guess 7 will listen politely. Probably 1 would make some changes (not give up the gun, but look for more secure ways to store it). That would be a good outcome; a friend kept his firearm locked, but his teenage son found the key and commited suicide, for example.

          But that last one is going to go off on a big 2nd amendment rant, and listening to that rant is going to negatively reinforce asking the question in the future.

          But I’m with you on the sunscreen thing. Here’s an article comparing sunscreen to margarine. At the end, they note that other countries have changed their recommendations on this issue.

          https://www.outsideonline.com/2380751/sunscreen-sun-exposure-skin-cancer-science?utm_medium=social&utm_source=twitter&utm_campaign=tweet&fbclid=IwAR1mUdaJ0QpTPA9oSmH126NzleZWOm1gBwOS7OUezffDKcWJcAUAKQnPy98

        • I think a lot of it comes down to how the doctor asks the question. If the doctor says something like “As a gun owner and a physician I really want to ensure my patients are safe from firearms accidents or misuse, do you have any firearms in your house, and are you confident that they are stored in a way that people won’t be harmed either by accident or misuse?” You’ll get a lot different response than if you say “do you have any guns, and have you thought about getting rid of them?” And I note that the doctor could safely lie about being a gun owner themselves. Patients are unlikely to go off on a 2nd amendment rant to a person who claims to be a gun owner themselves and wants responsible outcomes.

        • These factors a not really that important to life expectancy. You can try a crude exercise of adjusting total annual mortalities by the inter-country differences in suicide rate etc. and then re-calculate life expectancy to measure the impact (of course, such uniform adjustment is unrealistic, .i.e., suicide rate is not the same for teenagers and adults, but for the purpose of calculating impact this method is accurate enough).
          If you make such adjustment between Switzerland and US for road deaths (big difference), all suicides (small difference), and homicides (huge differences) you still get the impact on life expectancy of… about six-seven months. Which is not that surprising at all, if you explore and rank all the causes of all deaths in rich Western societies.

  4. I bought a carburator that saved 40 percent on gas, a timer that save 50 percent on gas, and spark plugs that save 30 percent. I drove 10 miles and my gas tank overflowed.

    (old joke, not sure of attribution)

  5. It’s the piranha problem – also related to the elementary design of experiments principle – that one should run a two-factor experiment instead of two sequential one-factor experiments. Despite the fact that each experiment supposedly isolates the effect of nutrient X (keeping everything “constant”), when you combine the effects of multiple experiments, we have failed to estimate the interaction between the treatments.

    I just used a similar example in my data science course. This happens in the business world via the ever popular A/B testing where you run lots of one-factor experiments. The dirty secret is if the boss is smart enough, s/he will ask: “if I add up all the individual effects you claim for each experiment, I should see that our revenues would have gone up by 50% but our revenues have gone up by 5%, how do you explain that?” The problem is the incremental customer could have come in with any of those successful interventions but certainly not all of them.

  6. The question on my mind is “instead od what?”. When you drink more coffee I get that it is healthy if it means drinking less Coke / Pepsi. But what about drinking less green tea?

    • Bingo! The challenge here is that not only is it all comparative to a general population- that is arguably unhealthy in many ways – but Intake of food and beverage is very tightly constrained, so that adding more of X tends to reduce Y. The endless disputes about macronutrients (e.g. is dietary fat bad? Maybe just saturated fat? Maybe just some saturated fatty acids in some contexts? Maybe it’s sugar all along etc) have given way to recognition that it is ^patterns^ that matter. But good luck studying that with all the standard noisy measurement instruments!

  7. Perhaps a more accurate reading of the implications of such literature is that life expectancy is increased // over the range of the life span involved in the study //. Ioannidis is using point estimates and then applying them to (actuarial) life expectancy curves over the full range. That obviously leads to crazy outcomes. Another thing to mind is that ‘all things’ health related are correlated, so pretty much any single measure is bound to overstate the individual effect.

    I understand that (health) researchers do not have very strong grips on actuarial tables, but so much of these discussions are grounded in misunderstanding that there is a person-level and a group-level distribution of ‘chance of dying’ and that individual actions marginally change your risk, but do not alter the baseline in any massive shift.

    It’s not crazy for actuaries to price with 50% shifts in chance of dying for certain populations (summarily defined by ‘high-income, high-status, good-health’). Those people live >10 yrs longer than average population. And yes, they drink coffee, eat nuts, drink less alcohol, are a lot less prone to drug related abuse. But al these effects work in tandem, and are probably hard to put in a prescriptive way. You just can’t prescribe someone to be in the top-10% of the income distribution.

    Most of the troubles in actuarial industries come from not-avoiding specifically what Ioannidis is doing: predicting death instead of life.

    • Correction: Ioannidis is pointing to flaws in the nutrition literature. He is not making the mistake, but following the mistakes of underlying papers to their logical conclusions.

  8. This quest for statistical significance suddenly reminded me medieval fascination with relics.

    Like, everybody understands there cannot be three genuine skulls of Saint John, but… it is fine as long as you only think about one at a time.

    Quoting The Name of the Rose:
    “And don’t succumb too much to the spell of these cases. I have many other fragments of the cross, in other churches. If all were genuine, our Lord’s torment not have been on a couple of planks nailed together, but on an entire forest.”

  9. There’s a nice literature on life table “entropy” that has analytic expressions on how to convert a uniform change in mortality by age into a change in life expectancy at birth.

    The basic result is that the proportional change in life expectancy is currently equal to about .1 to .2 times the change in mortality rates. So if some treatment causes mortality to drop by 10% at all ages, life expectancy at birth will increase by 1-2%.

    For those interested, here is our paper that reviews this classic result with some extensions to slowing the rate of aging: Goldstein, Joshua R., and Thomas Cassidy. “How slowing senescence translates into longer life expectancy.” Population studies 66.1 (2012): 29-37.

Leave a Reply to Toby Cancel reply

Your email address will not be published. Required fields are marked *