“Please make fun of this claim”

Jeff sent me an email with the above title and a link to a press release, “Nut consumption reduces risk of death,” which begins:

According to the largest study of its kind, people who ate a daily handful of nuts were 20 percent less likely to die from any cause over a 30-year period than those who didn’t consume nuts . . . Their report, published in the New England Journal of Medicine, contains further good news: The regular nut-eaters were found to be more slender than those who didn’t eat nuts, a finding that should alleviate fears that eating a lot of nuts will lead to overweight. . . .

For the new research, the scientists were able to tap databases from two well-known, ongoing observational studies that collect data on diet and other lifestyle factors and various health outcomes. The Nurses’ Health Study provided data on 76,464 women between 1980 and 2010, and the Health Professionals’ Follow-Up Study yielded data on 42,498 men from 1986 to 2010. . . .

Sophisticated data analysis methods were used to rule out other factors that might have accounted for the mortality benefits. For example, the researchers found that individuals who ate more nuts were leaner, less likely to smoke, and more likely to exercise, use multivitamin supplements, consume more fruits and vegetables, and drink more alcohol. However, analysis was able to isolate the association between nuts and mortality independently of these other factors. . . .

The authors noted that this large study cannot definitively prove cause and effect; nonetheless, the findings are strongly consistent with “a wealth of existing observational and clinical trial data to support health benefits of nut consumption on many chronic diseases.” . . .

The study was supported by National Institutes of Health and a research grant from the International Tree Nut Council Nutrition Research & Education Foundation.

The press release did not link to the study—what’s with that, anyway, how hard would it be to include a link???—but a quick google led to this article, “Association of nut consumption with total and cause-specific mortality,” by Ying Bao, Jiali Han, Frank Hu, Edward Giovannucci, Meir Stampfer, Walter Willett, and Charles Fuchs.

Here are my quick thoughts (in no particular order):

1. Unlike various other examples we’ve discussed recently, multiple comparisons and statistical significance were not major issues with this study. Sample sizes were huge, and the treatment and outcome variables seemed clear enough.

2. They report that, “as compared with participants who consumed nuts less frequently, those who consumed nuts more frequently were leaner, less likely to smoke, more likely to exercise, and more likely to use multivitamin supplements; they also consumed more fruits and vegetables and drank more alcohol.” The regression analysis controls for age, ethnicity, body-mass index, physical activity, smoking status, alcohol consumption, multivitaminin use, intake of total energy, red/processed meat, fruits, and vegetables, and a bunch of other things. Just to get a sense of things, I’d like to see the raw death rate for each nut-consumption group (adjusting for age but no other predictors) and then their regression estimates. Adjusting using regression makes sense, but I’d like to see what the adjustment is doing. It looks like the regression model is done on the log-hazard, which is standard and again makes sense.

3. The correlation-causation issues are real, and they are clear enough in the press release, which does not seem to say anything not supported by the study.

4. The Bayesian in me wants to do some partial pooling. The point estimate from the study is that a daily serving of nuts reduces the instantaneous mortality rate by 20%. So, if there really is an effect, my best estimate would be lower than 20%. Maybe 10% or 5%. A 20% reduction in risk from one food, that’s a lot! I’m not saying it’s impossible, I’m just saying that a bit of Bayes would pull our estimate toward zero.

As you can see, I can’t bring myself to mock this study, which seems much stronger than the notorious coffee study from a few months ago, and much much stronger than the various n=100 Mechanical-Turk-style psychology studies that we’ve been reporting on with depressing regularity over the past year or so.

Should I start eating nuts? I really don’t know what to believe. Much depends on the prior (that is, on the distribution of true health effects of various diets, as best we can estimate from existing scientific data and models).

One of the pleasures of blogging is that I am completely free to admit uncertainty, unlike in a research article (where there is pressure to demonstrate that some claim has been conclusively proved) or in the journalistic style in where there are clear heroes and villains.

44 thoughts on ““Please make fun of this claim”

    • Here’s how I would approach claims of massive reductions in overall deaths from some difference in diet:

      Rank order of the causes of death by how plausible they are that they are linked to diet. For example:

      1. Diabetes
      2. Heart attacks
      3. Strokes
      4. Cancer
      5. Genetic diseases
      6. Car accidents
      7. Drug overdoses
      8. Homicides
      9. Lightning strikes

      If this nuts-save-your-life finding is valid, then most of the effect should be found in causes of death near the top of the list (e.g., diabetes). But if it turns out that eating nuts only slightly reduces your chances of death from diabetes but makes you vastly less likely to be struck by lighting, then we’ve probably gotten a selection effect in which nut eaters are more careful people in general and thus don’t play golf during thunderstorms, or whatever.

      • Steve:

        This sort of thing is done sometimes—it came up in discussions of the risk of cancer—but it can be difficult because the number of cases gets pretty small as you go down your list, and the estimates get too variable to be useful.

  1. 1. “According to the largest study of its kind” and;

    2. “The authors noted that this large study cannot definitively prove cause and effect; nonetheless…”

    These two statements send my alarms ringing.

    First, claim 2 is a disclaimer but I would say the causal intent is there. It would be interesting to keep the same covariates but then replace nut consumption with any other food consumption in the survey. My guess is you will get similar results for a large proportion of them.

    Second, if there is causal intent then the size of the study is less important than the identification strategy (e.g. check for overlap, discontinuity, etc.).

    Third, there is a simple test they can do under the assumption that all covariates are non-descendants of nut consumption or the outcome: Search the space of covariates: if the outcome is orthogonal to the treatment conditional on some covariates then the relation is not causal (causation does imply correlation).

    Having said this I buy the direction of the results mainly bc nuts have low carbs, and are very filling. My sense is what matters is not how much you eat but what you eat. The amount you eat is endogenous to the composition of your diet.

    • PS 3. “The study was supported by National Institutes of Health and a research grant from the International Tree Nut Council Nutrition Research & Education Foundation”

      I guess that is another alarm bell. Doubt this study would have seen the light of day if results were null or, God forbid, negative. Did they pre-register?

    • Face validity is there, but without access to the data, protocol and domain experts to discuss the protocol/background with, its very hard to get further than that.

      Jamie Robins, made this point at a big Epi conference in Toronto almost ten years – with expressions like how are you supposed to know what someone means by adjusted for these covariates? The data has to be made available for other to _review_.

  2. I don’t think they addressed measurement error in the covariates, which will be considerable for many of them, such as total energy, individual food item consumption, and physical activity.

  3. Not mockery so much as doubt.

    Once you grant that nut-eaters are outliers in regards to a lot of clearly health-related covariates, you put a lot of pressure on the regression, and its specific functional form, to isolate the non-nut-eating effects. This reminds me of regressions I used to do for things like wage equality between men and women. Every variable you could find that made sense shrank the gap somewhat between men and women, though the gap never fully closed. The fact that every variable you find only made the gap smaller (never wider), and the fact that you knew there were lots of variables that you had no data for, led me to the conclusion (admittedly not entirely scientific) that unobserved covariates (and their correlation with gender) were responsible for much of the remaining difference.

    • I agree that this is the problem. It seems to me that the type of person who eats nuts has better health outcomes, and it’s hard to reach any other conclusion. Given that nuts are expensive and now considered “healthy”, it seems that nut eating is just a proxy for high SES and general “health awareness”. I haven’t read this paper’s text, but I imagine that it’s similar to the vegetarian study that came out this summer in JAMA Internal Medicine: vegans were the only non-overweight group and only ones where most engaged in recommended 150 minutes per week of exercise, and because the study sample were all 7th Day Adventists, vegans were the most involved/religious in the group, which could have independent impacts. I doubt that regression isn’t good enough to adjust for such fundamental baseline differences.
      http://www.ncbi.nlm.nih.gov/pubmed/23836264

      • Oops, double negative. I doubt regression can adjust for such fundamental baseline differences, either in this study or the vegetarian one.

  4. Well, there’s really not much to pick at, is there? I think I’m going to start eating nuts.

    The choice of restricted-cubic-spline is interesting; I’d like to see exactly how it was used, but with the p-value below the 0.001 range it doesn’t sound like they did a lot of shoe-horning, so that’s good.

    The only bias I can conceive of is that the entire subject group was health professionals, which makes me look a little harder for related causes (i.e. what do health professionals that decide to eat a lot of nuts ALSO do that may affect their longevity?) For example, low-carb diets (which I believe the health-care groups are somewhat conflicted over), tend to be nut-heavy. The survey indicates that they adjusted for Mediterranean-Diet effects, though, so if anything it seems they’ve been as thorough as one could expect, so I’m really reaching here.

    One last little thing I did notice… “Funded by the National Institutes of Health and the International Tree Nut Council Nutrition Research and Education Foundation.”… I always worry seeing research that looks great that is paid for by the people that come out looking the best. It SHOULDN’T affect the science, but it does beg for more scrutiny and an independently reproduced result.

    But overall, yep, I’m definitely going to eat more nuts.

    • FYI at these schools some 75% of a scientist’s salary comes from grants/consulting/etc…

      This does not imply they bend the data but it almost certainly affects what jobs are offered to them, and which they decide to take on. In essence anything that confirms their world view is legit.

      Here is an experiment I would love to see. Create two datasets and in one relabel consumption of nuts as consumption of red meat. Now hire N research groups at public health schools. Random assignment as follows:

      1. A quarter are randomly assigned the dataset with nut consumption labelled as red meat consumption, and asked to find an effect of red meat consumption on health.
      2. A quarter are given the intact data and asked to do the same.
      3. As above, but these scientists are asked to study the effect of nut consumption on health.

      Any bets what will happen?

      • Theory is important to narrow down the tests to perform and model choices. The fact that results would differ among each treatment is because theory matters. An it should matter. See for instance, the no-free-lunch theorems out there. http://www.no-free-lunch.org/

        The conclusion I take from these no-free-lunch theorems is that we neeed, somehow, to narrow down our space of models (or functions). And we use theory to help on this task.

  5. It’s wise to be skeptical about any claims that come about from the Nurses’ Health Study, not to mention other similar studies. First of all, it’s been shown that nutritional surveys are often wildly wrong (and biased) – e.g., see http://blog.cholesterol-and-health.com/2010/09/new-study-shows-that-lying-about-your.html.

    Second, other such major claims haven’t worked out. E.g.,

    “How did a 50% reduction in CHD (coronary heart disease) turn into a 30% increase in CHD?

    It’s because the initial data from 1991 was from the Nurses’ Health Study, an associative cohort study which could only answer the question “What are the health characteristics of nurses who choose to undergo HRT versus nurses who don’t?” The followup data from 2002 was from a randomized clinical trial, which answered the much more relevant question “What happens to two matched groups of women when one undergoes HRT and the other doesn’t?”

    From http://www.gnolls.org/2893/always-be-skeptical-of-nutrition-headlines-or-what-red-meat-consumption-and-mortality-pan-et-al-really-tells-us/

  6. http://scholar.google.com/scholar?q=%22nut%20consumption%22%20mortality

    Looks like this is far from the first time that nut consumption has been tied to reduced mortality or cardiac events. I don’t plan to increase my nut consumption, but that’s just because nuts are expensive.

    (And on the plus side, suppose this correlation turns out to be non-causal? What’s the worst that would happen? Some people ate more nuts than they would otherwise have eaten. It’s not like they’re being told to reduce intake of a vital nutrient like salt.)

    • The problem is that there are a zillion studies like this (blue berries, coffee, bananas, you name it) yet we still don’t know with confidence what is causing the obesity/diabetes/etc epidemic that is ravaging developed nations. And, the causal disclaimer notwithstanding, these studies inform policy.

        • When you grow a tumor you eat more bc the tumor stimulates your appetite so it can grow.

          When you start eating sugar and refined carbs, your body decides to grow fat, like a tumor, and hence estimulates your appetite. What you eat causes how much you eat, and fat build up.

          At least that is one theory. That’s where my money is.

  7. I read this when it came out, mostly because my wife eats a handful of cashews and almonds every day – and we have tubs of almond butter and peanut butter on the counter. I couldn’t see any clear way to separate the effect of the nuts from the other variables and took this to mean that nut consumption is a marker … but the strength of that marker would likely decrease if more people ate nuts.

    It’s easy to develop pet theories that don’t stand up. A personal one is that people who eat nuts (and legumes like peanuts) are getting more protein, but I think that because I believe the RDA for protein is low and that people tend to consume unbalanced diets too high in carbs, too low in protein, etc. True? Beats me. I try to remember we’re an imaginative species and that our need to make up stories is not the same as those stories being true.

  8. More subtle things are going on. Studies with really large N are reassuring, at least in a typical biostatistics culture, because all we really think about is sampling error on estimates. The error due to missing confounders will always be there, regardless of N, so the large N in these kinds of observational studies gives one false reassurance. Also, authors can give all the “we’ve just found association” warnings they want but human brains have a hard time dis-associating correlation from causation. The response will inevitably be, people concerned about their health will eat more nuts and doctors will tell people to eat more nuts. That’s the way we roll.

    • Jeff:

      Regarding your second point: Sure, but lots of authors don’t properly include these correlation/causation disclaimers. This happens all the time even in top journals, and we’ve discussed many such examples on the blog. So I appreciate it when authors do go to the trouble of saying it right. Giving the disclaimer is slightly awkward, and I think that’s a good thing, it that it reminds the readers of potential problems.

      • I think there is a difference between doing a causal study using observational data vs doing an associational (e.g. purely predictive) study. The latter needs no disclaimer.

        Here I am not sure what this study is trying to be. By the disclaimer I think it is attempting to estimate causal parameters. But there is much ambiguity, and some of the design features do not jive well with causal inference studies (e.g. emphasis on huge sample vs identification).

        I would prefer it if they said more explicitly “We find A causes B under assumptions Z”.

        • Anon:

          I respect where you’re coming from—indeed, it’s what Rubin or Hill might say—but, for me, I have a lot of experience with and respect for pure descriptive analyses (Red State Blue State, for example) and I’d be happy with a clean descriptive result, with causal analysis to follow. But, sure, if you’re going to make a causal implication, it’s a good and Rubinesque idea to state clearly the assumptions underlying it.

        • I too think descriptive analyses are useful. Indeed your descriptive work counters many preconceptions about political behavior w/o having to dwell on causality. E.g.some predicted correlations are not as expected, etc.

          What irks me with this study is the ambiguity. The way to deal w observational causal research is not to hide under a disclaimer but to put the identification assumptions upfront. Then do sensitivity analysis and so on.

  9. I’m surprised at the moderate response to this study and press release: the study is profoundly flawed, and the press release is misleading.

    The press release claims, “Sophisticated data analysis methods were used to rule out other factors that might have accounted for the mortality benefits.”

    What it should say is, “Sophisticated data analysis methods were used to rule out SELECTED, OBSERVABLE other factors that might have accounted for the mortality benefits.”

    Two extremely important determinants of health—income and education—are not among the covariates. These are such gross omissions that the study is basically worthless. A probably less important problem which may work in the other direction is “bad” controls, notably BMI, were included.

    It may be that nuts do confer important health benefits, but this study is does not provide convincing evidence supporting that claim.

    • Chris, you are exactly right. What people seem to miss about controlling for “confounders”, even ALL “known” confounders, is that controlling for those confounders can actually increase net confounding bias in the presence of unmeasured confounders (and I personally believe there are always unmeasured confounders). The presence of unmeasured confounding is readily apparent from the results in this paper and the MANY similar analyses Walter Willett’s Harvard nutritional Epi group did from these VERY SAME DATA. Just look at the quintiles, all of the health indicators improved monotonically…. Those who ate more nuts were inherently healthier than those who ate fewer nuts, and you simply cannot control for this in a statistical model.

      • Sorry I wrote this before actually looking at the paper, they didn’t actually use exposure quintiles in this paper, like they had in their umpteen other similar papers (most of which received serious media attention, but all of which are likely noise). They used frequency of nut consumption, and they presented many fewer descriptive analyses than they had in most of the other papers (just do a PubMed search on Willett). Otherwise my comment is still accurate.

  10. Shouldn’t one look at the absolute risks, first? In the reference group (zero portions a week), there were 3,343 deaths after 390,915 woman-years of observation, that is 86 in 10,000 women per year. In the highest group (more than six portions a week), they calculated a relative risk of 0.79, so that’s 68 deaths in 10,000 women. Absolute risk reduction: 18 deaths in 10,000 women per year. The advantages are smaller, of course, for the less extreme categories — and going from 1 portion a week to 5 or 6 portions a week gives 0 less deaths: it’s 75 per 10,000 in both groups. The numbers for men are a bit better (more men die), but not that much.
    Only 1.4% of the women eat >6 portions a week. Are these women really in all other aspects similar to women who never take a single nut? It looks a bit arbitrary to construct the categories like they did: 0 (18%), 6 (1.4%). Why not lump the last two or three together? Or take quintiles, like the same investigators did with the milk and hip fractures study they published a few days before this one?

  11. Hey, where did the categories go? New try: 0 (18%), less than 1 portion a week (46%), 1 (18%), 2 or 3 or 4 (15%), 5 or 6 (2.1%), more than 6 (1.4%).

  12. So researchers acknowledge that “eating nuts” is strongly correlated with a long list of observable factors that might also be positively affecting health outcomes (“leaner, less likely to smoke, more likely to exercise” etc.) What about the unobserved factors?

    This seems like a great place to apply the methods of Altonji, Elder and Taber, using the degree of selection on the observables to make a guess about the degree of selection on unobservables. The seminal paper (at least in economics) is here:
    http://faculty.smu.edu/millimet/classes/eco7377/papers/altonji%20et%20al%2005a.pdf

    More discussion of this idea with specific applications to public health is here:
    http://economics.yale.edu/sites/default/files/oster-130411.pdf

    I always thought this was a great idea and should be routinely included in studies of this type. I’d love to here what Andrew thinks of it.

  13. I get that a decrease of 20% is a lot and borders on implausibility, but I agree with Andrew. After giving the paper a somewhat quick read, the methodology seems pretty solid. I like that they did significant adjusting for confounders, and several sensitivity analyses. I would be surprised if adding SES and education as covariates would make a big difference as this group is more homogeneous on those variables than the general population.

  14. The researchers here did not ‘observe’ their subjects nut-eating behavior — they merely observed paper questionnaires completed by the subjects each 2-4 years.

    Self-Reporting is a huge error source, especially in diet studies.

    How often did you eat nuts in 2009 ? Do you recall what you had for lunch last Thursday ?

  15. As something of a health nut (did you see what I just did there?), I’ve looked a bit into the clinical research on nut consumption. The general consensus seems to be that the addition of nuts doesn’t lead to the expected weight gain from their caloric content for three reasons: (1) satiety: nuts make people feel full, and the nut-eaters spontaneously reduce calorie consumption later in the day (this offsets appx 65-75% of the calories); (2) increased energy output, possibly from the protein, the fatty acid profile, or both (appx 10% of the nuts’ calories): and (3) less bioaccessibility: relative to other foods, more of the energy in nuts is expelled in your fecal matter (5-7%).

    (Yes, nuts give you high-energy poop. Somehow I don’t think this will become a major marketing point for the Tree Nut Council.)

    Given this, I can see why eating nuts might have a minimal causal effect on weight (reducing it), under the assumption that the calories in nuts are replacing an equivalent number of calories from other, less fibrous, less protein-rich foods with worse fatty acid profiles. But, in an epidemiological study like Bao et al, shouldn’t this effect be absorbed in the adjustments for weight, BMI, etc?

  16. What about nut allergies among subjects, or even their kids? The allergies would reduce nut consumption and may reflect some underlying genetic or environmental risk factor.

  17. Another point I’d like to make here is that there was a recent link put up by someone in a comment somewhere on this blog to a randomized controlled study where they encouraged either nut consumption, or olive oil consumption, or standard “lose weight and exercise” as the three treatments, and the nuts or oil consumption group had the lowest heart disease outcomes.

    I think this might have been a comment on one of Phil’s diet/exercise posts. So, at the moment I’m not prepared to google up those results but someone else might be able to find that study quickly.

  18. One thing I’m confused about in this study is this sentence in the methods: “The hazard ratios from multivariate models in each cohort were pooled with the use of the random-effects model, which allowed for between-study heterogeneity.”
    http://www.nejm.org/doi/full/10.1056/NEJMoa1307352#t=articleMethods

    They don’t specify what their random-effects model is, and as far as I can tell there were only two cohorts (men and women), so how can you model that as a random effect?

    In addition to the question about whether it is possible to properly control for the healthy observer bias in studies like this, especially when all the measured confounders seem to be tracking each other, it seems that how they specified a random-effects component could have impacted their estimation.

  19. One way to analyze this question is to find out how how much of the 20% difference in deaths was due to things that wouldn’t seem to be plausibly caused by diet, such as car accidents and homicides. If people who go out of their way to eat nuts are less likely to, say, die of recreational drug overdoses, that might suggest that nut eaters are different on average from non-nut eaters.

Comments are closed.