Is coffee a killer? I don’t think the effect is as high as was estimated from the highest number that came out of a noisy study

Thomas Lumley writes:

The Herald has a story about hazards of coffee. The picture caption says

Men who drink more than four cups a day are 56 per cent more likely to die.

which is obviously not true: deaths, as we’ve observed before, are fixed at one per customer.  The story says

It’s not that people are dying at a rapid rate. But men who drink more than four cups a day are 56 per cent more likely to die and women have double the chance compared with moderate drinkers, according to the The University of Queensland and the University of South Carolina study.

What the study actually reported was rates of death: over an average of 17 years, men who drink more than four cups a day died at about a 21% higher rate, with little evidence of any difference in men.  After they considered only men and women under 55 (which they don’t say was something they had planned to do), and attempted to control for a whole bunch of other factors, the rate increase went to 56% for men, but with a huge amount of uncertainty. Here are their graphs showing the estimate and uncertainty for people under 55 (top panel) and over 55 (bottom panel) FPO-1 There’s no suggestion of an increase in people over 55, and a lot of uncertainty in people under 55 about how death rates differed by coffee consumption. In this sort of situation you should ask what else is already known.  This can’t have been the first study to look at death rates for different levels of coffee consumption. Looking at the PubMed research database, one of the first hits is a recent meta-analysis that puts together all the results they could find on this topic.  They report

This meta-analysis provides quantitative evidence that coffee intake is inversely related to all cause and, probably, CVD mortality.

That is, averaging across all 23 studies, death rates were lower in people who drank more coffee, both men and women. It’s just possible that there’s an adverse effect only at very high doses, but the new study isn’t very convincing, because even at lower doses it doesn’t show the decrease in risk that the accumulated data show. So. The new coffee study has lots of uncertainty. We don’t know how many other ways they tried to chop up the data before they split it at age 55 — because they don’t say. Neither their article nor the press release gave any real information about past research, which turns out to disagree fairly strongly.

I agree.  Beyond all this is the ubiquitous “Type M error” problem, also known as the statistical significance filter:  By choosing to look at statistically significant results (i.e., those that are at least 2 standard errors from zero) we’re automatically biasing upward the estimated magnitudes of any comparisons.  So, yeah, I don’t believe that number. I’d also like to pick on this quote from the linked news article:

“It could be the coffee, but it could just as easily be things that heavy coffee drinkers do,” says The University of Queensland’s Dr Carl Lavie. “We have no way of knowing the cause and effect.”

But it’s not just that.  In addition, we have no good reason to believe this correlation exists in the general population. Also this:

Senior investigator Steven Blair of the University of South Carolina says it is significant the results do not show an association between coffee consumption and people older than 55. It is also important that death from cardiovascular disease is not a factor, he says.

Drawing such conclusions based on a comparison not being statistically significant, that’s a no-no too.  On the plus side, it says “the statistics have been adjusted to remove the impact of smoking.”  I hope they did a good job with that adjustment.  Smoking is the elephant in the room.  If you don’t adjust carefully for smoking and its interactions, you can pollute all the other estimates in your study. Let me conclude by saying that I’m not trying to pick on this particular study.  These are general problems.  It’s just helpful to consider them in the context of specific examples.  There are really two things going on here.  First, due to issues of selection, confounding, etc., the observed pattern might not be real.  Second, even if it is real, the two-step process of first checking for statistical significance, then taking the unadjusted point estimate at face value, has big problems because it leads to consistent overestimation of effect sizes.

18 thoughts on “Is coffee a killer? I don’t think the effect is as high as was estimated from the highest number that came out of a noisy study

  1. Its all out fault – researchers do not try to be so daft on purpose.

    The first reference here http://en.wikipedia.org/wiki/Meta-analysis covers a basic approach for undertaking an adequate meta-analysis of observational studies _and_ the worked example is on coffee consumption!

    But researchers don’t have easy accesses to wiki or
    this popular intro epi text is not affordable or accessible or
    its too poorly written for most researchers to understand or
    the math (no calculus at all) is too hard for most researchers or

    it would make the likelihood of a publication and especially one with exaggerated claims too unlikely.

    p.s. If not already done, it would be nice if someone published a more refined approach using more advanced statistical methods.

  2. Has anyone tried feeding captive rats / monkeys etc. coffee & see if they die faster? Easier to control for smoking, skydiving, work(alc)oholism etc.

    • You’d probably want to do it on monkeys to be relatively valid. For example, if I remember correctly, much of the lab results on dietary cholesterol was based on rabbit studies. Turns out rabbits eat plants… who would have thought that they would have no tolerance for adding a new substance to their diet that they don’t normally consume? As far as I understand it, the amount of cholesterol in the human diet has absolutely nothing to do with cholesterol levels in the blood, that’s controlled by saturated fat intake where it’s converted in the liver. Similar problems could easily occur here.

  3. As a heavy coffee drinker, I tend to perk up when I see headlines like this, just to see what’s brewing. ;)

    Only tangentially related to the study being discussed: It’s hard not to think of all those old men, sitting in McDonalds all across the country (maybe the world) nursing their senior coffees. Are they running out of retirement savings and want to hasten their own demise with coffee drinking? Or, is this the type of social activity that keeps them connected to others and helps their overall social health by providing them with more social contact than watching baseball on TV alone in their home?

    I suspect the overall effect of coffee on older men is strongly positive due to the social context in which it is often consumed (so long as the group is nonsmokers).

    Now that I’m semi-retired, I’ve joined the ROMEOs (Retired Old Men Eating Out), an informal group of men at our church who meet once a week for lunch at a local restaurant. Yes, it’s a pleasant social activity, but it’s also a way to interact with some of the considerably older men (in their 80s) who I’d never visit with otherwise. I’m paying my attendance dues now, in the hopes that 20 years from now when I’m in my 80s I’ll have the same sort of opportunities for informal interaction.

  4. Andrew writes “Let me conclude by saying that I’m not trying to pick on this particular study.”

    I am, slightly.

    As a scientific paper, there’s nothing specially bad about it, as Andrew says, but this didn’t end up in the NZ Herald because our journalists are in the habit of reading Mayo Clinic Proceedings. It got there because someone, either at the journal or at one of the universities, with the cooperation of the researchers, deliberately tried to get it in the newspapers. That’s the threshold for me at StatsChat, and I do try to compare the story with the press release(s) when I can, to see where the distortions happen.

  5. On adjusting for smoking, this is buried near the end of the paper:

    “Fourth, residual confounding may still exist even though we adjusted for all the potential confounders available in the present study. Smoking is likely to be one of the most important factors to cause residual confounding in this investigation.
    We therefore stratified the analysis by smoking status and the results are shown in Supplemental Figures 1 and 2, available online at http://www.mayoclinicproceedings.org. We did not observe the significant association between coffee consumption and all-cause mortality both in current smokers and non-current smokers.”

    The graphs are here:

    http://download.journals.elsevierhealth.com/mmcs/journals/0025-6196/PIIS0025619613005788.mmc1.pdf

    In short: the partial correlation between coffee consumption and mortality entirely goes away when the models are estimated stratified by current smoking status. If instead smoking status is addressed only by use of a current smoker dummy, it appears coffee consumption and mortality are correlated.

    Why the abstract, intro, and conclusion fail to mention that the association is fragile to the manner in which smoking is included, and goes away entirely when less restrictive models are imposed, is a question that ought to be posed to the paper’s authors, referees, and editor.

    It’s also worth noting how limited the set of controls are: education, income, occupation, marital status, and employment are all conspicuously absent, for example. Variables that may be on the causal path from coffee consumption to health, such as hypertension, cholesterol, and BMI, are, however, included.

      • And unfortunately almost standard practice – at least the how limited the set of controls are and how this makes the results at best very tenuous and usually misleading.

        I really believe researchers likely can’t even face the real uncertainties of epidemiological studies or admit it privately to themselves and their family members _given_ little can be done without some real hardship to address that much of the uncertainty.

  6. Pingback: I refer the Honorable Member to the answer given some moments ago | Stats Chat

  7. I would have liked to also hear what you think is good about the study and what can be learned from it. You seem to have described only one side of your opinion.

    A young professor of psychology once told me that in graduate school she learned to think critically. I was surprised — this sounded naive. “You didn’t also learn to appreciate studies?” I asked.

    • A young professor of psychology once told me that in graduate school she learned to think critically. I was surprised — this sounded naive. “You didn’t also learn to appreciate studies?” I asked.

      I may be misunderstanding the intended point of this story, but it sounds naive to me. After all, isn’t critical thinking skill necessary in order to appreciate studies? Or do you mean to point out that critical thinking skill isn’t sufficient for the appreciation of studies (i.e., implying that the young psychology professor was claiming to have only learned to think critically in graduate school)? In that case, I agree, but don’t see the relevance to the preceding paragraph….

    • Seth:

      Given the attention this study received, I think I am contributing by discussing its limitations. I think that a better understanding of statistics should help people understand what is good about the study. When people go around hawking a study claiming a 56% increase in death risk, I think it’s useful to understand what went wrong to lead to such a big number. Understand mistakes can be a useful way to move forward.

Comments are closed.