Adam Pollack writes:
You and the blog readers might find this interesting: https://newsroom.heart.org/news/8-hour-time-restricted-eating-linked-to-a-91-higher-risk-of-cardiovascular-death.
Yesterday, my friend was very concerned for me after he found out I usually don’t eat breakfast. He told me it’s dangerous. I thought it was as simple as not being hungry for a few hours after I wake up.
He showed me the above press release from the American Heart Association newsroom. I have never seen the results of an abstract for a poster publicized like this. It even made it to CNN (https://www.cnn.com/2024/03/19/health/intermittent-fasting-pros-cons-wellness/index.html). Both the press release and the CNN article emphasize that the findings are preliminary. For example, press release says “As noted in all American Heart Association scientific meetings news releases, research abstracts are considered preliminary until published in a peer-reviewed scientific journal.”
This doesn’t make me feel better about the situation. Let’s pretend this analysis was conducted perfectly (whatever that means). How would the AHA newsroom & CNN report the results if this was peer-reviewed and published? From the newsroom quote above, I get the sense that if it’s in the peer-reviewed scientific journal the press release wouldn’t have any caveat. Maybe they’ll even recommend people change their lifestyles and diets?
I’m being a little disingenuous because the editor’s note from the date after the first press release tells readers they should always consult with their doctor before making changes to their health regimens. Wait, why is there an editor’s note the day after a press release that provides full poster presentation details?? I’m guessing this caused an uproar to some degree in the community. In general, there’s a lot to unpack from this about science communication and the role of science in informing decisions. I’d be most interested in a discussion on your blog about those points, though I’m sure that the poster could inspire some nice statistical discussion too (https://s3.amazonaws.com/cms.ipressroom.com/67/files/20242/8-h+TREmortality_EPI+poster_updated+032724.pdf). For example, the press release reports the authors “were surprised to find that people who followed an 8-hour, time-restricted eating schedule were more likely to die from cardiovascular disease” and it turns out that’s one of 4 effects (look like interaction effects) w/ p < .05 across all the comparisons they make.
The press release refers to “those who followed an 8-hour time-restricted eating schedule, a type of intermittent fasting,” but from the poster, we see that this “eating duration” variable is the average eating duration for the two dietary recall days in the survey. Of the 414 people in the study who reported less than 8 hours averaging those two days, 31 died of cardiovascular disease during the period of the study. In comparison, the reference group is 12-16 hours, which included 11,831 people, of whom 423 died of cardiovascular disease. (31/414)/(423/11831) = 2.09. The estimated risk ratio is 1.91, which they estimated from a a hazard regression adjusting for a bunch of variables including demographics, smoking, and drinking but also total energy intake, body mass index, and self-reported health condition status.
Looks like noise mining to me, but, hey, all things are possible.
Based on what I see in the paper, the statement, “people who followed an 8-hour, time-restricted eating schedule were more likely to die from cardiovascular disease,” does not seem like an accurate description of the data. How you ate in two days of a survey is hardly an “eating schedule.”
Also they say, “Our study’s findings encourage a more cautious, personalized approach to dietary recommendations, ensuring that they are aligned with an individual’s health status and the latest scientific evidence,” which sounds like gobbledygook. You don’t need a statistical analysis to know that, right?
The press release quotes someone else as saying, “Overall, this study suggests that time-restricted eating may have short-term benefits but long-term adverse effects.” B-b-but . . . if they only asked about how people ate for 2 days, in what sense is this telling us about long-term effects? He does follow up with, “it needs to be emphasized that categorization into the different windows of time-restricted eating was determined on the basis of just two days of dietary intake,” and I’m like, yeah, but then how do you get away with that first statement? OK, he is at Stanford Medical School.
I would guess that (1) the authors of the study know that it’s crap, but there was a poster session coming up, and you’ve got to present something, (2) the lead author* perhaps hasn’t actually seen the poster, or just wants publicity despite the crappy work, so responds to the email to conference attendees to let the AHA press office know about notable new findings, (3) the AHA press person has no understanding of science or data, and credulously buys the story, and (4) no one involved, even for a minute, thinks that real people like your correspondent’s friend will make decisions based on this work, given that they never think of such people at all, and there are therefore no qualms about the process.
* at Shanghai Jiao Tong University by the way, not Stanford. There’s one Harvard author on the poster list, which isn’t quite as disreputable as Stanford.
The poster is quite interesting and makes me wonder about their “adjustments” for different baseline characteristics (I can’t find any more details on the paper other than what Andrew says about “adjusting for a bunch of variables including demographics, smoking, and drinking but also total energy intake, body mass index, and self-reported health condition status.” The baseline characteristics of the <8 hour group are so dramatically different than for the other groups that I'm highly skeptical without seeing the data and analysis. For an observational study comparing groups on the dimension of their eating schedule, it seems doomed from the start when the two groups look like they come from different planets.
I’ve seen it suggested recently that studies with diet recall surveys (let alone two-freaken-day diet recall surveys) should be dismissed out of hand.
I think that’s an over reaction, but I’d certainly agree they should be taken with a grain of salt (although not between 12 AM and 12 PM).
I had an attack of deja vu when I read this (though I certainly do agree with your conclusions) – and then I remembered that the (UK) Science Media Centre had asked various people, including me, for comments on this when it was press released back in March this year. See https://www.sciencemediacentre.org/expert-reaction-to-conference-abstract-about-time-restricted-eating-and-cardiovascular-death/ . I must admit to being slightly surprised at the time how some of my fellow commenters made positive points about it, though most of us were pretty critical. It did get quite a lot of media coverage here in the UK< though at least some sources did put in critical comments too. But it got a news piece in JAMA Network, https://jamanetwork.com/journals/jama/article-abstract/2817556 .
For medical research, the average person needs to know the answer to two simple questions:
1) Is the study a direct replication that confirmed prior results?
2) Does the study compare an a priori quantitative prediction to new data?
If the answer to both is “No”, then they can go about their day better off never having heard about it. Why? Baseline replication rate (the easy part of science) is ~20-30% and correct interpretation of the observations (the hard part of science) occurs at some even lower rate. Thus there is 90+% chance whatever they are hearing is misinformation.
Science journalists should easily be able to provide this info with two check boxes at the top.
Why will we never see this? Because standards are so low this would filter out 99.999% of medical research. That leaves about ~10 papers per year to write about.
I don’t blame the science journalists, or even the researchers. The problem is funding standards that actively discourage “Yes” answers to both questions. Since most funding comes from the government, that ultimately lies on the politicians and voters to demand some minimum standard of research.
Eg, does anyone remember this from earlier in the year?
https://hms.harvard.edu/news/first-genetically-edited-pig-kidney-transplanted-human
Did you hear about the follow up and was this put into context of similar attempts going back decades? I bet unless you actually sought out that info, the answer is no.
“How you ate in two days of a survey is hardly an “eating schedule.””
True. But at least the research summary wasn’t: “If you skip food for 8 hours on just two days, your risk of death from cardiovascular disease goes up 91%!”
Or the classic “Participants reported being hungrier when they walked into the café (mean = 7.38, SD = 2.20) than when they walked out [mean = 1.53, SD = 2.70, F(1, 75) = 107.68, P < 0.001]."
Then the conclusions from the poster (and this is standard, not picking on this poster):
So, based on their NHST interpretation of the observations, there was no increase in all cause mortality but an almost doubling in cardiovascular mortality. Doesn’t that bring up an obvious question that requires addressing?
Like if we are to tentatively accept their first conclusion then…
Only mindlessly following bureaucratic rules can result in something that ignores that question. Or maybe now chatGPT which is much more efficient at making stuff up without any critical thinking than a bunch of humans.
At this point I think nutritional epi is just astrology for wellness influencers. I would lay a big chunk of the blame here on the research team, because I no longer trust the media to have even the most tenuous grasp on research.
Following the ‘if this study was done perfectly’ angle, from an observational perspective I suspect that a majority of the people who actually follow some kind of intermittent fasting regime are doing so for underlying health reasons. For example, I skip breakfast because I feel a bit ill if I try to eat too early, which could be due to some kind of health issue. Likewise, there are probably people out there who skip breakfast because they’re rushing to work and are stressed out.
There is a fun kind of guessing game to be had here where you can take any nutritional epidemiology study, any at all, and try to guess the actual underlying reason for the association, rather than whatever garbage is suggested by the authors.