How a clever analysis of health survey data became transformed into bogus feel-good medical advice

Jonathan Falk sends a message with the heading, “Garden of forking paths, p value abuse, questionable causality, you name it,” this link to an article in JAMA Internal Medicine, and the following remarks:

Unfortunately, I can only see the first page of this article, but it seems to contain all the usual suspects.
(a) Forking paths: what’s that “about the same” group doing there, a group insignificantly different than the reference group (once we control for health)?
(b) Everything I control for lowers the effect, so what should be my prior about things I can’t measure?
(c) Isn’t the causality here obviously backwards? People who know they are in poor health are far more likely to self-report that they feel older . . .
(d) A 1.4 (fully adjusted) hazard (relative to baseline) with a sample of 313 people? Really?

I replied:

Especially (c). I read the article and I assumed that this was their point, that clinicians should consider taking “perceived age” seriously as a quick and easy way to measure health status. Then their treatment of causality in the discussion threw me for a loop.

JAMA should know better than to publish papers by non-M.D.’s!

That last line was a joke but I’m serious about the rest.

In any case, Falk replied:

I would have assumed so as well if I hadn’t come across the article from reports like this.

Falk was pointing to a news article by Dana Dovey that goes like this:

“Acting your age” may actually be hazardous to your health. A team of British researchers have found that the key factor in living longer lies not in any vitamin supplement, but rather in your own mind. It seems that those who feel younger actually live longer. Although the cause for this remains unclear, it does mean that, according to science, being immature is healthy.

Wow. This is like a masterclass in bad science reporting. The paper said nothing at all about immaturity, nor did they identify subjective age as “the key factor” in living longer (whatever that means), nor did they say anything about vitamins.

That’s three serious errors in the first sentence of the article. OK, n=1 and all that, but I’m not planning to get my medical information from “Health Daily,” that’s for sure.

And then the report concludes with this:

If you do feel older than you are, don’t despair. The team hopes that their study will open up intervention opportunities. They suggest that unhappy feelings associated with aging may be the root of this early death and suggest that “health messages promoting positive health behaviors and attitudes toward aging” may help to avoid it.

Ummm . . . maybe. But I wouldn’t count on it. Again, it seems completely reasonable (and consistent with the data) to think of subjective age as a measure of existing health status without any causal effect to speak of.

The researchers in this study did address this issue. Here’s one of them quoted in a Reuters news article:

“The first thing we thought of is that people who feel older than their chronological age are sicker, and that is why they are at greater risk of dying,” Steptoe told Reuters Health by email.

To account for that, the authors measured pre-existing health conditions including cancer, heart disease, diabetes, stroke, arthritis and other illnesses, which explained some of the link.

“But when we had taken these illnesses into account in our statistical models, the relationship with perceived age remained quite strong,” Steptoe said. “We also measured mobility problems, lifestyle factors such as smoking, depression, and cognitive function. But none of these explained the relationship we saw.”

To me, this is all just evidence that perceived age is a measure of health status that captures information that is not included in pre-existing health conditions as traditionally measured. Again, this seems like it could potentially be of diagnostic value both to diagnosticians (if it’s a quick and easy way to add information about health status) and in large-scale health surveys.

But do I see evidence for the claimed causal effect? No, I don’t.

That Reuters article, by the way, does a pretty accurate job of reporting the available information:

Everyone dies, but a new study says feeling sprightly might suggest a person has more time left than people who feel their age or older.

Older people in the UK who felt at least three years younger than their chronological age were less likely to die over the next eight years than those who felt equal to or older than their actual age, researchers found. . . .

Using data from a previous study on aging, Steptoe and his coauthor Isla Rippon analyzed more than 6,000 adults who were at least 52 years old.

In 2004 or 2005, researchers asked the participants how old they felt.

More than two-thirds felt at least three years younger than their real age, while a quarter felt their real age and less than five percent felt more than a year older, according to the research letter in JAMA Internal Medicine.

Following the group through March 2013, the authors found that about 14 percent of those who felt younger had died, compared to about 19 percent of those who felt their age and about a quarter of those who felt older. . . .

People who feel older are less likely to go out and about, are lonelier, are less mobile and are less physically active, Steptoe noted.

People shouldn’t worry about how old they feel, he said.

“But it’s certainly something that we as medical researchers should try and understand,” Steptoe said. “Perhaps the beliefs and feelings that people have tell us something that our other measures of health and wellbeing do not capture.”

The news article does promote some of the causal claims uncritically, but I like that it is clear on the evidence from the study and that it ends on a point of measurement rather than causality.

P.S. Just to be clear: I have no problem with the study. “How old do you feel?” seems like a reasonable question to ask people, and it makes sense then to analyze these responses and find out what additional information it conveys. My problem is only with the interpretation, a bit in the conclusion in the research article and in the press release, and a lot in some of the news reports.

9 thoughts on “How a clever analysis of health survey data became transformed into bogus feel-good medical advice

  1. My dad used to say the most important question he’d ask a patient was “how are you feeling?” because the answer, which sometimes required a bit of prompting, spoke volumes about actual health. He wouldn’t say it this way but you are the only person inside you – at least if you keep the voices quiet. He would say that tests are hints and a diagnosis is a broad label that guides treatment. So that x feels their age is very telling because there’s truth in the phrase you’re as young as you feel. Not causative. Has society become that stupidly fixated on quack quick fixes?

  2. It may be an n of 1 for the article but if readership is high enough this may have given tons of people awful advice on their health. Unfortunately, this happens all to frequently in the education literature as well. There is even a framework for special education that has – what I believe to be – a horrible misnomer “Response To Intervention.”

  3. Andrew said in his PS:”I have no problem with the study. “How old do you feel?” seems like a reasonable question to ask people”

    This gave me a big surprise — I had been thinking, “What a silly question. I would have no idea how to respond to it.”

    Am I just weird, or do other people think it’s a weird question?

    • I agree with Andrew that it’s possibly a useful questions for clinicians to ask that might possibly capture some proportion of subjective health that is different than “How do you feel today?” and “How are you feeling in general?” But it is a very bizarre line of reasoning to say that convincing people to answer that question differently will magically do anything but reduce the observed correlation.

      So much wrong with the whole thing it is hard to know where to begin, but Lee is right in saying it is a weird question and that measurement is hard.

  4. It is a weird question. How could anyone 54 years of age (let’s say) know what it is like to feel 55; or 57 or 59? For that matter, how could anyone 54 know what it is like to feel 51? I can remember, more or less, how I felt three years ago, but I certainly cannot relate that feeling to age. At least the question is weird if the answer to it is taken to be quantitative.
    The question comes down to asking people how they feel, how they see their prospects, and so on. I have not seen the study, but I take it that they did not ask any other questions for comparison, e.g.,

    On the whole how are you feeling today?
    As you see it, has your health declined a lot in the past couple of years?
    If you drew a line showing the change in your health, the way you feel, over the past five years, how would the line look? (Think of a visual analog scale with slope as the metric)

    By the way, at what age does the attributed meaning of the question kick in? What to make of a 30 year-old who feels like 32? or an 85 year old who feels 87?

    I would agree that as a quasi-diagnostic question in an interview, used for heuristic purposes, that question (along with many others) might be useful.
    It is astonishing that such simple-minded, simple findings should be taken seriously.
    Seriously, folks, measurement in human domains is a difficult problem. It deserves to be taken seriusly.

    • Yes, the questions about declining health, or changes in health or in the way one feels, make much more sense to me than “How old do you feel?”

      I agree wholeheartedly that measurement in human domains is difficult and needs to be done carefully. I would not be surprised if this question (and possible alternatives) was not tried out on people before being used. Such testing out is important, but my impression is that it’s not done anywhere near as often as it should be.

    • Not sure I’d know how to interpret the response if a patient hadn’t previously responded to some anchoring vignettes about how one feels at various ages.

    • If you think this is a weird question then you probably feel about your age. I don’t think a 55 yo would answer, “I feel 57”, to use your example. I would think one notices that ages of people that engage in the activities that one does and this gives a sense of “feeling” a certain age. If a 55 yo moves from Florida to Maine and picks up snowboarding and is ripping double black diamonds and shredding the terrain park, they probably wouldn’t answer that they feel anything close to 55. More like 25. But if a 55 yo moves from Maine to Florida so they don’t have to shovel and they start attending bingo night after giving up golf because they cannot walk from the cart to the green without stopping for breath three times, then they will probably answer they feel 75 or 80.

Leave a Reply to Aaron Pallas Cancel reply

Your email address will not be published. Required fields are marked *