Nadav Shnerb writes:
I’m a Israeli physicist and an occasional reader of your blog. Recently I have noticed what appears to me as an essential failure in the way statistical facts are presented to the public, and I would be happy to learn about your opinion on that issue.
Here in Israel (and I’m sure also in the US) many facts regarding health hazards are presented in the media as “X Israelis are dying every year from Y”. For example, it is commonly stated that 8000 Israelis die every year from smoking. I’ve noticed that this number is, roughly speaking, simply the number of smokers that die every year (In Israel there are about 40000 death events yearly, and the percentage of smokers is 0.2).
This led me to realize that the statement “X Israelis are dying every year from Y” allows for two different interpretations and that the way the data is presented to the public right now is (I believe) almost completely meaningless.
Let us consider a country with 10 million people, 8 million non-smokers, and 2 million smokers. The population has no age structure, all non-smokers die at 80 and all smokers die at 70. The situation is thus clear: 20% of the citizens lose 10 years of their life due to smoking. No doubt. But what is the answer to the question “how many individuals are dying yearly because of smoking”?
What I believed to be the right answer is about 3600, which is THE DECREASE IN THE YEARLY DEATH TOLL ONE EXPECTS IF THERE WERE NO SMOKERS AT ALL. [Hey, no need to shout! — ed.] Without smokers we will have 1/80 of the population die every year, so for 10M citizens, it will be 125000. With smokers, we have to add 1/80 out of 8M (100000) to 1/70 out of 2M (~28600), so, as said, the answer is about 3600.
However, what I have gathered from the statistics literature I’ve read is that the answer presented in the media is the answer to a different question. They present the answer to “How many people who died this year would not have died (this year) if they had not smoked?”, and the answer to THIS [hey, chill out, dude! — ed.] question is simply 28600, because ALL the smokers that died this year would have stayed alive had they been non-smokers.
To me, this looks like a quite misleading statement. The answer to the second question delivers almost no information! Suppose the smokers die at 79 instead of at 70. The answer to the first question I have posed will change from 3600 to about 320, reflecting the fact that the damage due to smoking is ten times smaller. The answer to the second question, though, will stay almost the same (it will go down from 28600 to 25320). Isn’t that ridiculous?
Moreover, the answer to the second question depends very much on the segmentation of time. Suppose smoking shortens one’s lifetime only by one month, then the answer to the second question will by 1/12 of the number of dead smokers (only those that will survive this specific year), but if you count things monthly, then again all the dead smokers will be in the game.
I would have guessed that these issues are already known and discussed in the literature, but I did not find an appropriate source. Are you familiar with such work? Can you see any justification for that style of coverage of health hazards used in the media?
I agree this is a good example. I think the original sin here, as it were, is to speak of lives rather than life-years. We’re all gonna die, but we’d usually like to delay that time of reckoning. So, yeah, I recommend speaking of life-years or qalys.
Shnerb adds:
Additionally, I think any measure of the impact of smoking on mortality should be scalable. For example, if smoking is estimated to cause 12,000 deaths per year, it should also be estimated to cause approximately 1,000 deaths per month. On the contrary, if smoking is estimated to reduce life expectancy by precisely one month, say, then the answer to “How many people would have survived the YEAR if they had not smoked” and to “How many people would have survived a given MONTH if they had not smoked” are almost the same.
QALY’s forever! (note that ngrams says “QALY” over “Qaly” or “qaly” by enough to call the latter errors.)
I think QALY analysis is just clearly, overwhelmingly better, as you note. It also permits adjustments for non-fatal bad results, and generally considers fatalities that cost age 25-85 as more than six times worse one that cost age 90-100, which I think is correct.
To those pimping QALYs, I would suggest considering the impact of widespread adoption of this metric on individuals living with moderate-to-severe disabilities.
https://ncd.gov/sites/default/files/NCD_Quality_Adjusted_Life_Report_508.pdf
From the report: “[T]he QALY calculation reduces the value of treatments that do not bring a person back to “perfect health,”in the sense of not having a disability and meeting society’s definitions of “healthy” and “functioning” . . .”
This is intended as a criticism, which undermines the credibility of the source. Given two treatment options – one which restores perfect health and one which does not – a decision process that does not value the former more than the latter is clearly suboptimal.
Michael:
I think Qualy is a useful measure. Imperfect, like just about all measures, but useful. I’m not “pimping” the idea. Saying “pimping” is just rude.
@Andrew… I’m sorry the term “pimping” offended you. I believe you use similarly salty and provocative language from time to time. That’s not an excuse.
My choice of words distracted from my point (something I’m often guilty of, and trying to get better at), which is…
People living with moderate-to-severe disabilities are put at a significant disadvantage by government adoption of this metric. Mark Phariss seems to not consider that (1) almost no treatments restore perfect health, and (2) many treatments for individuals living with moderate to severe disabilities are (at least partly) palliative, and (3) the metric is applied even in the absence of a choice of treatments. If there is only one treatment, and it doesn’t provide a big QALY bump, it is not funded. But the definition of QALY is heavily skewed towards what is considered quality for people without disabling conditions. Hence, most treatments for those with moderate-to-severe disabilities fail the QALY test.
I’m not accusing anyone – even the QUALY “promoters” ;) – of being anti-disability. People without disabilities, or who don’t work with these populations, often just don’t think about the impacts of many actions or policies.
Aside from the ways that deciding that some people’s lives are worth less than others has been abused, I don’t see any way that the “Qualitative” part of QALY could be turned into numbers except making numbers up. Psychologists say that people are bad at guessing how a misfortune would affect their happiness, and there is no bank where you can exchange years of life for afflictions (“hey buddy, how many months of life could I get for a chronic depression?”). And using quantitative methods on made-up numbers is a common kind of quackery.
I can see that that approach might have value for specialists assessing cancer treatments, but I would be very uncomfortable with using it to communicate with the public or with elected officials.
Things like this always remind me of the Onion article, “World Death Rate Holding Steady At 100 Percent” https://www.theonion.com/world-death-rate-holding-steady-at-100-percent-1819564171 — so at least some journalists understood this, back in 1997!
Yes, your correspondent’s point is an excellent one.
Cause-deleted life tables are one of the interesting concepts of demography, and like everything, have some assumptions in their construction. My masters was (yikes) 31 years ago, so I don’t know if this is still current, but it represents the issues as far as I remember.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2822405/ or https://doi.org/10.4054/DemRes.2008.19.35
No. The point is to communicate clearly to the public. People reading this form would be horribly confused by it. Deaths per year is simple, straightforward, and understandable. This is not.
“this form?”
Anyway, the point is to communicate clearly *and* accurately. Clarity without sense is not something laudable. The following is also clear: “smoking causes zero additional deaths” because, of course, we’re all going to die. However, that would be a horribly misleading statement. People are, in fact, capable of understanding “years of life lost” — it comes up more or less spontaneously when people we know, or read about, pass away. Moreover, it should not be too much to ask of journalists writing about health issues.
> the point is to communicate clearly *and* accurately
Well, I’m more with total on this one. I don’t think the analysis as-presented is adding clarity.
For instance, smoking is presented in this hypothetical world where you live 10 years less than if you didn’t. Couldn’t the hypothetical media in that world just say that?
If the lesson is to speak in terms of life years, isn’t this all fixed in the hypothetical by just saying X people died due to smoking and on average we’d expect them to live Y years longer if they didn’t?
I don’t see how aggregating death rates from two groups adds clarity in this case. Imo you just gotta report multiple numbers somehow and there’s no real getting way from this. Given the media here are hypothetical, I don’t want to get too mad at them, though I don’t doubt it’d be possible to mislead on these issues.
“Anyway, the point is to communicate clearly *and* accurately”
Of course it is. So is the recognition that every measure is flawed and that sacrificing clarity for something only marginally better is silly.
“People are, in fact, capable of understanding “years of life lost”
I think you have the Pauline Kael problem.
Total:
Please don’t call it the “Pauline Kael problem.” I’m no fan of Kael’s movie criticism, but this particular thing is unfair to her, as it seems that she never made the error that was attributed to her regarding perception of public opinion.
See here for further background on this one. Maybe instead we could call it the Michael Barone problem?
“Please don’t call it the “Pauline Kael problem.””
Thanks, but no. I’m staying with — you knew immediately what I was talking about, which is what I needed.
In addition, Pauline Kael is dead, and cares not a whit for what some random Internet commenter has said.
Total:
Then at least call it the “unfairly attributed to Pauline Kael” problem. Dead people deserve fairness too.
Nope!
OK, not on this blog, then. If you want to spread misinformation elsewhere, that’s annoying, but I can’t do anything about it.
I’m not spreading misinformation — I didn’t tell the story, I used the commonly known cultural image to make a point about something else. You’re being as pedantic as someone who objects to saying “pure as the driven snow” to invoke innocence because driven snow isn’t actually that pure.
You’re derailing for no good purpose.
Total:
Blog comments threads are fun! Anyway, this will be my last reply here. My problem is not with driven snow or whatever, it’s that Kael was a real person who is mocked for something she didn’t say. I don’t think that’s cool. Kael said enough mockable things in her movie reviews—mock that all you want. But using her as an example based on something she never said . . . no, I don’t like that. The good news is there’s a whole wide internet where you can say whatever you want about her.
You want to know what kills people, and you also want to know what kills people prematurely. The things that kill people when they are already very old are still worth tabulating, even if we are mainly interested in *preventable* deaths.
Jonathan:
Fair enough. I guess the solution here is to report the number of deaths but break down these deaths by age and perhaps also by existing health conditions. Qalys can then be seen as one summary of this distribution.
That makes sense. The leading causes of death are things that affect older people more, and that is a good thing. Leading causes of premature death are of more concern.
Dr. Shnerb seems to be committing two classic errors common in physicists. First, deaths are something which empirically exists, whereas quality-adjusted life years are a calculated, counterfactual metric (yes, you can get philosophical about the “cause” part of cause of death). Calculated, counterfactual numbers are more vulnerable to fiddling to show what the person who generates them wants them to show. I hope we have all been studying the issues with excess deaths: they are a very useful metric for testing official statistics, but because they are hypothetical and calculated, there are all kinds of ways to make them higher or lower. The universe does not try to trick us, but people who present us with statistics usually do, so we have to take precautions.
Second, most people don’t think about QALY, but they do think “three of my friends have died or X, and one of them was really young.” If the purpose of your statistic is communication to the public, “deaths” is likely to be more meaningful to the public than life-years.
> First, deaths are something which empirically exists, whereas quality-adjusted life years are a calculated, counterfactual metric (yes, you can get philosophical about the “cause” part of cause of death).
Thanks for this framing.
On top of the statistical questions, and regardless of the problems with vulnerability with these metrics, I also find I have a visceral reaction that they just seem more abstracted. Maybe that feeling reflects a kind of bias or ignorance or irrationality – I’m not sure…but I do suspect that’s why speaking in terms of lives lost rather than in years of life lost or qalys (despite that they are ultimately more informative) is so much more common – it feels more easily accessible.
Maybe its not the best framing for smoking, because smoking causes a variety of health problems which can cause disability or death, but for something like automobile accidents or infectious diseases its relatively straightforward to decide “did she die of it or not?” (there are tricky cases eg. someone who is in an accident, gets sent to the hospital, acquires an infection in hospital, and dies of it).
But you can not observe that Aunt Lee would have lived another 14.2 years if she had never smoked, you can just observe that on average smokers die early and have more health problems than non-smokers. How long she would have lived if she had not smoked is a counterfactual. Sometimes people have unhealthy habits and live a long time, and sometimes they have healthy habits and die young, and sometimes a war or a disease drastically changes how many in a group die.
A related story:
But increasingly the American mortality anomaly, which is still growing, is explained not by the middle-aged or elderly but by the deaths of children and teenagers. One in 25 American 5-year-olds now won’t live to see 40, a death rate about four times as high as in other wealthy nations. </illk
It’s Not ‘Deaths of Despair.’ It’s Deaths of Children. https://nyti.ms/3KfOcpF
This sentence makes no sense to me
The speaker adjusts for the number of years someone has left, because that’s a large part of what people actually care about. Just reporting the death count is also misleading because readers are assuming some average QALY implicitly.
If I say pentobarbital kills 355000 dogs a year, is that really being more honest?
I agree that the analysis gives degrees of freedom to manipulate, and so the raw data and the collection process should be transparently available. But the unprocessed data is not in any way more honest and can be used for what essentially amounts to lies just the same.
The universe does not report data at all, processed or not. Only people do
somebody: many cultures of science have a strict culture of honestly reporting data. This speeds up their work, but it means that when they encounter people outside this culture, they are easily fooled. One of the best-known examples is the postwar parapsychologists who were fooled with simple sleight of hand, another is the Reinhart and Rogoff study of national debt and GDP where as soon as someone looked closely at the spreadsheet it was clear what had gone wrong (but too many people did not check, because they trusted that R&R had collected data correctly and done their arithmetic correctly).
Even if you collect the data yourself, quasars don’t try to fool the people measuring them. People do.
If I only report the spectrum of a quasar without a redshift correction, or if I report timestamps from GPS satellites without relativistic corrections, I am being more misleading than if I made those corrections.
If I report the curvature of the Earth as measured from raw satellite photos, without corrections for the wide angle lens distortion, people will conclude that the Earth is flat. Is that more honest?
somebody: honest scientists record any ways in which they have transformed or calibrated their data eg. reporting radiocarbon dates in years calBCE and explaining which curve they used.
I agree wholeheartedly. I still think it’s good to report the corrected values as well, and that corrected values can be more true.
” The universe does not try to trick us, but people who present us with statistics usually do, so we have to take precautions. ”
wins the thread
I am reminded of a dear friend of mine (still alive, BTW) who when told that every cigarette she smoked reduced her life by six minutes, would reply: “Sure. But those are the minutes at the end.”
I realize this is a tangent, but it’s provoked by the end of the OP. In some small corners of the world, you can get people agitated by promoting QALY’s over DALY’s or vice versa. If I had to choose I’d go with DALY. Are there strong opinions in this comment crowd?
Whist I agree with Andrew “I think Qualy is a useful measure. Imperfect, like just about all measures, but useful”, I still have a problem with them. The paper below seems to tease out and suggest ways forward.
How about Andrew and Peter D do a contest here, over years, on “Statistics, Reporting, Replication & Improvement of Life Measures?”. And reporting metrics as per Nadav Shnerb to enlighten “an essential failure in the way statistical facts are presented to the public, ” Thanks.
And I still want to know what outliers were dumped when developing QALYs & DALYs.
*
“Comparing the cost-per-QALYs gained and cost-per-DALYs averted literatures”
Version 2. Gates Open Res. 2018; 2: 5.
Published online 2018 Mar 5.doi: 10.12688/gatesopenres.12786.2
…
“Discussion
…
“Our data also indicate inconsistencies between literature coverage and disease burden. Some diseases and conditions (e.g., cardiovascular disease and mental health in Southeast Asia, South Asia and Oceania) are relatively “under-studied,” while other diseases and conditions (e.g., HIV and TB in all regions) are relatively “over-studied”.
“There is no clear explanation for these inconsistencies. As we have noted elsewhere, decisions to fund or conduct economic evaluations reflect not just the disease burden imposed by the targeted condition, but also the number of promising interventions or programs 19,20. Because specialty drugs for diseases such as cancer represent important new interventions in high-income countries, and because pharmaceutical companies have the resources and incentive to characterize value for those interventions, much of the cost-per-QALY literature has recently focused on specialty drug therapies. These financial incentives are less pronounced in the lower- and middle-income countries that are much more the focus of the cost-per-DALY literature. In addition to disease burden, priorities in the cost-per-DALY literature may reflect the visibility and emotional salience of diseases, the influence of advocacy groups, the vagaries of reimbursement decisions 19, and institutional priorities of the organizations sponsoring the research.
“In any case, the incongruities we observed between literature coverage and disease burden raise important questions about opportunities for the re-direction of future CEA research funding so that resources for such research can generate the highest return on investment.
…
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5801595/
*
Note re technical vs layperson jargon. The paper above mention CEAs.
“… QALY-based CEAs”
“cost-effectiveness analyses (CEAs)
“A CEA test measures the level of carcinoembryonic antigen (CEA)”
It is a tricky world.
Before newspaper headlines.
QALY’s and DALY’s are used by different constituencies. When I was in that world a while back, the former were in the World Bank stable, the latter in WHO. I was closer to WHO so I used DALY’s.
What the OP put in al caps “THE DECREASE IN THE YEARLY DEATH TOLL ONE EXPECTS IF THERE WERE NO SMOKERS AT ALL” is the population attributable risk fraction: p*(RR-1)/(1+p*(RR-1)) where p is the proportion of the population with the risk factor and RR the relative risk of death.
The denominator is total deaths, a baseline risk (1) plus excess risk (RR-1) among the proportion p that smoke; the numerator is the excess risk from smoking.
The RR of overall mortality in smokers is about 2.5 (from Doll’s British doctors cohort). If p=0.2, then PAF is (0.2*1.5)/(1+0.2*2.5) = 0.23
right, PAF is (0.2*1.5)/(1+0.2*1.5) = 0.23
the Doll reference: Doll R, Peto R, Wheatley K, et al.. Mortality in relation to smoking: 40 years’ observations on male British doctors. BMJ 1994; 309: 901–911.