The accidental experiment that saved 700 lives

Paul Alper sends along this news article by Sarah Kliff, who writes:

Three years ago, 3.9 million Americans received a plain-looking envelope from the Internal Revenue Service. Inside was a letter stating that they had recently paid a fine for not carrying health insurance and suggesting possible ways to enroll in coverage. . . .

Three Treasury Department economists [Jacob Goldin, Ithai Lurie, and Janet McCubbin] have published a working paper finding that these notices increased health insurance sign-ups. Obtaining insurance, they say, reduced premature deaths by an amount that exceeded any of their expectations. Americans between 45 and 64 benefited the most: For every 1,648 who received a letter, one fewer death occurred than among those who hadn’t received a letter. . . .

The experiment, made possible by an accident of budgeting, is the first rigorous experiment to find that health coverage leads to fewer deaths, a claim that politicians and economists have fiercely debated in recent years as they assess the effects of the Affordable Care Act’s coverage expansion. The results also provide belated vindication for the much-despised individual mandate that was part of Obamacare until December 2017, when Congress did away with the fine for people who don’t carry health insurance.

“There has been a lot of skepticism, especially in economics, that health insurance has a mortality impact,” said Sarah Miller, an assistant professor at the University of Michigan who researches the topic and was not involved with the Treasury research. “It’s really important that this is a randomized controlled trial. It’s a really high standard of evidence that you can’t just dismiss.”

This graph shows how the treatment increased health care coverage during the months after it was applied:

And here’s the estimated effect on mortality:

They should really label the lines directly. Sometimes it seems that economists think that making a graph easier to read is a form of cheating!

I’d also like to see some multilevel modeling—as it is, they end up with lots of noisy estimates, lots of wide confidence intervals, and I think more could be done.

But that’s fine. It’s best that the authors did what they did, which was to present their results. Now that the data are out there, other researchers can go back in and do more sophisticated analysis. That’s how research should go. It would not make sense for such important results to be held under wraps, waiting for some ideal statistical analysis that might never happens.

Overall, this is an inspiring story of what can be learned from a natural experiment.

The news article also has this sad conclusion:

At the end of 2017, Congress passed legislation eliminating the health law’s fines for not carrying health insurance, a change that probably guarantees that the I.R.S. letters will remain a one-time experiment.

But now that they have evidence that the letters had a positive effect, maybe they’ll restart the program, no?

33 thoughts on “The accidental experiment that saved 700 lives

  1. “It would not make sense for such important results to be held under wraps, waiting for some ideal statistical analysis that might never happens.”

    Agreed. And, it is commendable that the code is provided to reproduce the figures and the analysis appears to be done carefully and written up well and the authors have great pedigrees. The only thing missing is the data – not publicly available, of course. After all, it is IRS data and I can imagine all sorts of bad things that could happen if it were made available publicly. But when do we stop rewarding authors for what they did but not holding them responsible for what they didn’t do? The issues this paper addresses are important enough, and subject to enough forking paths, that surely the public has a right to see the actual data.

    I know I am over-reacting. This paper is better than most in a number of respects. However, I think we silently recreate the expectation that the actual data should not be expected. This has been complained about on this blog often – but perhaps not often enough.

  2. This conclusion is a typical NHST logical fallacy.

    And meanwhile, life expectancy peaked in 2014. Apparently due to the opiod crisis and blood pressure medication “poisonings”, both of which are funded by health insurance. This was discussed on the blog a few years ago.

    So I suspect it is also the wrong conclusion.

    • One of the reasons for exclusion is “the taxpayer was observed to die prior to the date of sample construction”. (That doesn’t explain why the mortality is not exactly zero in the first half of 2016 though.)

  3. Three issues come to mind when reading the paper.

    One, the authors claim that the study has a randomized design, for example, “Individuals in the sample were randomly assigned to receive a letter (86%) or to a control group (14%).” As far as I can see, the authors do not appreciate that being sent a letter is not the same thing as the letter being received or even delivered. This may bias the sample. Postal service is less effective in low-income urban and rural areas. Low-income people are less likely to have stable addresses. And even if the letter is delivered and received, low-income people may be less inclined to open it or pay much attention to it in case it is opened. What implications may this have for the study?

    Two, the authors appear to interpret frequentist results in Bayesian terms, “this point estimate is consistent with a range of effect sizes; the upper bound of our confidence interval implies that each month of coverage (on average) reduces baseline mortality among those who enroll in coverage by approximately 2.4%” Is this a reasonable interpretation?

    Three, the authors do not pay much attention effect size but focus on statistical significance. Given the sample size, well-nigh any difference would be statistically significant. Is the supposed effect size substantively meaningful or noisy?
    Relatedly, when the authors do pay attention to effect size, they do not do so in terms of substantive interpretation but in order to dismiss possible sources of upwardly biased magnitudes. They conclude that “Although we cannot rule out these possibilities, we note that neither would predict our observing an effect of the intervention on mortality (i.e., a non-zero intent-to-treat) if none was present” (p. 23). I find this an odd argument.

    • Additionally, if they want to stick with the frequentist interpretation, when they say “To further investigate variation in the observed mortality effect by age, Figure 4 plots the estimated effect of the intervention by 10-year age bin. The figure provides evidence that the coverage induced by the pilot reduced mortality even among the younger group of middle-aged adults we consider (45-54 year-olds)” they should actually say “the coverage induced by the pilot reduced mortality *only* in 45-54 year-olds”.

    • > One, the authors claim that the study has a randomized design, for example, “Individuals in the sample were randomly assigned to receive a letter (86%) or to a control group (14%).” As far as I can see, the authors do not appreciate that being sent a letter is not the same thing as the letter being received or even delivered. This may bias the sample. Postal service is less effective in low-income urban and rural areas. Low-income people are less likely to have stable addresses. And even if the letter is delivered and received, low-income people may be less inclined to open it or pay much attention to it in case it is opened. What implications may this have for the study?

      I’m not sure I understand this particular critique. Can you be more precise about “bias the sample”?

      A randomized controlled trial gives an unbiased estimate of the average treatment effect. The interpretation of this effect is “averaged over , if I send a letter to person X (instead of not send a letter), this will have a x% reduction in their odds of …”. The probabilities of individuals opening/not opening the letter, heeding/not heeding the advice of the letter, etc., are factored into this average treatment effect. (The implied “estimates” of these probabilities going into the average treatment effect are unbiased precisely because of randomization.)

      This is exactly why randomization is such a powerful tool — it allows us to marginalize out these confounding factors in a way that observational data simply cannot. However, we are indeed marginalizing over these factors (opening/not opening, heeding/not heeding, etc.) — therefore we cannot make causal statements about any particular individual, and we often can’t say useful things about other interesting parts of the causal generating process.

  4. As long as Sarah Kliff is mentioned, here is a more recent article of hers:

    https://www.nytimes.com/2021/08/22/upshot/health-care-prices-lookup.html?campaign_id=29&emc=edit_up_20210823&instance_id=38604&nl=the-upshot&regi_id=77532059&segment_id=66999&te=1&user_id=d7e3e90dc8fbbcc2d51df749fc62495f

    which shows how spectacularly consumer unfriendly the health system is in the U.S. For example,

    At Aurora St. Luke’s in Milwaukee, an M.R.I. costs United enrollees …
    $1,093
    if they have United’s HMO plan.
    $4,029
    if they have United’s PPO plan.

    Or,

    At the Hospital of the University of Pennsylvania, a pregnancy test costs …
    $18
    for Blue Cross patients in Pennsylvania.
    $58
    for Blue Cross HMO patients
    in New Jersey.
    $93
    for Blue Cross PPO patients
    in New Jersey.
    $10
    with no insurance at all.

  5. “how spectacularly consumer unfriendly the health system is in the U.S.”

    The comparison with the other industrialized countries is quite amusing. The fine print is different, but the bottom line is everyone (really, everyone) is insured and medical care is affordable and available. To everyone*. The US system is completely and totally insane.

    *: Yes, yes. There are some place with cracks to fall through. But nothing like the US. And not every country covers dental care, although Japan does. And, sure, some places, e.g. Japan, only covers things that are specified to be covered, meaning some of the latest, and all of the unproven quacky, things aren’t covered.

  6. For each 1648 ppl who bought Obama care paying the average cost of $283/mo, one life was saved.

    The cost of saving the one life was: $5,596,608, about $3400/yr per individual.

    Unlike the people skipping out on health care coverage, most of the readers of this blog likely earn well above median income. How many readers of this blog would voluntarily contribute $3400 every year to save the life of a stranger? How many people earning the median income would do so?

    • This framing makes no sense to me. You explain it like you’re paying $3400, knowing in advance that you’ll get nothing out of it but one stranger’s life will be saved. But in actuality, you’re paying $3400 a year for health care. It partially pays for whatever checkups and hospital visits you get, as well as the small probability that your life will the one saved.

      “What if, instead of what it is (paying for health care), it was this other thing (paying to save a stranger). Would that make sense to you?”

      To be clear, I’m not saying Obamacare and individual mandates are good policy, or that this is a reasonable price for health care.

      • +1
        Yes, the framing is exactly the way economists like to show the irrationality of many government policies. A full accounting of what is being paid and what the potential benefits are is necessary. Then, there is the whole matter of willingness to pay…..

      • Wow, you were both quick on that! :)

        Yes, you contribute $3400 for **your own** care.

        You and Dale are both right to point out that the payers in the case almost surely get some benefit from their $3400 cost for health care. But it’s hard to know how much because these are people who are otherwise foregoing health care, so, on average, they assess their health to be strong enough to survive without coverage. With that in mind, once they pay for services, they might use them but that doesn’t mean they get any benefit from them. For example they might get a physical just because they paid for it, but if they’re healthy, what’s the point?

        Let’s try this: we can say that, had that life not been saved, the mortality rate would have been 1/1648, or 60.7 / 100k; the CDC shows the national mortality rate among the same age group was 395.9 / 100k. [Did I do that right? I’m sure you’ll tell me if I didn’t!! :)]. So if I did this right, this group is healthier than average by a long shot, meaning they get a much smaller benefit for insurance than average. Again, I’m sure you’ll let me know if you have objections to this framing. :)

        On that basis, I object to your objections about my framing. As far as I can see it’s quite a bit better than the framing in the paper under discussion, which could have done substantially more to characterize the meaning of this data with ***very little*** extra work.

        BTW, according to the St. Louis Fed, real median income per person was $36,426. Assuming a 47yr working life (age 18-65) at median personal income, it takes the lifetime productivity of 3.29 ppl to save that one life.

        • this group is healthier than average by a long shot, meaning they get a much smaller benefit for insurance than average. Again, I’m sure you’ll let me know if you have objections to this framing. :)

          Yes this is a much better characterization than the previous one.

          BTW, according to the St. Louis Fed, real median income per person was $36,426. Assuming a 47yr working life (age 18-65) at median personal income, it takes the lifetime productivity of 3.29 ppl to save that one life.

          A problem here is that total compensation packages often already involve some kind of health care, and represent a very large share of the total compensation package for low-middle income earners. This is in fact a major driver of the decoupling of wage growth and per-laborer productivity, and income inequality growth. Some nice work on the subject

          https://www.mercatus.org/system/files/Warshawsky-Earnings-Inequality-v2.pdf

          Being that this is not typically an individual choice on the part of the workers, it does behoove us to ask whether or not we’re getting our money’s worth. But that back of the envelope calculation is not quite correct

        • I don’t want to get bogged down in the details – and I don’t want to defend the paper (I have my own issues with it). But I would propose a few “facts:”
          1. All health care policies will be redistributive to an extent. Healthier people will subsidize less healthy people if there is anything that can be legitimately called a public health policy.
          2. Mandatory purchase of health insurance is sure to make some people worse off. The proper accounting is quite complex – it should include potential direct benefits (and costs) of care received due to the insurance that would otherwise not be realized as well as indirect benefits attributable to helping others (altruistic motives). If such an accounting were possible, no doubt some people would pay more than the insurance is worth – that is why it would need to be required (otherwise these people would not make the purchase).
          3. The framing I would still object to is the reductionist logic that reduces the net value of the decision to $x/life or /life-year. This reductionist approach can sometimes be helpful, particularly when the financial impacts of a policy are far from clear. However, when used to make a political point (e.g., the policy is bad because you’d have to value each life saved at $Z for it to make sense), it ends up denying any consideration other than the narrow financial. It brings us back to the endless debate about whether it is necessary to reduce all benefits and costs to a single dimension in order to make “rational” decisions. I still do not agree with that position.

      • I don’t understand why people do this. If you saved the money in a bank account or just put it in some index fund/etf you would be far better off.

        Then when you go to pay cash the price will be 1-50% whatever the insurance would get “billed” (the chargemaster price used for your deductables, etc).

        The whole thing is a scam. I can see cheap catastrophic insurance but I’m not sure it exists anymore. It is like you have to pay $100/mo for cable just to get family feud.

        • And then there is the problem that the typical NNT of a treatment is ~10-100.

          So its like paying for $100/mo for cable just to get family feud when you only enjoy around one episode each season.

          People are impoverishing themselves over this, it seems totally irrational.

        • I don’t understand why people do this.

          It’s not really an option to not do that unless you’re self employed

        • 1. Many times you’ll be required to sign a waiver and later provide proof that you’ve actually obtained your own insurance.
          2. Many employers provide a “free” (PPO 500 or some crap) health insurance plan, so you don’t actually recoup any costs by declining anyways

        • Yes, health insurance as it works in the US is a bad system. That is my strongly held position, and you don’t need these back of the envelope calculations to see many of the problems. We have a system designed to promote private insurance through your employer – but the contract is between the employer and the health insurer, not with the insured employees. They, in fact, never see the contract at all, and only realize this when there is a dispute. The prices that exist in the system then vary – Medicare is dirt cheap and can only survive financially by shifting costs to those with private insurance. People paying out of their own pockets are worst off – unless they can negotiate their own prices (which they often can). Before the ACA there was not catastrophic insurance at all – policies has a maximum lifetime benefit. But the number of “routine” treatments included have continually been increased. We seem to love prepaying for routine care (which means we are paying the administrative costs of that system on top of the “expected” routine care costs that are covered (the same way that dental or vision insurance is not really insurance at all, but a prepayment system).

          The defense of this prepayment is that people would avoid getting this routine care (which, arguably, is relatively cheap and effective) if it were not covered. I guess there is truth in that, although this is one area where I am sympathetic to the libertarian viewpoint. Insurance should be for high consequence events that are individually unlikely but where the risks can be pooled across great numbers of people. The US system is almost the opposite – it “covers” risks that cannot really be pooled, and which are relatively affordable. That prepayment for service comes with a hefty price tag – all of the administrative costs involved with our complex system.

          The only rationale I can see for our competitive market approach to health insurance is the usual benefit of markets – to promote efficiencies and better serve consumer preferences. It is hard for me to see any evidence that these happen with health insurance. Insurers spend more energy competing by trying not to insure sick people than they spend trying to provide services more inexpensively. That is not to say that a centralized insurance system (i.e., single payer) works perfectly – it has its own problems, but at least they are different than the ones we have now.

        • 1. Many times you’ll be required to sign a waiver and later provide proof that you’ve actually obtained your own insurance.
          2. Many employers provide a “free” (PPO 500 or some crap) health insurance plan, so you don’t actually recoup any costs by declining anyways

          So the problem is people do not want to negotiate their employment contracts. Do they realize it can be the equivalent of a 5-10% raise?

          And in the 1st case I am sure people can find a loophole.

        • @Dale

          The way I look at it is that health insurance != healthcare != health.

          I haven’t had insurance for nearly a decade and had one somewhat major health issue I paid cash to deal with. I did get 50% off the insurance price, and would not have had that cash if I hadn’t avoided giving it to insurance companies. I was also told insurance would likely not have covered my needs anyway.

          Also, the subset of “healthcare” I would bother with is very short. It mostly consists of interventions that have barely changed since the 1940/50s. I don’t want the rest of it for myself or anyone else.

          I happen to be able to study the literature for myself, but even if you can’t you should still be able to understand. It amounts to getting health advice from people who have no idea what a p-value is but use them to decide everything, what are the chances they actually know what they are doing?

        • Anoneuoid
          I’m generally a very skeptical and pessimistic person – but you put me to shame. Your views are too extreme for me. I’ve had some good health care providers and I view many of the developments as real improvements. I may even end up attributing them with lengthening my life considerably (unless they cut it short due to unnecessary or flawed treatment!). So, yes, I believe the system is riddled with errors and waste – but I also see benefits and improvements. Your experience is just that – yours. We should be able to agree that an anecdote offers little in the way of evidence. My own anecdotes change over time – I used little of the health care system until I suddenly used it. If you had asked me 3 years ago about the value of some screening and treatment I would have given a different answer than today. Anecdotal evidence is particularly troublesome when it comes to health care.

        • It does not really have anything to do with anecdotes.

          Everyone should be free to make their own decisions about their own health. But centralizing it moreso will only force me to pay for what I consider to be largely a waste and often harmful.

          Eg, there is no reason for 80 million people in the US to be on blood pressure medication when *even the evidence provided by the advocates* shows it is only benefiting ~1 million of them.

          I have looked closely at nutrition (low salt/cholesterol/fat diets, the RDAs for various vitamins and minerals), Alzheimer’s, cancer, and stroke/TBI/spinal injury. Probably some others as well.

          In all cases I came to the conclusion that the wrong path has been followed leading researchers far away from the correct understanding. They then go on to (mis-)inform doctors about which intervention to choose.

          It is in everyone’s best interest to stop funding this current system.

        • the RDAs for various vitamins and minerals

          I was thinking more about how crazy this one is. The RDAs are based on the needs of (primarily young) healthy people.

          Vitamins and minerals are basically the raw materials your body needs to repair itself.

          Just like a town that gets hit by a tornado or whatever disaster, is it not obvious that the town now requires more than the usual amount of raw materials to repair the damage? And not just one raw material at a time but whatever is needed all together?

          The NHST/EBM paradigm appears to be incapable of figuring out obvious stuff like this. It would take thousands of years to brute force the problem of how to rebuild a town using NHST.

    • “For each 1648 ppl who bought Obama care paying the average cost of $283/mo, one life was saved.”

      That’s not even approximately what the paper said. The relevant quote is from p.2: “The rate of mortality among previously uninsured 45-64 year olds was lower in the treatment group than in the control by approximately 0.06 percentage points, or one fewer death for every 1,648 individuals who were sent a letter.”

      The treatment group consisted of those who were sent a letter, and the control group consisted of those who didn’t get sent a letter. Inevitably, a lot of people in the control group didn’t wind up getting insurance, and some of those in the control group did wind up getting it. If you could make a direct comparison between those who got insurance and those who didn’t, the difference in mortality would probably be much greater than 0.06 percentage points. On p.3, they mention a rough estimate of 0.17 percentage points per month of coverage. Yes, that’s per month of coverage rather than per year of coverage.

      Since they were looking at mortality over a two-year period, a crude extrapolation suggests that over a single year having coverage reduces mortality by 0.5*12*0.17 percentage points, i.e. by roughly 1 percentage point.

      Applying that to the Affordable Care Act and accepting your average cost number of $283 per month, it follows under your own reasoning that the cost of saving one life is around $283,000.

      • That cost isn’t quite correct in that you need to factor in any savings from actually having insurance in the insured group. If 1000 people get insurance, then some of them are going to get necessary medical care for a lot cheaper than they would have without the insurance. Our pet troll only got sick once in 10 years, but one car accident, deteched retina, or other emergency surgery (which there will be in those 1000 patient-months) will run through that $283,000 real quick in the US.

  7. “…the rate of mortality among previously uninsured 45-64 year-olds was lower in the treatment group than in the control by approximately 0.06 percentage points, or one fewer death for every 1,648 individuals in this population who were sent a letter.”

    Is this a reasonable scale? Back of the napkin math puts it at 1 in 1648 is 60 per 100,000. I’m assuming this was over the two year period, so call it 30 per 100,000 per year. That puts “uninsured” as the #5 leading cause of death for 45-54 year olds, between accidents at 50/100,00 and chronic liver disease and cirrhosis at 20.5/100,000. In the 55-64 age group, it would be #7 just edging out chronic liver disease (29.6/100,000) and a little behind cerebrovascular diseases at 32.5/100,000.

    That seems like an _extremely_ large response: Going uninsured for just two extra years results in enough excess deaths to put it well within the top 10 causes of death for that age category? Especially when the rest of the literature is finding no or only a small effect.

    I used the CDC’s 2015 LCWK2 table: Deaths, Percent of Total Deaths, and Death Rates for the 15 Leading Causes of Death in 10-year Age Groups, by Race and Sex: United States, 1999-2015, page 4 “All races, both sexes, 45-54 years” and “All races, both sexes, 55-64 years.” (Not sure if that is the best table, but 30 deaths per 100,000 is in the top ten for every table I’ve looked at except for maybe 85+ year olds. I also couldn’t find a chart with 20 year age bins.) URL: https://www.cdc.gov/nchs/data/dvs/LCWK2_2015.pdf

    • “That seems like an _extremely_ large response: ”

      Maybe. But that’s _all-cause death_ vs. specific causes you are comparing. (An accident that wouldn’t have been fatal might be without timely care, even for minor injuries. Maybe that tetanus shot at the ER was for a reason. Ditto for cancer.) In a cohort that knows enough to go to the doctor when they’re sick or injured and hesitant to do so when they don’t have insurance, that failure to get timely treatment (including preventative and screening sorts of care) is known to result in a lot of unnecessary morbidity and mortality. This is old news. So this study is just trying to put numbers on known stuff.

      Anyway, in principle, being able to get medical treatment when you need it at a price you can easily afford is a good thing that you’d think society would attempt to provide. One would think…

Leave a Reply to David in Tokyo Cancel reply

Your email address will not be published. Required fields are marked *