Pointing to this news article by Megan McArdle discussing a recent study of Medicaid recipients, Jonathan Falk writes:
Forget the interpretation for a moment, and the political spin, but haven’t we reached an interesting point when a journalist says things like:
When you do an RCT with more than 12,000 people in it, and your defense of your hypothesis is that maybe the study just didn’t have enough power, what you’re actually saying is “the beneficial effects are probably pretty small”.
and
A good Bayesian—and aren’t most of us are supposed to be good Bayesians these days?—should be updating in light of this new information. Given this result, what is the likelihood that Obamacare will have a positive impact on the average health of Americans? Every one of us, for or against, should be revising that probability downwards. I’m not saying that you have to revise it to zero; I certainly haven’t. But however high it was yesterday, it should be somewhat lower today.
This is indeed an excellent news article. Also this sensible understanding of statistical significance and effect sizes:
But that doesn’t mean Medicaid has no effect on health. It means that Medicaid had no statistically significant effect on three major health markers during a two-year study. Those are related, but not the same. And in fact, all three markers moved in the right direction. They just weren’t big enough to rule out the possibility that this was just random noise in the underlying data. I’d say this suggests that it’s more likely than not that there is some effect–but also, more likely than not that this effect is small.
The only flaw is this bit:
There was, on the other hand, a substantial decrease in reported depression. But this result is kind of weird, because it’s not coupled with a statistically significant increase in the use of anti-depressants. So it’s not clear exactly what effect Medicaid is having. I’m not throwing this out: depression’s a big problem, and this seems to be a big effect. I’m just not sure what to make of it. Does the mere fact of knowing you have Medicaid make you less depressed?
McArdle is forgetting that the difference between “significant” and “not significant” is not itself statistically significant. I have no idea whether the result is actually puzzling. I just think that she was leaping too quickly from “A is significant and B is not” to “A and B contradict.”
Also I’d prefer she’d talk with some public health experts rather than relying on sources such as, “as Josh Barro pointed out on Twitter.” I have nothing against Josh Barro, I just think it’s good if a journalist can go out and talk with people rather than just grabbing things off the twitter feed.
But these are minor points. Overall the article is excellent.
With regard to the larger questions, I agree with McArdle that ultimately the goals are health and economic security, not health insurance or even health care. She proposes replacing Medicaid with “free mental health clinics, or cash.” The challenge is that we seem to have worked ourselves into an expensive, paperwork-soaked health-care system, and it’s not clear to me that free mental health clinics or even cash would do the trick.
Other perspectives
I did some searching and found this post by Aaron Carroll. I agree with what Carroll wrote, except for the part where he says that he would not say that “p=0.07 is close to significant.” I have no problem with saying p=0.07 is close to significant. I think p-values are often more of a hindrance than a help, but if you’re going to use p=0.05 as a summary of evidence and call it “significant,” then, indded, 0.001 is “very significant,” 0.07 is “close to significant,” and so forth. McArdle was confused on some of these issues too, most notably by mixing statistical significance with a Bayesian attitude. I wouldn’t be so hard on either of these writers, though, as the field of statistics is itself in flux on these points. Every time I write a new article on the topic, my own thinking changes a bit.
I see some specific disagreements between McArdle and Carroll:
1. McArdle writes:
Katherine Baicker, a lead researcher on the Oregon study, noted back in 2011, “people who signed up are pretty sick”.
Carroll writes:
Most people who get health insurance are healthy. They’re not going to get “healthier”.
This seems like a factual (or, at least, a definitional) disagreement.
2. McArdle:
We heard that 150,000 uninsured people had died between 2000 and 2006. Or maybe more. With the implication that if we just passed this new law, we’d save a similar number of lives in the future. Which is one reason why the reaction to this study from Obamacare’s supporters has frankly been a bit disappointing.
Carroll:
This was Medicaid for something like 10,000 people in Oregon. The ACA was supposed to be a Medicaid expansion for 16,000,000 across the country. If 8 people’s lives in the study were saved in some way by the coverage, the total statistic holds.
(16,000,000/10,000)*8*7 = 90,000, so that’s in the ballpark of the claimed 150,000 in seven years. I’m guessing that McArdle’s would reply that there’s no evidence that 8 people’s lives were saved in the Oregon study. Thus, numbers such as 100,000 lives saved are possible, but other things are possible too.
The bottom line
What does this all mean in policy terms? McArdle describes Obamacare as “a $1 trillion program to treat mild depression.” I’m not sure where the trillion dollars comes from. A famous graph shows U.S. health care spending at $7000 per person per year, that’s a total of 2.1 trillion dollars a year. I’m assuming that the Obama plan would not increase this to 3.1 trillion! Maybe it is projected to increase annual spending to 2.3 trillion, which would correspond to an additional trillion over a five-year period? In any case, that sounds pretty expensive. Given that other countries with better outcomes spend half as much as we do, I’d hope a new health-care plan would reduce costs, not increase them. But that’s politics: the people who are currently getting these 2.1 trillion dollars don’t want to give up any of their share! The other half of McArdle’s quote (“mild depression”) sounds to me like a bit of rhetoric. If a policy will reduce mild depression, I assume it would have some eventual effect on severe depression too, no?
Beyond this, I can’t comment. I’m like many (I suspect, most) Americans who already have health insurance in that I don’t actually know what’s in that famous health-care bill. I mean, sure, I know there’s something about every American getting coverage, but I don’t know anything beyond that. So I’m in no position to say anything more on the topic. I’ll just link to Tyler Cowen, who, I assume, actually knows what’s in the law and has further comments on the issue.
Let me conclude where I began, with an appreciation of the high quality of statistical journalism today. In her news article, McArdle shows the sort of nuanced understanding of statistics and evidence that I don’t think was out there, twenty years ago. And she’s not the only one. Journalists as varied as Felix Salmon, Nate Silver, and Sharon Begley are all doing the good work, writing about newsworthy topics in a way that acknowledges uncertainty.
Note for the record, antidepressants have a very poor track record in decreasing depression over a placebo (and possibly extend it in the long run). The notion that the mechanism is Medicaid –> antidepressant scripts –> less depressed is highly unlikely. More likely is Medicaid –> less financial stress or worry about (other) health problems –> less depressed.
That sounds like at most an argument for catastrophic health insurance (i.e., the kind of coverage Obamacare discourages) or cash transfers to the poor, not comprehensive subsidies and coverage for everyone.
Although I have yet to read the academic articles I found the focus on health outcomes confusing.
First, the main goal of a health _insurance_ scheme is to protect from catastrophic expenditures, not to provide “free” health care. Most people should be on high deductible ($10,000 for family of 4?), very cheap insurance, that offers strong protection against catastrophic losses. For routine medical expenses pay out of pocket. My sense is for most people this is often cheaper than a gold plated, no deductible, high premium insurance — even one subsidized by your employer.
Second, you also want the in-network services contracted by insurance to be effective at improving health status. Here the focus on blood sugar, etc.. sets a very high bar. These conditions are often driven by life style choices that the current curative health care system is not good at dealing with. Sure, some innovative insurers are trying to induce behavior change to reduce the risk in their insured pool, but these are few. Besides the science of behavior change is, in my view, still nascent, certainly at scale (e.g. changing the behavior of millions of people at a time). Health insurance may affect health status in more nuanced ways that need to be adequately theorized, but overall I don’t expect huge effects except in very specific circumstances (e.g. someone who died bc could not afford an expensive treatment regimen for years, may have lived longer, and better, had she been adequately insured. But these are fewer cases for rarer conditions)
Third, you (e.g. government regulator, private providers) want patient satisfaction. People might give you credit at elections, or might retain your services.
From my superficial reading of the news it seems the program did well on risk protection and satisfaction, which is what I would expect.
I find the use of “lifestyle choices” an interesting diversion in these discussions. This caveat is brought up over and over especially when conservatives are discussing this issue. This also highlights vacuousness of “high deductible” cheap insurance for this socioeconomic class. They need healthcare,not protection from catastrophic loss. Basically that is the same argument against “lifestyle choices”; no money means no options when it comes to these choices. “The” choice is enough calories to sustain themselves. You try living on $15,000 per year and see how many choices you have.
@dswarts
I think you are lumping too many things together.
Here we are discussing whether a health insurance scheme is effective. My argument is that effectiveness should be measured relative to what the program is trying to do, namely, insure people against catastrophic loss. Another considerations is equity. For that you probably want income-based subsidies to pay the premium, which is what Medicaid does. Mandatory insurance makes sense due to adverse selection.
As you get lower down the income scale towards the bottom 14% of household that earn less than$15,000, people may not be able to afford out-of-pocket expenses, let alone insurance premiums. But this is an altogether different problem from the insurance one. As Jean Tinbergen said, “One target, one instrument”. Insurance is not the way to deal with this problem, and should not be judged in these terms. The problem you allude to is one of equity, and for that you need tax funded health care. I am ok with that, but that is quite different from an insurance program.
Finally, re lifestyle choices I would make the following conjecture. That even if the U.S. NHS style universal tax funded healthcare, it would have very similar rates of diabetes as it does today. Note well: This is not an argument against NHS, but simply a belief that such diseases are hard to deal with in a curative rather than preventive health system.
But we’re not discussing whether a health insurance scheme is effective, are we? We’re discussing whether expanding Medicaid is effective. Medicaid isn’t a catastrophic plan, it’s an all-in plan which just happens to cover people for whom fairly modest levels of expense are fairly catastrophic. Thus, your distinction collapses… catastrophically.
@Jonathan
Thanks for your comment. From Wikipedia:
However, you are right that Medicaid is not purely an insurance program. The access and equity component almost trump any insurance aspects. But, at the margin, as it expands to cover other groups with higher incomes, it is more like insurance, and the effects we see dovetail with that.
I suppose one could say that Medicaid is a “gold plated” (in your all-in sense) insurance scheme for low-income people; or an NHS for low income households contracted out to private “insurers” or providers dressed up as insurance.
However, in my view it is important to keep these distinctions. For example, one can support the expansion of catastrophic insurance for all the population, without at the same time having to support the expansion of the NHS aspect of Medicaid beyond the most financially needy population. Two targets, two instruments.
Finally, to get to Andrews point. What this discussion shows is that even with RCTs is hard to leave the politics out. Among other politics determines how you conceptualize the intervention, what measures are used, how these are interpreted, and whether the evidence is relevant to a particular discourse.
Critics of Obama — from left and right — will criticize the (null) health outcome aspects. Supporters will point to the beneficial insurance aspects.
A marginally statistically significant outcome may not be too different from a marginally insignificant one, but my understanding was that the depression results were not merely marginally significant, but surprisingly large.
Wonks:
Results that are statistically significant tend to be too large; that’s the statistical significance filter (also called Type M errors).
A couple of comments on problems with the study:
1. of the 10,000 lottery winners, only about 6,400 actually ended up on Medicaid. The others presumably decided that they didn’t really need to be on Medicaid that badly. Thus the population that did end up on Medicaid might have had more serious health issues. There are always issues in randomized clinical trials with people dropping out of the study, but this is pretty extreme compared to most studies.
2. With respect to diabetes (they didn’t specify type 1 or type 2, but type 2 diabetes is much more common these days and probably represents 90% of the cases in this data set), the outcome was a binary outcome of “A1c below 6.5%” Even with good care few type 2 diabetics (and even fewer type 1’s) reach this goal. Many more of the type 2 diabetics in the Medicaid group were receiving treatment than type 2 diabetics in the non medicaid group. If you argue that “A1c under 6.5%” is the right outcome to measure, then you’re really arguing that there’s no substantial value in treating type 2 diabetics unless you can get their A1c under 6.5%. Very few physicians would agree with this. (Disclosure- I’m not a physician but I am a person with type 2 diabetes and I’ve been involved in public health efforts relating to diabetes.)
Many other studies have shown that reducing A1c has substantial benefits for patients even at levels far above 6.5% (e.g. 7% is much better than 8% or 9%) You can certainly expect that over the long run (years and decades) you’d see substantially higher rates of heart attack, stroke, blindness, kidney disease and other complications of diabetes among the untreated patients.
Rather than using a binary outcome, the study should have looked at A1c as a continuous outcome.
Brian:
I don’t know from diabetes, but in general I very strongly agree with your point against the practice of taking a continuous outcome and making it binary.
According to today’s NY Times, 38,364 people died from suicide in 2010.
http://www.nytimes.com/2013/05/03/health/suicide-rate-rises-sharply-in-us.html
Presumably, reducing depression could save a fair number of lives.
Responding to your quotes:
“1. McArdle writes:
Katherine Baicker, a lead researcher on the Oregon study, noted back in 2011, “people who signed up are pretty sick”.
Carroll writes:
Most people who get health insurance are healthy. They’re not going to get “healthier”.”
One statement is a statement about the marginal case, ie. people *who signed up* were really sick, vs the other statement about the average case people *who get health insurance* (in general) are healthy. It’s perfectly possible for both to be true, in fact it makes good sense that healthy people can get health insurance cheaper than people who already have severe illness (at least prior to the Obamacare thingy when existing conditions issues were the norm) so that the people who remained uninsured probably did it largely because their cost was very high. When you make the cost equal for all, some of these sick people can now afford it so they sign up.
I read the NEMJ article before realizing it was going to draw so much attention. Some points:
1. It’s not a study. It’s a progress report. A report doesn’t have the same structure as a study. I would hope that’s obvious.
2. If you go through the medians – which are what is reported – you see only the very basic before and after (or rather current) stats for a few measures. Take a look. Blood pressure median was 119/76. Cholesterol was total 201, hdl about 47 or8 (I forget). See the issue? These are good numbers. No matter what you do medically, you aren’t going to lower 119/76 for a reasonably sized group. That’s where you want to be – which raises a bunch of questions about the health going in of the people in the pool.
3. As to what has happened, we can only see that about 6% were identified as hypertensive. And about the same were identified as having high cholesterol. It’s entirely possible – I’d say likely – you’d identify people with worse than the median, treat them and not move the median in this case. Why? Because 201 is a good number, just as 119/76 is a good number. I take blood pressure and cholesterol meds and those are roughly my numbers because that’s about where you can get for most people.
4. There seems to be an assumption – mentioned in the post – that people must have been unhealthy. The data we see, which isn’t much, indicates they aren’t as a group unhealthy and/or that unhealthiness can be reduced without moving the median much. That makes basic numerical sense. I can imagine a large group of people clustered near 119/76 and 201 with not much of a range below but a relatively larger range above – again, because you don’t see many people with better than 119/76, etc. That means improvements in the unhealthy wouldn’t show in the median. I’d love to see more representations of the data.
Since I believe in corrections, I was looking at the article again and it hit me that I referred to median in this comment. I meant mean. There is a difference but the basic point remains. I should read things more carefully. But I took the time and went through the supplemental appendix, which contains much more data. The mean is basically the median, which didn’t surprise me. Not an excuse for misreading.
I note the dominant medications taken in the group are the ones I take, lisinopril and simvastatin, the generics for hypertension and high cholesterol. I still wonder why people expect radically different results.
That was a bit of a joke; I was tweaking the folks who in 2011 were confidently predicting that this round would show major health improvements, and are now claiming to be excited that we’ve finally proven Medicaid doesn’t kill you.
The projected 10-year cost of Obamacare is currently somewhere north of $1 trillion (though as this was offset by Medicare cuts, total spending won’t increase that much. Though many of those cuts may not take place, so perhaps it will.) Total annual spend on Obamacare is expected to be somewhere in the range of $150-200 billion a year after 2014, if I recall correctly.
It’s very hard to tell how many lives you’d save. There was no significant mortality improvement, but with a population under 65 in only two years, I’m not sure we could get that number. If you look at the effects of these, given that they’re the three major chronic diseases, and assume that they’re close to the true figures in the population, I don’t think you could get to 8 people out of the 6400. For hypertension, the number needed to treat to prevent one stroke in five years is between 29 for severe and 118 for mild hypertension, and as with most conditions, mild hypertension is far more common than severe. 85 people in the treatment group were well controlled, compared to the rate in the control group. So (with the caveat that I Am Not A Health Care Statistician) the most likely result is that we prevented 0 to 1 stroke over a five year period. And not all strokes are fatal. There are also heart attacks, but NNT is higher, so again, 0 to 1. That’s about the same as for cholesterol treatment–and again, in both cases, we mean all heart attacks, not just fatal ones.
(I am assuming “well controlled” is a proxy for “treated” which is not quite right, but I think an okay kludge given that we lack more granular measures).
Nor would we expect to see 8 people in this population die of diabetes every year absent insurance. 5% of the control group had high blood sugar, or <300 people. The treatment group fell to 4.1%. That's a difference of around 60 people. Obviously, they couldn't possibly have a diabetes mortality rate enough to significantly contribute to a death rate of 8 people a year. I mean, not for long, anyway.
Of course, the more numerous, but less common, diseases might swamp that. So Austin Frakt is right in one sense: we cannot rule out the possibility that 8 people a year in this group would be kept a live by their Medicaid. But given that they found modest improvements on major indicators, and modest improvements on Framingham risk scores, and that the one year results showed no significant mortality improvement, I wouldn't say this study suggests such a result is likely.
I actually blogged about this debate but I did want to concede one point here. The precise link between insurance and outcomes for chronic conditions involving routine care isn’t as well established as one might think. I suspect that we would find survival rates after major events to be improved by people seeking care more quickly, but it is unclear how many people that might mean. Still, it is concerning that the confidence intervals (interpreted as credible intervals) include a broad range of clinically significant changes. That suggests important things about the data invovled.
I do want to point out that Medicaid is the most cost-efficient means we have of delivering care in the United States. Given that I am skeptical about major reform in the face of entrneched interests, it does make some sense to consider expanding this class of care. it is definitely making a financial difference in the lives of people on marginal incomes.
Finally, I think Aaron Caroll picked up on a point by Tyler Cowen that these results do mean that we should incorporate them into our policy debates.
And, it does seem to finally be good evidence agaisnt the odd claim that medicaid kills people. :-)
The more nuanced discussion of the statistics here and at Daily Beast isn’t reflected in the introduction to the piece: “Bombshell news out of Oregon today….”. If anything, the latter half of the Beast article makes it clear that the study is not even close to a knockout blow against Obamacare, even in regards to overconfident/over-hyped predictions of reduced mortality. “Bombshell” how lame.
I didn’t call it a knockout blow against Obamacare. This is bombshell news simply because it’s a really big study, only the second RCT ever done on the subject, and it didn’t show what folks were expecting . . . and what intuitions would lead almost anyone to expect. I’d call today’s pair of studies suggesting bacterial infections may cause the majority of chronic back pain a “bombshell” too, even though it’s not even a knockout blow against back surgery.
If you get past the misleading summary statement in the paper, the study found better physical outcomes on most, if not all, measures for the medicaid group. It’s just that it’s possible the effects were 0 or negative–not that it’s likely.
Further, as the authors note, the uncertainty is large enough to include clinically significant effects. Your argument that no statistical significance + big study means the effects were small for sure is not correct. Part of the problem is that some of the underlying diseases are relatively rare, so the study isn’t as big as it looks.
The statistics appear to be based on intention to treat, which, given that less than half the people in the intention to treat group ended up enrolling in medicaid, is bound to dilute any measured effects (since you are measuring the effect on a group some of whose members didn’t actually receive the treatment).
There’s ongoing discussion on details of these issues at http://theincidentaleconomist.com/wordpress/power-calculations-for-the-oregon-medicaid-study/.
I didn’t say that they were small for sure. I said that looking at this study, and other studies which have had similar findings, and comparing them to the studies that got larger findings, the overall weight of the evidence makes me think that the effect is small. Otherwise, it would show up more consistently in high-quality studies.
As for intention to treat, I’m going to blog about this tomorrow, but this argument doesn’t make a whole lot of sense to me. Failing to enroll is one way that the treatment (giving people insurance) might be less effective than we’d like. There are all sorts of ways in which the folks who enrolled in Medicaid might be different from the rest of the population; it doesn’t follow that if we’d somehow gotten the remainder enrolled, everyone would be enjoying similarly large effects which would then reach significance.
Some of the discussion in the paper suggests they may have been estimating the effects for people who actually enrolled (first column p 1715), i.e. the numbers are not based on intention to treat. OTOH, some people trying to match their numbers only get there doing a classic intention to treat analysis.
You are correct that it is not safe to assume those who won the lottery (=intention to treat group) and did not end up enrolled would have shown the same pattern if they did enroll. Equally, it is not safe to assume the effect would have been zero for them.
The authors note “the subgroup of lottery winners who ultimately enrolled in Medicaid was not comparable to the overall group of persons who did not win the lottery.” They must have been poorer because of the income screen, and it wouldn’t be surprising if they were sicker. This makes inference quite challenging; I can’t tell from the paper how they dealt with this. (p 1715)
Surely the fact that 40% of lottery winners turned out to be ineligible for Medicaid is not much evidence that “giving people insurance might be less effective than we’d like”. In fact, they weren’t given insurance.
As I said, the point may be moot if the analysis was not a conventional intention to treat analysis.
1. It is not safe to say that the lack of statistical significance + sample size -> effects were small. From p. 1719 of the article “The 95% confidence intervals for many of the estimates of effects on individual physical health measures were wide enough to include changes that would be considered clinically significant.”.
A much better summary of the evidence is that is quite likely enrolling in Medicaid improved people’s health outcomes. The point estimates are positive across the board, and interpreting the CI as a posterior probability gives substantial probabilities of positive effects
The discussion of this has mostly taken a finding which really means “we can’t prove the effect isn’t zero” as “we proved the effect was zero.” Which is ridiculous.
2. The meaning of “significant” is up for grabs. The paper’s abstract just says “significant” even though “statistically significant” is what they mean. The discussion quoted above mentions “clinical significance”. In matters of health policy, there is also “financial significance.” Virtually any effect will be financially significant given our huge spending on health care.
3. As others have noted, none of the physical health “outcomes” are really outcomes that matter, such as death, sickness or disability. The measured “outcomes” only matter because they predict such problems. From that point of view, the mental health outcome is also a predictor of physical health, since depression is associated with physical problems (leaving aside the issue that it is itself a physical problem). So even if one looks only at statistically significant effects, there is a positive effect on physical health through depression (if one accepts the measure of depression).
4. Finally, the outcomes highlighted, blood pressure, diabetes, and cholesterol, are all related to chronic diseases, and US health care is pretty bad at managing those. Treatment of depression is also of limited value (though probably not 0, as some posters claim). If there really is an effect it may be through some channel like insurance -> stress reduction or insurance -> more money -> stress reduction.
‘When you do an RCT with more than 12,000 people in it, and your defense of your hypothesis is that maybe the study just didn’t have enough power, what you’re actually saying is “the beneficial effects are probably pretty small”.’
This is good, not great. It’s implicitly a statement about average treatment effects *in the entire sample*. That’s an interesting thing to estimate, but not relevant for every outcome. When we have to slice and dice to smaller populations (e.g. people with diabetes or hypertension) N can get small quickly. So you might actually be saying that the relevant subpopulation is smal or underrepresented in the sample. And the effect could get worse if we’re looking at improvements over baseline, since the relevant population becomes even smaller – i.e. does it make sense to include diabetics whose disease is well-controlled at baseline, since they can’t really improve? Or is the right estimand something like the average improvement in people whose disease is not well controlled at randomization? Probably both are of interest, I suppose.
In any event, they seem to have enrolled more healthy people than I’d have expected. That’s an important point that doesn’t seem to be geting a lot of play outside wonky outlets.
The more I think about it, the more I am convinced the study is woefully under theorized.
The study is not an indictment of Medicaid expansion, so much as an indictment of poorly theorized RCTs.
Did theyregister a protocol t
The more I think about it, the more I am convinced the study is woefully under theorized.
The study is not an indictment of Medicaid expansion, so much as an indictment of poorly theorized RCTs.
Did theyregister a protocol t
The more I think about it, the more I am convinced the study is woefully under theorized.
The study is not an indictment of Medicaid expansion, so much as an indictment of poorly theorized RCTs.
Did they register a protocol, complete a CONSORT statement checklistk?Those might have higlighted conceptual, theoretical issues early on.
My understanding is that this was not a RCT but a natural experiment that has a strong resemblance to a RCT because of random assignment. I’m not even sure it would be considered ethical as a RCT since control groups are supposed to get the usual standard of care (and the argument that the usual standard of care is no care has not flown for trials in Africa, even though it’s true).
This might be why there apparently aren’t baseline measures, which would certainly be helpful in assessing some of the sample selection concerns people have raised.
Well … Rand did it just fine. And there you had a much broader sample (i.e., not just the poor & sick)
One can write protocols, and use checklists (e.g. STROBE), for observational studies too.
The control roup is not a no care group, they can access care by walking into any Emergency room, they can access free clinics or pay for care or OTC medications. The assumption that no insurance equals no health care is false.
The comparison is between, lets call it, base line health care available in america and that provided by medicaid.
I don’t know that the design would be considered unethical, but I think it’s a possibility. I’ve asked at our school’s ethics blog, http://accelerate.ucsf.edu/blogs/ethics. I don’t think my post has gone up yet.
Ryan: Wasn’t the RAND study in the 70’s? Research protocols have got a lot stricter since then. I imagine the specifics were different; standards may be higher for the poor and the sick, and obviously health care has gotten a lot more expensive. These are significant because it’s considered very important to avoid coercion in inducing people to participate.
Fernando, true, one doesn’t need an RCT to attempt a disciplined study. But if it wasn’t an RCT it’s not quite right to criticize it as a bad RCT.
Leroy: I didn’t assume no medicaid = no care, though you seem to be making fairly rosy assumptions about the alternatives. The control group doesn’t even mean no health insurance; it just means didn’t get Medicaid through the lottery. The question is whether it would be ethical to have a study in which some people got insurance (through the study) and some didn’t. My point in bringing up the African studies was precisely that a condition that is baseline care for a population may be considered unethical even for a control group.
My comment was intended to clarify that the study was probably not an RCT, and secondly to point out the irony that the design might not be considered ethical as an RCT.
Apologies for spam. Writing from phone, small buttons, big thumbs…
I imagine if you did a study of the effects on health/mortality of air pollution reduction over 2 years with a sample of 12,000 you would get no significant differences in any area in the US (maybe in Beijing). By analogy with the argument made against Medicaid from applying this study, we should abandon pollution abatement efforts. A real test will come when states expand eligibility to large number of citizens, and other, comparable states, refuse.
If only the authors had thought of that! Oh, wait… they did. That’s why the specifically selected indices that would be expected to change in the time they had available. But maybe I’m wrong… maybe advocates like Jonathan Gruber really wanted a test that would fail. Not surprising they left that motivation section out of the paper.
Indeed, (16,000,000/10,000)*8 = 128,000 …
Huh?
Oof! That’s pretty embarrassing. I fixed it now; thanks.
I do agree that the journalism and debate are at a higher level than usual. However, there are 2 things that bother me about this study that have not been mentioned (issues about power, sample size, etc. do not bother me – that is part of the elevated statistical debate). First, why is this data not publicly available? Indeed, why is the NEJM willing to publish this without the data being released? There are no privacy concerns with this data, only the authors’ wish to milk the data for all they can (publication-wise). Given the potential importance of this study for public policy, I believe it is irresponsible to publish it without the data being made available.
The study did include extensive supplementary material, but not the raw data. Given the number of questions the study raises, this data is necessary for informed debate to occur. Otherwise, the assumption must be “trust us, we’re smart and famous.” This is hardly a step forward in the public use of statistical analysis.
Second, after reading some of the extensive supplementary material, I was struck by the fact that only 60% of the lottery “winners” actually filled out the paperwork to receive the Medicaid benefits. Some of them dropped out because they were ineligible (income or assets too high) but most apparently just did not complete the application. My hypothesis would be that these people would generally be healthier and that the sample that actually was studies had a selection problem in which their health was generally poorer. This could have important implications for interpreting the study results. Yet, the study appears to not even raise this issue or discuss it. If the data had been released, some further analysis of this issue would be possible – that is part of the reason why it is necessary for the data to be available. From what I can see in the reported materials, the Medicaid sample does appear to have higher rates of diabetes (prior to the study frame) than the non-Medicaid sample. But there is not enough information to say much more than this.