Statistical Modeling, Causal Inference, and Social Science

Bill Harris points to this press release, “Almost 400 medical practices found ineffective in analysis of 3,000 studies,” and asks:

The intent seems good; does the process seem good, too? For one thing, there is patient variation, and RCTs seem focused on medians or means. Right tails can be significant.

This seems related to the last email I sent you (“What if your side wins?”).

From the abstract of the research article, by Diana Herrera-Perez, Alyson Haslam, Tyler Crain, Jennifer Gill, Catherine Livingston, Victoria Kaestner, Michael Hayes, Dan Morgan, and Adam Cifu:

Through an analysis of more than 3000 randomized controlled trials (RCTs) published in three leading medical journals (the Journal of the American Medical Association, the Lancet, and the New England Journal of Medicine), we have identified 396 medical reversals.

I’m not sure what to think about this! I’m sympathetic to the aims and conclusions of this article, but I can see there can be problems with the details.

In particular, what qualifies as a “medical reversal”? From the linked article:

Low-value medical practices are medical practices that are either ineffective or that cost more than other options but only offer similar effectiveness . . . Medical reversals are a subset of low-value medical practices and are defined as practices that have been found, through randomized controlled trials, to be no better than a prior or lesser standard of care. . . .

The challenge comes in when making this judgment from data. I fear that pulling out conclusions from the published literature will lead to the judgment being made based on statistical significance, and that doesn’t seem quite right. On the other hand, you have to start somewhere, and there’s a big medical literature to look through: we wouldn’t want to abandon all that and start from scratch. So I’m not quite sure what to think.

Jon Baron points to this news article by Christopher Rowland:

Pfizer had clues its blockbuster drug could prevent Alzheimer’s. Why didn’t it tell the world?

A team of researchers inside Pfizer made a startling find in 2015: The company’s blockbuster rheumatoid arthritis therapy Enbrel, a powerful anti-inflammatory drug, appeared to reduce the risk of Alzheimer’s disease by 64 percent.

The results were from an analysis of hundreds of thousands of insurance claims. Verifying that the drug would actually have that effect in people would require a costly clinical trial — and after several years of internal discussion, Pfizer opted against further investigation and chose not to make the data public, the company confirmed.

Researchers in the company’s division of inflammation and immunology urged Pfizer to conduct a clinical trial on thousands of patients, which they estimated would cost $80 million, to see if the signal contained in the data was real, according to an internal company document obtained by The Washington Post. . . .

The company told The Post that it decided during its three years of internal reviews that Enbrel did not show promise for Alzheimer’s prevention because the drug does not directly reach brain tissue. It deemed the likelihood of a successful clinical trial to be low. A synopsis of its statistical findings prepared for outside publication, it says, did not meet its “rigorous scientific standards.” . . .

Likewise, Pfizer said it opted against publication of its data because of its doubts about the results. It said publishing the information might have led outside scientists down an invalid pathway.

Rowland’s news article is amazing, with lots of detail:

Statisticians in 2015 analyzed real world data, hundreds of thousands of medical insurance claims involving people with rheumatoid arthritis and other inflammatory diseases, according to the Pfizer PowerPoint obtained by The Post.

They divided those anonymous patients into two equal groups of 127,000 each, one of patients with an Alzheimer’s diagnosis and one of patients without. Then they checked for Enbrel treatment. There were more people, 302, treated with Enbrel in the group without Alzheimer’s diagnosis. In the group with Alzheimer’s, 110 had been treated with Enbrel.

The numbers may seem small, but they were mirrored in the same proportion when the researchers checked insurance claims information from another database. The Pfizer team also produced closely similar numbers for Humira, a drug marketed by AbbVie that works like Enbrel. The positive results also showed up when checked for “memory loss” and “mild cognitive impairment,” indicating Enbrel may have benefit for treating the earliest stages of Alzheimer’s.

A clinical trial to prove the hypothesis would take four years and involve 3,000 to 4,000 patients, according to the Pfizer document that recommended a trial. . . .

One reason for caution: another class of anti-inflammatory therapies, called non-steroidal anti-inflammatory drugs (NSAIDS), showed no effect against mild-to-moderate Alzheimer’s in several clinical trials a decade ago. Still, a long-term follow-up of one of those trials indicated a benefit if NSAID use began when the brain was still normal, suggesting the timing of therapy could be key.

Baron writes:

I bet this revelation leads to a slew of off-label prescriptions, just as happened with estrogen a couple of decades ago. My physician friends told me then that you could not recruit subjects for a clinical trial because doctors were just prescribing estrogen for all menopausal women, to prevent
Alzheimer’s. I’m still not convinced that the reversal of this practice was a mistake.

That said, off-label prescribing is often a matter of degree. It isn’t as if physicians prescribed hormone replacement for the sole purpose of preventing Alzheimer’s. Rather this was mentioned to patients as an additional selling point.

Here’s the bit that I didn’t understand:

“Likewise, Pfizer said it opted against publication of its data because of its doubts about the results. It said publishing the information might have led outside scientists down an invalid pathway.”

Huh? That makes no sense at all to me.

Baron also points to this blog by Derek Lowe, “A Missed Alzheimer’s Opportunity? Not So Much,” which argues that the news article quoted above is misleading and that there are good reasons that this Alzheimer’s trial was not done.

Baron then adds:

I find this issue quite interesting. It is not just about statistics in the narrow sense but also about the kind of arguments that would go into forming “Bayesian priors”. In this sort of case, I think that the structure of arguments (about the blood-brain barrier, the possible mechanisms of the effect, the evidence from other trials) could be formalized, perhaps in Bayesian terms. I recall that a few attempts were made to do this for arguments in court cases, but this one is simpler. (And David Schum tried to avoid Bayesian arguments, as I recall.)

It does appear that the reported result was not simply the result of dredging data for anything “significant” (raising the problem of multiple tests). This complicates the story.

I also think that part of the problem is the high cost of clinical trials. In my book “Against bioethics” I argued that some of the problems were the result of “ethical” rules, such as those that regard high pay for subjects as “coercive”, thus slowing down recruitment. But I suspect that FDA statistical requirements may still be a problem. I have not kept up with that.

Conflict of interest statement: I’ve done some work with Novartis.