JJ Levine writes:
I wanted to share a thought about the expected changes at the FDA and related agencies in the new administration. I am wondering if many of the controversies in health are really a problem of data aggregation.
Here is an article from the Washington Monthly that talks about the new proposed head of the FDA and some of his controversial opinions.
Aside from one erroneous line (“In statistics, ‘significance’ refers to the likelihood that the results are correct, not that there is a major effect.”), the article seems to be mainly about how to put together evidence from various different studies.
What I am wondering is what can be done to foster decision making at the highest levels that is not based on pull quotes from selected studies but is based on the weighing of evidence from different sources? How would a Bayesian framework deal with these issues?
The linked article, by Merrill Goozner, quotes Marty Makary, the potential incoming leader of the FDA, as writing in his book “Blind Spots” that “Home deliveries triple the risk of infant mortality.”
Goozner follows up on this triple-the-risk statement:
That shocking claim . . . sent me rushing for the footnotes. There were none. So, I searched the medical literature and found a 2010 meta-analysis (an analysis using pooled data from a group of studies on the same subject) in the American Journal of Obstetrics and Gynecology that made that claim. It reviewed data from over one-half million planned hospital or planned home deliveries. However, only 1 of the 12 studies used for the meta-analysis was a randomized controlled trial. The rest were observational studies, some with some without matched cohorts. The low quality of the evidence was blasted by letter writers to the Journal.
I [Goozner also found a subsequent review of that meta-analysis that warned, “the authors’ conclusions should be treated with some caution as they did not reflect all the evidence presented in the review.” . . . the meta-analysis showed no statistical difference in the two groups in perinatal infant mortality (deaths up to 7 days after childbirth).” The alleged “tripling” was only in neonatal infant mortality (deaths up to 28 days after childbirth), which logic suggests would include more deaths not associated with childbirth itself. The critique also pointed out that the data on neonatal infant deaths came from trials that included fewer than 50,000 births — less than a tenth of the meta-analysis’ total review population.
In a long book, anyone can make mistakes, and Goozner writes that many of Makary’s points are very reasonable. Goozner is concerned with Makary’s blind spots:
While Makary digs deep in the medical literature to find studies that justify his opinions, he avoids mention of the hundreds of medical journal studies published throughout his quarter-century career that document the pernicious effects of industry’s role in financing medical research, clinical practice guidelines, patient advocacy groups, continuing medical education, and physician marketing. He ignores how industry-funded research supported the extensive use of drugs and devices beyond the indications included on the FDA-approved label, often causing great harm in the process.
At this point, I should note my own conflicts of interest, that I’ve collaborated with colleagues in the pharmaceutical industry and I’ve received research support in those collaborations.
I agree with Levine that combining information is key here. I don’t see any easy answers. The conflicts of interest in health care are huge. That said, we don’t just have to talk about health care here. The conflicts of interest in military procurement, funding of police departments and prisons, etc., are even bigger, right? And, as Goozner notes, political pressures and conflicts of interest regarding the FDA are not new.
I don’t think this is a statistical matter. There are probably various ways to combine information from different sources, but it is unlikely that a single method dominates and could receive sufficient support from the research community to be established as a standard. Instead, this seems like a political problem. Researchers and experts (in the legal sense) are always selective in their citation of other work. What is required is a process through which poor/biased decisions can be exposed. To some extent this exists in regulatory proceedings. In the current administration, it seems to be gone. What we have is biased and selective choices made by Administration officials, with critiques from the academic community – very evident in both health and environmental policies. There is no forum where the differences are adjudicated – that used to happen in either legislative or judicial processes but seems to have mostly disappeared. What is replacing it is the court of public opinion. I guess that is what happens when the elites are dethroned (which they somewhat deserve, but is replaced by something worse in my view).
Missing bracket after Goozner
I [Goozner also found a subsequent review of that meta-analysis that warned, “the authors’ conclusions should be treated with some caution as they did not reflect all the evidence presented in the review.”
Medical stuff is tricky and that is why I rely on Max Shulman’s all-purpose edict: “Starve a cold and stuff a fever.” So, just to keep up to date, I went to Wikipedia which indicates Shulman had it completely backwards–an in-joke I missed for decades.
https://en.wikipedia.org/wiki/Feed_a_cold,_starve_a_fever
The thing to realize is that NHST measures a wealth/influence weighted collective opinion. So, far from there being some kind of quality control in place to address your concerns, the problematic behavior is being actively incentivized.
Afaict, there is no interest in fixing this. The practitioners rely on guidelines (they have no time for anything else), the patients have no way of knowing anything, and the guidelines are corrupted every year with more NHST misinformation.
Too many people who should know better will stay silent on, or defend, practically anything coming from these organizations.
The progression is clearly away from replication, prediction, and evidence a treatment actually works (rather than only affects a proxy). It is toward removing basic precautions (eg, aspiration after injection) due to “no evidence”. The most recent news was that control groups are unneccesary if they cost too much.
Makary is rather infamous for writing that medical error is the 3rd leading cause of death in the US. The claim was not based on a formal methodology. “Gold standard science” indeed.
I have sparred with Goozner in the past when I worked at PhRMA where I was VP for Regulatory Affairs (and part of that included drug safety). What Goozner and others fail to recognize (or maybe they do and are blind to it) is that FDA receives all the data for all the trials the company has conducted in support of a New Drug Application. Companies spend quite a lot of money in assuring compliance with good clinical practices which are regulatory requirements. Trials are monitored by both the company and spot monitored by FDA. Ultimately, it is the quality of the data that supports whether a drug is approved or not. Most companies will pull the plug early in clinical development if it looks like the drug won’t work or has a safety profile that cannot be managed. Of course companies pay investigators to conduct clinical trials. There is a cost to all of this and one cannot expect otherwise.