Bill Harris writes:
Thanks for posting my question the other day.
Here’s another, somewhat related question.
What if “your side” wins? What if, starting today, every analysis is done properly? Null hypothesis significance testing is something you read about only in history of statistics books. When binary decisions are made, they are supported with real decision analyses, not NHST.
There’s a lot of current research based on NHST. What would you do with that?
Researchers, I presume, would feel free to redo any NHST work they thought was both important and realistic for them to work on.
But what about practitioners? For example, in medicine and healthcare, what do you propose doctors should do with drugs and procedures justified with NHST? What should patients who read about this transition in statistical practice do?
I suspect current research results are not all bad (I certainly hope that’s the case), but it seems that some certainly is. How do practitioners and patients discern the difference? Are there useful rules of thumb?
My response: Whatever or not is done by medical research in the future, the problem you describe is already here. The quick answer is that what’s important from each experiment are the design, measurement protocols, and raw data, not the published conclusions. To the extent that all we have are the conclusions and some incomplete descriptions of measurement and design, we kind of need to reconstruct what the data could have looked like. From a purely practical perspective, my answer to your question is that we’ll do what we always do, which is to trust the best judgment of doctors and authors of medical textbooks and review articles.
I like that Harris confronts you/us with a purposely provocative question. Here’s another one: Is it researchers’ role to determine how practitioners make their decisions? Put another way, it’s hard enough to reform attitudes and methods among researchers, must we also take it upon ourselves to change how those outside our field use that research? Is that even a realistic option?
If NHST suddenly disappeared, the demand for it would not. The FDA would still require generics to pass a TOST (two one-sided statistical test) of equivalence. Organizations like Cochrane and the Campbell Collaborative wouldn’t shut down, nor would the Dept of Ed’s What Works Clearinghouse. Indeed, once we start publishing raw data, these and similar practitioner-focused orgs will be able to conduct their own NHST analyses and meta-analyses. New journals will pop up–or old ones will be co-opted–to “translate” scientific findings for practitioners.
Which is not much different from the hard sciences–quantum physicists don’t dictate how Intel uses their research to build semiconductors. But the consequences of using good science badly are different in the hard sciences: if Intel relies on p < alpha to design a chip that doesn't work, they'll figure out that it doesn't work before mass producing it, or else they'll go out of business when its failure rate far exceeds alpha. (hence "six sigma") But in the social and health sciences, it's often not obvious when policies fail or even do harm, sometimes for decades after their initial implementation, and getting them undone can be a quagmire.
This conundrum is another reason we can't reform in a vacuum (the other being that researchers depend on funding from institutions that demand NHST results). We have to somehow persuade practitioners to champion related reforms in their own fields, while at the same time prioritizing reform within our locus of control. It's a messy, but worthwhile, process.
Real clinicians need to understand NHST and other tools in order to achieve certain goals. One has to be able to grasp the value of an outcome and not be mislead by p-hacking and modest positive results hyped to the skies. One must overcome that wonderful feeling that hits when one is a 28 year old in the second year that one knows everything, and everything is coincidentally congruent with one’s economic interest. Then one has to be able to convey the story, the whole story, to a patient who is often frightened and carries a bundle of prejudices based on rumors, fears, and misinformation.
None of this is easy.
Yeah, second year syndrome is definitely a thing.
There’s also the clinician-researchers seduced into over-signifying their own results (early goal directed therapy for sepsis as an old example, tranexamic acid for haemorrhage as a current one).
Not forgetting public health policy makers who’ll jump on a band-wagon before they know where it’s going (sepsis bundles).
And the patients who having spent an hour or two googling arrive with the most outlandish ideas (anti-vaxxers).
But it does tend to come out alright in the end. There is a core who want to do the best got the patients (rather than the best for the drug company/their bank balance/their career), who carefully and logically sift the evidence. I come across these clinicians in hallways and on blogs but I rarely see them writing editorials in The Lancet or keynoting conferences.
I’m skeptical about a lot of research but not about most people.
” I’m skeptical about a lot of research but not about most people.”
+1 but bad research will pollute the efforts of good people.
Everyone would win. Its like saying there is a safe, simple, cheap method for curing all cancer with proponents vs skeptics who would be embarrassed that they were going about things so wrong.
When all cancer is cured, all will benefit. Even the naysayers.
Imagine you are a private cancer hospital currently handling about 300 patients who on average last about 1 year before dying and there is a continuous turnover so you always have about 300 patients at any time. you do a variety of procedures, these cost about $1M per patient, and you make about 25% profit on this. So you are making about $75M/yr on this business.
Now Dr Miri Werker invents an immune system based treatment, and because she is a devoutly religious person she licenses this patent for $1 to any organization that wants to produce the drug. An Indian company begins producing this miracle drug for $200/dose. After 5 doses given one per month, 90% of cancer patients experience complete remission, regardless of type.
Your hospital goes from a $75M/yr profitable business to a business that bleeds money like someone in a horror film.
What incentives do you think there would be for organizations like this to try to discredit this miracle cure? To make it seem dangerous, maybe some might even go so far as to intentionally give tainted doses or lobby the government to eliminate the possibility for the Indian company to produce this drug on the basis that it’s not well enough controlled in manufacture, no we need to require only one special company here in the US with an ultra fancy expensive manufacturing facility to produce this drug and sell it at $800,000 a dose… etc etc.
I’d say DMSO and vitamin C both got that treatment. Essentially if something appears to show promise for being cheap, safe, and effective a sham clinical trial is run to “debunk” it (eg, alter the protocol to give vitamin c once a day orally… which no one ever thought would work… vs IV) or make a big deal out of a minor side effect at extremely high doses to shut down further research along that path.
https://www.ncbi.nlm.nih.gov/pubmed/1055563
https://www.ncbi.nlm.nih.gov/pubmed/7321921
Also, regarding:
Think of it all as preliminary data and speculation.
+1