Is it ok to pursue underpowered studies?
This question comes from Harlan Campbell, who writes:
Recently we saw two different about commentaries on the importance of pursuing underpowered studies, both with arguments motivated by thoughts on COVID-19 research:
COVID-19: underpowered randomised trials, or no randomised trials? by Atle Fretheim
and
Causal analyses of existing databases: no power calculations required by Miguel HernánBoth explain the important idea that underpowered/imprecise studies “should be viewed as contributions to the larger body of evidence” and emphasize that several of these studies can, when combined together in a meta-analysis, “provide a more precise pooled effect estimate”.
Both sparked quick replies:
https://doi.org/10.1186/s13063-021-05755-y
https://doi.org/10.1016/j.jclinepi.2021.09.026
https://doi.org/10.1016/j.jclinepi.2021.09.024
and lastly from myself and others:
https://doi.org/10.1016/j.jclinepi.2021.11.038and even got some press.
My personal opinion is that there are both costs (e.g., wasting valuable resources, furthering distrust in science) and benefits (e.g., learning about an important causal question) to pursuing underpowered studies. The trade-off may indeed tilt towards the benefits if the analysis question is sufficiently important; much like driving through a red light on-route to the hospital might be advisable in a medical emergency, but should otherwise be avoided. In the latter situation, risks can be mitigated with a trained ambulance driver at the wheel and a wailing siren. When it comes to pursuing underpowered studies, there are also ways to minimize risks. For example, by committing to publish one’s results regardless of the outcome, by pre-specifying all of one’s analyses, and by making the data publicly available, one can minimize the study’s potential contribution to furthering distrust in science. That’s my two cents. In any case, it certainly is an interesting question.
I agree with the general principle that data are data, and there’s nothing wrong with gathering a little bit of data and publishing what you have, in the hope that it can be combined now or later with other data and used to influence policy in an evidence-based way.
To put it another way, the problem is not “underpowered studies”; it’s “underpowered analyses.”
In particular, if your data are noisy relative to the size of the effects you can reasonably expect to find, then it’s a big mistake to use any sort of certainty thresholding (whether that be p-values, confidence intervals, posterior intervals, Bayes factors, or whatever) in your summary and reporting. That would be a disaster—type M and S errors will kill you.
So, if you expect ahead of time that the study will be summarized by statistical significance or some similar thresholding, then I think it’s a bad idea to do the underpowered study. But if you expect ahead of time that the raw data will be reported and that any summaries will be presented without selection, then the underpowered study is fine. That’s my take on the situation.
Some of the best papers I’ve ever read are n=one/some data. It allows you to really carefully observe the detailed dynamics of what *can* go on.
Those detailed observations also tell you what needs to be controlled and what kinds of measurements should be taken in any larger study.
Eg, check out this one where the researcher gave himself scurvy and carefully monitored the timecourse of symptoms and various blood tests. He even has some one else (surgically) wound him and they monitor the healing:
https://www.nejm.org/doi/10.1056/NEJM194009052231001
I find that paper 10,000 times more informative than a large RCT that reports the percent of people with scurvy in each group or whatever. Such studies are just such fertile grounds for scientific and medical progress.
Barry Marshall famously did an n=1 study where he infected himself, which gave him an ulcer. It was a key part of his discovery of the true cause of ulcers, for which he won a Nobel.
A few years ago I read a book about historical examples of self-experimentation and other low-n experiments. Very interesting. And yes, it’s often possible to learn from it, although of course there’s the issue of how much one can generalize.
I’ve made this point a few times in posts about exercise and weight loss. There are many studies about the effects of exercise on weight loss, although as a commenter pointed out most of them are small and have various methodological problems, and the current consensus seems to be: if you get a bunch of slightly overweight people to exercise, on average they will not lose a substantial amount of weight. And yet, if I stop doing high-intensity exercise, I start gaining weight at about a pound every week or two, and once I resume high-intensity exercise — getting my heart rate very close to its maximum a few times per session, at least two sessions per week — I lose about a pound a week until I’m back in my moderate-fitness window. The cautionary tale here works both ways: if I generalized from my own experience, I’d be making an error: on average, people don’t lose weight by exercising. But if I assumed the average experience applied to me, I’d be making the opposite error, falsely assuming that _I_ would not lose weight by exercising.
My old school friend Charles Farthing was an early AIDS researcher and in 1997 he declared publicly that he was going to inject himself with the virus.
https://www.scotsman.com/news/obituaries/obituary-dr-charles-farthing-medical-practitioner-1535793
I see it was actually that he said he would inject an inactivated HIV vaccine. But couldn’t find out when this claim was made. In 2014, he passed at 60 from a presumably unrelated heart attack.
Did he actually do it? If so that is very respectable. I have the opinion that people running and funding clinical trials (and/or their family members) should also be participants if it makes sense.
This has played a big role to ensure vaccine safety in the past. The contaminated salk vaccine was covered up until one of the main proponents publicly vaccinated his grandchildren. One died and the other was paralyzed for life: https://www.nytimes.com/1955/05/05/archives/bulbar-polio-kills-doctors-grandson.html
Note the date in bold:
https://en.wikipedia.org/wiki/Bernice_Eddy
Eg, for the covid vaccine there was no reason for the researchers and funders couldn’t also participate. In fact they were in the best position to report the most detailed description of the effects.
In general it would engender trust by the public if this was more common.
I would argue that a confidence interval in these high noise conditions is a perfectly reasonable thing to use. The one caveat is that it isn’t used as an alternative version of a test deciding whether the effect crosses 0 or not. A large effect with a very large CI should be looked at skeptically by the authors and allow them to perhaps claim an effect but necessarily concede that it could be very small and the data are too noisy to reach a strong conclusion.
Should the rare case occur of a study that should be very noisy yielding a small large CI, it behooves the author to look at prior literature and discuss the representativeness of the sample.
For that reason I really don’t like it when such studies are called under-powered when they should really be called high noise or high uncertainty studies. Power only exists as a concept if there really is an effect and all you’re doing is making a decision about it’s presence.
Psyoskeptic:
Agreed. I’m using the phrase “low power” because it’s a standard term, but I agree that “high uncertainty” would be be better.
I actually think the big problem is underpowered models. This is probably particularly true in medicine.
If you have a good predictive model, you can get away with N=1 or some small number of studies. This is pretty obvious in say mechanics. You have an equation for the deflection of a beam under load. You get one beam, measure it carefully, put one or two or three different loads on it, and measure its deflection as a function of distance from the support. You’ll find very good agreement with theory. Do you need to repeat this with a wood beam and a steel pipe and an aluminum bar and a PVC pipe? Maybe, but by the time you’re done with those 15 experiments you’ve validated the model to within epsilon.
That’s not the way we do medicine for the most part these days. Compare that to Louis Pasteur and the invention of the rabies vaccine… he grows the virus in rabbits, weakens it through drying, creates an injectable and treats several people successfully. No randomized controlled trial of 30,000 people etc, just actual science. I certainly think today we should be doing more substantial *safety* tests than he did (there’s some evidence that he gave someone rabies before treating the child with the first successful cure according to Wikipedia, but yeah, at the time they hadn’t any idea what germs really were)
The problem is we’ve taught people that the world **is** a random number generator. That doing experiments in rabbits is the same as generating random numbers and drawing boxes of bolts out of a shipping pallet and determining the distribution and average breaking strength.
It’s NOT, just STOP IT.
With the bolts there’s a definite finite population with fixed characteristics, breaking strength of bolts doesn’t change without fairly dramatic application of some cause (like acid or intense heat), and the population is the 10000 bolts on the pallet and not in general some broader “population” of “future bolts” or whatever.
With people being exposed to disease causing agents or whatever, there’s a dynamic process which is affected by time-evolution. There’s nothing fixed about the strength of bones in a living human for example. Through time more exercise, changes in diet, exposure to viruses, or osteoporosis can change the strength of bone from day to day including trending downward or upward for potentially years to decades.
The idea that we can’t build mechanistic models of medical issues because “it’s too complex” is the worst kind of copout. In fact, every real important advance in medicine has been through mechanistic models. H Pylori and ulcers, HIV antiretroviral drugs, RNA vaccines, anti-osteoclast drugs for osteoporosis, etc etc etc.
These underpowered studies are underpowered precisely because there’s no model involved, and therefore there’s no idea what to expect, what to measure, or what success really looks like at the mechanistic level. Shooting in the dark can be an OK method to generate hypotheses, but then … flesh them out mechanistically and figure out what *causes* the outcomes, at the cellular and physiological level.
I’m just going to list a bunch of important medical advances created by observing things and then finding out their causal mechanistic models:
1) Mold kills bacteria -> Penicillin is the causal agent
2) Boiling broth stabilizes it from spoilage -> microorganisms in the broth cause spoilage, heat kills them
3) X is caused by a virus/bacteria/prion etc (Influenza, Rabies, Cholera, Salmonella, HIV, etc)
4) Stroke is caused by obstruction of blood vessels to the brain
5) Cancer is caused by mutation of the bodies own cells leading to out of control growth
6) Cancer mutations can be caused by viruses etc (HPV virus + cervical cancer)
7) Exposure to carcinogens can cause mutations leading to cancer (asbestos, carbon tetrachloride, various industrial chemicals)
8) Exposure to pollen causes immune system to react as if infected producing allergies, atopic dermatitis, systemic inflammation, asthma, etc
9) An important portion of the immune system is mediated through the generation of antibodies specific to invading disease causing organisms, etc.
10) Immune response to parasites is mediated through the IgE group antibodies, and these are responsible for most allergy responses.
Etc etc etc. Without mechanism we are pre-scientific talking about “miasma” or whatever. Even the Miasma theory of disease was scientific relative to bloodletting and drilling holes in the head to release demons or whatever.
Fortunately we still have biologists who actually work on disease mechanism. Unfortunately what you find out when you look into it is that it’s much easier to get funding for a bunch of quick and hopeless studies testing lots of garbage than it is for a concerted program of research to tease out the specific mechanism of say bone regeneration (what my wife works on) or how RNA can be induced to cause immune response in a controlled manner (the RNA vaccine researchers had a very hard time remaining in business over decades).
I’d be careful about assigning a mechanism to diseases that have not been cured (like scurvy):
It really depends what you mean by mutation and where you put it on the distal-proximal cause spectrum.
Eg, since the 1800s it was known (near) every cell in your body has what is essentially an “antenna” (ie, the primary cillium). This was ignored until the 1990s as some vestigial structure. Now there is the idea it plays a huge role in all kinds of diseases including cancer.
When the cell is not copying DNA or dividing (G0/1 phase), this antenna is normally attached to the centriole near the edge of the cell. This then defines the apical “top” side of the cell. Without it, the cell cannot orient itself. It doesn’t know up from down or left from right.
The centriole is a structure made of microtubles/etc (proteins) that the chromosomes all attach to when dividing. When it is at the base of the antenna that prevents inappropriate division. When the cell is dividing the proper arrangement of these proteins is crucial to correct splitting up of the chromosomes between the daughter cells.
Thus it could be loss or malfunction of this “antenna” (for whatever reason) as the ultimate cause of cancer.
For sure, one reason is mutations to certain genes (which could be anything to a single basepair to loss/duplication of multiple chromosomes). But it can also be lost due to inappropriate signals from the environment or even just get broken off due to mechanical stresses. It could turn out the genetic mutations are only a minor cause of cancer. Instead they are mostly a result of a defective or lost primary cillium.
A good review is here:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6396132/
I was more thinking at a earlier stage of understanding. Cancer comes from the bodies own cells, not from a foreign organism. At some point that must have been new information.
Daniel, are the good models you are considering mostly ODE models?
Not necessarily. In medicine it could be as simple as something like “Helicobacter pylori causes ulcers” mentioned above by Steve Sailer.
Or maybe something like “long COVID occurs when COVID infection induces X type of biochemical response which leads to Y type of long term damage”
There’s not really a need for describing explicit rates of change to get the benefits of causal mechanistic models. Though I do think ODE models are a good idea. But also discrete time models or even just DAG type models would be better than anything which assumes randomness.
But is it ever that simple? e.g. about half of the world’s population is infected by H pylori, and most do not have a gastric ulcer. In some people H pylori causes gastritis or
gastric ulcer. Most health problems are multifactorial, unlikely to be neatly solved by a simple monocausal model. That is where large epidemiologic studies can help.
ahem – gastric cancer
Infections are like a seed growing into a plant. The mere presence of a seed without fertile soil, etc does not cause a plant.
The same for cancers. The presence of a given set of mutations appears not sufficient for cancer. You can replace the nucleus of an egg cell with one from a cancer cell (that has all the mutations) and then cells with a cancer genome start behaving normally:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC517407/
@Daniel:
The earlier explanation was “imbalanced humors” afaik, eg excess black bile. Possibly caused by miasma (“bad air”). This doesn’t sound too conceptually different to me than “something is wrong with the body, possibly caused by exposure to viruses or chemicals.”
imbalanced humors and miasma etc were basically just names for what had been observed, namely that when exposed to poor ventilation people often got sick, and when they were sick, they tended to trend away from health and either die, or come into some chronic illness state.
But until you investigate what about the poor ventilation actually causes the illness, such as for example the tuberculosis bacterium, or influenza virus, or carbon monoxide poisoning or whatever, you’re just renaming the symptoms… and doctors love to do that. Once my doctor friend had pain in his foot, he went to get it evaluated, he was in his mid 70’s and of course had some fear it might be metastatic cancer or something serious like that… his doctor told him “you have metatarsalgia”. Of course being a doctor himself he immediately joked to us while telling the story “what a load of horseshit, I went to him telling him my foot hurt, and he diagnosed me with “pain in the foot””
This process of giving a special name to a syndrome as if you’ve now discovered something is rampant in medicine.
But what’s the alternative? For example perhaps his “metatarsalgia” was caused by immune system inflammation of the connective tissue due to micro physical damage from being overweight and standing and walking too much or in bad shoes?
Or perhaps his metatarsalgia is caused by excessive calcification of the tendons from having a genetic variant that induces calcification, and causes things like heel spurs? (He was also prone to kidney stones for example) Or some other cause…
Giving things a name is not the same as saying what causes them. And that’s especially true when there are multiple causes for the same thing. Is “Alzheimers” one disease? Or is it 11 different contributing causes any several of which are sufficient to cause notable disease? We don’t know, because we don’t have a proper causal model of Alzheimers. Actually it doesn’t even seem like we’re close to one, though I think the best thing I’ve read is that it may all be related to breakdown of blood-brain barrier, and for all we know maybe that’s a VitC deficiency leading to low level chronic problems in collagen formation? Or any number of other things.
If we give a medication to 300 people all diagnosed with Alzheimers, but Alzheimers is caused by any combination of at least 3 out of 11 different causes (as a simplistic model) then perhaps the medication really helps those people who have exactly 3 of the causes one of which is cause 7 because the meds help with cause 7… so it will help some few percent of the population. the rest will not be helped… we can try out a bazillion of these RCTs, find “average effect was not statistically significant” and even if in the right combination of 3 treatments we could cure *all* the alzheimers patients, we’d never discover it.