I happened to receive two emails on the same day on two different topics, both relating to how much to trust claims published in the medical literature.
1. Someone writes:
This is the follow up publication for the paper that was retracted from preprint servers a few months ago, the language has changed but the results are the same: patients treated with cacifediol had a much lower mortality rate than patients who were not treated:
This follows three other papers on the same therapy which found the same results:
– Small pilot RCT
– Large propensity matched study
– Cohort trial of 574 patientsI continue to be bewildered that this therapy has been ignored given that it’s so safe with such a high upside.
This led me to an interesting question which I thought you may have an answer for: “What are the most costly Type II errors in history?”
2. Someone else writes:
Do you think these two studies are flawed?
– Serum Phospholipid Fatty Acids and Prostate Cancer Risk: Results From the Prostate Cancer Prevention Trial
– Plasma Phospholipid Fatty Acids and Prostate Cancer Risk in the SELECT Trial
I said that I don’t know, I’ve never heard of this topic before. Why do you think they might be flawed?
And my correspondent replied:
I don’t understand the nested case cohort design but a very senior presenter at our Grand Rounds mentioned the studies were flawed. He didn’t go into the details as his topic was entirely different. I am trying to understand whether fish oil leads to increased risk for prostate cancer. I take fish oil myself but these studies shake my confidence, although they may be flawed studies.
I have no idea what to think about any of these papers. The medical literature is so huge that it often seems hopeless to interpret any single article or even subliterature.
An alternative approach is to look for trusted sources on the internet, but that’s not always so helpful either. For example, when I google *cleveland clinic vitamin d covid*, the first hit is an article, Can Vitamin D Prevent COVID-19?, which sounds relevant but then I notice that the date is 18 May 2020. Lots has been learned about covid since then, no?? I’m not trying to slam the Cleveland Clinic here, just saying that it’s hard to know where to look. I trust my doctor, which is fine, but (a) not everyone has a primary care doctor, and (b) in any case, doctors need to get their information from somewhere too.
I don’t know what is currently considered the best way to summarize the state of medical knowledge on any given topic.
P.S. Just to clarify one point: In the above post I’m not saying that the answers to these medical questions are unknowable, or even that nobody knows the answers. I can well believe there are some people who have a clear sense or what’s going on here. I’m just saying that I have no idea what to think about these papers. So I appreciate the feedback in the comments section.
Trusted medical sources, in person or on the internet, are indeed hard to find. On Alex Jones’ website there are many seemingly legitimate medical doctors offering advice opposite to customary reality. Another way to find a good medical guru is to interrogate people who have lived a long and productive life. With that in mind, note that today(!!) is the 95th birthday of Tom Lehrer. And, if you prefer a different nuanced perspective, Henry Kissinger will be 100 years old on May 27.
“advice opposite to customary reality”
Nicely said.
A doctor acquaintance of mine once recommended “UpToDate”, saying that half her work is just looking things up in that database. https://www.wolterskluwer.com/en/solutions/uptodate
My wife takes fish oil, so I looked up one of the Omega-3 papers on Google Scholar. It has 193 citation, most of them dealing with prostate cancer). But, I did find a 2021 systematic review in the same journal that published the original paper that concluded that: “The most extensive systematic review to assess the effects of increasing PUFAs [Omega-3 and Omega-6 fatty acids] on cancer risk found increasing total PUFA may very slightly increase cancer risk, offset by small protective effects on cardiovascular diseases.” I take from this that a fair number of people have looked at the issue and probably any effect is small.
The cited Covid/vit D paper is a raises some questions. They looked at 930 patients admitted to a hospital in Barcelona with Covid. They excluded the 93 who were already taking vit D. 447 were given vit D supplements and 391 were not. It does not appear that this was a pre planned randomization to treatment/no treatment. The patients given vit D did better, but why were about half not treated? On Pubmed, if you filter for completed randomized trials, the results are not impressive.
Googling NIH instead of Cleveland Clinic may give better results:
https://www.covid19treatmentguidelines.nih.gov/therapies/supplements/vitamin-d/
https://ods.od.nih.gov/factsheets/Omega3FattyAcids-HealthProfessional/#h7
For English-language info I think I’d generally trust NIH, NICE, and Cochrane reviews, especially if they agree
These types of studies simply make no sense.
A certain amount of raw material (wood/screws/etc) is required for routine maintenance of a house. So you keep a supply around, but not too much because the house fills with clutter. No two houses are exactly the same. An older house requires more repairs, a bigger house needs more supplies, and so on.
If a tornado hits the house, then you need orders of magnitude more than usual to do the repairs. Then perhaps there are some chronic leaks, so after the damage the baseline supplies required remains elevated.
These vitamins and minerals are the raw materials for your body (house). It is ridiculous to act like there is a one-size-fits-all beneficial dose.
What we need are cheap and easy ways to measure the levels of these various raw materials so we can tell when there is a shortage.
“I don’t know what is currently considered the best way to summarize the state of medical knowledge”
The fundamental problem is that neither observational studies, nor small RCTs, actually yield medical (i.e. casual) knowledge – just correlations fishing in noisy confounded data. Published reports on such make reviewers and readers believe the opposite, naturally.
This is the kind of thing I have learned from this blog that makes me sound like a total crank if I try to explain it to “well-informed” friends.
> The fundamental problem is that neither observational studies, nor small RCTs, actually yield medical (i.e. casual) knowledge.
Casual or causal?
If causal, perhaps a bit categorical?
I meant causal, and perhaps I sounded a bit too categorical (i.e. definitive).
Carefully analyzed large observational studies may give some supporting evidence for sound medical conclusions, so do good meta-analyses encompassing a multitude of well defined smallish RCTs. But these are few and far between (and swamped by meaningless “association” results published in the literature).
In particular, the 3 pieces of evidence listed in 1) seem to offer rather meager support (e.g. cohort studies are notorious for their lack of accounting for confounds, however large they are).
Makes sense to me.
For a specific medial research question, a systematic review of the existing literature with or without a meta-analysis synthesizing the results of multiple studies (at the level of the study or sometimes with patient-level data), is considered the best way to summarize the current research on a topic. These can be very helpful for some things, but 1) if the included studies are of poor quality the systematic review will be, too, plus 2) it can be difficult and expensive to do a truly systematic and thorough search of the literature; 3) it is often complicated or impossible to do a meta-analysis given heterogeneity of the studies; and 4) a systematic review only addresses a single question (does drug X reduce acute exacerbations in patients with asthma) rather than a whole clinical scenario (how can I prevent acute exacerbations in this patient with asthma). There are some other types of reviews that get at some of these issues — scoping reviews identify literature on a broader topic with very little synthesis and network meta-analyses that compare different interventions addressing the same outcome. But they all have limitations.
UpToDate is different, but as a clinical resource it is pretty great. It has narrative (not systematic) evidence summaries for a huge number of clinical topics written by experts along with expert opinion to fill gaps in evidence.
I agree that UpToDate is the best but it is not entirely reliable. Also it is generally biased toward over-investigating and over-treating when the evidence is weak. I use it to look up rare diseases I haven’t encountered before but if I look up something I am familiar with I will not infrequently disagree with some of its advice.
As Andrew says, a competent medico operating within their scope of expertise is the best source of information. There are some fools and charlatans out there though (and Covid flushed a lot of them out of the woodwork).
I think it is very difficult for a layman to dive into the medical literature and come up with all pearls and no seaweed – background and context are important. For instance, consider the Vitamin D paper. I know that therapeutic use of vitamins (in the absence of a deficiency) as been studied many times over the last 40 years with very un-impressive results. This includes vitamin C, D and E and various para-vitamins. Therefore my prior for Vitamin D to help Covid is very skeptical.
But most of the covid patients did have a deficiency at baseline. That is the reason for giving the vitamin.
If you want to argue low serum levels don’t always reflect a deficiency, that is perfectly legitimate. So lets figure out what does then.
Nobody would deny treating deficiency. The question is about treating Covid19. And the prior is very low–no plausible mechanism and like all the other purported indications that Vit D is good for XXX (insert everything but Vit D deficiency), nothing stuck when well done trials were done.
Covid patients were typically deficient in vitamin D though. From the paper in the OP:
So give them vitamin D. And the same was seen for vitamin C, so give them that too. And make sure it is enough because both are antioxidents that can get consumed orders of magnitude quicker in diseased tissue. Do the same for anything they appear to be deficient in.
Like I said, you can argue these serum levels are a poor proxy for deficiency. Fine, then it is of utmost importance to find good ones.
I explained above why these studies are all junk. These patients are apparently lacking (according to the standard measure of deficiency: serum concentation) in multiple raw materials their body needs.
The studies are like trying to use the same amount of water to put out any sized fire, or giving someone wood but no nails or screws to fix their house after a tornado damaged it. They make no sense, if we tried to solve other problems this way no one would be able to figure out how to fix a house or put out a fire.
Results from Bing Chat. I’m impressed by the quality of the literature references identified by the “chat” search. Less so by the “chat” text that is offered as response, even though it can compete with some of the contributions here.
Quelle: Unterhaltung mit Bing, 10.4.2023(1) :
According to COVID-19 Treatment Guidelines, the results from several cohort studies, clinical trials, and meta-analyses on the use of vitamin D for the prevention or treatment of COVID-19 have been published in peer-reviewed journals or have been made available as manuscripts ahead of peer review². A study published in Ärzteblatt shows that 96% of the total therapy studies on Vitamin D and COVID-19 show positive effects³. However, according to Nature, demonstrating a meaningful clinical benefit of supplementation over existing standards of care is likely to become…⁴.
It’s important to note that you should always consult with your doctor before taking any supplements.
I hope this helps!
Vitamin D | COVID-19 Treatment Guidelines. https://www.covid19treatmentguidelines.nih.gov/therapies/supplements/vitamin-d Zugegriffen 10.4.2023.
(2) COVID-19 und Supplementierung: Vitamin D – Ärzteblatt. https://www.aerzteblatt.de/archiv/219362/COVID-19-und-Supplementierung-Vitamin-D-in-der-Pandemie-hinterfragt-und-doch-empfohlen Zugegriffen 10.4.2023.
(3) Vitamin D for COVID-19: where are we now? – Nature. https://www.nature.com/articles/s41577-022-00765-6 Zugegriffen 10.4.2023.
(4) Vitamin D: The truth about an alleged Covid ‘cover-up’. https://www.bbc.com/news/health-56180921 Zugegriffen 10.4.2023.
Not “Big Pharma” but “Big-Vitamine” strikes again I think.
Attempting to make everyone buy vitamine D for a few bucks, instead of getting it for free when walking outside in the sunshine.
That’s why they told everyone to stay inside at the beginnings of the Covid 19 stuff I bet.
This whole Covid 19 stuff was probably designed by vitamine C and D manufacturers to get governments to buy super expensive vitamine C and D supplements and give it to as many the people as possible…
Thank heavens my government mostly talked about masks, vaccination, and booster number X instead of talking about the possible importance of making sure I had enough vitamine C and D in my system.
In fact, I can’t remember them ever saying anything at all about vitamine C and D (or other stuff like Zinc, and Magnesium).
Maybe I could have been a victim of “Big Vitamine” and this whole possible scheme if it were not for my government’s possibly exellent knowledge, wisdom, judgment, and ethics.
I’m going to take a hopefully brief sidebar from the scientific/statistical side of these questions to, at least in part, their relevance in medical practice.
Several replies refer to ‘talking with your doctor’ or ‘competent medico’ for trustworthy domain info. I may very well get flamed for this, but a patient’s trust in the knowledge a single caregiver possesses without doing something (even when its ‘noisy’) to challenge particular assumptions and viewpoints within that caregivers knowledge domain (i.e I’m referring to active vs passive patient involvement in their care), can lead to nasty outcomes down the road. And frankly, in the vast majority of cases, no one would be the wiser in the end.
In this forum we discuss the role of poor data –> poor questions –> poor methods in smaller studies which lead to same in metastudies. Nothing new to see here.
BUT, given the sea is so wide and deep in the medical literature in any number of subdomains (), why would we expect noise in studies not to drive differences in clinical diagnostic and treatment opinions? A further aside: I agree that UpToDate is at least a step in the right direction to help tackle this. Still, “Noise in. Noise out.”.
An interesting way to demonstrate this, to see medical interpretations truly diverge, is with rare diseases (and amusingly, sometimes, with political footballs). While more research in rare diseases (and indeed, in wider clinical domains) *might* help stabilize consistency in diagnostic and treatment opinions between clinicians, that pendulum can swing just in the opposite direction as well.
Sometimes it’s the lay patient who has to weigh differences in opinion – that’s a heavy rock to carry for many. Getting multiple clinical opinions by patients is perfectly fair. But it can sometimes just add to patient chaos. Additionally this is also more costly (for both patients and payors), when raised barriers between clinicians could simply be lowered –> we don’t need to charge a fee for every single time we pick up a @#$% phone, anymore than every opinion requires a referral). Finally, this sort of process delays resolution on important clinical treatment decisions for the patient – especially in potentially life-threatening cases where time waits for no one (for my peer clinicians, just TRY going for second/other opinions into practices where you’re inevitably seen as a new patient – and scheduled as such. The word ‘delay’ will certainly take on new meaning for you.)
This is where a convergence of more than one mind on a patient is warranted (and must be coordinated – i.e. through one’s PCP, a generalist). I believe this approach is both informative and creates value for each individual clinician responsible for a patient’s care (generalist OR specialist, as much as one would easily dismiss this thought).
Each clinician, if we are to define ‘competency’, should naturally have a strong handle in understanding and interpreting quite the compendium of literature that affects their domain (for the generalist, this is arguably the most daunting. Funny how it’s also the most undercompensated relative to their specialist peers). But it is equally important how well one recruits a reasonable network of other clinicians in their domain and other specialties to place other eyes on the patient in a coordinated way (i.e. for patient clinical profiles that may be more (even deceptively) complex) with the aim of positively affecting a person’s life. Note I didn’t merely refer to ‘clinical outcomes’ – they’re of course important, but just one facet of the whole.
I think what we’ve come to learn in both medical research and, applicably, in clinical practice is that one medical study (as landmark as it might be) or one clinical opinion alone, in isolation, is likely to be fraught with difficulty, error, and subsequent mis(sed)diagnosis or mis(sed)treatment of the patient. However, we have been reluctant as a medical community to truly own this and take strong steps to fix it. But that’s another complicated story for perhaps another blog.
Generally patients are loath to fix this. Instead they choose one clinician as their oracle.
Very well put. I toast not roast.