No, I don’t like talk of false positive false negative etc but it can still be useful to warn people about systematic biases in meta-analysis

Simon Gates writes:

Something published recently that you might consider blogging: a truly terrible article in Lancet Oncology.

It raises the issue of interpreting trials of similar agents and the issue of multiplicity. However, it takes a “dichotomaniac” view and so is only concerned about whether results are “significant” (=”positive”) or not, and suggests applying Bonferroni-type multiplicity adjustments to them. I found it amazing that such an approach was being seriously suggested in 2019 – it’s like decades of systematic reviews, the Cochrane Collaboration and so on never happened.

The first author has a podcast called Plenary Session that is widely listened to by doctors, so he’s quite influential. From what I’ve heard it’s usually pretty good, and raises some really important points about research and clinical trials. But I think he’s got it badly wrong here. The consequences concern me; this is in one of the top journals in oncology; it’s read by doctors treating people for very serious conditions, who make life and death decisions, so it’s really important to get these issues right and not mislead ourselves. I’ve written a letter to the journal (just heard it was accepted this morning) but I don’t think they really have any impact. Most people who read the paper won’t see the letter.

I guess most people who read the paper won’t see this blog either, but here goes . . .

I followed the link and I didn’t think the paper in question was so terrible. I mean, sure, the method they propose is not something that I would ever recommend—but it seemed to me that the main purpose of the paper was not so much to recommend Bonferroni-type multiplicity adjustments (a bad idea, I agree) but rather to warn people not to take published significance tests and confidence intervals seriously when there are forking paths in the data processing and analysis.

So, yes, I disagree (see here and here) with their statement, “The rare so-called positive trial within a sea of negative studies is more likely a false positive than a true positive.”

And I disagree with their statement that “we need to correct for the portfolio of trials not within a single pharmaceutical company but across all companies”—I think it’s good to analyze the whole portfolio of trials, not to skim the statistically significant results and then try to use statistical methods to try to estimate the rest of the iceberg.

But I agree with their conclusion:

To deliver high-quality cancer care in a sustainable health ecosystem, clinicians, investigators, and policy makers will need to identify therapies that offer benefits that are substantial in magnitude and not statistical artifacts.

So I would not call this a “truly terrible article.” I’d say that it’s a reasonable article that is imperfect in that it is operating under an outdated statistical framework (unfortunately a framework that remains dominant in theoretical and applied statistics!). But, as a whole, it still seems reasonable for doctors to read this sort of paper.

P.S. I wrote this post in part to avoid selection bias. I get lots of emails from people pointing out things they read and liked, and I share these with you. And I get lots of emails from people pointing out things they read and hated, and I share those too. So when I get such emails and I disagree with the reader’s assessment, I should let you know that too!

12 thoughts on “No, I don’t like talk of false positive false negative etc but it can still be useful to warn people about systematic biases in meta-analysis

  1. I haven’t read the article in question; paywall and all that. I do listen to the podcast, “Plenary Session”, a couple of times a month. My profession has a very strong pro-treatment attitude, and a little “slow down and look at the evidence carefully” is useful. Our enthusiasm has led to quick adoption of treatments. This can be good when it leads to prompt dissemination of worthwhile treatments, but it’s bad when we go overboard for the latest and greatest that turn out not to be that great. Dr. Prasad is no more infallible than any other doctor, but he provides useful cautions. I do remember sitting at a national meeting where a presenter announced that their trial in pancreatic cancer was the only one to reach statistical significance out of many trials using a 5-FU based combination. The uniqueness of the combination was touted as the source of this result. I sat there thinking that if enough trials were done and some p value was the goal, then eventually you’d get the p < 0.05 that you’re looking for. I was disappointed when I was the only one who groaned after the claim was made.

  2. I remember Andrew mentioning that in low prevalence situations false positives are more important than false negatives. However, for clinicians FN is really an important issue. When patients present with signature symptoms of, say, covid, but multiple tests show negative, what are they supposed to do?

    I understand the probability part of why in low prevalence FN is not such a big deal, but in their practice it is.

    On the other note, I wonder why FN/FP would be an issue in diagnostic tests where we have pre and post measures, with post measure being able to verify the presence or absence of something 100%.

    If you perform 100 X rays and then biopsy on all 100 patients, it is possible to exactly confirm FN/FP rate for a given test.

    What’s not to like there?

    Thanks.

    • “When patients present with signature symptoms of, say, covid, but multiple tests show negative, what are they supposed to do?”

      Simple. If it isn’t covid, it is likely to be a garden variety cold/upper respiratory infection which will resolve on its own. In terms of the patient’s well being, if that diagnosis is wrong and the patient develops severe COVID, time will tell and one can intervene at that point. In terms of risk to others, people with covid symptoms should self isolate even if an initial test i”If you perform 100 X rays and then biopsy on all 100 patients, it is possible to exactly confirm FN/FP rate for a given test.s negative, and get re-tested in a few days. False negatives are often due to the test being done too early or too late. Yes, there is a very small risk of serial negative tests, but the probability of that is low enough that the public health consequences are negligible–in fact, false negative tests are probably associated with decreased ability to transmit the infection to others.

      “If you perform 100 X rays and then biopsy on all 100 patients, it is possible to exactly confirm FN/FP rate for a given test.”
      Not so simple. If the x-ray is negative, how do you do a biopsy? Where do you put the needle/make the incision? And biopsies are not always 100% sensitive and specific either. With needle biopsies, in particular, the lesion can be off the needle track. Even with excisional biopsies, the slides can be misread if the findings are subtle or the lesion is a needle in the tissue haystack.

      Not to mention that the FN and FP rates estimated from a sample of 100 will be subject to at least sampling variation, and perhaps also to selection bias or various kinds of forking paths issues.

      Although nobody likes to admit it, medical decision making is associated with enormous amounts of uncertainty.

  3. Andrew, could you clarify why you dislike, “The rare so-called positive trial within a sea of negative studies is more likely a false positive than a true positive.” I’m imagining it’s because you think the real state of the situation is not that there’s a false positive but that there’s a small effect with the occasional study overestimating and getting the correct decision, if incorrect estimate.

    • I second that. Prasad elsewhere makes a persuasive argument for the proposition that whatever the trial data may imply statistically, once approved most new cancer drugs do no better than older generics. And of those that “work” too many simply add a couple of weeks to the life, such as it is, of a dying patient. The promise made by approving these drugs is thus “false”.

  4. A lot of people don’t realize “Bonferroni” is from the original Latin, where “bon” means “good” or “positive” and “ferroni” means “false.” Hence, “false positive.”

    The more you know.

Leave a Reply to Navigator Cancel reply

Your email address will not be published. Required fields are marked *