James Watson (this one, not this one or that one) writes:

You might be interested by the latest WHO guidelines on the use of hydroxychloroquine (HCQ) to prevent COVID-19. HCQ is the COVID-19 pariah and the guidelines recommend stopping all research into it. They pooled 6 COVID-19 prevention studies and found that HCQ did not reduce mortality. This conclusion is given a 4-star high GRADE rating, which means “The authors have a lot of confidence that the true effect is similar to the estimated effect”. However, there were only 13 deaths in the six studies, all of which occurred in one trial, with 5 out of 1116 in the HCQ arm and 8 out of 1198 in the placebo arm.

Apart from this being obviously silly (these studies were not designed to look at an effect on mortality), I think there is some weird thing going on whereby they are thinking about this from the perspective of absolute reduction in mortality. Even if there had been no deaths in the HCQ arm, the “number needed to treat” would be too high according to their assessment, so what’s the point. This doesn’t make sense to me. Suppose (unlikely but still possible) that HCQ reduced mortality by 20%. There would clearly be a huge difference in giving it to a young healthy person at risk of infection (absolute risk of death quite close to 0) versus an old frail person at risk of infection (maybe 10% chance of dying). The clinical trials they summarized were predominantly in young healthy people so even the best drug in the world wouldn’t look good under their framework.

One also has to downsides of the drug, notably toxicity. They say that there is evidence of increased numbers of adverse effects (significant! But this only gets 3-stars), but unfortunately they mis-coded one of the studies so their p-value doesn’t actually cross the magical 5% threshold.

It’s hard for anyone doing COVID-19 therapeutics research to think dispassionately about HCQ given the enormous mess this drug has caused. But that’s not a reason to make strong recommendations based on extremely limited data and unstable estimates. We wrote about these problems in a letter on the platform Wellcome Open Research. We are still running a prevention trial which started a year ago, and it would be a huge waste of money to have to stop it prematurely just because someone decided that 13 deaths equated to high quality evidence.

One thing I don’t know here is what is the result of putting a recommendation on the WHO guidelines. They say, “The panel considers that this drug is no longer a research priority and that resources should rather be oriented to evaluate other more promising drugs to prevent covid-19,” but I’m not sure what this recommendation will do. I don’t think it has legal force, but I suppose that many research funding organizations will be likely to follow the WHO guidelines? In any case, with vaccines available now, perhaps this is all less of an issue.

I also noticed that the above-linked article with the WHO guidelines has 53 authors, among which 8 are listed as methodologists. I guess it was a big project. One difficulty here is that with all these names, it’s hard to figure out who was in charge of the statistical analysis. That doesn’t mean they did anything wrong, it’s just not clear where the decisions were made and who had responsibility for them. We saw an extreme example of this with the Surgisphere scandal. It can take a bit of unraveling to figure out who did what on a paper.