Dying children and post-publication review (rectal suppositories edition)

James Watson (this guy, not the guy who said cancer would be cured in minus 22 years) writes:

Here’s something that you may find of interest for your blog: it involves causal inference, bad studies, and post publication review!

Background: rectal artesunate suppositories were designed for the treatment of children with suspected severe malaria. They can be given by community health care workers who cannot give IV treatment (gold standard treatment). They provide temporary treatment whilst the kid gets referred to hospital. Their use is supported by (i) 1 moderately large RCT; (ii) our understanding of how severe malaria works; (iii) good pharmacological evidence that the suppositories do what they are supposed to do (kill parasites very quickly); and (iv) multiple large hospital based RCTs showing that artesunate is the best available drug.

The story: A group at Swiss TPH got a very large grant (19 million USD) to do a `deployment study’: basically look at what happened when the suppositories were rolled out in three countries in sub-Saharan Africa. This study (CARAMAL) asked the question: “Can the introduction of pre-referral QA RAS [quality-assured rectal artesunate] reduce severe malaria case fatality ratio over time under real-world operational circumstances in three distinct settings?” (see clinicaltrials.gov: NCT03568344). But they saw increases in mortality after roll-out, not decreases! In Nigeria, mortality went up 4 fold (16.1% versus 4.2%)! In addition, the children who got the suppositories were more likely to die than those who didn’t. These results led the WHO stopping deployment of these suppositories in Africa earlier this year. This is a really serious decision which will probably result in preventable childhood death.

The authors put their findings online last year as 10 pre-prints. In July we wrote a commentary on the overall decision to stop deployment and the reported analyses in the pre-prints. The main points were:
– No pre-specified analysis plan (lots of degrees of freedom of exact comparisons to make, and what to include as baseline variables). This is unusual for a big clinical study.
– Temporal confounding, COVID-19 being one small difference in post versus pre roll-out world….!
– Confounding by indication: comparisons of who got the suppositories versus who didn’t post roll-out are probably due to health workers giving them to the sicker children.
– Mechanistic implausibility of having a massive increase in mortality within 48 hours of giving the drug compared with not giving the drug (this just doesn’t fit with our model of the disease and the pharmacology).

Unsurprisingly (?) the authors did not comment on our piece, and their main paper was published in October with no apparent changes from the pre-print version…

The now published paper (BMC Med) uses pretty strong causal language in the abstract: “pre-referral RAS had no beneficial effect on child survival in three highly malaria-endemic settings”. Given the design and data available in the study, this causal language is clearly not justified. I emailed the authors to ask them for their study protocol (some things are unclear in the paper, like how they exactly recorded who got the suppositories and whether there was a possibility of recall bias). I also wrote my main concerns as a thread on twitter.

They answered:

“We saw that you have just publicly on Twitter implied that we conducted the CARAMAL study without a study protocol nor an analysis plan. You are essentially publicly accusing us of conducting unethical and illegal research. This was less than 24 hours after sending this email requesting us to share the study protocol.

Your Tweet adds to the tendentious and poorly-informed commentary in BMJ Global Health. We fail to see how this style of interaction can lead to a constructive discussion of the issues at the heart of the CARAMAL project.

We have provided all necessary information including the study protocol and the analysis plan to a panel of international experts gathered by the WHO to conduct a thorough evidence review of the effectiveness of RAS. We look forward to their balanced assessment and opinion.”

Basically they refused to share the study protocol. They also admit to reading the previous commentary which discussed the lack of analysis plan. I didn’t accuse them of not writing a study protocol or analysis plan, but not posting it with the paper (which is a fact). Most medical journals make you post the protocol with the publication.

Why this is important: pushback from various people has made the WHO convene an expert group to see whether the moratorium on rectal artesunate deployment was justified. They will be making a decision in February. The underlying study that caused this mess is poorly thought out and poorly analysed. It’s quite urgent that this gets reversed.

This seems pretty wack that they say they have a study protocol and an analysis plan (perhaps not a preanalysis plan, though) but they’re not making it public. What’s the big secret?

To put it another way, if they want to keep key parts of their research secret until the panel of international experts gives “their balanced assessment and opinion,” fine, but then what does it mean for them to be publishing those strong causal claims? Once the big claims have been published, I can’t see a good reason for keeping the protocol and analysis plans secret.

Also what about the issues of temporal confounding, confounding by indication, and mechanistic implausibility? These all seem like a big deal. It always seems suspicious when researchers get all defensive about post-publication criticism and then don’t even bother to address the key concerns. Kids are dying here, and they’re all upset about the “style of interaction”???

Or maybe there’s more to the story. The authors of the published article or anyone else should feel free to provide additional background in the comments. If anyone has the study protocol and analysis plan (or pre-analysis plan), feel free to post that too!

54 thoughts on “Dying children and post-publication review (rectal suppositories edition)

  1. Am not sure how relevant this is to the particular subject at hand, rectal artesunate suppositories, but this appeared in the Washington Post this very day:

    https://www.washingtonpost.com/wellness/2022/11/08/microbiome-fiber-immunotherapy-cancer/

    “Some researchers are trying to overcome resistance to immunotherapy by doing fecal transplants. They take stool samples teeming with gut microbes from patients who responded to the drugs and transfer it via colonoscopy to another patient. In one recent trial, scientists gave fecal transplants to 15 people with advanced melanoma who didn’t respond to immunotherapy.”

    As to “artesunate”, a word new to me, https://www.merriam-webster.com/medical/artesunate

    “Medical Definition of artesunate

    an antimalarial drug C19H28O8 that is a semisynthetic, water-soluble, derivative of artemisinin taken orally or administered by intramuscular or intravenous injection Artemisinin is made from the leaves of the Chinese wormwood shrub, and the drugs artesunate and artemether are derivatives known to kill the parasites that cause malaria.— Janet Raloff, Science News

    Note: When used orally, artesunate is typically combined with another antimalarial drug, such as mefloquine or amodiaquine.
    — compare artemether”

  2. Observational data can be used to check predictions or to generate hypotheses, used like this you can conclude whatever you want based on your biases.*

    Just look at how equally crappy observational data on HCQ/IVM was treated vs masks/vaccines. People just correctly questioned one set of studies while letting the other slide (which set depended on your bias). Total waste of time and money.

    * RCTs limit this only somewhat, the real solution is to start checking theoretical predictions

    • I think that’s largely an issue of priors, plus some mechanistic understanding (which I guess can be considered part of the prior). There was reason to expect the vaccines to work pretty well. Masks are a tougher one: obviously a really good mask, worn properly, provides a lot of protection but in practice if you simply tell people to wear good masks properly it was not at all obvious what that was going to do…although it would certainly do much less than getting everyone to actually do it.

      I’m very very far from being an expert but it’s my understanding that people who _are_ experts thought Ivermectin and hydroxychorloquinine were long shots, since (unlike masks and vaccines) there wasn’t an obvious mechanism for them to help a lot.

      And of course eventually we did get enough data on three out of these four to have decent estimates of efficacy.

      I’m not sure we will ever really know about masks at the societal level. At the individual level you are certainly less likely to contract COVID if you wear a good mask properly when in the company of others — most cases are from inhalation — but it would be almost impossible to quantify this without challenge trials. I think we should have been doing challenge trials for the past two and a half years, but evidently bioethicists have decided it’s better to let hundreds of thousands of people die through inaction than to kill a very small number of people in experiments on volunteers.

      • :Phil –

        > but evidently bioethicists have decided it’s better to let hundreds of thousands of people die through inaction than to kill a very small number of people in experiments on volunteers.

        In the very least, that seems like quite an uncharitable characterization of what some unspecified bioethicists (was it all of them?) “decided.”

        Seems to me that conducting challenge trials, say of the use of masks, would be exceedingly difficult to conduct, particularly during an ongoing pandemic, and particularly if you want to have good control for a very long list of confounds and compliance (beyond highly problematic self-report data). And it seems to me that there would be obvious ethical implications that any bioethicist would rightfully have some concerns about (not to say that I think that challenge trials would have necessarily been unethical).

        I dunno. Don’t really want to be argumentative, and maybe you have explicit evidence at least with respect to some particular bioethicists, but in addition to being uncharitable that statement strikes me as not entirely unlike the brand of mind-probing we see regularly from certain other unnamed commenters.

        • Joshua:

          Echoing what you say: I doubt that “bioethicists” have much power. The people who make these rules are regulators and politicians with some mix of technical and political motives, including concerns about backlash resulting from bad outcomes.

        • Certainly the highly charged political atmosphere is detrimental to an open persuit of evidence.

          The way the investigation of the “lab leak hypothesis” has become politicized is depressing as all get out and doesn’t bode well.

          That said, Musk has made it clear that the new Twitter is going to straighten everything out of we can be patient enough.

          https://twitter.com/elonmusk/status/1589413653190938624?t=rYY8B6i5aVm1M8l6ZSrwWw&s=19

          https://twitter.com/elonmusk/status/1589403131770974208?t=nOF4H-anem6Clh1JRrGt8w&s=19

        • “The people who make these rules are regulators and politicians…”

          Regulators and politicians don’t act in vacuum. They need crutches for fall protection. While bioethicists don’t have any direct power, they could be important crutch-makers – especially given that, without the scientific constraints that circumscribe scientists’ statements and actions, they have total creative freedom over the crutch. That allows them to craft the policy to meet the need if they so desire, just as journalists do.

        • Joshua,
          You’re right, many bioethicists are fine with challenge trials for COVID and for other diseases, and some have been strong advocates for them. But my somewhat-informed opinion, or perhaps ‘observation’ is a better word, is that the ones who are involved in “human subjects protection” are very reluctant to allow subjects to be exposed to significant risks even if the subjects are informed of the risks and willing to accept them. I recall seeing news articles in the first few months of the pandemic that reflected this viewpoint. That is, although from a numerical standpoint there may be plenty of bioethicists who are fine with challenge trials for COVID and other dangerous diseases, I think that when it comes to influence, the nays have it.

          On the other hand — or perhaps this does support my view, I dunno — I just did a search to find some of those articles and the first thing I found is a New York Times article that I don’t recall reading: https://www.nytimes.com/2020/07/01/health/coronavirus-vaccine-trials.html

          It says “In a draft report published last month, the World Health Organization said that challenge trials could yield important information, but that they would be daunting to run because of the potential of the coronavirus “to cause severe and fatal illness and its high transmissibility.”

          The report, by a 19-member advisory panel, provided detailed guidelines about the safest way to conduct challenge trials, recommending that they be limited to healthy people ages 18 to 25 because they have the least risk of severe illness or death from the virus. The virus — in a dose carefully calculated to produce an infection but unlikely to cause severe illness — would be dripped into their noses.

          But the panel also said its members split nearly in half over several major issues. They were divided over whether trials should be carried out if no highly effective therapy had been identified to treat participants who got sick; over whether studies in healthy young adults could predict the efficacy of a vaccine in older people or other high-risk adults; and over whether challenge trials could really speed vaccine development.”

          So there we go, half of bioethicists in that group were in support of challenge trials for vaccine development. But I note that that was a World Health Organization report prepared by an international group of researchers. Perhaps the US is more conservative.

          Eh, I could quote more articles and more bioethicists but without a measure of representativeness or influence I don’t know what the point would be.

          Anyway you’re right that I was unfair to treat bioethicists as monolithic on this issue, but I have not changed my mind that human subjects review panels in the US are far too conservative, i.e. they weight the health of the subjects too highly compared to the healths of the non-subjects who might benefit.

          And I disagree that challenge trials involving masks would have been difficult to conduct, judged on a relevant scale of difficulty. At this point it would be almost silly, but think back a couple of years, to before there were vaccines and when the worldwide economic cost of the pandemic was in the tens of billions of dollars per month (which of course is just a partial gauge of the human suffering that was being caused. I think it would have made sense to get thousands of volunteers to do some tests. For masks, you could assign people randomly into groups of “no mask”, “poor mask”, “good mask” and have them sit in a room with some infected people. See how many of each group get sick. Do tests with 10, 30, 90 minutes of exposure, for starters. All such experiments are harder than they seem, and I don’t mean to imply that this would be easy on an absolute scale, but considering that thousands of people were dying of COVID every day, and many more were losing livelihoods, I think it would have been worth overcoming the experimental difficulties.

        • Phil –

          > You’re right, …

          Well, blind squirrels and stopped clocks and all that, right? I think I was right once before. But I could be wrong about that.

          Anyway, thanks for doing the work!

          > it is that the ones who are involved in “human subjects protection” are very reluctant to allow subjects to be exposed to significant risks even if the subjects are informed of the risks and willing to accept them.

          I could imagine where that would get a bit tricky. I mean at some point you’d have to ask yourself if there’s an unethical aspect of letting someone harm themselves. I’m not weighing in one way or the other, and there’s the whole trolly car problem aspect of trading uncertain harm to fewer people to prevent certain harm to lots more people – but I can imagine where it’s not always an easy call.

          Maybe there’d be a difference calculus in challenge trials for vaccines and challenge trials for masks? Maybe because the expected return from a vaccine would be greater, and last beyond the study period? I wouldn’t even know where to begin.

          > but I have not changed my mind that human subjects review panels in the US are far too conservative, i.e. they weight the health of the subjects too highly compared to the healths of the non-subjects who might benefit.

          Maybe. But again, kind of like the trolly car problem, I think that an abstract scenario might be different than if you’re the one who has to actually throw the switch. I get your logic but I’d be reluctant to say that I have a strong impression one way or the other without knowing all the factors that go into a decision in a particular context. I mean if people are being conservative, I tend to give them the benefit of the doubt that they have good reasons for being conservative. Of course, there’s always CYA as a motivation or other less that noble reasons, but I tend to be circumspect about judging how other people make those kinds of decisions if I don’t really have all the relevant information.

          > For masks, you could assign people randomly into groups of “no mask”, “poor mask”, “good mask” and have them sit in a room with some infected people. See how many of each group get sick. Do tests with 10, 30, 90 minutes of exposure, for starters.

          Yeah. Ok. I wasn’t really thinking of lab experiments. I was thinking of how difficult it would be to conduct informative real world experiments. Of course, lab experiments would have limited utility in terms of generalizing for obvious reasons – but I guess it’s reasonable to conclude that if masks have something approaching zero benefit in a lab setting there isn’t much reason to expect they’d be efficacious in a real world setting.

          So now that you point it out – I go along with the idea that it would have made sense to conduct lab-based challenge trial mask experiments. It’s not too late for that, actually – especially since Omicron seems to be significantly less deadly and treatments are better, so the risk of the challenge would be considerably less. Since Omicron is so much more infectious, such experiments might not be that beneficial looking backwards but could be instructive looking forward.

        • Joshua,
          We let people volunteer to go to war. We let people volunteer to become firefighters. We allowed people to work in New York hospitals with inadequate safety gear during the first months of the pandemic. I think we should let a person take a 0.01% risk if the resulting knowledge has a decent chance of saving even a few lives. We allow bigger risks than that in other contexts.

        • Phil, indeed we should have settled the vaccine effectiveness question in about 5 weeks, get 20-30 volunteers in the armed forces to take the vaxx, then after the full vaccination period, challenge them all with a calibrated dose of virus. I honestly don’t even personally think a control group was needed, provided you had a sufficiently well designed titration of the viral challenge (so maybe do monkey based experiments to figure out an ID50 or something similar, test the infectious dose on multiple animal models). Some might argue that no control group would be wrong, but if we have other external evidence to suggest that the dose is something like 50% infectious and we see 0 or 1 out of 20 volunteers getting infected, it would have been more convincing than what we did do which took tens of thousands of injections and like 6 months. If we didn’t see that kind of result it’s only 4-5 weeks, and followup experiments could have been done and still have a faster result than what we did.

        • Phil –

          > We let people volunteer to go to war. We let people volunteer to become firefighters. We allowed people to work in New York hospitals with inadequate safety gear during the first months of the pandemic. I think we should let a person take a 0.01% risk if the resulting knowledge has a decent chance of saving even a few lives. We allow bigger risks than that in other contexts.

          Sure. And those are interesting analogs. And maybe I’m reaching too hard here, but it feels to me that there are pretty important differences in the decision-making domain for those other forms of challenge volunteering. Seems to me the decision with the medical treatment challenge trials is a very explicit decision to say to someone “Yes, we want you to risk your life” and then the risk is, in a sense, more directly applied by the decision-makers themselves. Effectively, it’s more like the decision-makers are volitionally administering the shot to the volunteer. With the solider, the firefighter, the medical worker, the application of risk is more mediated, contextual. Essentially, I think, the responsibility for applying the risk is more directly on the shoulders of the decision-makers.

          Intellectually, I could say that there’s no real difference once you boil everything down. And in that sense I agree there *should* have been more trials. But I also could understand why having that decision placed on their shoulders would naturally, and I think understandably, incline at least some folks towards being more conservative.

          But applying your analogy in the other direction….at least with the sending volunteers off to risk their lives in war – I wish the responsibility for the decision were more distinctly and discretely placed on the shoulders of the decision-makers as I think, then, that maybe there would be more efforts made towards alternative ways to get through conflict.

        • Daniel –

          > Phil, indeed we should have settled the vaccine effectiveness question in about 5 weeks, get 20-30 volunteers in the armed forces to take the vaxx, then after the full vaccination period, challenge them all with a calibrated dose of virus,

          What generalizable information do you see as coming out of that sort of trial – relatively quite healthy people, exposed to the virus in an unnatural setting? I mean it would be information, of some use but of pretty limited utility for generalizing, no?

        • Joshua,

          They’re calculating vaccine effectiveness as essentially the reduction in the number of exposed people who get sick. The real trials randomized some huge number of people, let’s call it 30k into two groups vaxx and placebo. Then they let them go around the community until they saw N people who had symptoms and tested positive… I don’t remember the exact number maybe it was around 150 people, then once they had sufficient N who were sick they unblinded the people and calculated essentially (1 – nvaxxed/(0.5*N)) and if it was sufficiently high and there were sufficiently few serious side effects, and the antibody levels responded in the appropriate way, they approved the vaccine.

          Now, in a challenge trial with a prior of 50% will get infected because of the dose given, you calculate essentially (1 – n_infected/(0.5*N)) as vaccine effectiveness and if its high enough and there is the appropriate antibody response then you move to a trial with 30k to measure rare side effects and 3 weeks after the 30k injections you approve for widespread use. Basically you could cut 3 or 4 months of waiting around for 150 people to randomly get sick in the community.

          We have strong mechanistic ideas about how these vaccines work, and we have the ability to measure antibody levels, and other mechanistic response variables that confirm the mechanism. Yes, the question of whether the vaccines would be equally effective in elderly people would be open after the initial challenge trials, but the 30k injections to check for rare side effects could also measure antibody and other mechanistic response variables and include more elderly people.

          Let’s say the timeline is 5 weeks from initial injections to challenge trials, and then 5 weeks from initial broad scale injections to confirmation that side effects were sufficiently rare and mechanistic progression of the vaccine response was sufficiently close to expected… We had these vaccines designed and available for initial testing in I think mid January 2020 (yes, that soon https://nymag.com/intelligencer/2020/12/moderna-covid-19-vaccine-design.html ). That means we could have approved the vaccine by say end of March or April 2020, and of course we’d have to ramp up production, but let’s say we could have rolled out the vaccine as soon as maybe July 2020 in small quantities and in larger quantities by say Sept and I’m guessing at least in the US we could have avoided maybe 500k deaths.

        • Joshua,
          Although challenge trials were suggested for vaccine effectiveness, and might have been useful there, I was thinking of other questions like the effectiveness of masking, and how long an exposure does it take for someone to get sick, and other things that are nearly impossible to learn any other way but that have huge implications for the public health response. With masks, for example, how would you even do a ‘real world’ experiment? You could recruit people and ask them to wear masks or not to wear masks, but 1. there’s no way the people with masks would behave the same as people without masks, so that already leads to problems, 2. since you don’t know who gets exposed to what, you’ve got this enormous source of noise that would require a huge sample size to reduce, 3. There’s no way to check compliance. Some people are supposed to wear masks but maybe they don’t always, or don’t wear them right, or don’t wear the kind you’re trying to test; while other people aren’t supposed to wear them but maybe they sometimes do. I think most masking-related questions can only reasonably be answered through challenge trials. And although the answers may not be really important now, a couple of years ago if we had known (for example) that a properly worn N95 mask provides enough protection that you can work all day next to an infected co-worker with very low chance of getting sick, that would have been extremely useful information.

          Yes there’s a difference between trial participants and people like firefighters and doctors and nurses who risk their lives to help others… although I think the difference isn’t all that big in a way. In the early weeks of the pandemic, health care workers knew they were being exposed to the virus but went to work anyway, even as co-workers were getting sick (and some were dying), that’s not so different from deliberately being exposed to the virus. Yes, with the health care workers they know they’re being exposed but they don’t _intend_ to be exposed, and I agree that matters psychologically, but I’m not sure it should matter ethically.

          Daniel, although I’m fine with challenge testing for vaccine effectiveness too, I think it only would have been useful if they had worked their way up the age ladder. The main point of vaccines is to prevent infections from having bad health effects, not to prevent infection. Lots of vaccinated people get infected, but the death rate is many times higher for unvaccinated than for vaccinated people. Even I would not be so cavalier with people’s lives as to recommend challenging elderly diabetic patients who had and hadn’t been vaccinated. I think you’d start with people in their twenties, then thirties, and try to draw conclusions based on disease severity rather than fraction of people infected.

        • Daniel –

          Ok, now I get it. It’s not replacing the large-scale with trials with challenge trials, but a sequenced staging including both types of trials, that would make the process more efficient.

          Makes sense to me.

          I think that it would be hard to convince everyone that the shorter trial period with older people to test for side-effects, and an accelerated approval time frame, would be trustworthy, but certainly some would have been convinced. Given the nature of the vaccine hesitancy I’ve seen, I think hesitancy in some ways would have been increased. One of the frequent complaints about the process as it was was that “it happened too quickly to be trustworthy.” So having it happen yet more quickly, no matter if there’s a logical reason for making it quicker, could have exacerbated that problem. I’m also reflexively skeptical about the 500k lives saved, but certainly an accelerated approval process would have resulted in many lives saved.

          But yeah, it does seem that structure would make sense going forward as a methodology for testing vaccines during a medical emergency/ongoing pandemic.

        • Phil –

          Again, I agree that lab-based challenge trials for masks would have made sense. Not something I’d thought of before.

          Also for studying the length of exposure it takes someone to get infected, or the titer levels, etc.

          But I also think that the level of variability with all that at individual levels (say one person’s immune system versus the next person’s) and across contexts (lab-based to out of the lab) is so high, that those trials would have quite limited utility

          As for real world testing of masks, I agree that would be extremely hard to do very well enough to base policies on. But I think also that community-based challenge trials would also not be terribly robust with all the variables to content with, and there would be big problems trying to generalize from lab-based challenge trials. All of that is why I think it’s better to accept that we can’t really know about mask-wearing, and look at the issue as high damage low probability risk management in a context of high uncertainty. Especially when you consider that individual level risk reduction could possibly compound at a population level.

          But more information of the sort that could come from challenge trials in either setting couldn’t hurt, and would likely help to some degree.

          > Yes there’s a difference between trial participants and people like firefighters and doctors and nurses who risk their lives to help others… although I think the difference isn’t all that big in a way. In the early weeks of the pandemic, health care workers knew they were being exposed to the virus but went to work anyway, even as co-workers were getting sick (and some were dying), that’s not so different from deliberately being exposed to the virus. Yes, with the health care workers they know they’re being exposed but they don’t _intend_ to be exposed, and I agree that matters psychologically, but I’m not sure it should matter ethically.

          Again, I’m looking at it at the level of the decision-maker. A level which exists for challenge trials but doesn’t really exist for firefighters or healthcare workers. There is no group of people who really get to make the decision in those other contexts as there would be for challenge trails. And I’m also not sure that the ethics are really that different. But maybe I’m just inclined to give more leeway for the difficulty of making that decisions regarding challenge trials, and I don’t think it’s really as you described with: but evidently bioethicists have decided it’s better to let hundreds of thousands of people die through inaction than to kill a very small number of people in experiments on volunteers.

          It seems more complex to me.

      • Yep, agree 100%.

        All that happened was people stuck with their priors. Additional evidence of this kind does not meaningfully affect the posterior because there are too many near equally plausible explanations.

        I have been saying that NHST only measures the collective prior for years, it has been on full display for all to see the last few years.

      • And of course eventually we did get enough data on three out of these four to have decent estimates of efficacy

        Excep this. There is no way to get valid estimates without accounting for testing rates, which afaik was never done for any observational study regarding those interventions. If you know of one I would like to see it.

        Even the RCTs are confounded by side effect profiles. I also never saw any do an exit poll to see what percent guessed correctly.

  3. Watson above: “I didn’t accuse them of not writing a study protocol or analysis plan, but not posting it with the paper (which is a fact).”

    Watson in tweet: “More importantly: There is no study protocol. There is no statistical analysis plan.”

    Maybe Watson meant they weren’t posted, but he said they didn’t exist. Oops.

  4. I think an important feature of this study (CARAMAL) is that the research question it was asking was doomed from the outset. The main question was fundamentally causal: “does deployment of this drug reduce mortality?”. The idea was that temporal changes in mortality could be interpreted as causally linked to the deployment (before versus after). But unless you have a very good idea on who is presenting to the community health workers at different times in the study, and that these patients groups are comparable, and that case ascertainment is the same, you just can’t say anything causal. Having a once in a century epidemic in the after deployment period didn’t help either (OK that wasn’t their fault!). Another point is that one of the goals of the study was to actually change health seeking behaviour in the places where rectal artesunate was being deployed! As stated in the end-of-grant report: “The CARAMAL project worked closely with the Ministries of Health to design context-specific strategies with a goal to improve care-seeking behaviour among community members, promote the use of RAS among CHWs [community health workers] and encourage completion of treatment among caregivers”
    https://unitaid.org/assets/CARAMAL_Evaluation_Final-Report_BroadImpact_May-2021.pdf

    Now I can understand that the investigators didn’t really want to go back to their funders and say: “err, remember that 19 million USD you gave us? Well, turns the study we designed was fundamentally flawed relative to its aims and so we can’t actually say anything about the effectiveness of rectal artesunate. Sorry…”

    Much easier to just completely ignore any criticisms regarding design and interpretation and stick with your interpretation.

    • Thanks James for raising this issue publicly. This seems like an important issue and whether or not your analysis is right I think it should be discussed widely and not just behind closed doors at the WHO. I had some questions / comments I was hoping you might have some thoughts on (addressing this to James, but invitation is to anyone including Jon Doe since he also seems familiar with the field):
      1. Do you have an estimate and/or a framework for thinking about the implications of this moratorium on the number of fewer/more lives saved by adopting it? I am struggling because I don’t know what the next best alternative is.
      2. Is my understand correct that the WHO’s initial recommendation of prereferral rectal artesunate draws primarily from this the results of the RCT described in Gomes et al., 2009? Do you know if there are any documents describing their deliberations / any cost-benefit analysis done at the time?
      3. From the Jan 2022 WHO information note, referring to Gomes et al., 2009: “administration of RAS reduced mortality by about 25% in children under 6 years but was associated with a doubling of mortality in older children and adults.” Putting aside the selective use of causal language here, this sort of pattern makes me a bit suspicious (I realize you are not an author, but hopefully he can speak to some of this). Is there some causal explanation in terms of the mechanism of action or differences in physiology or whatever that would explain such a drastic difference between 6 and under and the rest, especially if this is observed only for rectal artesunate and not i.m. or IV ACT? Does the absolute magnitude of risk increase for the older children and adults group even seem plausible for this type of drug? Does the original data was there a somewhat continuous change in the effect size with age or is there a discrete break ~6? Has there been any follow up work investigating the difference?
      4. Even if we grant that the conclusions of the CARAMAL papers are correct, the moratorium on rectal artesunate alone seems a bit wonky. The argument seems the be that because rectal artesunate is pretty effective temporarily or that because afterwards caretakers feel reassured and shift focus to other home priorities, we tend to see lower rates of referral follow through and this leads to worse outcomes overall. From the Oct 2017 note, regarding the recommended pre-treatment referral options for children under 6 with suspected severe malaria, “i.m. artesunate > rectal artesunate > i.m. artemether > i.m. quinine”. I get that CARAMAL looked at rectal artesunate only, but if that’s the proposed causal story, shouldn’t the effect be even stronger for i.m. artesunate since it is a more effective treatment? Changing guidance on rectal artesunate alone seems either myopic or to be coming from a spirit of CYA. Am I missing something? Have similar “real-world” studies been conducted on i.m. artesunate? (This is all putting aside my general skepticism about this kind of story which borders on nudgelord territory).

      Sorry for the dump of questions – hopefully answers are of broad interest to others. And I also apologize if some of them are a bit leading – feel free to disagree with the premises.

      I also can’t resist ending with a terrible joke – when the WHO retracts previously issued guidance, do they play “Won’t Get Fooled Again” in the background?

  5. Can you find a similar study you think is not fatally flawed though? Not on malarial drugs per se, but anything.

    In real life they all are. I mean this one resulted in apparent 4x more harm in the treated group (according to your interpretation due to confounds), it wasn’t even close to seeming beneficial.

  6. Reading the authors’ response to Watson, it is perfectly understandable why they do not want to share anything with him (they are not forced to) and do not want to engage in discussion. Maybe (probably!) Watson is right that the CARAMAL study is flawed, but he is also clearly being an ass writing such tweeter threads and implying a lot of things.

    From what I understand the decision making will be the responsibility of a WHO group who will have access to the study protocol and will review the available evidence – including the commentary of Watson et al. I do not see any issue with that. Quite the opposite in fact: reconsidering recommendations based on emerging evidence is pretty healthy. Let us just hope that they will make the right decision and be aware of the CARAMAL study’s limitations.

    • Jon:

      1. Why do you say that Watson is “being an ass”? The article was published; it’s public. Watson is doing a service by commenting on it. If the authors didn’t want public comments, they shouldn’t have published their data. Also, he’s not “implying a lot of things”; he’s accurately noting that the observational study has major potential confounding and that the protocol and analysis plan are not publicly available. If that bothers the authors, there’s nothing stopping them from sharing the protocol and analysis plan right now. Ideally they would share the anonymized data too, although I understand that could be more difficult.

      2. Your statement that the authors “are not forced” to share anything with outsiders is correct. Indeed, they are not forced to share anything; also, we are not forced to believe anything they say. Trust goes two ways.

      3. As I said above, I don’t see any evidence that Watson is at all “being an ass” by expressing criticism of a high-profile study on a life-and-death topic. But, even if he is, so what? It’s good science to share information with the general public, “asses” included. Also beware the Javert paradox.

      • Thanks for your reply. Let me elaborate what I meant.

        Post-publication review is always very delicate. Commentaries are fine, but trashing somebody’s work on twitter and insinuating misconduct (Watson has now deleted that tweet) pretty much fits my definition of “being an ass”. Imagine someone did that to you, would you be keen on sharing information with them? I personally would probably not.

        Now you should perhaps also know a little bit about the background of this story. Rectal artesunate was initially recommended following a trial led by N. White – who is no other than Watson’s boss and also the last author of Watson et al commentary. This trial also seems to have received serious criticism in commentaries (see e.g. https://link.springer.com/article/10.1186/1745-6215-12-188) . It is then perhaps not such a bad thing that the WHO independently reviews the totality of evidence to make the best recommendations. See also the WHO note explaining their thought process: https://apps.who.int/iris/bitstream/handle/10665/351187/9789240042513-eng.pdf.

        • > Imagine someone did that to you, would you be keen on sharing information with them? I personally would probably not.

          That would never happen to me personally because it would be absolutely a requirement imposed by me on whoever wanted me to perform the study that the study protocol be public, and that the participants sign off on making the anonymized data public as well. The only people being asses are the assholes who think that science is something done behind closed doors with nothing but a backroom deal with specially selected experts involved in the decision making. If people looked at my public data analysis and sent me an email saying they had serious concerns that I hadn’t accounted for important confounders and could be coming to the exact opposite conclusion of what the data warranted, I would immediately contact them and request their help in understanding their concerns. Because I WANT TO SAVE PEOPLES LIVES so I really want to get it right.

          These backroom dealing people can get stuffed. It’s unethical, it’s immoral, it’s extending the time that children are dying of potentially preventable diseases. Watson and others like him want to provide a public service to the world by donating their time to reanalyzing data in the hopes of saving children’s lives, and the establishment wants to backdoor deal to further their own careers. Stick it where the sun don’t shine. Private data and private protocols funded by public money makes me angry as hell. The same with the vaccine trials by Pfizer and Moderna, where are the CSV files? They took billions of public money, they should provide the data to the public.

        • To Daniel:

          “Watson and others like him want to provide a public service to the world by donating their time to reanalyzing data in the hopes of saving children’s lives, and the establishment wants to backdoor deal to further their own careers. ”

          What you need to realize is that Watson and his boss ARE the establishment here. They were e.g. part of the RECOVERY trial for covid that has also often denied access to raw data for reanalysis. In the commentary I shared, one of the criticism on White’s trial was lack of access to data (it needs to be checked though!)…

          Here the Swiss and Oxford teams are essentially competitors who have reached opposite conclusions based on different methodologies, and hopefully independent experts will be able to sort this out. The point of my initial comment was that given Watson’s behaviour, I can totally understand why the Swiss folks are not particularly willing to share information and engage in conversation.

        • Watson can get stuffed too if he’s withholding info on his trials. What it comes down to is this: publicly funded research does not “belong” to the researcher, the data and protocols and all of it belongs to the public.

        • Interesting take on this… It is true that Study 13 (the only big randomised trial of the rectal artesunate suppositories) was not very convincing. It wasn’t very well thought out, mortality was lower than expected and in the end there was no definitive result. The main result reported (considerable reduction in mortality in children who took more than 6 hours to reach hospital) was a post hoc subgroup analysis. In our commentary in BMJ Global Health, we note this limitation. The unconvincing results of Study 13 are probably the main reason that it has taken so long to get rectal artesunate rolled-out in Africa. BUT: the randomised trial data are not the only data we have to support the use of this drug! As I very explicitly said in the above text, we also have a strong mechanistic understanding of the disease and pharmacological data that show good absorption of the drug and rapid parasite clearance. Looking at the totality of the evidence, i don’t think there is any doubt that this drug works, i.e. buys extra time for a child with severe malaria. Also good evidence that it is safe at the recommended doses.

          I personally don’t think I have been “an ass”. The papers have resulted in a change of WHO policy. This is a huge deal. We have not been attacking the authors personally, but only raising serious concerns about the interpretation and analysis of the data. I see this as a core part of my job. Our first commentary was published in July. The authors completely ignored it and made no changes to their main paper (published in October). Twitter is a useful way to reach out to the wider academic infectious diseases community. I should have been more careful about wording regarding protocol, but happy to admit that mistake.

          John Doe writes:
          “What you need to realize is that Watson and his boss ARE the establishment here. They were e.g. part of the RECOVERY trial for covid that has also often denied access to raw data for reanalysis. In the commentary I shared, one of the criticism on White’s trial was lack of access to data (it needs to be checked though!)…”

          Please note:
          -Nick White is not my boss (I have my own fellowship and work in Viet Nam now).
          -Interesting to hear that I am part of the establishment! We were peripherally involved in RECOVERY advising on hydroxychloroquine dosing (chloroquine is a malaria drug so our unit knows a lot about the pharmacology). I have no control over the RECOVERY data.
          -As for Study 13, Nick White was brought in towards the end (trial was already up and running). It was not an Oxford run trial and Oxford do not own the data (so no control there either!). Happy to admit flaws in that study.

          For all the studies that we now do, we aim to publish all data and code necessary for replication of results (see for example our currently running COVID drugs platform trial: https://github.com/jwatowatson/PLATCOV-Remdesivir-Regeneron).

        • Jon:

          You write, “I can totally understand why the Swiss folks are not particularly willing to share information and engage in conversation.”

          Sure, I understand too! They spent a multimillion-dollar grant, they have a high-profile publication, and someone’s questioning their methodology. From a “business” perspective, it’s natural for them to throw as many stumbling blocks as possible in front of the criticism to make it harder to succeed. But from a scientific perspective, it’s a terrible idea to suppress criticism. As to sharing information: if the authors don’t want to give it to Watson in particular, then they should just post it all on the web, so it’s free for all to access.

          From a scientific perspective, the idea that the two research teams are “competitors” is irrelevant. This is public health we’re talking about. If the government gives you millions of dollars to do some research, then the results are not yours to hoard; they’re for the public.

          For more background, see our ladder of responses to criticism.

        • I realize I’m stepping into the middle of some kind of dispute between different researchers about which I have no knowledge. But I will add that the study in BMC Med states:

          “De-identified individual participant data that underlie the results reported in this article are available at zenodo.org (DOI: 10.5281/zenodo.5548261) and access is provided upon reasonable request [40].”

          So, I went to that cite and put in a request for the data. It has now been several days with no response yet. I’ll let you know if I ever get a response, but frankly I’m not optimistic.

        • Dale:

          “Upon reasonable request,” huh? I just went to the website and requested the data too. I can’t remember exactly what was the reason I gave, but it was something like, “The paper is on an important topic so the data should be available.” We’ll see what happens. Given that millions of tax dollars were spent to perform this study, it seems close to criminal for the data not to be shared.

          If I get the data (or do not), I’ll report back.

  7. To Daniel:

    Phil, indeed we should have settled the vaccine effectiveness question in about 5 weeks, get 20-30 volunteers in the armed forces to take the vaxx, then after the full vaccination period, challenge them all with a calibrated dose of virus

    I must be out of touch with what people believe these days.

    Is it still not acknowledged
    that the response depends on age/frailty? And that immunity not only wanes over the course of a few months to years but will select for escape mutants?

    Also, that the vaccine manufacturers never checked for infection/transmission because it was obvious that wasn’t going to be a selling point. No IM vaccine in the past induced mucosal immunity and this was verified for these vaccines with the animal trials. Then later we got confirmation in humans when everyone got covid within a year anyway.

    Is this not yet common knowledge?

    • The “vaccine effectiveness” numbers used for approval were calculated based on preventing infection. I think I and most people realize that the most important part of the vaccine is that it prevents serious downstream consequences, but the ability to prevent infection in the few months after initial injection was high, and is a reasonable proxy for prevention of serious cases. Approving on the basis that shortly after IM injection there was a period during which infection was substantially less probable than without was a sufficiently good proxy for “this vaccine saves lives overall” that it was a reasonable choice of decision variable.

      We’re not going to prevent continued transmission and continued mutation, but we sure as hell are preventing a lot of deaths, for some reason you don’t seem to be able to acknowledge that writing on the wall.

    • Also while the response depends on age and fragility, if we had a vaccine that only worked in people under age 60 would we not have approved it? We weren’t going to give millions of 30 year old healthcare workers the vaccine that worked super well on them because older people couldn’t benefit as well? Ridiculous. If it worked in 20-30 year old military volunteers we’d roll it out to the entire military and all healthcare workers and utility maintenance workers and teachers and food delivery people and whatever… if it didn’t work well in older people we’d continue research into trying to improve the effectiveness for them.

      I’d certainly recommend doing more continued research on the vaccine after approving using a challenge trial on volunteers of course, just do it in parallel with the roll out.

      Remember they didn’t approve the vaccine in children for like a full year after it was approved for adults. That was also really dumb. Expected side effects from COVID exceeded expected harms from the vaccine right from the start, and by large margins. Everyone focused so much on death, which was very low in children, but weren’t focusing on serious illness enough. The much touted rare heart side effects of the vaccine were even worse from actual COVID, and there were a lot of other bad things COVID could do as well. The decision making around all of that was very dumb.

      • My other response is held up because I included two links (to both the FDA and Pfizer CEO agreeing they did not look at infection or transmission).

        But my point is you can’t “settle the vaccine effectiveness question” by looking at a small subset of the population for a short period of time.

        The very idea of a single number to measure effectiveness was nonsense. In what population, and for how long? I said that on here before they even came out.

        Also, you can’t get an accurate number by looking at a snapshot of exponential growth. You have to wait for the plateau phase.

        Start with 100 infected with doubling time of one week. Then after 8 weeks you stop the study and have 100*2^8 = 25,600 cases in control. If the vaccine slows the spread so it doubles every two weeks (50% effective) that would instead be 100*2^4 = 1,600 cases. Thus a 50% effective vaccine appears to be 93.75% effective.

        If you wait to see the percent of pop infected after both plateau, that will give you the number most people think was measured. But it is impossible to know that without waiting for the plateau.

        This is the same error made in those angiostatin “cancer cure” studies touted by the other James Watson in the 90s/2000s.

        • The thing we really should care about is if everyone in the world is infected after receiving the vaxx as compared to before, then what will be the ratio of health burden… Let’s call it QALY loss or some other similar stat…

          But what we can measure in the time frame needed to make this worthwhile is something like how much antibody response will there be and is the antibody response actively effective against the virus.

          That’s all you’re going to get before the decision has to be made whether to roll out the vaccine.

          As such the reduction in risk of infection from a challenge is a decent proxy. You can either challenge with a well controlled dose or you can challenge via a community random exposure. The well controlled dose is the better faster and mechanistically more well controlled protocol.

        • As such the reduction in risk of infection from a challenge is a decent proxy

          There appeared to be increased risk of infection for the first few weeks (due to lymphocytopenia: white blood cells migrating to the lymph nodes and/or dying).

          Then decreased infections for a month or so (as about 1/1000 of the IgG leaked from the blood into the mucosa).

          Then risk of infection returned to baseline and perhaps even was higher after 3-6 months (as the IgG waned to non-neutralizing levels in the mucosa, which is worse than nothing as this increases the rate immune cells get infected).

          In the frail, the response was something like 0-50% of the healthy so cut the timeframes above by about half.

          So how is looking at effectiveness in young healthy people after a few weeks a decent proxy? It would be interesting information but hardly settle the matter. Prior information from previous respiratory viruses (and associated vaccines) gave a far better picture of what was going to happen than such a trial ever could.

          * And of course, that is all highly confounded by testing rates which could easily swamp any true vaccine effect.

        • Again the thing we care about is burden of disease in vaccinated people vs unvaccinated. The ability to prevent infection for a couple months indicates that the antibody response is effective it doesn’t cause antibody mediated enhancement of symptoms and most likely innate immune response to the vaccine is relevant to the disease rather than mistargeted.

          Literally millions of people got COVID all at once around new years 2023 when omicron hit all at once yet rates of hospitalization in vaccinated people without immunocompromise stayed low as did deaths in that group. Why? Because the immune response in vaccinated individuals was successfully targeted at the disease. How could we determine that would likely be true in a short timeframe without exposing lots of people of varying ages to direct challenges and then measuring hospitalization and death rates? By exposing a small number of healthy individuals to a calibrated dose of virus and seeing if the antibody response in the initial weeks after vaccination (say 5 weeks after first dose) was sufficient to noticeably reduce infectivity of the calibrated viral dose.

          That’s all I’m saying. If during the relevant time period people are much less likely to be infected then the immune response is sufficiently well targeted.

        • Literally millions of people got COVID all at once around new years 2023 when omicron hit all at once yet rates of hospitalization in vaccinated people without immunocompromise stayed low as did deaths in that group. Why?

          You’d have to share your data source because according to the CDC data overall deaths and hospitalizations were pretty much the same as the previous year.

          I don’t want to post multiple links but here’s hospitalizations for example:
          https://covid.cdc.gov/covid-data-tracker/#new-hospital-admissions

          All cause mortality was also about the same.

          So this overall data needs to be reconciled with whatever hospitalizations/deaths *in the vaccinated* data you are referring to. I guess the argument is the rates in unvaccinated coincidently rose to approximately cancel out the drop in vaccinated?

          I did expect longer lasting protection against severe disease (eg, T-cells against SARS were still detected a decade later *after infection*) but going with that explanation requires a pretty big coincidence. Seems more likely the data you are talking about was cherry-picked or otherwise biased in some way. But let’s see the data and methods.

        • > overall deaths and hospitalizations were pretty much the same as the previous year.

          With a much, much more infectious variant on the scene.

          Of course, there’s also evidence that it was less virulent, less likely to cause serious disease in lower respiratory tract – which would confound the data on hospitalizations and death. So many other confounding aspects too, like testing rates, in identical infections in those hospitalized, etc.

          You can probably read whatever you’d like into the population level epidemiological data – as is indeed a rather common feature for some people.

        • https://covid19.ca.gov/state-dashboard/#postvax-status

          California Trax hospitalization and death rates in unvaccinated and vaccinated and it’s very clear that in the last few months the hospitalization and the death rates in the unvaccinated was three times as high as among those with at least the primary series they don’t break it down by boosters but it’s very clear that the boosters reduce risk even further they used to plot that data and it was like another 1/3 of the primary vaccinated

        • Anoneuoid –

          It’s still amusing to see how you play plausible deniability games with vaccine data.

          Beyond the population-level epidemiological data, (which I personally think is pretty problematic) there are many studies that show biological evidence that the vaxes AND the monovalent boosters enhanced T cell response and even some that show enhancement of neutralizing antibodies that do more than just fight infection but also enhance immunity against severe covid.

          And there’s even some evidence of similar benefit from the bivalent booster – although seems to me the benefit there seems less uniformly supported.

          I have to say I think it’s bizarre that you’re always implying (but not outright saying) that the vaxes didn’t save any lives. What’s up with that?

  8. I still thinking that it was a huge mistake : planning to stop Rectal artesunate , is this drug safe ? Yes,
    is it feasible to introduce this drug in remote area using CHW? yes
    Alone does it going to strongly decrease deaths? No
    What shall we do to make it work as better we planned ? let us listen to the field

  9. California Trax hospitalization and death rates in unvaccinated and vaccinated and it’s very clear that in the last few months the hospitalization and the death rates in the unvaccinated was three times as high as among those with at least the primary series they don’t break it down by boosters but it’s very clear that the boosters reduce risk even further they used to plot that data and it was like another 1/3 of the primary vaccinated

    Thanks Daniel. I’m travelling atm so not at a PC.

    But here is what I see from eyeballing those graphs along w all cause mortality and covid deaths for California.

    Comparing Mar 2020 – Feb 2021 vs Mar 2021 – Feb 2022, call that 2020 vs 2021:

    – Approx 40-50% fewer covid deaths in 2021
    – Approx 30-40% fewer excess deaths in 2021

    Then:

    – Approx 70-80% vaccinated in the older age groups in 2021
    – Approx 10x more deaths in unvaccinated vs vaccinated in 2021

    Those seem to be the facts requiring explanation, if I made an error please correct it.

    The question then, is how 20-30% of the population dying of covid at 10x the rate of the
    rest in 2021 results in an overall 40-50% reduction in covid deaths and 30-40% fewer excess deaths.

    Shouldn’t we see more like 60% fewer deaths?

    And california is an exception since it is closer to consistent than the US as a whole.

Leave a Reply

Your email address will not be published. Required fields are marked *