This one’s important: Designing clinical trials for coronavirus treatments and vaccines

I’ve had various thoughts regarding clinical trials for coronavirus treatments and vaccines, and then I came across thoughtful posts by Thomas Lumley and Joseph Delaney on vaccines.

So let’s talk, first about treatments, then about vaccines.

Clinical trials for treatments

The first thing I want to say is that designing clinical trials is not just about power calculations and all that. It’s also about what you’re gonna do with the results once they come in. The usual ideas of design (including in our own books, unfortunately) focus on what can be learned from a single study. But that’s not what we have here.

Hospitals have lots of coronavirus patients right now, and they can try out whatever treatments are on the agenda, starting with the patients that are at the highest risk of dying. This should be done in a coordinated fashion, by which I don’t mean a bunch of randomized trials, each aiming for that statistical-significance jackpot, followed by a series of headlines and maybe an eventual meta-analysis. When I say “coordinated,” I mean that all the studies should put their patient-level information into an open repository using some shared format, everything gets registered, all the treatments, all the background variables, all the outcomes. This shouldn’t be a burden on experimenters. Indeed, a shared, open-source spreadsheet should be easier to use, compared to the default approach of each group doing their own thing.

Ok, now that I wrote that paragraph, I wish I’d written it a couple months ago. Not that it would’ve made any difference. It would take a lot to change the medical-industrial complex. Sander Greenland et al. have been screaming for years, and the changes have been incremental at best.

Let me tell you a story. A doctor was designing a trial for an existing drug that he thought could be effective for high-risk coronavirus patients. He contacted me to check his sample size calculation: under the assumption that the drug increased survival rate by 25 percentage points, a sample size of N = 126 would assure 80% power. (With 126 people divided evenly in two groups, the standard error of the difference in proportions is bounded above by √(0.5*0.5/63 + 0.5*0.5/63) = 0.089, so an effect of 0.25 is at least 2.8 standard errors from zero, which is the condition for 80% power for the z-test.) When I asked the doctor how confident he was in his guessed effect size, he replied that he thought the effect on these patients would be higher and that 25 percentage points was a conservative estimate. At the same time, he recognized that the drug might not work. I asked the doctor if he would be interested in increasing his sample size so he could detect a 10 percentage point increase in survival, for example, but he said that this would not be necessary.

It might seem reasonable to suppose that a drug might not be effective but would have a large effect if it did happen to work. But this vision of uncertainty has problems. Suppose, for example, that the survival rate was 30% among the patients who do not receive this new drug and 55% among the treatment group. Then in a population of 1000 people, it could be that the drug has no effect on the 300 of people who would live either way, no effect on the 450 who would die either way, and it would save the lives of the remaining 250 patients. There are other possibilities consistent with a 25 percentage point benefit—for example the drug could save 350 people while killing 100—but I’ll stick with the simple scenario for now. In any case, the point is that the posited benefit of the drug is not “a 25 percentage point benefit” for each patient; rather, it’s a benefit on 25% of the patients. And, from that perspective, of course the drug could work but only on 10% of the patients. Once we’ve accepted the idea that the drug works on some people and not others—or in some comorbidity scenarios and not others—we realize that “the treatment effect” in any given study will depend entirely on the patient mix. There is no underlying number representing “the effect of the drug.” Ideally one would like to know what sorts of patients the treatment would help, but in a clinical trial it is enough to show that there is some clear average effect. My point is that if we consider the treatment effect in the context of variation between patients, this can be the first step in a more grounded understanding of effect size.

I also shared some thoughts last month on costs and benefits, in particular:

When considering design for a clinical trial I’d recommend assigning cost and benefits and balancing the following:

– Benefit (or cost) of possible reduced (or increased) mortality and morbidity from COVID in the trial itself.
– Cost of toxicity or side effects in the trial itself.
– Public health benefits of learning that the therapy works, as soon as possible.
– Economic / public confidence benefits of learning that the therapy works, as soon as possible.
– Benefits of learning that the therapy doesn’t work, as soon as possible, if it really doesn’t work.
– Scientific insights gained from intermediate measurements or secondary data analysis.
– $ cost of the study itself, as well as opportunity cost if it reduces your effort to test something else.

This may look like a mess—but if you’re not addressing these issues explicitly, you’re addressing them implicitly. . . .

Whatever therapies are being tried, should be monitored. Doctors should have some freedom to experiment, and they should be recording what happens. To put it another way, they’re trying different therapies anyway, so let’s try to get something useful out of all that.

It’s also not just about “what works” or “does a particular drug work,” but how to do it. . . . You want to get something like optimal dosing, which could depend on individuals. But you’re not gonna get good discrimination on this from a standard clinical trial or set of clinical trials. So we have to go beyond the learning-from-clinical-trial paradigm, designing large studies that mix experiment and observation to get insight into dosing etc.

Also, lots of the relevant decisions will be made at the system level, not the individual level. . . . These sorts of issues are super important and go beyond the standard clinical-trial paradigm.

Clinical trials for vaccines

I haven’t thought about this at all so I’ll outsource the discussion to others.

Lumley:

There are over 100 potential vaccines being developed, and several are already in preliminary testing in humans. There are three steps to testing a vaccine: showing that it doesn’t have any common, nasty side effects; showing that it raises antibodies; showing that vaccinated people don’t get COVID-19.

The last step is the big one, especially if you want it fast. . . . We don’t expect perfection, and if a vaccine truly reduces the infection rate by 50% it would be a serious mistake to discard it as useless. But if the control-group infection rate over a couple of months is a high-but-maybe-plausible 0.2% that means 600,000 people in the trial — one of the largest clinical trials in history.

How can that be reduced? If the trial was done somewhere with out-of-control disease transmission, the rate of infection in controls might be 5% and a moderately large trial would be sufficient. But doing a randomised trial in setting like that is hard — and ethically dubious if it’s a developing-world population that won’t be getting a successful vaccine any time soon. If the trial took a couple of years, rather than a couple of months, the infection rate could be 3-4 times lower — but we can’t afford to wait a couple of years.

The other possibility is deliberate infection. If you deliberately exposed trial participants to the coronavirus, you could run a trial with only hundreds of participants, and no more COVID deaths, in total, than a larger trial. But signing people up for deliberate exposure to a potentially deadly infection when half of them are getting placebo is something you don’t want to do without very careful consideration and widespread consultation. . . .

Delaney:

One major barrier is manufacturing the doses, especially since we decided to off-shore a lot of our biomedical capacity in the name of efficiency (at the cost of robustness). . . . We want an effective vaccine and it may be the case that candidates vary in their effectiveness. There are successful vaccines that do not grant 100% immunity. The original polio vaccines were only 60-70% effective versus one of the strains, but that still led to a vast decrease in the number of infections in the United States once vaccination became standard.

So, clearly we want trials. . . . Now we get to the point about medical ethics. A phase III trial takes a long time to conduct and there is some political pressure for a fast solution. . . . if the virus is mostly under control, you need a lot of people and a long time to evaluate the effectiveness of a vaccine. People are rarely exposed so it takes a long time for differences in cases between the arms to show up. . . .

Another option is the challenge trial. Likely only taking a few hundred participants, it would have no more deaths than a regular trial. But it would involve infecting people, treated with a placebo(!!), with a potentially fatal infectious disease. There are greater good arguments here, but the longer I think about them the more dubious they get to me. Informed consent for things that are so dangerous really does suggest coercion. . . .

Combining these ideas

Organizing clinical trials for treatments . . . I just don’t think this is gonna happen.

But organizing clinical trials for vaccines? Maybe this is possible. Based on the above discussion, it seems like it’s likely we’ll soon be seeing vaccine trials based on infecting healthy people with the virus and then seeing if they fight it off. If so, I have a few thoughts:

1. I don’t see why you need to give anyone placebos. If we have several legitimate vaccine ideas, let’s give everyone some vaccine or another. If they all work, and nobody gets sick, that’s great. If we’re testing 100 vaccine ideas, then we can guess that most of them won’t be so effective, so we’ll get placebos automatically.

2. As discussed above, coordinate all of these. Certainly no need for 100 different placebo groups.

3. Multilevel modeling all the way. Bayesian inference. Decision making based on costs and benefits, not statistical significance.

Can we make this happen?

P.S. Zad informs us that the above cat is exhausted from quarantine and wants a vaccine immediately if not sooner.

106 thoughts on “This one’s important: Designing clinical trials for coronavirus treatments and vaccines

  1. > signing people up for deliberate exposure to a potentially deadly infection when half of them are getting placebo is something you don’t want to do without very careful consideration and widespread consultation. . . .

    Why would you EVER give people a placebo in such a trial? The answer is because you haven’t a clue how science works and you’re stuck in a NHST theory of how to “test if there’s a difference between this vaccine and nothing”.

    We have many many vaccine candidates (like at least 20 have been in the news) which we can test on animals, say monkeys or something… only among those which show some protective effect in monkeys, and are given to small groups of volunteer humans to show that serious side effects are not a problem… you can then use in a competitive trial design to simply find the best vaccine (measured as something like a combination of effectiveness and produceable at low cost in large volume).

    there is NO reason to give people a placebo and then infect them with a deadly virus when you can give them something where we have biological mechanistic evidence in animals that it works, then infect them with a deadly virus and attempt to figure out which of the things that work work *best*

    Stop thinking binary “this works/this doesn’t” and start thinking continuously “some of these things protect more effectively than others…”

      • The placebo effect hasn’t been shown to be that strong in clinical trials with more objective outcomes (biomarkers), see some references below, but you certainly do need a control group to account for the placebo effect and regression to the mean among other things, and for that, you could utilize an active control group, which could be the standard of care, etc.

        Hr’objartsson A, Gøtzsche PC. Is the Placebo Powerless? N Engl J Med. 2001;344(21):1594-1602.

        Senn S. Francis Galton and regression to the mean. Significance. 2011;8(3):124-126. doi:10.1111/j.1740-9713.2011.00509.x

    • I disagree.

      The fact that a vaccine is effective in animals, even animals that are closely related to humans, is only modestly informative about whether it will be effective in humans. Immune systems vary a lot from species to species.

      Now, if Andrew’s proposal for coordinating these trials were adopted, and all of the vaccine candidates are tried under sufficiently similar conditions and randomly assigned, it is likely that we can identify one that yields the best results, and it is likely that whichever produces the worst results is, in effect, a placebo. But even that is not a certainty. If all of the candidate vaccines prove roughly equally effective, we might never know if it’s because all of them work, or because none of them work and they are all placebos. (Given the past history of difficulties in developing vaccines for other coronavirus infections this scenario is not altogether far-fetched even if the number of candidate vaccines being studied is pretty large.)

      • CLyde –

        >If all of the candidate vaccines prove roughly equally effective, we might never know if it’s because all of them work, or because none of them work and they are all placebos. (Given the past history of difficulties in developing vaccines for other coronavirus infections this scenario is not altogether far-fetched even if the number of candidate vaccines being studied is pretty large.)

        Thanks. That answer my question. I wondered about the “equally effective” scenario and then how would you tease out the magnitude of a placebo effect – but then I thought to myself “That’s probably very unlikely.”

        See Zad’s response above:

        https://statmodeling.stat.columbia.edu/2020/05/19/this-ones-important-designing-clinical-trials-for-coronavirus-treatments-and-vaccines/#comment-1340778

      • Certainly you can’t just test in animals and assume the human vaccine will work. But that’s why we’re testing in humans. The question is does the *infection* procedure work. If the infection procedure doesn’t work, then even placebo would protect you from it, so you couldn’t distinguish between “vaccine works” and “infection procedure doesn’t work”.

        But infection procedures are very simple: swab some controlled dose of virus into the nasal passages. If you show that it works in monkeys, mice, and cell culture you’re extremely likely to see it work in humans. But then once we have vaccine effectiveness data, we can ramp up the infectious dose, and show that protection is conferred even at doses which would otherwise be extremely probable to cause infection.

        If we’re concerned to show the infection procedure works in humans, a small trial of the infection procedure might be justified. It’d NEVER be justified to give matched placebo vaccines to each real vaccine.

        • Daniel –

          > But then once we have vaccine effectiveness data, we can ramp up the infectious dose, and show that protection is conferred even at doses which would otherwise be extremely probable to cause infection.

          Is it possible that the vaccine might work for a highly efficient infection process, but not work (or be less effective, or be less effective with a certain person’s profile) with a more real-world infection scenario? Again, seems maybe unlikely?

        • It’s certainly the case that under real world scenarios where you might get truly massive infectious doses, (like a nurse rushing in to treat a patient who is having severe symptoms, and the face shield falls off just as they get a full force cough right in the face, where you could imagine maybe 1 Billion virions in a single dose no problem) well even the vaccine wouldn’t necessarily protect that person.

          So, yes, the probability of infection is a function of both the vaccine and the real world dose.

          But it would be hard to imagine that a vaccine which does *worse* in the controlled dosing scenario does substantially *better* in the uncontrolled one. So you’re still probably identifying the “best available choice”.

          In reality the main thing is to reach herd immunity and have the virus die out. For that even a 60% effective vaccine would be great.

        • > It’s certainly the case that under real world scenarios where you might get truly massive infectious doses,

          Actually, I was imagining a hypothetical scenario where a vaccine of relatively limited sensitivity might be effective when the individual is exposed to a heavy dose, but not responsive by a less heavy (real world) dose. In that case, presenting a dose that is much more infectious than a typical real world dose might have limited value?

          Of course, I have absolutely no idea of the realisticness of that hypothetical.

        • I realize, of course, that it isn’t the vaccine itself that is fighting the virus. But maybe the antobodies produced by a given vaccine might be more readily activated when responding to a more concentrated or heavier dose?

        • Is there evidence of that? I don’t really think that is how the immune system usually works… there is often a minimum dose of infectious particles to get sick at all, and higher dose sometimes means worse infection. But I’m not an immunologist (undergraduate biology major only).

        • “So, yes, the probability of infection is a function of both the vaccine and the real world dose.”

          Not to mention it is also a function of the most important unknown variable – individual differences in immune system and how it reacts to COVID.

          Besides the obvious fact that it affects elderly and fragile worse, this virus has exhibited some very weird patterns that probably boil down to how much we don’t know about human immune system.

          It would be really great to study immune system of older population for which antibodies were found, but who didn’t experience severe symptoms, consistent with their cohort.
          But considering how much chaos there is out there and the lack of anti-body testing and other issues, it’s a long shot.

      • “But even that is not a certainty. If all of the candidate vaccines prove roughly equally effective, we might never know if it’s because all of them work, or because none of them work and they are all placebos.”

        Excactly.
        However that problem could be mitigated by comparing to the current data (length to recovery in absence of vaccine). We really need good data on those who have recovered so far, to make this non-control type of study meaningful.

    • I think the placebo (almost certainly not randomized 1 to 1) is needed because we are concerned that vaccines could enhance the coronavirus. So the concern is some vaccines cause harm and some do nothing. I don’t think we ever need a placebo again once there is a vaccine with ANY efficacy established.

      Derek Lowe points out this risk:

      “You can see how important these details are – depending on what happens, you could have an infection that doesn’t set off enough of a response to leave behind B and T cells that will remember what happened, leaving people vulnerable to re-infection. Or you could set off too huge a response – all those cytokines in the “cytokine storm” that you hear about?”

      Once you have one candidate that is better than nothing, obviously you never use a placebo again

  2. The ring trial design that was used for Ebola vaccines solves some of these issues (https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(16)32621-6/fulltext#%20). Instead of randomizing huge populations and just waiting for them to get sick, which is both inefficient and unethical, the idea is to wait for a new case, identify all the people in contact with him (and contacts of contacts) and assign the entire group of contacts randomly to either immediate vaccination or delayed vaccination.

    So 1) you increase the power by selecting people at high risk of infection (since they are in contact with a case) and 2) you solve some of the ethical issues by providing the vaccine to everyone, only with a delay in one of the groups.

    • Good points — but where (if at all) does informed consent from contacts and contacts of contacts come in? Also monitoring to see if there might be other relevant factors in play (e.g., if contacts decide to try something else in addition to the assigned treatment)?

  3. Thought I would add two thinks quickly.

    1. I did work in vaccine evaluation/approval, for instance the H1N1 vaccine. The usual phase III trail is usually skipped as showing an antibody effect is considered adequate – especially in emergency situations. And in Canada the approval process itself was about to be skipped, but we made it just under the wire. Now I have been out of vaccines since 2011 so hopefully someone can update folks. The big delays are safety and manufacturing with quality control.

    2. As for wide scale collaboration there are finally moves in that direction – for instance N3C in the US https://covid.cd2h.org/ and the main value of the OHDSI Covid19 study group may well be to inform clinical trial design with observational data and perhaps even trial conduct (i.e. if recruited subjects are the data base you already have all their prescription and medical history data.)

  4. I don’t see why you need to give anyone placebos. If we have several legitimate vaccine ideas, let’s give everyone some vaccine or another. If they all work, and nobody gets sick, that’s great.

    What if there’s a problem with your infection procedure so that nobody would have gotten sick even if they had merely gotten a placebo?

      • assuming you’re doing something like taking a swab with active virus and swabbing it in subjects noses to infect them, it’d be simple to use the same procedure to swab plates of cultured cells and show that they are infected. You only need to show that the virus exists on the swab and its active / infectious and in doses known to be sufficient to cause infection from background information (we have estimates of the ID50 of viruses like SARS-1 for example, it’s about 250 virions for mice for example). Most likely we should use a titration scheme, where you’re starting out by giving something approximating the ID50, so that there’s only ~ a 50% chance any given person would even get sick without treatment. If the vaccine is effective you’d observe many fewer than 50% of the subjects getting infected. Then when you have initial estimates of effectiveness from that, move on to giving the more effective vaccines to a next round of subjects and increasing the infectious challenge to something like ID90, and then 2x ID90. By the time you’re seeing swabs that reliably infect cells in culture at viral loads that are 2x the dose that produces 90% infection in untreated monkeys, you’d be pretty damn confident your vaccine was working. You could do this trial in a few months this way with very manageable risk. if you were doing volunteers in the 20-30 year old age range, the estimated uncontrolled IFR is already maybe 0.1% so with controlled doses it’d probably be even lower. You could get very decent estimates of effectiveness by infecting as few as say 20 patients in each batch, say 10 vaccines, building up to 3 different dosage levels, with ~ 600 or 1000 patients you could get very good data on 10 vaccines and you’d have well below 1/1000 death risk. The probability you’d see even one death if done well would be small.

        on the other hand, you could take 500 people and give them the vaccine and 500 people and give them the placebo, and send them out to be nurses at COVID wards and get almost no information and potentially multiple deaths. Which seems more ethical?

        • This point…. Having been both in the clinical and research wings of this blech, *viral load* is IMO an extraordinarily important endpoint. Reason being as there are 2 types of SARS_Cov2 patients – those who are symptomatic and those who are not. We still don’t have a firm grasp on the latter due to to-date fairly mediocre testing overall (refer to noted issues on testing kits, how many reasonably solid testing kits are available, and some idea of the proportion of infected but asymptomatic patients).

          Reducing pathogen ‘bio-load’ is the first step to demonstrating you can really do anything. The rest is left to the response of the patient themselves (where monitoring and repeated testing and viral load estimated is critical).

          In the clinical setting, patients who are received, in general, either 1) didn’t need to be tested in the first place (think cold/allergy discrimination by the patient from what they perceive could be early onset covid-19 symptoms), or 2) they come in nearly dying. The middle of the pack, those who are mildly symptomatic, may or may not even be tested, from the simple behavior that a ‘covid-aware’ vs ‘non-covid-aware’ patient have been recommended simply to stay home – some later reaching the ER due to severe symptom onset. This could be one explanation why the ER setting may be rather binary in patient types 1) and 2) above.

  5. Human challenge (intentional infection) trials are scary, but the statement that ” Informed consent for things that are so dangerous really does suggest coercion.” seems extremely dubious (verging on totally wrong) to me.

    Of course it depends on what population you use, but the suggestions I’ve seen involve healthy young adults. The US Navy has seen 2284 infections among military personnel, with 1 death and 1,082 recoveries. (And this is a male-biased population, so the risk may even be a bit higher than for a comparably young and healthy 50/50 male-female population. Nor are *all* Navy personnel particularly young.)

    A 1/2000 or even 1/1000 risk of death for something with a huge public benefit is NOT irrational to accept, unless you consider joining the military to be inherently irrational. Frankly it is probably less dangerous than some things people do for no objective benefit (eg BASE jumping, climbing Mt Everest…)

    • confused –

      > A 1/2000 or even 1/1000 risk of death for something with a huge public benefit is NOT irrational to accept, unless you consider joining the military to be inherently irrational.

      The problem being that only gives you limited information. Eventually, you have to know the safety level among a more varied sampling, including people of much higher risk. So you run into the same question about how to speed up the process with “challenge trials’ in that sense also.

      • I agree with Joshua here.

        In WWII, people didn’t just volunteer to fight and then get sent off to storm Omaha Beach, they were _conscripted_ to fight and sent off to storm Omaha Beach. Most people don’t have a problem with that, and didn’t at the time either. In the past people have been asked, and sometimes forced, to take much much higher risks than they’d experience in a vaccine challenge trial like this.

        The infection fatality rate is very low among otherwise-healthy people under 40 years old. Explain the risks; move them into nice quarters — there are lots of empty luxury hotels right now; get them great on-site medical care and a generous life insurance policy and a generous compensation policy in the event that they develop a chronic health issue; give them the vaccine; and try to infect them.

        It’s tempting to say they should be generously compensated for participating, but I’m afraid you would then end up with a trial full of poor people, which would be ‘bad optics’ and possibly ethically questionable, so probably participants should not be paid, or should be paid a relatively token amount (maybe the equivalent of minimum wage or something).

        • I actually wasn’t thinking of WWII, but current conflicts, where (at least in the US) the military is all-volunteer. (Society’s acceptance of risks – and government orders – is not really comparable between the 1940s and today.)

          And yeah, I agree that lots of compensation could make things questionable. But they should absolutely be guaranteed free medical care if they end up hospitalized from it.

      • You can do vaccine *safety* trials with a less healthy, broader population, but do the challenge trial for *effectiveness* in a young healthy population.

        I’m not sure it’s really needed to show effectiveness, though. If the vaccinated people produce neutralizing antibodies at a good level (comparable or better than what’s seen in recovered patients), that might be good enough (for a crisis situation).

        • confused –

          > You can do vaccine *safety* trials with a less healthy, broader population, but do the challenge trial for *effectiveness* in a young healthy population.

          I don’t understand your argument. The problem is, you need to test safety *and* effectiveness with less healthy populations as well. That can take a very long time unless you do challenge trials.

        • >> The problem is, you need to test safety *and* effectiveness with less healthy populations as well

          I don’t know. If you can prove:
          – it is safe (in old and young)
          – it produces good levels of neutralizing antibodies (in old and young)
          – it actually protects against disease (in the young)

          shouldn’t that be good enough?

          It might even be enough just to show that it produces good enough levels of neutralizing antibodies (same or better than naturally recovered people), in a crisis situation like this. I’m not sure we really need trials vs. actual infection. Neutralizing antibodies aren’t going to just do nothing, I wouldn’t think.

  6. Andrew –

    Interesting post.

    > In any case, the point is that the posited benefit of the drug is not “a 25 percentage point benefit” for each patient; rather, it’s a benefit on 25% of the patients.

    How is “benefit” being defined there? Is work/didn’t work a clear distinction? It could be extended life for a certain period of time. It could be led to a less serious infection among those who survived. It could mean a shorter period of infectiousness, etc. There is a longitudinal component and a dose-dependent effect aspect that, it seems to me, can sometimes get lost in treatment/effect analyses.

    Seems to me that an important part of this should be a focus on the mechanism of causality. And yes, to the main point of your post – it seems that collecting a comprehensive set of data on the patients in the trial should be a really big part of the efforts, and should actually be folded into an assessment of “power” and the best way to made the data a large set is to standardize the data and pool them into an easily accessible database. The standardization of the data piece seems very important to me.

    But it seems to me that a clear elucidation of mechanism of causality should also be an important part of considering the “power” of a study. Yes, it gets kind of messy and subjective and speculative – but I think that often the subjectivity of more easily measured “clinical” data can often be covered in a relatively simplistic look at the associations between variables only. There can be an illusion of non-subjectivity, that you think you’re avoiding the subjectivity of speculating about mechanisms – but in reality you aren’t.

    > Once we’ve accepted the idea that the drug works on some people and not others—or in some comorbidity scenarios and not others—we realize that “the treatment effect” in any given study will depend entirely on the patient mix.

    As there can almost be an infinite list of possibly important data to collect, a focus should be on the mechanism of causality – as that should inform the data collection criteria. Perhaps you chunk the data according to p
    alternative theories on the mechanism of causality.

    And then there’s also the question of a dose-dependent relationship, and the possibility of drug interactions or mediating/moderating effects from other treatment conditions – again all looping back to (1) a lot of focus on the mechanism of causality and (2) a lot of focus on coordinated data collection.

    > but in a clinical trial it is enough to show that there is some clear average effect.

    Do you mean specifically in the current situation when there’s such a compelling public health crisis at hand? I wonder about the real benefit of finding an average effect without a clear idea of the mechanism of causality and without a really comprehensive dataset as you describe.

    • I dunno … I’m not sure what you mean by ‘mechanism of causality’. In epi parlance, we generally use ‘mechanism of action’, which granted, is somewhat easier to understand in antibiotics (bacterial and viral agents) than in other areas of medicine. But to second Andrew’s point, ‘fast’ is a rather critical need (though yes, you don’t want the agent to produce more harm than intended effect). Leave it to ‘Operation Warp Speed’.

      I do agree though that a very thorough covid-19 patient registry that has been anonymized and captures a shittonne of variables, is absolutely warranted. Current datasets available are kinda thready (for too many reasons to mention: ownership and ease-of-access by clinical/research associations, different registries tied to a single specialties with different foci), or what data are more easily accessible may simply not be sufficiently robust. I’m not saying registries don’t exist, but to paint some broader brushstrokes in the clinical domain, they do require being stitched together in some resonably informative way for analyses.

  7. “The other possibility is deliberate infection. If you deliberately exposed trial participants to the coronavirus, you could run a trial with only hundreds of participants, and no more COVID deaths, in total, than a larger trial. But signing people up for deliberate exposure to a potentially deadly infection when half of them are getting placebo is something you don’t want to do without very careful consideration and widespread consultation.”

    I’m afraid this is just nutty. With deliberate exposure, some of the deaths would be avoidable. Why not drop the number of participants even more by only using the highest risk population?

    Andrew and commenters picked up on the ongoing confusion about placebos. Placebos are essential when patients are asked to fill out a questionnaire at the end of a trial. If on the other hand the data is all about whether and when someone tests positive, what does a placebo accomplish? We don’t care whether someone feels better or not, that would just add pure noise to the study.

    • In a well designed study building up from small infectious doses using volunteers at low risk (20-30 year olds no comorbidity) you’d have very close to zero deaths from intentional infection trials. It’d be more dangerous to work as a grocery store clerk or a nurse than participate in a well designed intentional infection trial.

  8. I’m not keen on this ditching a placebo idea.

    1. Well firstly half the sample doesn’t need to be a placebo.

    2. If the idea is that “well we have 100 candidates, at least some of them will suck and they will be a placebo for us”, in that case the relative increased risk of running the trial with 101 candidates – one of which you know does not work – would be smaller. Further you would be able to answer questions like “well, is the worst performing one actually useless, or simply not as good as all the others”, which might help a lot if it turns out the worst one is also the safest and cheapest and quickest to manufacture.

    3. Our infection method is unlikely to be that reliable, nor is our test method. If we are doing stuff like ring trials – where we only know that the patients have been in contact with an infected – then we really don’t know for sure everyone will be infected. If we are using antibody tests, we might also get a bunch of false positives and/or false negatives. A placebo will help pin that down. Even more if our measure is something more subjective, like symptom scoring or admission to ICU.

    • The real concern is to unconfound the infection procedure with the protectiveness. If you try all the drugs and get essentially no-one infected, no matter what infectious dose you use… then you might choose to run a small placebo trial to show that the infection procedure works… but if you run the vaccines and some people get infected under some conditions… then you’d know your infection procedure worked and the placebo trials would be unnecessary.

      The outcome we’re optimizing for should be “expected global reduction in morbidity and mortality from choosing this candidate and running with it” which involves cost to produce and time to produce inherently counting into the objective function.

      • But if you run the placebo trial as a separate experiment, it’ll be hard to ensure the new experiment actually replicates the conditions of the first, so you might see infections in the second experiment but that’s just because some systematic issue in the first experiment didn’t recur.

        • I think the key here is understanding the scientific process as information building rather than “proving a statistically significant difference”. There’s no point in exposing the placebo group if the information is unnecessary… so you do the vaccine groups first and see what information you get out of it. If you don’t get sufficient information, you devise experiments to add to your information.

          This is all totally illegitimate in a frequentist philosophy, at least without some kind of tortured design. Corey has posted blog posts showing how this kind of sequential adapted design can never really work under strict frequency based analysis. But it’s totally natural under a Bayesian analysis in which you are learning simultaneously about your infection procedure, the infectious dose, and the protective effect of each vaccine, and using partial but not complete pooling across trials.

  9. There are three steps to testing a vaccine: showing that it doesn’t have any common, nasty side effects

    Which vaccine trials are really checking for this?

    To evaluate the efficacy of existing vaccines against infection with SHC014-MA15, we vaccinated aged mice with double-inactivated whole SARS-CoV (DIV). Previous work showed that DIV could neutralize and protect young mice from challenge with a homologous virus14; however, the vaccine failed to protect aged animals in which augmented immune pathology was also observed, indicating the possibility of the animals being harmed because of the vaccination15. Here we found that DIV did not provide protection from challenge with SHC014-MA15 with regards to weight loss or viral titer (Supplementary Fig. 5a,b). Consistent with a previous report with other heterologous group 2b CoVs15, serum from DIV-vaccinated, aged mice also failed to neutralize SHC014-MA15 (Supplementary Fig. 5c). Notably, DIV vaccination resulted in robust immune pathology (Supplementary Table 4) and eosinophilia (Supplementary Fig. 5d–f). Together, these results confirm that the DIV vaccine would not be protective against infection with SHC014 and could possibly augment disease in the aged vaccinated group.

    https://www.nature.com/articles/nm.3985

    A couple weeks ago I looked into every vaccine listed here:

    Ad5-nCoV (CanSino Biologics)

    * 18-60 in general good health

    * https://clinicaltrials.gov/ct2/show/NCT04341389

    piCoVacc (Sinovac):

    * 18-59 healthy individuals

    * http://www.sinovac.com/?optionid=754&auto_id=897

    ChAdOx1 nCoV-19 (U of Oxford, Vaccitech):

    * 18-55 healthy adults

    * https://clinicaltrials.gov/ct2/show/NCT04324606

    BNT162 (BioNTech, Pfizer):

    * 18-85 healthy participants

    * https://clinicaltrials.gov/ct2/show/NCT04368728?term=BNT162&draw=2&rank=1

    mRNA-1273 (Moderna):

    * 18-55 healthy adults

    * https://clinicaltrials.gov/ct2/show/NCT04283461

    INO-4800 (Inovio):

    * 18-50 healthy volunteers

    * https://clinicaltrials.gov/ct2/show/NCT04336410?term=INO-4800&draw=2&rank=1

    NVX-CoV2373 (Novavax, Emergent Biosolutions):

    * Healthy adults

    * https://ir.novavax.com/news-releases/news-release-details/novavax-identifies-coronavirus-vaccine-candidate-accelerates

    ad26 SARS-CoV-2 (Johnson & Johnson)

    * Healthy adult volunteers

    * https://www.hhs.gov/about/news/2020/03/30/hhs-accelerates-clinical-trials-prepares-manufacturing-covid-19-vaccines.html

    Unnamed (Sanofi, GSK):

    * NA

    * https://www.gsk.com/en-gb/media/press-releases/sanofi-and-gsk-to-join-forces-in-unprecedented-vaccine-collaboration-to-fight-covid-19/

    You can see all are only in healthy people and only one plans to include people over 60. It seems they only plan to check for safety in the population at risk of an adverse response after wasting 6-12 months testing it in young/healthy people. That makes me think most of these vaccines are meant as some kind of investment scam.

    • And the other problem is that the main safety issue is antibody-dependent enhancement, where the later response to a similar virus becomes pathological only after the antibodies have waned:

      Low levels of antibody did not enhance disease, intermediate levels exacerbated disease, and high antibody titers protected against severe disease. These findings have major implications for vaccines against flaviviruses. Indeed, recent vaccine trials have shown evidence of severe disease in some recipients who were previously exposed to virus.

      https://science.sciencemag.org/content/358/6365/929

      So to establish the safety correctly animals/volunteers must be challenged at various points after vaccination until the antibodies are undetectable. This would probably take a few years at least for humans but they should be able to do this in animals quicker (but all the animals studies have also been in young/healthy rodents/primates that do not get very ill, if at all, from the virus).

      • And also note that an additional exclusion criteria for these vaccine trials is a history of covid-19 diagnosis or detection of antibodies. That is another at-risk group being excluded by these safety trials.

      • We can’t really wait for a few years, though.

        Antibody-dependent enhancement might be an issue (though I think there are promising signs for this virus, we obviously don’t know what happens years later, since nobody’s had antibodies for more than 4-5 months). But if we want a vaccine in an useful timeframe, that will mean accepting more risk than the usual vaccine.

        (It may be largely irrelevant, though. While this isn’t a flu virus, flu pandemics tend to end on their own in say 12-20 months. COVID was already spreading outside China by January, so we’re ~4 months in already. The pandemic may be over before a vaccine can be distributed.)

        • “The pandemic may be over before a vaccine can be distributed.”

          Color me amused. Please elaborate. Somehow Eric Trump is beginning to metastasize my spleen with such comforting claims.

        • >>Color me amused. Please elaborate.

          Optimistic timelines for availability of a vaccine are 12-18 months (from when serious work started, several months ago).

          In the 2009-10 H1N1 “swine flu” pandemic, a vaccine was widely available to the public by November, ~10 months in … but the second wave was already declining by then.

          So if this pandemic follows a timeline similar to 2009-10 (or a spring wave / fall-winter wave pattern more generally) it will be over in 12 to 18 months (from the beginning … ~8-14 months from now).

          If a third wave occurs later in 2021, it might be available in time to help with that.

          Either way, a vaccine would still be useful in the “endemic” phase after the pandemic is over, so it’s still worth developing even if it can’t be ready in time for the pandemic itself.

        • I also don’t see this as a particularly “comforting” claim, because it basically means that – if true – we will probably have a pretty large proportion of the population infected.

    • No, they only mean to risk the lives of high-risk people after having shown a protective benefit in low-risk people.

      Do you really want them to start with those at highest risk?

      Separately, a vaccine that is safe and effective only for the young and healthy is still of significant potential benefit to the old and immunocompromised. Such a vaccine would not be used to innoculate the aged and immunocompromised, but would still protect them by keeping infection prevalence way down, decreasing the chance they ever get infected. This is the real benefit of herd immunity, and the moral imperative behind those who are candidates for a vaccine actually getting themselves vaccinated — to protect those who can’t. We see that argument with measles all the time.

      • Do you really want them to start with those at highest risk?

        If they plan to vaccine that type of person (~30% of the population) then yes.

        Such a vaccine would not be used to innoculate the aged and immunocompromised, but would still protect them by keeping infection prevalence way down, decreasing the chance they ever get infected. This is the real benefit of herd immunity, and the moral imperative behind those who are candidates for a vaccine actually getting themselves vaccinated — to protect those who can’t. We see that argument with measles all the time.

        I haven’t seen anyone running these trials say they plan on only vaccinating the young and/or healthy. I think the idea would get a bit more scrutiny if they did.

        And herd immunity means the infectious agent dies out. Measles is clearly still with us. Btw, vaccinating at just below the eradication threshold (as is the policy for measles) is the dumbest thing you can do when it comes to vaccines. The susceptible accumulate in the population due to waning antibodies, vaccine failure, missed vaccinations, etc. Eventually a tipping point is reached and the honeymoon period ends, then you have a huge rebound pandemic. In the case of measles this will primarily be in adults who have a much worse illness than children while also having parental/career responsibilities.

        • Measles could be eliminated but the percentage of the population required to be vaccinated to achieve herd immunity is very high (more than 95% maybe) because it is so infectious (R~25).
          Influenza vaccination is only cost-effective when targeted at high risk groups (elderly, pregnant) and of course cannot be eliminated due to its animal reservoir.

        • Measles could be eliminated but the percentage of the population required to be vaccinated to achieve herd immunity is very high (more than 95% maybe) because it is so infectious (R~25).

          Indeed, the original plan was to “eradicate” (not eliminate, which is a lesser goal of no local chains of transmission persisting for over a year: https://www.ncbi.nlm.nih.gov/pubmed/15106085) measles in the US by 1967 just “because it was there”.

          The availability of potent and effective measles vaccines, which have been tested extensively over the past 4 years, provides the basis for the eradication of measles in any community that will raise its immune thresholds to readily attainable levels. Effective use of these vaccines during the coming winter and spring should insure the eradication of measles from the United States in 1967.

          https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1919891/

          To those who ask me, “Why do you wish to eradicate measles?,” I reply with the same answer that Hillary used when asked why he wished to climb Mt. Everest. He said, “Because it is there.” To this may be added, “. . and it can be done.”

          https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1522578/

          Langmuir later wrote about this “blunder”:

          The Center for Disease Control (CDC) led in mounting the program with a formal paper at the American Public Health Association annual meeting in Miami in the fall of 1966. Two colleagues and I wrote the “official statement” which outlined in detail unqualified statements about the epidemiology of measles and made an unqualified prediction. My third position in the authorship of this paper did not adequately reflect my contribution to the work.14 I will make but two quotes:
          1. “The infection spreads by direct contact from person to person, and by the airborne route among susceptibles congregated in enclosed spaces.” (Obviously the ideas of Perkins and Wells had penetrated my consciousness but not sufficiently to influence my judgment). 2. “Effective use of (measles) vaccines during the coming winter and spring should insure the eradication of measles from the United States in 1967.” Such was my faith in the broad acceptance of the vaccine by the public and the health professions and in the infallibility of herd immunity.
                  […]
          There are many reasons and explanations for this rather egregious blunder in prediction. The simple truth is that the prediction was based on confidence in the Reed-Frost epidemic theory, in the applicability of herd immunity on a general basis, and that measles cases were uniformly infectious. I am sure I extended the teachings of my preceptors beyond the limits that they had intended during my student days.

          In the relentless light of the well-focussed retrospectiscope, the real failure was our neglect of conducting continuous and sufficiently sophisticated epidemiological field studies of measles. We accepted the doctrines imbued into us as students wikout maintaining the eternal skepticism of the true scientist.
                  […]
          Clearly we must revise our theory and recognize that these outbreaks must be airborne in character involving exposure to aerosols presumably created by the rare super-spreader who contaminates a large populated enclosed space such as a school auditorium or gymnasium. These have happened sufficiently often to prove the far sightedness of Perkins and Wells when the rest of us were smugly secure in our epidemic theories, our traditional faith in contact infection and herd immunity.

          https://www.ncbi.nlm.nih.gov/pubmed/6939399

          Now the plan has somehow morphed into vaccinating just underneath that threshold.

          The first achieves coverage of 95%, which is above the critical proportion for eradication (91% when R 0 = 11). The number of infections immediately plummets, and no further infections are seen. The effective reproductive ratio is depressed below 1 after the introduction of vaccination and never rises above it again. Eradication is achieved. The second scenario represents the impact of a vaccination programme that reaches high levels of coverage (85% of all new-borns) which are, nevertheless, not high enough to lead to eradication of the agent. However, for the first 15 years after the introduction of vaccination, it appears as if eradication has been achieved, there are no infections. Then, suddenly, a new epidemic appears as if from nowhere. This is an illustration of a phenomenon known as the ‘honey- moon period’. This is the period of very low incidence that immediately follows the introduction of a non-eradicating mass vaccination policy. Thishappens because susceptible individuals accumulate much more slowly in a vaccinated community. Such patterns were predicted using mathematical models in the 1980s 6 and have since been observed in communities in Asia, Africa and South America 7 . Honeymoon periods are only predicted to occur when the newly introduced vaccination programme has coverage close to the eradication threshold.

          https://pubmed.ncbi.nlm.nih.gov/12176860/

        • I think 95% is not really ideal, but the best you can hope for these days, with the level of resistance to vaccines many people have.

        • I think 95% is not really ideal, but the best you can hope for these days, with the level of resistance to vaccines many people have.

          “Anti-vaxxers” are a boogeyman made up by the media. The vast majority of people without measles immunity are either poor, have some condition that precluded vaccination, got faulty vaccines, had their antibodies wane, or it simply didn’t take for whatever reason.

          Don’t feel like digging up sources again today but I’d guess there are 50-100 million susceptibles in the US right now, at least half of whom were vaccinated 30-50 years ago.

        • There is significant anti-vaccination sentiment in some parts of the US, but it is strongly associated with certain subcultures & therefore not noticeable everywhere. I have personal experience – not just from the media. It definitely exists.

        • There is significant anti-vaccination sentiment in some parts of the US, but it is strongly associated with certain subcultures & therefore not noticeable everywhere. I have personal experience – not just from the media. It definitely exists.

          It exists but in negligible numbers compared to other reasons for lack of immunity to measles.

        • ““Anti-vaxxers” are a boogeyman made up by the media.”

          https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6657116/

          I was hoping there was data in there about the number of people susceptible due to “anti-vaxxers” vs other reasons but there wasn’t anything like that.

          That article contributes to the “boogeyman” image I am talking about. They do cite this paper though:
          https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5997312/

          That study showed that vaccination rates were 90%+ in all states that allow non-medical exemptions except Colorado. I couldn’t find what source they used for MMR vaccination rate, but the overall vaccination rate is reported to be ~91% in 2017. So that means all those states were vaccinating at higher rates than average:

          Percent of children aged 19-35 months receiving vaccinations for:
          Measles, Mumps, Rubella (MMR) (1+ doses): 91.5%
          Source: Health, United States, 2018, table 31 pdf icon[PDF – 9.8 MB] (data are for 2017)

          https://www.cdc.gov/nchs/fastats/immunize.htm

        • The World Health Organization in 2019 listed Vaccine hesitancy as one of the top 10 threats to global health.
          https://www.who.int/news-room/feature-stories/ten-threats-to-global-health-in-2019

          Of course they listed a flu pandemic rather than a cororavirus pandemic as another top ten candidate. Close but no cigar.

          Given the US President’s prior anti-vaxination attitudes and the fact that in some parts of the world WHO polio vactination teams get murdered, I would not discount the anti-vaccination movement.

        • Well, we’ve had 4 flu pandemics in about a hundred years (1918, 1957, 1968, 2009)… it was a more likely thing to worry about. We knew 100% that influenza could and did cause pandemics; people were concerned about coronaviruses with pandemic potential since SARS, but it wasn’t a known fact the way influenza was.

        • I don’t think you *start* with the more at-risk people (in phase I), I think that happens in a later phase (II or III, can’t remember).

          But I think it would necessarily be tested in them before general distribution.

          >>And herd immunity means the infectious agent dies out.

          Not necessarily. It means the epidemic wanes, but not necessarily that the pathogen becomes extinct. (Especially since herd immunity can happen in one place, but not another. If the whole world hit herd immunity simultaneously, maybe it would be different.)

          Over time people are born who haven’t been exposed, and (for some pathogens) immunity wanes. So, pre-vaccination, there were cyclical epidemics of many diseases. Measles worked this way, IIRC – it’s super contagious, so it would tear through the population and infect basically everybody, but N years down the road, those born since the last outbreak had no immunity, so there was another outbreak…

        • I don’t think you *start* with the more at-risk people (in phase I), I think that happens in a later phase (II or III, can’t remember).

          But I think it would necessarily be tested in them before general distribution.

          We already know this virus and similar viruses induce a completely different response in young/healthy vs old/sick animals and people. So we know we can’t extrapolate between the two groups. It could even be that some vaccines induce a pathological response in the young/healthy, but not the old/sick…

          And why can’t they test it in old/sick mice or macaques?

        • >>We already know this virus and similar viruses induce a completely different response in young/healthy vs old/sick animals and people. So we know we can’t extrapolate between the two groups. It could even be that some vaccines induce a pathological response in the young/healthy, but not the old/sick…

          Unless it is a weakened live virus vaccine, the effects of the virus itself shouldn’t correlate with any potential side effects of the vaccine, unless I’m missing something fundamental.

        • Unless it is a weakened live virus vaccine, the effects of the virus itself shouldn’t correlate with any potential side effects of the vaccine, unless I’m missing something fundamental.

          The severe illness occurs primarily later than one week after symptom onset. PCR tests are still positive for another couple weeks but after about a weekits reported no more virus can be isolated and people are no longer infectious.

          Whereas the virus was readily isolated during the first week of symptoms from a considerable fraction of samples (16.66% of swabs and 83.33% of sputum samples), no isolates were obtained from samples taken after day 8 in spite of ongoing high viral loads.

          https://www.nature.com/articles/s41586-020-2196-x

          Assuming an incubation period distribution of mean 5.2 days from a separate study of early COVID-19 cases1, we inferred that infectiousness started from 2.3 days (95% CI, 0.8–3.0 days) before symptom onset and peaked at 0.7 days (95% CI, −0.2–2.0 days) before symptom onset (Fig. 1c). The estimated proportion of presymptomatic transmission (area under the curve) was 44% (95% CI, 25–69%). Infectiousness was estimated to decline quickly within 7 days.

          https://www.nature.com/articles/s41591-020-0869-5

          This is also why all the reports of drugs that are supposed to slow viral replication (hydroxychloroquine et al) in people already hospitalized aren’t really checking if it works. People are usually sick for a week before showing up at the hospital, at that point the virus is already winding down.

        • That makes sense and is consistent with what I’ve read before about viral load/infectiousness, but I don’t see how that relates to potential vaccine side effects/adverse effects.

          If the vaccine is a weakened live virus type, then one possible adverse effect is actually getting the disease or a version of it. (There’s a form of polio derived from the live-virus polio vaccine.) But the Moderna vaccine (the one that just reported a phase I trial) is an RNA vaccine – no actual virus included, I don’t think – so wouldn’t that be different?

        • That makes sense and is consistent with what I’ve read before about viral load/infectiousness, but I don’t see how that relates to potential vaccine side effects/adverse effects.

          If the severe illness is occurring after peak viral load (it may be even after there is no actual viral particles left at all), that indicates it is the body’s response to the virus that is responsible for that pathology rather than the virus itself. Ie, the vaccine could trigger the same response.

        • They plan to vaccinate everybody. Start the testing with those most able to fight off potential complications to see if it has any benefit. If it does, move on to the rest of the population.

          Herd immunity is a localized phenomenon. Getting R below 1 in a given population says nothing about eradicating a disease from the globe.

          I have no idea where you came up with your idea of the “policy for measles”.

          And after 30+ years, doesn’t a honeymoon simply become a happy marriage?

        • I have no idea where you came up with your idea of the “policy for measles”.

          Measles vaccination rates in the US is ~90% by two years old (and has been for awhile): https://www.cdc.gov/mmwr/volumes/68/wr/mm6841e2.htm

          It’s usually said you need to vaccinate ~95% of the population to get herd immunity: https://www.who.int/immunization/sage/meetings/2017/october/2._target_immunity_levels_FUNK.pdf

          It is apparent the policy is to vaccinate at just below the vaccination threshold.

          And after 30+ years, doesn’t a honeymoon simply become a happy marriage?

          It may take another 50 years, but its coming:

          For instance in the case of a transmission potential of 10%, re-emergence will not be a problem until almost 70 years after the be- ginning of large scale vaccination which in Fig. 2 was sup- posed to be in 1980. Assuming a transmission potential of 30%, the virus would have re-emerged already before the year 2000. Furthermore, Fig. 2 shows that with an as- sumed transmission potential of 20% the circulation of the virus would re-emerge after 30 years and will not disappear anymore. With a mean duration of protection of 20 years, re-emergence would occur approximately 10 years earlier for the scenarios where the transmission potential is between 10 and 30% (see Fig. 1).

          More importantly, the model assesses the change of the immunity profile in the population following the start of the vaccination programme. In Fig. 3, we can see how this shift from a population whose immunity was induced by classical infection is replaced by vaccine-induced immunity, which is less protective. For these simulations an average duration of vaccine protection of 25 years and a vaccine-modified transmission potential of 10% was assumed. It is clear that while high routine vaccine coverage achieves a temporary elimination of measles infection in the population, the pro- portion of vaccinees becoming susceptible to the vaccine modified form of infection increases slowly. As shown in Fig. 3, 25 years after starting vaccination, 28% of the total population will be vaccinated, 18% will still have protective vaccine-derived immunity and 10% will be susceptible to vaccine-modified infection. Fifty years after the beginning of the vaccination programme, 55% of the total population have vaccine-induced immunity, and 31% will be susceptible to vaccine-modified infection. It is clear that as time passes, the pool of susceptible individuals will increase until a critical threshold is reached where sustained vaccine-modified measles transmission occurs.

          https://www.sciencedirect.com/science/article/pii/S0264410X03004493

        • You’ll have to support your claim that the outcome is equivalent to the policy. Policy is to get everyone vaccinated.

          If delaying outbreaks for 50 years isn’t success, then I don’t know what is. Nothing in your cite suggests the “huge rebound pandemic” you’re afraid of. In fact, the Discussion says:
          “One of the principal insights gained from this model is that waning of immunity and subsequent mild subclinical infection in vaccinees would not necessarily result in a rapid re-emergence of measles, but that the re-emergence is realistic and essentially depends on parameters for which no good estimates exist.”

          Non-rapid re-emergence of mild infection, 50 years later? If that manner of outcome is what you get from “the dumbest thing you can do when it comes to vaccines”, then sign me up for anything and everything you can do when it comes to vaccines.

        • You’ll have to support your claim that the outcome is equivalent to the policy. Policy is to get everyone vaccinated.

          Yes, this is de facto a policy of vaccinating just under eradication threshold. Do you ever hear about wiping out measles anymore? No, because they gave up.

          If delaying outbreaks for 50 years isn’t success, then I don’t know what is.

          Eradicating the virus, which was the original stated purpose of the vaccination.

          I don’t think you comprehend what is going to happen when this measles outbreak happens. Like a third of the US adult population is going to be infected before anyone even realizes what is going on, most of them having been vaccinated years ago.

          Covid is nothing compared to what that is going to look like, and look how ridiculous the response to that illness has been. And if there are ever supply chain breakdowns for some other reason it is going to happen much sooner.

          This is not comparable to little kids getting sick and some tiny percent having a severe reaction. This situation is absolutely unprecedented, probably the closest comparison is the decimation of the Americas when the Europeans showed up in the 15-16th centuries.

        • Are you sure that many adults have lost all immunity to measles?

          There have been measles outbreaks in the last five years or so in the US, among populations that didn’t have high vaccination rates, and they didn’t spread to adults like that (I don’t know, a couple of adults may have gotten it, but the outbreaks were mostly among kids and quite local — absolutely nothing like measles in an “immunologically naive” population like Native Americans or Pacific Islanders at contact.

        • Are you sure that many adults have lost all immunity to measles?

          Let’s see. About 90% of people get vaccinated: https://www.cdc.gov/mmwr/volumes/68/wr/mm6841e2.htm

          Only about 85% of those last vaccinated in 1987 still had detectable antibodies 20 years later and 18% of more recently vaccinated children had low or undetectable levels only 5 years later: https://pubmed.ncbi.nlm.nih.gov/22966129/

          That indicates circulating measles acted as a booster for the people vaccinated in the 1980s, so the rate of waning is increasing.

          Another study reports 92% of people vaccinated in 1971 still had detectable antibodies in 1997-199: https://pubmed.ncbi.nlm.nih.gov/15106101/

          From that we get about 20% of people vaccinated 20-30 years ago are still immune, and this must be higher in those vaccinated more than 30 years ago. So basically about 20% of people 30-40 yrs old are not immune and this rate should increase the longer ago the vaccination, at least 10% (from the vaccination rate) under 30, and lets say 20-40% of the population over 40 (I haven’t seen any data on this).

          Under 30: 125M * 0.1
          30-40: 40M * 0.2
          40-60: 85M * [0.2 to 0.4]
          https://en.wikipedia.org/wiki/Demographics_of_the_United_States

          Over 60 should mostly have life long immunity. That gives 12.5 + 8 + [17 to 34] = 37.5 to 54.5 Million susceptible (10-20% of the total population).

          There have been measles outbreaks in the last five years or so in the US, among populations that didn’t have high vaccination rates, and they didn’t spread to adults like that

          I’d need to see the details but if it was in an “populations that didn’t have high vaccination rates” it sounds like the adults probably had measles as a child so have lifelong immunity? Also, diagnosis of measles is less likely if the doctor has been told you are vaccinated.

        • Typo:
          > From that we get about 20% of people vaccinated 20-30 years ago are still immune

          This is not 20% of people vaccinated, but people from that cohort (0.9*0.9)

        • Anoneuoid said,
          “a third of the US adult population is going to be infected before anyone even realizes what is going on, most of them having been vaccinated years ago.”

          What about people who actually had measles? Do they lose their immunity after enough years? (I could imagine this might happen if successive mutations change the disease so much that immune responses no longer work on the mutated version.)

        • What about people who actually had measles? Do they lose their immunity after enough years?

          This doesn’t seem to happen.

          Group 4 (n = 50), also collected from the same set of residual sera, were presumably naturally infected 50- to 59-year-olds not covered by the MMR vaccination program.

          […]

          There were no measurable measles IgG antibodies for 15.5%, 10.4%, 4%, and 0% of groups 1, 2, 3, and 4, respectively.

          https://pubmed.ncbi.nlm.nih.gov/22966129/

        • There was an outbreak in California (associated with Disneyland) about 5 years ago… it was associated with parents’ refusal to vaccinate, and shortly afterward California put stricter limits on exemptions from vaccination requirements for schools.

          I think there’s been at least one other in the US … Minnesota or somewhere in that area… but don’t remember the details.

          But I misread your original post, you were not talking about our immunity being gone now, but decades in the future. I mean… maybe… I don’t know enough about the topic to be certain. I kind of would have expected to have heard of this, since I do read about the politics and issues surrounding vaccine uptake, but obviously I haven’t read everything.

          If it is true, I would expect people on the public health / medical side to do something about it, like promote measles vaccination for adults, before it becomes a major crisis (can’t boosters be given to adults when there is reason e.g. medical personnel?)

          This pandemic has rather lowered my opinion of the CDC/WHO/etc. (masks vs. no masks, delay in calling COVID a pandemic, etc.) but things like vaccination programs is well in the CDC’s wheelhouse. I think their issues with COVID don’t speak to general incompetence but a bureaucratic blind spot (this disease was just different enough from the flu pandemic they’d planned for to mess up their response).

        • That CDC page seems to say that you only need it as an adult if you didn’t have it before, or if you’re in a high-risk group for exposure. I meant that if losing immunity from previous vaccination proves to be a major threat, I’d expect them to recommend it for all adults over a certain age / X many years after the first vaccination.

        • There was an outbreak in California (associated with Disneyland) about 5 years ago… it was associated with parents’ refusal to vaccinate, and shortly afterward California put stricter limits on exemptions from vaccination requirements for schools.

          But were the parents vaccinated?

          But I misread your original post, you were not talking about our immunity being gone now, but decades in the future.

          We don’t have enough info to say when it will happen.

          If it is true, I would expect people on the public health / medical side to do something about it, like promote measles vaccination for adults, before it becomes a major crisis (can’t boosters be given to adults when there is reason e.g. medical personnel?)

          Yes, everyone needs to get a booster and they need to start vaccinating infants earlier because the maternal antibodies wane faster when the mother was vaccinated. It causes some issues politically because it involves

          1) Admitting that measles vaccine immunity isn’t life long like they promised

          2) Putting an end to the myth that vaccination is protecting babies “too young to get vaccinated”. The reason measles vaccination is delayed is because the maternal antibodies are already protecting the infant for about a year (but shorter in the case of a vaccinated mother).

  10. Challenge trials are ethically problematic in absence of effective treatment. If a miracle anti-sars drug becomes available then why not. The issue is not the total number of deaths, but risk for any individual who would volunteer to participate. Enrolling high risk contacts doesn’t have the ethical hurdle.

    Randomizing 1:1 does not make much sense if a high degree of effectiveness is expected. It would be more efficient to aim for comparable numbers of cases in the 2 groups if the disease is not too common. So if one expects vaccine efficacy of 75% wouldn’t it be better to randomize 4:1? (vaccine:placebo)

    • >>The issue is not the total number of deaths, but risk for any individual who would volunteer to participate.

      Sure, but if you choose your population well, the risk for any individual would not be so high that it would be irrational to accept.

      The US Navy military personnel numbers seem to show that for that population, the risk of death is between 1/1000 (there are over 1000 recoveries) and a bit less than 1/2000 (there are 2294 total cases known).

      And I am sure there are some people in the Navy who have hypertension for example… And the one death was age 41. So if you set the age range at say 18-35 and excluded anybody with hypertension, diabetes, etc. you could probably get a significantly lower risk.

        • >>So would you be ok with an 18-35 year old from your own family being in a challenge trial?

          I’m in that age range myself, though possibly not quite healthy enough to fit my own criteria for who would be chosen :)

          I’d certainly consider participating if it was offered, depending on the details (has the vaccine previously been demonstrated in phase I / II trials to produce neutralizing antibodies? what adverse effects, if any, turned up? what kind of viral dose is being used for the challenge?)

          Given neutralizing antibodies, there’s really not much chance the vaccine would provide *no* protection, so the risk would be less than a normal infection. It’s not totally clear participating in that trial would be a greater risk than otherwise (i.e. getting the vaccine earlier might compensate for a 100% chance of being exposed to the disease, depending on what the chance of being exposed before the vaccine became generally available would otherwise be).

          And there would certainly be a huge psychological benefit in being *done* with the whole thing and not having to worry about it any more.

  11. I think the argument against placebo is just play-acting at ethics here.

    If your prior is that out of 100 candidates there is enough heterogeneity that you can expect some to function as placebo, then ethically you are in fact including placebo in your challenge trial.

    Which means that ethically, you are _comfortable_ including placebo in your challenge trial. In which case you should include actual placebo in your challenge trial, in order to to account for the possibility that your prior is incorrect.

    And as I think about it more, there’s another unstated aspect of that prior, which is that the worst-acting candidates will perform like placebo. I’m not sure that’s a good assumption; given the nature of vaccines there’s a pretty decent chance that the candidate itself infects people who would not otherwise have been infected by the infection procedure. Without true placebo, you don’t have firm footing for determining that any of the outcomes were better than otherwise would have been expected.

    • Joseph:

      I’m not play-acting. I might be wrong, but I think the default is that if there are 100 vaccines being tested, that each would be compared to placebo. At least, that’s what I thought Lumley and Delaney were saying. It’s hard to organize 100 different big trials, and so I think it would make sense to coordinate these, in which case there’s no need, or at least much less need, for placebo arms.

      • If you’re comfortable giving what’s effectively a placebo, why are you not comfortable giving an actual placebo?

        I agree that coordination is a fantastic idea, and doing so could decrease the number of placebo arms dramatically while still having effective tests.

        But going with no placebo seems to be taking it too far, given the potential issues with the prior that I printed out.

      • My view was FDA descriptions of Phase 3 trials:

        “In Phase 3 studies, hundreds or thousands of volunteers participate. Vaccinated people are compared with people who have received
        a placebo or another vaccine so researchers can learn more about the test vaccine’s safety and effectiveness and identify common side effects.”

        https://www.fda.gov/media/83528/download

        In this case, it is possible that vaccines could cause harm, so a pure head to head trial could conclude the vaccine that does nothing is better than the one that does harm. Nor do we know the actual infection rate after challenge, but it is unlikely to be 100%.

        Obviously if the rules for phase 3 trials are altered that would be different.

      • My guess is that before all 100 potential vaccines make it into Phase III, one or more will be successful in a Phase III study and get approved. Once that happens, it will be difficult, if not impossible, to run any further placebo controlled Phase III studies. They’ll have to be active controlled. When that happens, the number of remaining candidates will dry up fast.

        There is a place for more rationale Phase III development of a number of potential drugs/vaccines. One thing that makes this difficult is that centers don’t usually run a large number of similar studies at the same time. So subjects at that center don’t have access to all possible candidates. In rare diseases, there is a push toward umbrella trials that test a large number of drugs simultaneously within one protocol. Very difficult to organize because of the conflict between various sponsors.

        • Running 100 two-arm trials using different protocols at different times and different places seems to me orders of magnitude easier than running one single 100-arms trial. Multi-arms trials are not rare (I just saw a reference to 2% of trials having five or more) but I wonder what’s the maximum number of arms ever done.

        • I really doubt it would be 100, my guess is 15 to 30. The safety and effectiveness at inducing antibodies screenings would probably drop it to 10-20 for the actual challenge trials.

    • >If your prior is that out of 100 candidates there is enough heterogeneity that you can expect some to function as placebo,

      The prior is that some are more effective than others, but that all would be pre-screened to be better than placebo in animal models. The purpose of the human studies is to pick the winner, not to prove that there’s a statistically significant difference between the vaccine and placebo.

      if you’re testing one single thing you obviously need a placebo to compare it against, if you’re testing 20 or 40 vaccines then some will be more effective than others. If they’re all 100% effective then you’ll need to try a small placebo arm to ensure that your infection procedure is effective. But the reduction in total number of people exposed to virus with placebo treatment is HUGE.

      • Why is it huge? The appropriate placebo size could be relatively small for a coordinated set of trials.

        I don’t see the benefit to foregoing that small set as outweighing the costs of the prior being wrong. Having to retest with placebo if there isn’t the expected dispersion in outcomes is problematic. Further, the uncertainty created by the possibility that positive relative performance was the result of the top performer having no effect while the poor performer infected people who would not have been otherwise, also seems problematic.

        The idea that you’re running without placebo seems like “ethical theater”, providing cover from the ethical concerns of a challenge trial with placebo … but without actually resolving them.

        Personally, I don’t see the “coercion” angle that Delaney mentions. I can tell you that my wife and I, healthy in early middle age and voracious consumers of information on Covid outcomes, would sign up to be infected even with possibility of receiving placebo. I struggle to come up with an element of coercion in that choice.

        I think the ethical considerations are important – critical in fact. But I think on balance challenge trials with placebo are justified.

        • Yeah, I agree with this. The cost of the placebo is additional people exposed to the virus. But the cost of potentially having to re-run the study is time, and losing that time can be very lethal indeed. I think that in this circumstance it is better to run with the placebo in one go.

        • I guess it all comes down to what your priors are on the various parameters describing the vaccines and infection protocols.

          My assumption is this, you’d do an initial effectiveness study in say monkeys. Then you’d do an initial safety study in humans who aren’t exposed to COVID, and determine both that the vaccine was safe enough to move forward, and that it caused the humans to produce antibodies. Now you’re practically guaranteed to have some level of effectiveness (as Kieth said elsewhere on the page, just proving that the antibodies are generated was enough for the Canadians under some circumstances). it *is* possible for vaccines to actually cause harm of course, make the reaction worse. This is a risk that’s unavoidable. On the other hand, it’s certainly the case that infecting unprotected people will cause harm. So that’s not really a risk at all, it’s a practical certainty. And, it has no value to the decision rule, which is choose the best performing vaccine. (I guess it would have some value if it turned out that *all* the vaccines caused actual harm, but this seems to be a relatively low risk if you’re testing 20 vaccines).

          But, you’re likely to have variable effectiveness across the candidates. So you might expect that say people exposed to enough virus to infect 90% of placebo treated people would result in infections ranging from almost 0 up to 40 or 50% for a vaccine that didn’t work particularly well, with perhaps single digit percentages as the mean (so a bit of a long tailed distribution). The main risk is that somehow all the vaccines are effective, in which case you’d have to prove that your infection technique would indeed have infected a large fraction of placebo treated people.

          But if you wound up in the “unfortunate” position where *all* of the vaccines were 100% effective, your best bet at that point would be to begin large scale trials in the general population.

          If things really were like this, then taking a study where you’re doing say 100 people getting each of say 10 vaccines and a 100 person placebo group would expose 100 people to a deadly disease with *no* protection.

          we honestly don’t care how much better the vaccine does than placebo. We can estimate how badly COVID affects the general population through our experience at hospitals and clinics and in the general population.

          If we’re in the situation where we suspect that none of the vaccines do substantially better than placebo, we can probably discern this from the trial by comparing to the general population’s experience with COVID. The point is to get something that cuts infections by a LARGE percentage, like with 90% infective dose only 3% of vaccinated people are infected.

        • I wonder how likely it is that we’ll be testing 20 vaccines. I mean, as usual, I would be delighted to be wrong. But the Phase 1 and Phase 2 trials will winnow the field and there are a limited number in advanced phases of development. I mean we could wait until there are 20, but what if there are 2 ready to go?

          We would like to distinguish between:
          1) vaccine A promotes cytokine storm (increases death rate) and vaccine B
          2) vaccine A promotes does nothing and vaccine B is 50% effective (which would be a notable win)

          The long discussion about “what is the IFR” makes it hard to generalize to highly selected samples and, generally, the FDA requires a pretty high standard of proof (which is the ultimate hurdle, we can talk at some point whether this is the ideal).

          If we were positive that harm wasn’t a risk . . . but that is a hard thing to be positive about given the properties of the virus (although it is reassuring that researchers know about the biggest risk and are looking for it in the animal models).

          Clearly, once you have scenario 2, and want a better vaccine, the ethics of a superiority trial start seeming less concerning.

        • I agree that if there are only 2 or 3 candidates that make it past safety screening and antibody results… that you would want to use a placebo. If you are testing 20 candidates, the probability that *all* of them are worse than placebo seems very low. If you are testing 3 or 4 the probability that all of them are worse than placebo is non-negligible and you should compare to placebo in this case. It’s probably worth doing some analysis at what point do you consider the risk low enough that you could leave placebo out. But this is just speculation at this point, without some kind of idea of how many candidates we should be testing, it’s not worth doing the math.

        • Yeah, some of this definitely depends on the number of vaccines. I am a skeptic as to how many will hit phase III at the same or similar times, but I would be delighted to be incorrect. My personal bet is that we need to be extremely incremental as early Phase I data is already hinting at possible trade-offs we may need to make, and it isn’t obvious that the final winner will be in the initial group.

  12. 1. The effect size in a sample size calculation should be the minimum clinically important difference, not the effect size you expect to observe. Stephen Senn has explained this very clearly in a post on Deborah Mayo’s blog. If you truly expect an effect size much greater than the minimum clinically important difference then you should employ a sequential analysis design.

    2. Just about all treatments in medicine have the potential to cause harm rather than benefit, and vaccines are certainly no exception. Animal studies are a first step but the list of treatments that have appeared to work in animals but subsequently failed in humans is huge. So you need safety trials in humans.

    3. Of course we need a control group – how else can you reliably measure the effect size? Ethical issues can be avoided by offering vaccination to a large group of people once safety has been established. A reasonable proportion will decline (anti-Vaxers, religious objectors, the dubious) and these form a control group. Sure, there won’t be randomisation, allocation concealment or blinding but it will still be better than no control group at all.

    4. How could you meta-analyse a bunch of trials on different vaccines? Answer: you couldn’t.

    5. Vaccination against corona viruses that affect animals has been disappointing in the past. Vaccine development remains hit and miss because we do not fully understand the immune system. I would not be surprised if all these vaccines in development turn out to be ineffective. Conversely, I would be surprised if none of them end up having adverse effects.

  13. I am worried about the safety issue that Anoneuoid referred to above. A vaccine causing adverse effects even in a small portion of population could make things worse given that the population death rate of this disease is most likely below 1% even without any mitigation. A mismatch between the test sample and the at risk population combined with a rushed timeline could lead to bad outcomes.

  14. > [L] But signing people up for deliberate exposure to a potentially deadly infection when half of them are getting placebo is something you don’t want to do without very careful consideration and widespread consultation. . . .
    > [D] There are greater good arguments here, but the longer I think about them the more dubious they get to me. Informed consent for things that are so dangerous really does suggest coercion. . . .
    > Based on the above discussion, it seems like it’s likely we’ll soon be seeing vaccine trials based on infecting healthy people with the virus and then seeing if they fight it off.

    Given their concerns it doesn’t seem so likely, if you mean actively infect people rather than wait for them to get infected.

  15. Relevant: “Key criteria for the ethical acceptability of COVID-19 human challenge studies”

    https://apps.who.int/iris/bitstream/handle/10665/331976/WHO-2019-nCoV-Ethics_criteria-2020.1-eng.pdf

    They point to challenge studies as a way to shortlist the most promising candidates to be tested in field trials: “multi-arm trials of vaccines can be particularly complex and demanding to conduct, and challenge studies could be used to prioritize experimental vaccines for inclusion therein (thereby reducing the total number of comparators and overall study complexity).”

  16. Andrew, any chance you want to publicize this platform for collaboration across planned covid trials: covidcp.org? Addresses some but not all of the issues you discuss.

  17. Phase 3 studies are designed to assess not just efficacy, but also safety and duration of protection in a large sample under wild type conditions. I think that a short-term, small scale challenge study in a select subgroup provides less evidence for vaccine effectiveness than a typical phase 3 study. Given that the primary benefit of a challenge study is quicker deployment to a large population, why not veer off to the other extreme and do a city scale phase 3? The benefit would be far greater evidence for safety and efficacy than a challenge or typical phase 3, but also potential herd immunity which could directly save lives (unlike a challenge or typical phase 3 study).

    • Jonathan:

      Not 25%. 25 percentage points. He seemed to really think that 50 percentage points was reasonable and that 25 percentage points was a conservative guess. Of course, this was in reference to a subpopulation with a very low survival rate, and this was right after everything had happened in Wuhan and everyone was scared of bodies piling up in the streets.

  18. “I don’t see why you need to give anyone placebos. If we have several legitimate vaccine ideas, let’s give everyone some vaccine or another. If they all work, and nobody gets sick, that’s great.”

    Not sure if someone mentioned this already (I perused through the comments but may have missed it) but I’d be concerned about a Peltzman effect happening. Especially if the participants know they’re not getting a placebo, and expect the vaccine to work, they may behave more riskily. If the average participant ‘behaviorally’ doubles his risk of infection, you’re not going to detect a 50% reduction by the vaccine when comparing with the unvaccinated population.

Leave a Reply to Nick Adams Cancel reply

Your email address will not be published. Required fields are marked *