Rob Trangucci, Yang Chen, and Jon Zelner write:
In order to meet regulatory approval, pharmaceutical companies often must demonstrate that new vaccines reduce the total risk of a post-infection outcome like transmission, symptomatic disease, severe illness, or death in randomized, placebo-controlled trials. Given that infection is a necessary precondition for a post-infection outcome, one can use principal stratification to partition the total causal effect of vaccination into two causal effects: vaccine efficacy against infection, and the principal effect of vaccine efficacy against a post-infection outcome in the patients that would be infected under both placebo and vaccination. Despite the importance of such principal effects to policymakers, these estimands are generally unidentifiable, even under strong assumptions that are rarely satisfied in real-world trials. We develop a novel method to nonparametrically point identify these principal effects while eliminating the monotonicity assumption and allowing for measurement error. Furthermore, our results allow for multiple treatments, and are general enough to be applicable outside of vaccine efficacy. Our method relies on the fact that many vaccine trials are run at geographically disparate health centers, and measure biologically-relevant categorical pretreatment covariates. We show that our method can be applied to a variety of clinical trial settings where vaccine efficacy against infection and a post-infection outcome can be jointly inferred. This can yield new insights from existing vaccine efficacy trial data and will aid researchers in designing new multi-arm clinical trials.
Sounds important. And they use Stan, which always makes me happy.
Looks like this post is referring to this article…
https://arxiv.org/abs/2211.16502
Link added; thanks.
I didn’t follow the entire argument but they seem to make three questionable assumptions:
1) That vaccine effectiveness can be meaningfully described by a single value, even for some specific population. Due to waning and mutations this is never true. Even for viremic viruses like measles the lifelong immunity is due to a series of “natural booster” exposures. Under low prevalence conditions immunity wanes much more quickly.
2) High specificity tests:
This was shown to be false by the challenge trial. About half the uninfected participants (no symptoms, consecutive positives, or culturable virus) tested positive on PCR within two weeks (however, two consecutive positives 12 hours apart appeared very specific for recent, but not necessarily still active, infection). So they also need to consider specificities of ~50%.
3) There are three independent misclassification assumptions:
Due to side effects, many people may be aware of the treatment and adjust their behavior accordingly. Eg, test more often.
2) High specificity tests:
For example, PCRs for COVID-19 have very high specificity, but tend to have sensitivities in the range of 0.6 to 0.8 due to variation among patients in how the virus populates the nasal cavity, variation in swab quality, and viral RNA dynamics (Kissler et al., 2021; Wang et al., 2020)
This was shown to be false by the challenge trial. About half the uninfected participants (no symptoms, consecutive positives, or culturable virus) tested positive on PCR within two weeks (however, two consecutive positives 12 hours apart appeared very specific for recent, but not necessarily still active, infection). So they also need to consider specificities of ~50%.
That’s not true. The subjects you mention in the study were infected. For relevant measures of specificity, one can look at FDA’s website. The specificity of these tests is generally 98-99%.
These were the results:
https://www.nature.com/articles/s41591-022-01780-9
“Uninfected” was negative PCR 12 hrs both before and after the positive test, no symptoms, and negative culture for the duration of the study.
They can only be considered infected using the circular logic that a single positive PCR means infected.
And it isn’t surprising this happened if you know how PCR works, eg Ive been saying the tests were never validated for real world use since spring 2020. And their cycle threshold for positive was 33.5 (which is still high), but some places were using 40.
I spent 3 months a few years ago reviewing validations for various tests of another virus. The specificity and sensitivity of a test are determined in the validation experiment, not in the user community. The validation uses pre-planned in-vitro experiments with various levels of virus. There are basic rules of validation but they do vary between companies and diagnostic products. The results of a study such as this challenge do not determine the specificity and sensitivity. This study was not designed for that purpose but to assess viral patterns in-vivo.
If you want to use the test to decide how likely someone is of being at risk of illness or transmitting to others, then you need to do a study like that. Two different PCR assays that both amplify some RNA trapped in a booger do not tell us how often the positives are infected (or not).
There should be a label on the test that it has not been validated for real world usage.
Quick question, if you test an uninfected population weekly (as was the OHSA requirement for unvaccinated at one point), how long until half the population has “had covid”? The simplest model is geometric distribution, right? But that is assuming independence between the tests… so really we don’t know.
Like NHST, these numbers do not tell people what they want to know.