Kaiser writes:
After over a year of navigating the peer-review system (a first for me!), my paper with Mark Jones and Peter Doshi on observational studies of Covid vaccines is published.
I believe this may be the first published paper that asks whether the estimates of vaccine effectiveness (80%, 90%, etc.) from observational studies have overestimated the real-world efficacy.
There is a connection to your causal quartets/interactions ideas. In all the Covid related studies I have read, the convention is always to throw a bunch of demographic variables (usually age, sex) into the logistic regression as main effects only, and then declare that they have cured biases associated with those variables. Would like to see interaction effects in these models!
Fung, Jones, and Doshi write:
In late 2020, messenger RNA (mRNA) covid-19 vaccines gained emergency authorisation on the back of clinical trials reporting vaccine efficacy of around 95%, kicking off mass vaccination campaigns around the world. Within 6 months, observational studies report[ed] vaccine effectiveness in the “real world” at above 90% . . . there has (with rare exception) been surprisingly little discussion of the limitations of the methodologies of these early observational studies. . . .
In this article, we focus on three major sources of bias for which there is sufficient data to verify their existence, and show how they could substantially affect vaccine effectiveness estimates using observational study designs—particularly retrospective studies of large population samples using administrative data wherein researchers link vaccinations and cases to demographics and medical history. . . .
Using the information on how cases were counted in observational studies, and published datasets on the dynamics and demographic breakdown of vaccine administration and background infections, we illustrate how three factors generate residual biases in observational studies large enough to render a hypothetical inefficacious vaccine (i.e., of 0% efficacy) as 50%–70% effective. To be clear, our findings should not be taken to imply that mRNA covid-19 vaccines have zero efficacy. Rather, we use the 0% case so as to avoid the need to make any arbitrary judgements of true vaccine efficacy across various levels of granularity (different subgroups, different time periods, etc.), which is unavoidable when analysing any non-zero level of efficacy. . . .
They discuss three sources of bias:
– Case-counting window bias: Investigators did not begin counting cases until participants were at least 14 days (7 days for Pfizer) past completion of the dosing regimen, a timepoint public health officials subsequently termed “fully vaccinated.” . . . In randomised trials, applying the “fully vaccinated” case counting window to both vaccine and placebo arms is easy. But in cohort studies, the case-counting window is only applied to the vaccinated group. Because unvaccinated people do not take placebo shots, counting 14 days after the second shot is simply inoperable. This asymmetry, in which the case-counting window nullifies cases in the vaccinated group but not in the unvaccinated group, biases estimates. . . .
– Age bias: Age is perhaps the most influential risk factor in medicine, affecting nearly every health outcome. Thus, great care must be taken in studies comparing vaccinated and unvaccinated to ensure that the groups are balanced by age. . . . In trials, randomisation helps ensure statistically identical age distributions in vaccinated and unvaccinated groups, so that the average vaccine efficacy estimate is unbiased . . . However, unlike trials, in real life, vaccination status is not randomly assigned. While vaccination rates are high in many countries, the vaccinated remain, on average, older and less healthy than the unvaccinated . . .
– Background infection rate bias: From December 2020, the speedy dissemination of vaccines, particularly in wealthier nations, coincided with a period of plunging infection rates. However, accurately determining the contribution of vaccines to this decline is far from straightforward. . . . The risk of virus exposure was considerably higher in January than in April. Thus exposure time was not balanced between unvaccinated and vaccinated individuals. Exposure time for the unvaccinated group was heavily weighted towards the early months of 2021 while the inverse pattern was observed in the vaccinated group. This imbalance is inescapable in the real world due to the timing of vaccination rollout. . . .
They summarize:
[To estimate the magnitude of these biases,] we would have needed additional information, such as (a) cases from first dose by vaccination status; (b) age distribution by vaccination status; (c) case rates by vaccination status by age group; (d) match rates between vaccinated and unvaccinated groups on key matching variables; (e) background infection rate by week of study; and (f) case rate by week of study by vaccination status. . . .
The pandemic offers a magnificent opportunity to recalibrate our expectations about both observational and randomised studies. “Real world” studies today are still published as one-off, point-in-time analyses. But much more value would come from having results posted to a website with live updates, as epidemiological and vaccination data accrue. Continuous reporting would allow researchers to demonstrate that their analytical methods not only explain what happened during the study period but also generalise beyond it.
I have not looked into their analyses so I have no comment on the details; you can look into it for yourself.
Testing. It was all over the news that if you were vaccinated you didn’t need to get tested. Not just rate, the reasons for the tests diverged (eg, with symptoms vs screening) in a way that drastically affected the accuracy. Essentially, different tests were being used.
From the UK challenge trial we saw ~50% of people with no symptoms, culturable virus, antibody development, or consecutive positive pcr test meet the definition of “confirmed case” within a couple weeks. So in real world conditions the tests were only really good for measuring base rate anyway (false positive and negative rates both a little better than 50%). Two consecutive positive PCRs 12 hrs apart seemed accurate, so that should have been the definition.
And that window bias is even worse, since the first dose caused lymphocytopenia for a week making them more susceptible to infection. Higher infection rates during that window were then seen during the trials and across the world in observational studies. And some even counted the vaccine-caused cases as in the
“unvaccinated.”
I also didn’t see healhy vacinee effect or dealing with “unknown” vaccination status in the summary. But I’ll definately read to see if there is anything new I missed in that complete fiasco.
Seems to me your comments on a potential healthy vacinnee bias leave much to be desired.
But at any rate, you’d likely find this interesting:
https://twitter.com/jsm2334/status/1682450942623469585?s=20
I never claimed a healthy vaccinee effect, the CDC did based on some bizarre methods. This has been discussed here multiple times.
These are all excellent points. Case-counting window bias can be counter-acted by case to control temporal matching: then take the 7 (or 14) days for both groups. This also helps with the background infection rates. And if you match by age, then it helps there too. We often do this type of matching in cancer cases to non-cancer controls. But, it is not easy to do.
Wouldn’t age bias tend to make the vaccine look bad because older, and especially older and sicker, people are more likely to get the vaccine while the unvaccinated cohort would be enriched by younger and healthier old people? In oncology trials real world, all comers use generally looks worse than the selected populations in randomized trials.
> Wouldn’t age bias tend to make the vaccine look bad because older, and especially older and sicker, people are more likely to get the vaccine while the unvaccinated cohort would be enriched by younger and healthier old people?
One of the arguments is that the really, really sick don’t get vaxed because of the risks involved.
But there’s been a lot of gaming on this issue. Seems to me assumptions about a healthy vaccinee effect based on other vaccines (like flu vsccines) are of limited value given the explicit focus in many countries in vaccinating older and more vulnerable people for COVID.
Part of the context for this paper is that Doshi is rather well known for his stance as a “covid contrarian.” Make of that what you will (I don’t think in itself, it necessarily means anything) but it’s relevant context. It’s rather like the context for a paper from Ioannidis.
> – Case-counting window bias: Investigators did not begin counting cases until participants were at least 14 days (7 days for Pfizer) past completion of the dosing regimen, a timepoint public health officials subsequently termed “fully vaccinated.” . . .
The UK ONS data are a good comparitor for this, as in the ONS they count someone as vaxed as soon as they get vaxed. To my understanding, their data show this potential bias doesn’t make the vaxes look more effective than they are (and their analyses show vaxes as very effective). I would suspect the effect of this potential bias would be minimal – but regardless, it would make sense to me to do a sensitivity analysis to see how time from first vax over a few weeks affects the outcomes results.
Like in so many things we often treat time-series as if they were static, when we shouldn’t.
We know that vaccines have a very short term effect (1-10 days) which could be negative (side effects, lymphocytopenia, whatever). A medium term effect in which antibodies are high and infection is less likely, and a long term effect in which even if you are infected the severity is milder and possibly likelihood of long term effects are reduced. And of course, this is a continuous process.
If you measure “vaccine effectiveness” as preventing infection, and you do it at 6 months you will get a very different measure of vaccine effectiveness than if you do it at 1-2 months.
The thing is, vaccine effectiveness as 1 month is really a proxy for what we care about, which is the entire area under the “risk” curve through time. Arguing about the specific number at 1 month-ish is not as important as estimating the overall value of the vaccine.
If I search it I can find my own comments saying the same, probably before the vaccines were released. Ie, makes no sense to talk about a single “effectiveness” number to begin with.
But what we have learned since is that there basically is no reliable method to determine vaccine effectiveness or safety currently being used. Like VAERS, it may as well be shut down if 1000% increase in reports can be dismissed. Then that database is worthless.
What we know is the number of reported cases and all-cause deaths increased after the mass vaccination campaign. What see is cases and deaths increase drastically when NPIs started, then drop to near normal when all the NPIs stopped. Which makes sense, because stopping vulnerable people from having family/friends around has been considered abusive since forever.
> , it may as well be shut down if 1000% increase in reports can be dismissed. Then that database is worthless.
Vaers has been useful in the past to identify signals of vaccine harm.
That doesn’t mean that spikes in reports are always informative.
Again, binary thinking.
> What we know is the number of reported cases and all-cause deaths increased after the mass vaccination campaign. What see is cases and deaths increase drastically when NPIs started, then drop to near normal when all the NPIs stopped. Which makes sense, because stopping vulnerable people from having family/friends around has been considered abusive since forever.
What an interesting collection of comments you make. Some quite sophisticated and interesting. Some just bizarrely inane.
It’s like a pressure gauge at a nuclear power plant.
“Oh sometimes it just spikes into the extreme danger zone, but that doesn’t necessarily mean anything because there are these other reasons it could happen.”
OK, how about you need to design a better gauge or also measure/deal with those other reasons then.
Only in medicine are such low standards accepted as useful. All people need to do is demand higher standards and they would get them.
It’s not binary!
There are different forms of surveillance.
The case-counting window bias seems like a nice example of the value of target trial emulation, where observational studies are designed to emulate particular hypothetical clinical trials, particularly in accounting for these kind of issues.
I’m not sure if there is supposed to be some interesting point here about the age bias except perhaps to say that nonparametric adjustment for age (via, e.g., matching) is preferable to adjustment with just linear or logistic regression.
Dean: Regarding age bias, I’d say we have a few points: a) the papers should always report the model equations and coefficients and not just verbally describe the class of model; b) they need interaction terms not just main effects of age – or use other methods like matching; c) the model structure constrains the interpretation e.g. main-effects only models and treatment effect being the same across all age groups