Skip to content
 

The Pfizer-Biontech Vaccine May Be A Lot More Effective Than You Think?

Ian Fellows writes:

I [Fellows] just wrote up a little Bayesian analysis that I thought you might be interested in. Specifically, everyone seems fixated on the 90% effectiveness lower bound reported for the Pfizer vaccine, but the true efficacy is likely closer to 97%.

Please let me know if you see any errors. I’m basing it off of a press release, which is not ideal for scientific precision.

Here’s Fellows’s analysis:

Yesterday an announcement went out that the SARS-CoV-2 vaccine candidate developed by Pfizer and Biontech was determined to be effective during an interim analysis. This is fantastic news. Perhaps the best news of the year. It is however another example of science via press release. There is very limited information contained in the press release and one can only wonder why they couldn’t take the time to write up a two page report for the scientific community.

That said, we can draw some inferences from the release that may help put this in context. From the press release we know that a total of 94 COVID-19 cases were recorded. . . .

We do get two important quotes regarding efficacy.

“Vaccine candidate was found to be more than 90% effective in preventing COVID-19 in participants without evidence of prior SARS-CoV-2 infection in the first interim efficacy analysis

The case split between vaccinated individuals and those who received the placebo indicates a vaccine efficacy rate above 90%, at 7 days after the second dose.”

How should we interpret these? Was the observed rate of infection 90% lower in the treatment group, or are we to infer that the true (population parameter) efficacy is at least 90%? I [Fellows] would argue that the wording supports the later. . . . the most compatible statistical translation of their press release is that we are sure with 95% probability that the vaccine’s efficacy is greater than 90%. . . .

Assuming my interpretation is correct, let’s back out how many cases were in the treatment group. Conditional on the total number of infections, the number of infections in the treatment group is distributed binomially. We apply the beta prior to this posterior and then transform our inferences from the binomial proportion to vaccine effectiveness. . . .

There is a lot we don’t know, and hopefully we will get more scientific clarity in the coming weeks. As it stands now, it seems like this vaccine has efficacy way above my baseline expectations, perhaps even in the 97% range or higher.

I [Fellows] could be wrong in my interpretation of the press release, and they are in fact talking about the sample effectiveness rather than the true effectiveness. In that case, 8 of the 94 cases would have been in the treatment group, and the interval for the true effectiveness would be between 81.6% and 95.6%. . . .

It is important to have realistic expectations though. Efficacy is not the only metric that is important in determining how useful the vaccine is. Due to the fact that the study population has only been followed for months, we do not know how long the vaccine provides protection for. There is significant evidence of COVID-19 reinfection, so the expectation is that a vaccine will not provide permanent immunity. If the length of immunity is very short (e.g. 3 months), then it won’t be the silver bullet we are looking for. I’d be happy to see a year of immunity and ecstatic if it lasts two. . . .

I’ve not tried to reconstruct this analysis, but I’m a fan of the general idea of trying to reverse-engineer data from published reports. We had a fun example of this a few months ago.

72 Comments

  1. Interestingly, StatNews reports that Pfizer (and BioNTech), in discussion with the FDA, decided in October to skip the first Interim Analysis planned in their published clinical protocol for 32 cases and wait for the second (62 cases).

    But, oddly, they shut down lab processing of evidence, putting nasal samples in cold storage until the day after the election, by which point, when they resumed processing, they had 94 cases. So they wound up skipping Interim Analysis 2 as well and went right to Interim Analysis 3.

    From Matthew Herper’s article in StatNews:

    “Gruber said that Pfizer and BioNTech had decided in late October that they wanted to drop the 32-case interim analysis. At that time, the companies decided to stop having their lab confirm cases of Covid-19 in the study, instead leaving samples in storage. The FDA was aware of this decision. Discussions between the agency and the companies concluded, and testing began this past Wednesday. When the samples were tested, there were 94 cases of Covid in the trial. The DSMB met on Sunday.

    “This means that the statistical strength of the result is likely far stronger than was initially expected. It also means that if Pfizer had held to the original plan, the data would likely have been available in October, as its CEO, Albert Bourla, had initially predicted.”

    https://www.statnews.com/2020/11/09/covid-19-vaccine-from-pfizer-and-biontech-is-strongly-effective-early-data-from-large-trial-indicate/

    • Andrew says:

      Steven:

      Yes, the whole interim analysis thing is a mess. In my opinion a big part of this problem is the null hypothesis significance testing attitude, where the goal is to get statistical significance rather than to directly learn about efficacy. The rules being what they are, sometimes a study can be destroyed by stopping earlier, and given that this is so high stakes I can see why they’d rather err on the side of going to long and making sure they get a strong and statistically significant result. On one hand, if they’d held to an earlier stopping plan, the data could’ve been available earlier; on the other hand, if they’d kept the earlier plan and the data had not looked so good, it could’ve messed up their chance for a statistically significant result. The whole thing is very frustrating, as the incentives for getting approval can get in the way of the incentives for public health. (Not that switching to a Bayesian approach would necessarily make things better, as then I’m sure new problems would arise.)

      • Kert Viele says:

        My understanding is that the trial was originally designed prior to the FDA’s requirement of “median 2 month followup for safety”. After this requirement, they needed to wait until that safety database was obtained, delaying their planned interims. Given their interim schedule was disrupted, I suspect (don’t know, but based on experience) they wanted to avoid seeing any data until they could create a pre-specified amended plan. It would be useful if they would indicate what that was, but I would suspect it essentially said to keep the same interim requirements (declare efficacy if the posterior probability vaccine efficacy exceeded 30%, Pr(VE>30%), exceeded 99.5%). Given their result, they would have declared efficacy with any plan. FDA provided clarity on the requirements for safety and Pfizer had to react to them.

      • OV says:

        Something I wanted to point out: There is a very interesting game theory phenomenon with these kinds of interim analyses in the current pandemic climate. Presumably, the first to get authorized has a first-mover advantage. As a result, early “significance” may have a bigger payoff than later “significance.” If that is the case, you would expect to see more aggressive “alpha spending.” Indeed, analyzing Pfizer’s protocol with a frequentist alpha spending approach (this is analysis done by David Benkeser at Emory), you find that Pfizer’s Bayesian analysis equated to early, aggressive alpha spending vs. Moderna’s analysis.

        In the end, Pfizer decided to be a bit more lax and waited to do their interim. Perhaps they realized they would be first even if they weren’t aggressively alpha spending and wanted to therefore minimize the risk of a null result. Or perhaps it’s a completely different reason.

        In any case, things appear to have worked out. It looks like we have an effective vaccine in our hands.

      • Carlos Ungil says:

        > (Not that switching to a Bayesian approach would necessarily make things better, as then I’m sure new problems would arise.)

        They are doing a Bayesian analysis! Nevertheless, at some point the FDA has to take a decision and that’s why the statistical analysis plan defines decision rules.

  2. I came up with the same interval that Ian did, although the distribution is smoother.

  3. Steve Sailer says:

    If you are interestested in the nuts and bolts, here is Pfizer’s published clinical protocol for the vaccine trial, with the schedule for interim analyses and their justifying math on pp. 102-104.

    https://pfe-pfizercom-d8-prod.s3.amazonaws.com/2020-09/C4591001_Clinical_Protocol.pdf

    • Luke says:

      What an interesting “minimally informative” prior they use beta(0.700102, 1).

      • Ian Fellows says:

        Yeah, the .700102 looks like a very particular number. I think they got it by starting with beta(1,1), which is standard “non-informative.” Then they altered the first parameter so that the mean vaccine efficiency was a round value they found plausible. After all, we don’t expect the vaccine to make you MORE susceptible to being infected, though they did put significant prior weight on that possibility.

        The difference between weakly informative priors doesn’t matter much. You’ll get pretty much the same bounds from the beta binomial as you get from the poisson.test function.

          • Carlos Ungil says:

            I may be wrong, but the “So it’s Bayesian in name, but calibrated for a frequentist” comment doesn’t seem right to me.

            From the protocol: “The success threshold for each interim analysis will be calibrated to protect overall type I error at 2.5%.”

            The thresholds are calibrated, the prior is just part of the model used. (Of course one could also say that the whole model has been selected to get the thresholds they want.)

            • Seems to be the most agreeable position for regulators to take at this point in time.

              After last JSM meeting where this was discussed, Don Berry forwarded this guidance to me https://www.fda.gov/regulatory-information/search-fda-guidance-documents/interacting-fda-complex-innovative-trial-designs-drugs-and-biological-products

              Essentially Bayes is OK if error rates are adequately calibrated. There is a “never say never” clause for an exception but it would likely be hard to get.

              • Carlos Ungil says:

                Sounds reasonable:

                “2. Decision Criteria

                “For many clinical studies conducted using a frequentist statistical approach, hypothesis tests at a one-sided alpha level of 0.025 are used to establish support for the efficacy of a product. When Bayesian approaches are used, it is necessary to specify alternate decision criteria. The choice of decision criteria has a large impact on both the design and the quality of inference from a study.

                “For example, a study protocol may state that a conclusion of effectiveness will be supported if the probability that the response rate in Group A is greater than the response rate in Group B exceeds 99% (in mathematical notation, Pr(πA > πB) > .99). Another study protocol may state that a demonstration of effectiveness requires that the probability that the response rate in Group A is at least 10 percentage points higher than the response rate in Group B exceeds 95% (Pr(πA – πB > .10) > .95).

                “Sponsors should propose decision criteria in study protocols for all primary and secondary endpoints that would be included in product labeling if approved. These proposals should include a rationale for the choice of criteria. FDA will evaluate these proposals during IND review, and final determination will be based upon agreement with the review division.

                “For some Bayesian designs, it is possible to use simulations to estimate the frequentist operating characteristics of power and Type I error probability. Calibrating to frequentist operating characteristics in this way can provide a basis for comparing a given Bayesian proposal with previous studies or development programs that used frequentist inference, as Bayesian and frequentist inference are not otherwise generally directly comparable.”

  4. Rahul says:

    This 90% figure is based on actually how many individuals not getting covid who would be expected to have otherwise in absence of this vaccine?

  5. Alex Birch says:

    The reinfections may be due to different strains of Covid: https://www.thelancet.com/journals/laninf/article/PIIS1473-3099(20)30764-7/fulltext not necessarily being reinfected by the same strain.

  6. Dan F. says:

    A different sort of Bayesian analysis, the sort that fixates on things like the benefits of being first out of the gate and the stock dump by Pfizer’s CEO on the same day of the announcement, indicates that efficacy is likely a lot less than claimed.

    • Matt Skaggs says:

      “…the stock dump by Pfizer’s CEO on the same day of the announcement, indicates that efficacy is likely a lot less than claimed.”

      Darn tootin’. In fact, with mass manufacturing and distribution on the horizon for this miracle drug, the stock should go up and up. So why sell now?

      If I understand this process correctly, it works like this:

      1. To avoid an appearance of insider trading, CEOs of public traded companies schedule their stock sales well in advance.
      2. To reap the benefits of insider trading, these CEOs tweak their production schedules and information releases to ensure that stock prices will be high on a future date – the one when they scheduled the stock sale.

      So the Pfizer CEO had a bit of a dilemma. Making a killing on a vaccine announcement was no doubt baked in all the way back in January, but how to schedule it? Both the election and the competitors made it a bit tricky.

      Apparently the final decision was to delay the announcement and stock sale to the day after the election, not to avoid influencing the election, but to provide the perfect cover for the stock sale.

    • Fred says:

      Pfizer CEO stock sale might look fishy but he basically placed a sell stop order three months ago.
      It wasn’t ‘I’ll sell X number of stocks on 11/9’ but rather ‘I’ll sell X number of stocks once the stock price reaches $Y.’
      Unless you want to forbid CEOs from trading stocks completely, this is as legitimate as it gets.

      • Dan F. says:

        That the sell order was placed well in advance, as a function of a target sell price, in no way argues against what I am suggesting, as there’s no other effective way to legally benefit from the announcement (if he were to place the sell order after making the announcement, that would clearly be problematic legally). Precisely the fact that he placed a sell order months in advance suggests that he anticipated in the future making an announcement of the sort he made … so, long in advance he had reasons to expect to be able to announce efficacy with no side effects … Precisely this makes one question to what extent the announcement responds to results, or the results respond to expectations.

        Academics with comparatively trivial motivations such as looking to inflate citation counts exaggerate their results all the time. That someone planned in advance to benefit economically from an announcement of hopeful results should at least generate skepticism vis-a-vis the integrity of the announcement.

        • Carlos Ungil says:

          > (if he were to place the sell order after making the announcement, that would clearly be problematic legally)

          How so? (Unless you mean any order placed at any time is problematic as a matter of principle, which is why pre-scheduled trading plans are a thing.)

  7. Carlos Ungil says:

    I would be very surprised if this analysis is correct and the observed split is 3/91.

    The press release has to be interpreted looking at what is said but also at what is not said.

    Why would they say just that the split “indicates a vaccine efficacy rate above 90%” leaving out qualifiers like “strongly” or “with 95% probability” or pointing out to the median estimate “above 96% with 50% probability”?

    If the split is 3/91 the average efficacy is 95.9%, the median efficacy 96.3%. But what if the vaccine is “half as good” and the split is 6/88? In that case the average efficacy is 92.4% and the median efficacy 92.8%.

    Would they be wrong in saying that the case split 6/88 “indicates a vaccine efficacy rate above 90%” if the probability is 80%?

  8. MaximB says:

    The results are promising, but it’s way too early to talk of “true” efficacy.

    Any discussion of vaccine efficacy needs to take into account the mechanism of protection and the mechanism of infection.
    Were those protected, protected “forever” or protected only temporarily? Most likely the 2nd, because the virus keeps mutating,
    like the Flu. If they are protected temporarily, how does this work? How would the efficacy change if the protected would get more frequently into contact with infectious people or if the virus dose would be higher? At only 7 days after the last dose it’s reasonable to think the conditions of exposure didn’t vary much yet and the induced immunity is close to theoretical high.

    There are lots of other hurdles left before jumping from a more-or-less instantaneous estimate of vaccine efficacy to true vaccine effectiveness. It also depends on who were the vaccinated and who were the protected. What was the VE in the older age groups and in co-morbidities groups? Also, vaccine trials usually exclude some categories like immunocompromised people who are at higher risk.

    At this moment I think there are more chances the published estimate is an upper margin of the true VE.

    • confused says:

      >>because the virus keeps mutating,
      like the Flu.

      Not an expert, but I believe influenza viruses are rather unusual in how quickly/significantly they mutate, thus requiring annual shots.

      SARS-COV-2 does mutate, but *everything* mutates. I don’t think it’s anything like the same scale.

      Also, immunity/protection isn’t a yes/no binary. Even with the flu vaccine, there can be partial protection — you can get the vaccine, still get the flu, but it’s not as severe as it otherwise would be.

  9. I recently read a news about the availability of the Covid-19 vaccine is due in the Q1 2021. And the most concerning part of the news is that rich countries are pre-booking the stock of the cure. https://myksa.net/covid-19-vaccine-saudi-arabvia-among-the-very-first-countries-to-receive/
    Well there is no problem in that. but I think the poor third world countries should be given the preferences as they don’t have resources to develop or buy the vaccine.
    Any how if the news is true its a great push to the deadly virus and a great new of mankind.

  10. Joshua says:

    Andrew –

    > How should we interpret these? Was the observed rate of infection 90% lower in the treatment group, or are we to infer that the true (population parameter) efficacy is at least 90%? I [Fellows] would argue that the wording supports the later. . . . the most compatible statistical translation of their press release is that we are sure with 95% probability that the vaccine’s efficacy is greater than 90%. .

    Have you been able to assess their sampling? Was the control group a fully representative national sample?

    • Joshua says:

      Including participants across the age-spectrum, who have comorbidities, etc.

    • Navigator says:

      “Was the control group a fully representative national sample?”

      Joshua, I don’t think so. I believe that in the first stages it wouldn’t be ethical to experiment on very sick participants
      (ironically, those are the ones who will benefit most).
      It’s really a sample of willing participants. I didn’t volunteer. Did you? I bet nobody on this blog didn’t either.

      • anon e mouse says:

        I actually did sign up for the trial that was tested in my region, which may not have been Pfizer, I can’t remember. They never called me. I’m young-ish (at least in terms of COVID risk), white, and healthy, though, which I am guessing is the easiest category to recruit. But I tried.

  11. Navigator says:

    90% sounds high, until we realize that 10% of the world population is about 80 million (the population of Germany or Vietnam roughly).
    Too bad that no stats in the world can assure any individual whether he/she will be part of those 90%.

    I’m really curious what the penetration of the vaccine will be once introduced, given the mRNA issues. Considering that COVID is a global problem,more people will be immunized than usual(for flu and such), possibly leading to quite a few side effects or other issues being reported, as the situation develops.

    In any case, we shall see…

    • confused says:

      A 90% reduction in symptomatic cases should be enough to drop the IFR of COVID to that of seasonal flu, or lower (though neither COVID IFR nor seasonal flu IFR is terribly well-constrained…)

      • Joshua says:

        confused –

        You gonna update us on how your Covid predictions are working out?

        • confused says:

          I think I already admitted I was wrong about deaths not rising before Election Day.

          At this point, I have no idea what is going on. Not sure how much of what we’re seeing is seasonality vs people giving up on distancing vs the “first wave” hitting communities (eg Midwest/Great Plains) not yet hit, and without that knowledge, prediction isn’t really possible.

          But what I said above isn’t really a *prediction* in that sense — if the IFR of COVID is 10x that of flu*, then reducing it by 90% will make it the same.

          *CDC suggests 0.65% for COVID, and ~0.1% is often quoted for seasonal flu, though the latter may be an overestimate due to missed asymptomatic cases.

          • Navigator says:

            Confused,

            Reducing the IFR so it is on par with flu is not enough. The danger of COVID is the manner in which it kills or disables. It’s an expensive disease (multiple days/weeks in ICU and such).

            That’s why the comparisons with car crashes, etc. are meaningless. It’s not the absolute number, IFR, that matters, but all the other things involved to get to that number.

            • confused says:

              I’m not sure I agree that that is more important than the deaths, but the 90% is supposed to be reduction of symptomatic infection. So it should be a 90% reduction of hospitalization and ICU as well as death, roughly*, shouldn’t it?

              *I think one could argue that if it is less effective in the very elderly or very unhealthy, it might reduce hospitalizations *more*, since a smaller percentage of hospitalizations in the young lead to death.

              • Brent Hutto says:

                You can’t possibly extrapolate hospitalizations and deaths from a sample of volunteers (all presumably from the lowest-risk groups) to the real-world population. At this point evaluate their claims about reduction in infections (or in symptomatic infections if that’s what they are claiming). But you can’t possibly evaluate population impacts on mortality and serious illness until much, much later in the process when it has been administered to the folks who are likely to die or become extremely ill if infected.

              • confused says:

                Yeah, it’s very early in the game.

                But still a reduction in symptomatic infection/illness ought to reduce hospitalizations, ICUs, and deaths more or less comparably*, as it goes through illness before reaching hospitalization/ICU/death.

                *At least in a fairly healthy population — not sure what we can know about effectiveness in the very oldest/least healthy groups, which do seem to be an outsized share of COVID deaths.

    • Dan F. says:

      80 million is 1% not 10% of the world’s population.

  12. Michael Weissman says:

    Yeah, it’s frustrating that this simple piece of info wasn’t included. So my guess is that it’s >90% point estimate, now lower end of CI.
    On some of the other comments, remaining questions do include whether some groups (elderly?) get much less benefit. An even bigger question is to what extent is the vaccine sterilizing, i.e. preventing infection well enough to greatly reduce contagiousness. If it and the others are sterilizing enough, then once they’re widespread, the pandemic will mostly go away, at least until immunity fades. That also affects who should get it. Sterilizing but not very effective in the vulnerable would mean it should go to high-contact people first. Effective in the vulnerable but not very sterilizing means the opposite.

  13. Carlos Ungil says:

    Another week, another highly efficacious vaccine:

    “The primary endpoint of the Phase 3 COVE study is based on the analysis of COVID-19 cases confirmed and adjudicated starting two weeks following the second dose of vaccine. This first interim analysis was based on 95 cases, of which 90 cases of COVID-19 were observed in the placebo group versus 5 cases observed in the mRNA-1273 group, resulting in a point estimate of vaccine efficacy of 94.5% (p <0.0001)."

    https://www.businesswire.com/news/home/20201116005608/en/Moderna’s-COVID-19-Vaccine-Candidate-Meets-its-Primary-Efficacy-Endpoint-in-the-First-Interim-Analysis-of-the-Phase-3-COVE-Study

    No opportunity for debating the adequacy of priors in this case: "Statistical Hypotheses: For the primary efficacy objective, the null hypothesis of this study is that the VE of mRNA-1273 to prevent first occurrence of COVID-19 is ≤ 30% (ie, H0efficacy: VE ≤ 0.3). (…) A stratified Cox proportional hazard model will be used to assess the magnitude of the treatment group difference (ie, HR) between mRNA-1273 and placebo at a 1-sided 0.025 significance level."

    (a link to the protocol can be found in the news release)

    • Anoneuoid says:

      I took a look at it, they still havent checked in the at risk group…

      The primary endpoint of the Phase 3 COVE study is based on the analysis of COVID-19 cases confirmed and adjudicated starting two weeks following the second dose of vaccine. This first interim analysis was based on 95 cases, of which 90 cases of COVID-19 were observed in the placebo group versus 5 cases observed in the mRNA-1273 group, resulting in a point estimate of vaccine efficacy of 94.5% (p <0.0001).

      A secondary endpoint analyzed severe cases of COVID-19 and included 11 severe cases (as defined in the study protocol) in this first interim analysis. All 11 cases occurred in the placebo group and none in the mRNA-1273 vaccinated group.

      The 95 COVID-19 cases included 15 older adults (ages 65+) and 20 participants identifying as being from diverse communities (including 12 Hispanic or LatinX, 4 Black or African Americans, 3 Asian Americans and 1 multiracial).

      […]

      The Phase 3 COVE trial is a randomized, 1:1 placebo-controlled study testing mRNA-1273 at the 100 µg dose level in 30,000 participants in the U.S., ages 18 and older.

      […]

      The Phase 3 COVE study was designed in collaboration with the FDA and NIH to evaluate Americans at risk of severe COVID-19 disease and completed enrollment of 30,000 participants ages 18 and older in the U.S. on October 22, including those at high risk of the severe complications of COVID-19 disease. The COVE study includes more than 7,000 Americans over the age of 65. It also includes more than 5,000 Americans who are under the age of 65 but have high-risk chronic diseases that put them at increased risk of severe COVID-19, such as diabetes, severe obesity and cardiac disease. These medically high-risk groups represent 42% of the total participants in the Phase 3 COVE study.

      https://investors.modernatx.com/news-releases/news-release-details/modernas-covid-19-vaccine-candidate-meets-its-primary-efficacy

      More info here:

      The placebo is 0.9% sodium chloride (normal saline) injection, which meets the criteria of the United States Pharmacopeia (USP).

      […]

      Unblinded personnel, who will not participate in any other aspect of the study, will perform IP accountability, dose preparation, and IP administration.

      […]

      Participants ≥ 65 years of age will be eligible for enrollment with or without underlying medical conditions further increasing their risk of severe COVID-19.

      https://www.modernatx.com/cove-study

      It looks like 15k vaccinated and 15k got saline. Then 58% of the total was young/healthy (under 65 with no comorbidities), and ~25% of each group was healthy 65+ year olds while 17% were 18-65 years old with diabetes (36%; of that 17%), severe obesity (25%), significant cardiac disease (19%), chronic lung disease (18%), or liver disease (2%). Its not clear if these percentages include overlapping comorbidities.

      Anyway, that breaks down to 8.7k young/healthy, 3.75k old/healthy, and 2.55k young/comorbid for each group. From above we saw:

      This first interim analysis was based on 95 cases, of which 90 cases of COVID-19 were observed in the placebo group versus 5 cases observed in the mRNA-1273 group

      […]

      The 95 COVID-19 cases included 15 older adults (ages 65+) and 20 participants identifying as being from diverse communities (including 12 Hispanic or LatinX, 4 Black or African Americans, 3 Asian Americans and 1 multiracial).

      First of all, this makes it sound like there were zero cases in the young/cormobid group, but that isnt made 100% clear. Second of all, they dont tell us if those old/healthy cases were all in the placebo group, or if all 5 of the vaccinated cases were over 65 then the other 10 were placebo, or something in between.

      But we can say 10-15 out of 3.75k old/healthy placebo subjects got sick, which is 0.27% – 0.4%. Likewise, 75-80 out of 7.7k young/healthy subjects got sick, which is 0.97% – 1.0%.

      Therefore (in placebo group) the young/healthy subjects got infected at a rate 3x the old/healthy and it seems none of the young/comorbid subjects got infected.

      So I dont think this tells us much about healthy 65+ year olds and nothing at all about younger people with comorbidites. Definitely says nothing about older people with comorbidities. But, as expected, it does seem to be effective and safe for at least a few months in young/healthy people.

      Also, they only started counting cases two weeks after the second dose of vaccine (28 days after the first dose). What happened during these six weeks? Were there any cases?

      • confused says:

        >>Therefore (in placebo group) the young/healthy subjects got infected at a rate 3x the old/healthy

        Is that unexpected? It seems pretty logical to me that older people are at much higher risk so may take greater precautions. Especially as a vaccine trial population is going to exclude people who think COVID is a hoax/just a cold (who might not take precautions even if elderly/high-risk).

        And I don’t think “haven’t checked in the at risk group” is an accurate statement — age seems to be the overwhelmingly primary risk factor, it makes *orders of magnitude* difference, much more than obesity or other comorbidities.

        • Anoneuoid says:

          Is that unexpected? It seems pretty logical to me that older people are at much higher risk so may take greater precautions. Especially as a vaccine trial population is going to exclude people who think COVID is a hoax/just a cold (who might not take precautions even if elderly/high-risk).

          Agreed, that is probably what is going on. The only way to really check the at risk population before mass distribution is a challenge trial. Obviously no one is going to do that in humans for ethical reasons. So the only way to get an indication was animal trials with aged/obese/diabetic animals. It is now mid-Nov and that still hasnt been done.

          And I don’t think “haven’t checked in the at risk group” is an accurate statement — age seems to be the overwhelmingly primary risk factor, it makes *orders of magnitude* difference, much more than obesity or other comorbidities.

          Age just correlates with comorbidity, it is a very easy to measure proxy. Why would an extremely fit 70 year year old be at higher risk than an obese 50 yr old with heart failure or whatever?

          For 6% of the deaths, COVID-19 was the only cause mentioned. For deaths with conditions or causes in addition to COVID-19, on average, there were 2.6 additional conditions or causes per death.

          https://www.cdc.gov/nchs/nvss/vsrr/covid_weekly/index.htm

          • confused says:

            >>Age just correlates with comorbidity

            I really don’t think so. Age per se does seem to have an effect as well — otherwise one wouldn’t see a significant change in risk between the younger age categories, and we do see that in the death data.

            And this isn’t really surprising; the 1918 flu had a very different curve of risk by age, but it *did* have one that wasn’t explainable by comorbidities (an odd peak at 30 or so).

            • Anoneuoid says:

              Age per se does seem to have an effect as well

              Age is just the number of years since birth. In general people are more likely to die at older ages due to accumulated damage and genetic errors leading to cancer or organ failure.

              Your tissue stem cells can only divide so many times before enough mutations accumulate that a choice must be made between high cancer risk or senescence. In the latter case that means the body needs to keep the differentiated cells around longer (which, instead of being replaced accumulate various junk and mutations). Then when the cells finally die after a period decreasing functionality fill in the space with connective tissue.

              The exact rate at which this occurs depends on genetics and lifestyle, but it shows up as various organ dysfunctions we are calling comorbidities.

              If you can decrease the base mutation rate or increase the rate at which “junk” (eg, amyloids) accumulate or increase the rate at which cancer cells are cleared then you can change the number of years before the chronic conditions appear.

              Age is just a convenient proxy for this process.

              • confused says:

                On the cellular level, you are likely right that aging and poor health are not two separate things (although I think the causes/mechanisms for aging may be a bit more complex than that, and are not yet terribly well understood).

                But nonetheless the observed effect of COVID risk by age does not seem to be explainable by increased prevalence of the listed COVID “comorbidities” – eg obesity, diabetes, heart problems, asthma, etc.

                Things like general performance of the immune system, and decreased capacity in various ways that don’t rise to the level of a diagnosed condition, are not really measured by those listed “comorbidities”.

              • Anoneuoid says:

                But nonetheless the observed effect of COVID risk by age does not seem to be explainable by increased prevalence of the listed COVID “comorbidities” – eg obesity, diabetes, heart problems, asthma, etc.

                How so? Even if we remove the stuff that may be sequelae to SARS2 infection, it still adds up to 370k total cases of the comorbidities vs 220k covid deaths. That means theres about 1.7 comorbidities per covid patient who died:

                I’m sure this will format horribly but this is what Im looking at:

                Condition N
                Chronic lower respiratory diseases 19486
                Other diseases of the respiratory system 8441
                Hypertensive diseases 47455
                Ischemic heart disease 25274
                Cardiac arrhythmia 14400
                Cerebrovascular diseases 11046
                Other diseases of the circulatory system 12920
                Sepsis 20864
                Malignant neoplasms 10443
                Diabetes 36204
                Obesity 8366
                Alzheimer disease 8253
                Vascular and unspecified dementia 24787
                Intentional and unintentional injury, poisoning, and other adverse events 7919
                All other conditions and causes (residual) 114459

              • confused says:

                Counting comorbidities doesn’t mean much — there are few really elderly people with zero comorbidities.

                My point is the opposite — if the effect of age was merely increasing comorbidities, we’d see far more COVID deaths among young people with comorbidities. Some of these things (especially obesity!) are not rare among the younger population of the US.

                Yet we observe a difference of ~10,000x in the proportion of the population that has died of COVID by age group:

                https://www.cdc.gov/nchs/nvss/vsrr/covid_weekly/index.htm

              • Anoneuoid says:

                Counting comorbidities doesn’t mean much — there are few really elderly people with zero comorbidities.

                Yes, the older you are the more likely you are to have various health issues and so die of covid.

                More underlying health issues -> more likely to die of covid. If you only have healthy people get exposed to the virus in your vaccine trial you arent going to see what happens in those people. And thats where youll see ADE and less robust immunity.

              • confused says:

                >>Yes, the older you are the more likely you are to have various health issues and so die of covid.

                But someone who is 50 with no comorbidities is much more likely to die of COVID (though the risk still isn’t *that* high) than someone who is 30 with no comorbidities.

                Indeed, someone who is 30/no comorbidities is more likely to die than someone who is 10/no comorbidities. It’s not even purely “aging” in the “senescence” sense, or you wouldn’t see *that* effect.

                (I also remain extremely skeptical that ADE will be relevant with this disease, but I think we’ve argued that to death before…)

              • Anoneuoid says:

                But someone who is 50 with no comorbidities is much more likely to die of COVID (though the risk still isn’t *that* high) than someone who is 30 with no comorbidities.

                I doubt this, maybe if you mean *detected* comorbidities. Even then what data are you basing it on? How many people without any comorbidities died at different ages? That isn’t in this cdc data. It looks to me like almost all at any age had some kind of reported comorbidity.

              • Anoneuoid says:

                This doesn’t answer that question but is another interesting way of plotting that data:
                https://i.ibb.co/zrZN0yH/covidcomorbid2.png

                It shows the % of covid deaths at a given age for each comorbidity (actually comorbidity or sequela).

              • Anoneuoid says:

                Btw, I realize it says *Adult* Respiratory Distress Syndrome instead of *Acute*. That is how the CDC had it for some reason. The ICD-10 code is for ARDS (Acute Respiratory Distress Syndrome).

                It was also surprising to see the plots of ARDS vs age. First of all the max was only 20% for the 45-54 age group, dropping to 8% for 85+. I was under the impression ARDS was considered to be the “canonical” cause of death from covid, it certainly was early on.

              • confused says:

                Well, here’s half of what I was referring to: https://www.cdc.gov/coronavirus/2019-ncov/covid-data/investigations-discovery/hospitalization-death-by-age.html

                It’s not presented in a terribly helpful way, as it’s all relative to risk in 18-29 year olds, but it does show a roughly ~10,000x variation in risk by age (between “16x lower” and “630x higher”).

                Note that these are “crude” ratios – not adjusting for prevalence of infection by age group.

                I believe there was a similar chart showing risk based on specific comorbidities, but I don’t see it there now.

              • Anoneuoid says:

                It’s not presented in a terribly helpful way, as it’s all relative to risk in 18-29 year olds, but it does show a roughly ~10,000x variation in risk by age (between “16x lower” and “630x higher”).

                And probably nearly all of them had at least one known comorbidity. That still doesn’t tell us anything about people without comorbidities.

                Also, from Table 1 here:
                https://www.cdc.gov/nchs/data/nvsr/nvsr61/nvsr61_03.pdf

                Probability of an 85 year old dying in the next year from any cause is ~1000x that of an 11 year old (.08685/0.000094).

              • Martha (Smith) says:

                Confused said,
                “It’s not presented in a terribly helpful way, as it’s all relative to risk in 18-29 year olds, but it does show a roughly ~10,000x variation in risk by age (between “16x lower” and “630x higher”)”

                Sure makes me wish I were younger!

              • confused says:

                The reason I don’t think the effect of age can just be “increase in incidence of comorbidities” is that the US population isn’t *that* healthy. Things like asthma, obesity, and diabetes aren’t rare enough at younger ages to allow so dramatic an effect.

                Also, COVID deaths without comorbidities aren’t *that* rare. For example it’s 5% of COVID deaths in my county (which has a larger population than some states).

                So I don’t think comorbidities alone can explain a ~10,000x difference in risk.

                There are a lot of biological effects of age that aren’t diagnosable, distinct conditions, so this is IMO not terribly surprising. If there is something odd about COVID’s risk profile by age, it’s the relative mildness to the youngest (the CDC data does suggest a *slightly* higher risk <1 year, but not very much).

              • Anoneuoid says:

                The reason I don’t think the effect of age can just be “increase in incidence of comorbidities” is that the US population isn’t *that* healthy. Things like asthma, obesity, and diabetes aren’t rare enough at younger ages to allow so dramatic an effect.

                The older you get the more concurrent chronic health issues you have.

            • Anoneuoid says:

              I plotted all the comorbidities by age from that cdc covid site:

              https://i.ibb.co/Wp88Cc6/covidcomorbid.png

  14. confused says:

    Here is something very heretical, and I’m far from an expert.

    But… can we really argue that at least fairly good efficacy (if not this good) wasn’t “expected” from prior biological knowledge?

    If it was, why wasn’t the better balance of risk/harm to start vaccinating people as soon as a vaccine showed that it produced antibody/T-cell responses, rather than going through the trials? If that had been done, this current surge of COVID might not be happening.

    “1977 swine flu” level of side-effects would simply be irrelevant in comparison to the amount of harm from COVID. Is it really plausible that the side effects could be, say 100x-1000x worse?

    Or would it have been impossible to scale up manufacturing/distribution that quickly even if the FDA had approved, say, Moderna to start vaccinating people in June?

    It might not be terribly relevant in this case, as public trust might be too damaged to see the benefits. But for a future pandemic, it might really matter — the political environment 20 or 30 years from now is unlikely to be so polarized, IMO, and as biotechnology improves, public trust in it may also increase.

    • Antibody Dependent Enhancement is a definite thing. They tried to make a vaccine for respiratory syncytial virus and it make the effects worse https://en.wikipedia.org/wiki/Respiratory_syncytial_virus

      “Attempts to develop an RSV vaccine began in the 1960s with an unsuccessful inactivated vaccine developed by exposing the RSV virus to formalin (formalin-inactivated RSV (FI-RSV)).[39] Unfortunately, this vaccine induced a phenomenon that came to be known as “vaccine-associated enhanced respiratory disease” (VAERD), in which children who had not previously been exposed to RSV and were subsequently vaccinated would develop a severe form of RSV disease if exposed to the virus itself, including fever, wheezing, and bronchopneumonia.[39] Some eighty percent of such children (vs. 5% of virus-exposed controls) were hospitalized, and two children died lethal – lung inflammatory response during the first natural RSV infection after vaccination of RSV-naive infants.[39] This disaster hindered vaccine development for many years to come.[39]”

      So, if we gave out a COVID vaccine widely without checking that it doesn’t produce enhanced COVID symptoms, it’s entirely possible that we’d be seeing millions of deaths with the new cases.

      Luckily it doesn’t seem to be the case that it causes enhanced symptoms among the people it’s been tested on so far.

      • Anoneuoid says:

        You expect to see it when there are low quantity and/or affinity antibodies towards the spike protein:

        However, when viruses infect cells expressing Fc receptors, such as Raji, K562, or primary immune cells, the antibody at suboptimal neutralizing concentration promotes virus entry into cells through interaction between antibody and Fc receptors (Figure 9). We found that amino acid substitutions F342L and E516A on RBD allowed the virus escape from the neutralization by 7F3 without reducing binding affinity to antibody.

        […]

        These results also suggest that ADE may be more likely to occur at later time points after recovery from COVID-19 when the concentration of neutralizing antibodies elicited by the primary SARS-CoV-2 infection have waned to suboptimal neutralizing level.

        https://www.medrxiv.org/content/10.1101/2020.10.08.20209114v1.full-text

        To evaluate the efficacy of existing vaccines against infection with SHC014-MA15, we vaccinated aged mice with double-inactivated whole SARS-CoV (DIV). Previous work showed that DIV could neutralize and protect young mice from challenge with a homologous virus14; however, the vaccine failed to protect aged animals in which augmented immune pathology was also observed, indicating the possibility of the animals being harmed because of the vaccination15. Here we found that DIV did not provide protection from challenge with SHC014-MA15 with regards to weight loss or viral titer (Supplementary Fig. 5a,b). Consistent with a previous report with other heterologous group 2b CoVs15, serum from DIV-vaccinated, aged mice also failed to neutralize SHC014-MA15 (Supplementary Fig. 5c). Notably, DIV vaccination resulted in robust immune pathology (Supplementary Table 4) and eosinophilia (Supplementary Fig. 5d–f). Together, these results confirm that the DIV vaccine would not be protective against infection with SHC014 and could possibly augment disease in the aged vaccinated group.

        https://www.nature.com/articles/nm.3985

        We found that higher concentrations of anti-sera against SARS-CoV neutralized SARS-CoV infection, while highly diluted anti-sera significantly increased SARS-CoV infection and induced higher levels of apoptosis. Results from infectivity assays indicate that SARS-CoV ADE is primarily mediated by diluted antibodies against envelope spike proteins rather than nucleocapsid proteins. We also generated monoclonal antibodies against SARS-CoV spike proteins and observed that most of them promoted SARS-CoV infection. Combined, our results suggest that antibodies against SARS-CoV spike proteins may trigger ADE effects. The data raise new questions regarding a potential SARS-CoV vaccine, while shedding light on mechanisms involved in SARS pathogenesis.

        https://www.sciencedirect.com/science/article/pii/S0006291X14013321

      • confused says:

        True — though I’m not entirely sure one can compare these mRNA vaccines with 1960s technology.

        But if one started with the assumption that the vaccine must have some biological effect — IE, that the possibilities include harm (because of ADE) or positive effect, but not “antibodies do nothing” — would you end up running the same kind of standard trials?

        I feel like there’s a problem with the “null hypothesis” here.

        • Martha (Smith) says:

          Interesting possibility.

          • confused says:

            Yeah, I wish I knew enough about the situation to really evaluate it. But I feel that there is a standard FDA “course” to go through which is probably insufficiently adaptable to the “facts on the ground” — both vast improvements in biological understanding, and the fact that COVID is a different sort of issue.

            Most drug development fails, yes — but currently all the “easy” problems already have good solutions. So you either have to do better than an already good solution, or deal with a genuinely hard problem like Alzheimer’s.

            COVID is different because the lack of treatments/vaccines is because it’s new, not because decades of efforts have been tried and failed.

            The RSV vaccine problem Daniel Lakeland mentioned above may indeed be relevant! But it troubles me when the usual problems cited to defend the level of caution in medical approvals (thalidomide, 1977 swine flu vaccine, etc.) date from the 60s and 70s — it’s hard to overstate the degree of improvement in understanding of biology & available techniques in just the last 20 years, much less the last 50.

          • confused says:

            IE – in 1962 a null hypothesis of “it does nothing” probably made sense, because biological/biochemical knowledge was pitiful. It probably makes sense today for Alzheimer’s drugs, since the mechanism of the disease is not really understood.

            But when we are dealing with a vaccine that has already been shown to produce an immune response, I think a null hypothesis of “it does nothing” is no longer relevant.

            The remaining issues might be more safety-related than proving an effect.

            What would show I’m wrong here is if a couple of the major/well-funded/reputable COVID vaccine development programs *didn’t* work. That would suggest that efficacy wasn’t predictable from what we knew in, say, March/April.

            • Steve Sailer says:

              Interestingly, during the Spanish Flu of 1918, the US Army came up with a vaccine for the bacterial pneumonia that was killing off a lot of Spanish Flu patients in just four months. The vaccine was 70% effective, which is not as good as these vaccines, but pretty good for 1918 technology in 4 months!

              • Nick Adams says:

                I find this implausible. Do you have a source? This publication claims none of the Spanish flu vaccines were effective:

                Public Health Rep. 2010; 125(Suppl 3): 27–36.
                The State of Science, Microbiology, and Vaccines Circa 1918
                John M. Eyler, PhDa

              • confused says:

                There certainly was no flu vaccine (vs. influenza virus) that early — not until WW2 — it wasn’t even known to be a virus in 1918.

                I would indeed like to see a source for that: I’d gotten the impression that attempts aimed at the bacterium then believed to cause influenza (Haemophilus influenzae) were not effective, since it wasn’t actually the cause.

                I have read that many of the deaths were due to bacterial pneumonia, but not that much of anything useful could be done about that then (before antibiotics).

  15. Carlos Ungil says:

    Final results: https://investors.biontech.de/news-releases/news-release-details/pfizer-and-biontech-conclude-phase-3-study-covid-19-vaccine

    Primary efficacy analysis demonstrates BNT162b2 to be 95% effective against COVID-19 beginning 28 days after the first dose; 170 confirmed cases of COVID-19 were evaluated, with 162 observed in the placebo group versus 8 in the vaccine group

    Efficacy was consistent across age, gender, race and ethnicity demographics; observed efficacy in adults over 65 years of age was over 94%

    Safety data milestone required by U.S. Food and Drug Administration (FDA) for Emergency Use Authorization (EUA) has been achieved

    Data demonstrates vaccine was well tolerated across all populations with over 43,000 participants enrolled; no serious safety concerns observed; the only Grade 3 adverse event greater than 2% in frequency was fatigue at 3.8% and headache at 2.0%

Leave a Reply