Dean Eckles writes:
I thought you might “like” this COVID-vaccine-related example of a case where one contrast is statistically significant (at p = .048) and another is not. So the authors make a big deal out of how one thing worked and the other didn’t.
And this seems like a product of having run a clearly underpowered study. After all, it was barely able to “detect” a 85% lift!
I agree with Eckles that this is a problem. Here’s the abstract of the linked paper:
To increase COVID-19 vaccine uptake in resistant populations, such as Republicans, focus groups suggest that it is best to de-politicize the issue by sharing five facts from a public health expert. Yet polls suggest that Trump voters trust former President Donald Trump for medical advice more than they trust experts. We sought to test the efficacy of these alternative approaches for effective vaccine communication – facts delivered by an expert versus political claims delivered by President Trump endorsing the vaccine. We conducted an online, randomized, national experiment among 387 non-vaccinated Trump voters, using two brief audiovisual artifacts from Spring 2021. Relative to the control group, Trump voters who viewed the video of Trump endorsing the vaccine were 85% more likely to answer “yes” as opposed to “no” in their intention to get fully vaccinated (RRR = 1.85, 95% CI 1.01 to 3.40; P = .048). There were no significant differences between those hearing the public health expert excerpt and the control group (for “yes” relative to “no” RRR = 1.14, 95% CI 0.61 to 2.12; P = .68). These findings suggest that a political speaker’s endorsement of the COVID-19 vaccine may increase uptake among those who identify with that speaker. Contrary to highly-publicized focus group findings, our randomized experiment found that an expert’s factually accurate message may not be effectual to increase vaccination intentions.
This is pretty much pure noise. Also, yes, the difference between the two studies is easily explained by sampling error—and this would be the case even if the p-value in the first study was 0.001 rather than 0.048.
The good news is that this is just a preprint. It doesn’t seem to be published anywhere so there should be plenty of time for the authors to correct the paper, making clear that this is basically a pilot study with results that are way too noisy to tell us anything at all about the effects of political or expert messaging. They could perhaps rewrite it as a short cautionary note as to how small experiments can be minsinterpreted.
I can’t really blame the authors here; they’re doing what they’ve been taught and what has been common practice in social science. It will take awhile before statistics education catches up; maybe it would help to see a note by the authors discussing how they got confused, warning future researchers not to fool themselves in this way.
Eckles adds:
I [Eckles] also have some vaccine-acceptance related work, linked below, so I’ve been following this area. Sometimes people seem to think the effects in our paper (from an experiment with N > 400k) are small. I think they are substantial, but realistic, and that many social scientists have totally distorted views of plausible effect sizes.
This last issue points us to a vicious circle in social science:
1. Someone does a noisy study and finds a comparison that is statistically significant and thus, by necessity, corresponds to a huge estimated effect size.
2. Future noisy studies are done in this area. Researchers expect to see large effect sizes, so they manage to find what they’re looking for.
3. When new studies are being designed, there is the expectation of large effect sizes, so no effort taken to design a precise study.
4. Naive worship of identification strategies with a focus on purported unbiasedness and a lack of interest in measurement.
Also all the stuff about the difference between significance and non-significance, forking paths, etc., which enables step 2 above.
P.S. Maybe the above-linked study can be featured in the second edition of Noise. It’s a whole new continent!
Question for Andrew: I’ve seen loads and loads of statistical errors pointed out on this blog. For practitioners, what would you say is the most effective training we could engage in to protect ourselves from them? For example, is there a book that publishes a nice list of common statistical errors and how to avoid them?
Thanks!
Adrian:
The best thing I’ve got for you now is Regression and Other Stories.
Andrew: Thanks for the recommendation of your own book. Are you familiar with “Common Errors in Statistics (and How to Avoid Them)”, by Phillip I. Good and James W. Hardin? I just found it on Amazon. Is it worthwhile?
You didn’t ask me. But you might wanna check out Martha Smith’s website of stat mistakes: https://web.ma.utexas.edu/users/mks/statmistakes/StatisticsMistakes.html
Another issue with this study is that the measured outcome variable is reported intent to get vaccinated. That doesn’t really support their conclusion “These findings suggest that a political speaker’s endorsement of the COVID-19 vaccine may *increase uptake* among those who identify with that speaker. ” [Emphasis added.] There is a long and slippery chain of events between intent to get vaccinated and actual uptake, not to mention the possibility that the reported intent may not be the actual intent. Even if the increase in reported intent really is 85%, this likely would translate into, at most, a very modest increase in actual vaccine uptake.
Although I’m sympathetic to the general point that these things are noisy (this study especially) and we can’t conclude differences without statistical tests, I don’t find it obvious that “This is pretty much pure noise…this would be the case even if the p-value in the first study was 0.001 rather than 0.048.” Indeed, “the difference between the two studies is easily explained by sampling error” is an easily testable hypothesis. The authors should have done it (!), but since it’s published in PLoS ONE and the authors shared the data, we can pull the data from OSF – unadjusted (because I don’t want to recode everything into my preferred syntax) rates of vaccination intention (yes/no) are 44%/35% in control (n=126), 50%/24% in Trump video (n=144), and 45%/34% in Frieden video (n=117). A multinomial logistic here yields RRs of 1.69 and 1.06, similar to the headline numbers. Is the difference between the two coefficients consistent with sampling error? I get p=.11. Maybe their contrast would have been p<.10, so weak evidence against the null, but which direction do you update?
Daniel:
Yes, the study provides very weak information, consistent with noise. If journals want to publish papers with very weak information, that’s fine, but then I think this should be labeled as such. I think the above-quoted presentation in terms of significance and non-significance is a bad way to present the results as it implies a level of certainty or strength of conclusion that is inappropriate given the data and analysis shown.
It’s already published in PLOS One: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0257988
One- False, it has been known since very early on that viremia is rare. In most cases the virus was contained in the respiratory tract.
Two- Maybe, there have been reports of long-term term expression of the spike mRNA and this was theorized to occur when it got into a cell expressing endogenous retrovirus (eg, HERV-K, HERV-W, etc). About 5-10% of the human genome consists of these and expression rises in fast proliferating tissue (children, cancer patiints, etc).
Anyway, the issue is the hoped-for “immune priming” is not robust because it is limited to a single sequence of only 1/29 viral proteins. Further the IM injection does not trigger immunity at the site of infection/transmission (respiratory/GI-tract mucosa).
Three- Ok, I have no idea but it sounds like a “95% of stats are made up on the spot” kind of claim.
Four- False. For whatever reason, after the vaccination campaign there were more cases and excess deaths than in the year before. It clearly did not accomplish this goal.
Five- Maybe, but see number four.
As for Trump’s claims, obviously he doesn’t know what hes talking about.
tldr; These “facts” are sketchy at best, either way people shouldn’t be trained to make decisions based off soundbites.
Anoneuoid –
> Four- False. For whatever reason, after the vaccination campaign there were more cases and excess deaths than in the year before. It clearly did not accomplish this goal
Well, that doesn’t quite negate the argument. The relevant comparison is to what would have happened had their not been vaccines. That there were more cases and excess deaths post- compared to pre-vaccine isn’t really the most relevant comparison to the claim that was made. Even if there were more deaths post- compared to pre-vaccine, there might have been fewer post-vaccine than there would have been during that same time period than there would have been with no vaccines – and hence a quicker recovery than there would have been otherwise.
Counterfactual reasoning is hard.
> Four- False. For whatever reason, after the vaccination campaign there were more cases and excess deaths than in the year before. It clearly did not accomplish this goal
This is not true, even if we use a very early date for what constitutes “after the vaccination campaign”, which of course is still ongoing at some low level.
For instance, in mid-May 2021, half of Americans had received at least one dose of a covid vaccine, at which point about 40% were “fully vaccinated.” A lot of people got vaccinated after that date — more than 75% of Americans have now been vaccinated — but I’ll use that date for now. According to https://ourworldindata.org/covid-deaths, at that point about 580K Americans had died of covid (meaning covid was listed as a contributing factor on their death certificate). The current number is 1.04 million, meaning 460K Americans have died since then. So 120K more Americans died of covid prior to mid-May 2021 than have died of covid since then.
I have used the time at which 50% of people had received at least one dose in the above calculation, but I think it would be very odd to call that “the end of the vaccination campaign.” The plot of cumulative vaccinations as a function of time got very flat starting in mid-January 2022, so I think it would make more sense to call that “the end of the vaccination campaign.” Of course that means that deaths are even more heavily weighted towards the time “before the vaccination campaign.” But of course that’s potentially misleading; if I defined the end of the vaccination campaign” to be today at midnight, we’d have pretty much no deaths after the end of the vaccination campaign but that would tell us nothing about the vaccine effectiveness.
I think a better way to convey the effectiveness of the vaccines is to directly speak to the probability of dying if infected, or perhaps the probability of hospitalization if infected. Such data exist, e.g. the data that are featured in https://www.cdc.gov/media/releases/2022/s0715-COVID-VE.html or the data that were used for https://aspe.hhs.gov/sites/default/files/documents/c5d0dde224c224dd726694367846b609/aspe-covid-medicare-vaccine-analysis.pdf
Phil:
What about the “excess deaths” that I mentioned though? Ie, not the subset that was tested for covid and ended up positive.
Anon,
You confused the issue by mentioning covid cases, and then two words later you’re talking about deaths but not covid deaths. Why bring up cases in the first place?
But you seem to be wrong about all-cause excess deaths as well, at least according to https://ourworldindata.org/excess-mortality-covid
I did not confuse the issue. Excess deaths (all cause mortality) is what people actually care about. Covid deaths are too confounded by testing rates to be meaningful.
This is the data I was looking at:
https://www.usmortality.com/excess-mortality/yearly-percentage-cumulative
It used to have a nice paragraph yearly summary below the chart but I can’t find it now, it looks like the layout of the site changed.
Anoneuoid –
> Excess deaths (all cause mortality) is what people actually care about. Covid deaths are too confounded by testing rates to be meaningful.
I’d say that using excess deaths as a measure of efficacy of the vaccines is certainly not un-confounded.
Anoneuoid –
> Covid deaths are too confounded by testing rates to be meaningful…
Are you saying that excess deaths is un-confounded by testing rates?
It is less confounded than covid deaths. But testing leads to anxiety and hysteria, so I would expect a positive relationship between the two.
That said I do agree with basic sanitary measures like ventilation and if you are sick (with any respiratory virus) stay home.
Anyway, if you all want to keep pretending these vaccines were some kind of great success it is easy to see that in the future this low standard will continue.
I’d think we all benefit from an intervention that actually reduced the number of cases and excess deaths, but there is no real incentive to create one when people think this is acceptable.
Anoneuoid –
> I’d think we all benefit from an intervention that actually reduced the number of cases and excess deaths…
Again, you’re making assumptions about counterfactuals.
How many excess deaths would there be if there were no vaccines? You don’t actually know but you’re arguing as if you know.
Excess deaths is a noisy statistic. Perhaps better than others but just plucked out of the box without context, it’s not very useful.
Further, as I was getting at above. Some % of the “excess deaths” might be people infected and recovered (who wouldn’t currently test positive) who are dying sooner because their bodies were weakened as a result of the infection. And maybe there are people who were infected and recovered (and wouldn’t test positive now) so aren’t dying now because with a vaccination their infection caused less damage.
There’s a ton of noise.
You’re arguing as if you know these numbers. But you don’t.
And on top of all of that, there are the issues related to timing if vaccines and case rates and variants.
I thought this article dies a good job of introducing some complications even if it leaves behind much uncertainty.
https://erictopol.substack.com/p/immunity-walls
And by the way…
> Anyway, if you all want to keep pretending these vaccines were some kind of great success it is easy to see…
That is a classically bad faith argument at many levels. As long as you assign guilt by association, build around straw men, rely on binary thinking., argue from personal incredulity,
assume you can mind-probe, etc., you can’t get very far in to meaningful exchange.
In additon to what others have said, #4 doesn’t say too much about vaccine efficacy against mortality without taking into account vaccine take up. It’s striking (to me) how high the proportion of Covid deaths post-vaccine is in the US compared to the UK for example. Looking at the Worldometer Coronavirus updates it seems that around 43% of US covid-related deaths occurred post-vaccine (I’m using June/July 2021 when all the most/moderate vulnerable offered a vaccine could have been double-vaccinated should they have wished). In the UK around 28% of total deaths occurred post-vaccine. Similar post-vaccine proportions in Italy/France..
That likely relates to vaccine take up (approaching 90% in the UK; around 67% in the US). Looking at individual states is revealing. In NY with lots of early deaths but highish vaccine take up (78%) around 15% of total deaths occurred post-vaccine (NY may be a special case with a real onslaught of early deaths). Compare Arizona (63% vaccine take-up and around 43% of deaths were post-vaccine) similar for Mississippi (53% take-up and 40% deaths post-vaccine).
I expect this has been looked at more rigorously than my cursory look (!) but it does seem like the very low vaccine take up in some states is associated with high post-vaccine mortality. Brown University recently did a study modelling vaccine-preventable deaths that seems to accord with this ( https://globalepidemics.org/vaccinations/ )
That is now two people who responded talking about covid deaths rather than excess deaths. You guys know better than this.
Anon, my point about post-vaccine Covid deaths and likely relation to vaccination rates (see Brown Uni study linked to above) also applies to excess deaths; i.e. high post-vaccination excess death rates in US where vaccination rates lower, much lower post-vaccination excess deaths in Western Europe (UK, France, Germany, Italy, Spain) where vaccination rates high/highish.
You can follow this through some of the Eastern European countries where vaccination rates lower (e.g. Romania, Bulgaria) and post-vaccine excess deaths high.
(using https://ourworldindata.org/excess-mortality-covid link from Phil’s post and https://vaccinetracker.ecdc.europa.eu/public/extensions/COVID-19/vaccine-tracker.html#uptake-tab for vacination rates)
Measures of Covid cases is obviously massively dependent on testing rates! A question is how one assigns causality to excess deaths. Clearly a large chunk of this is likely to be associated with Covid deaths (i.e. a high level of Covid deaths contributes to high level of excess deaths as described in the Brown Uni study linked to above). Other excess deaths relate to diversion of resources e.g. away from surgical interventions, acute responses, cancer treatments, access to hospital care etc., and presumably in countries where lockdowns and mask-wearing etc. were followed especially over winter periods there was some negative contributions to excess deaths due to lower levels of seasonal flu and so on. So not straightforward to deconvolute but presumably there are some studies on this.
Comparing across time is confounded enough, and even across the US so many things differ. IMO comparing across countries is just adding more confounds. So I am just talking about the US.
And let us not forget that the moderna/pfizer RCTs reported ~15% higher all-cause death rate in the vaccinated groups. That is the best evidence out there that the vaccines had little impact on excess deaths.
Anon,
Thank you for informing me that I was not confused about what you were saying above covid cases and deaths. I could have sworn I was confused! But I guess I wasn’t!
I find many of the plots at https://www.usmortality.com/excess-mortality/yearly-percentage-cumulative very confusing — I don’t know exactly what they show — but I think I do understand what I’m seeing when I click on ‘mortality’ in the top menu bar, and then turn off all of the lines in the uppermost plot except ‘baseline’ and ‘mortality.’ Is it your claim that the area between the mortality line and the baseline is larger after May 2021 than it was between January 2020 and May 2021? I have not added the numbers but that does not appear to be correct.
Moving on, you say “Anyway, if you all want to keep pretending these vaccines were some kind of great success it is easy to see that in the future this low standard will continue.” Well, vaccines are absolutely a great success at reducing mortality from covid among people who have been infected. https://www.scientificamerican.com/article/how-to-compare-covid-deaths-for-vaccinated-and-unvaccinated-people/ for example but you hardly need one more example. I’d be interested in a clear statement from you: do you agree or disagree with the following statements:
(a) The vaccines are safe in the sense that getting vaccinated is extremely unlikely to kill you.
(b) If you are vaccinated and get covid, you are much less likely to die than if you are unvaccinated and get covid.
I believe both of those statements are true. What about you?
I also think that when someone gets vaccinated they are more likely to get covid than if that same person hadn’t been vaccinated, because people know the vaccine confers some protection against the worst outcomes and therefore are willing to behave in riskier behavior. I know I am much more willing to be in crowds than before I was vaccinated, and I’m very sure I’m not the only one. In principle one could agree with a and b while still believing vaccines have not made people safer. Perhaps that is your belief. But for now I’m curious about whether you believe (a) and (b).
RE: Phil’s last paragraph, “I also think that when someone gets vaccinated they are more likely to get covid than if that same person hadn’t been vaccinated, because people know the vaccine confers some protection against the worst outcomes and therefore are willing to behave in riskier behavior. I know I am much more willing to be in crowds than before I was vaccinated, and I’m very sure I’m not the only one. In principle one could agree with a and b while still believing vaccines have not made people safer. Perhaps that is your belief. But for now I’m curious about whether you believe (a) and (b).”
I do believe your points A and B but I did not become one bit more comfortable around crowds than before I was vaccinated. For that matter, in the first year of the pandemic I was never more comfortable around crowds with myself and others wearing masks than I would have been without.
I don’t think that the vaccination campaign has made me materially less at risk of catching COVID. There’s neither biological plausibility (to the extent I can evaluate such) or evidence that it has done so. Just like I don’t think there was any clear and convincing evidence that wearing masks while out and about made me less likely to catch it.
So I still avoid crowds. Not to an extreme. I still live my life, visit with my extended family on celebratory occasions, move about in stores or offices, etc. But I can’t imagine a time in the near to medium future where I’d be comfortable sitting on an airplane for eight hours or sitting shoulder to shoulder at a concert or sporting event or hanging around in a crowded, busy bar or restaurant. It’s just an unnecessary risk no matter how many people end up geting whatever kind of vaccine.
I do hold out a vague hope that somewhere the distant future (a decade hence maybe?) some form of vaccine that DOES confer immunity to infection might be developed. And if so I certainly hope the vast majority of people I encounter will join me in taking that vaccine. But for now we can’t avoid catching the darned thing if we’re going to frequent crowded places, current vaccines or not.
Phil –
> I also think that when someone gets vaccinated they are more likely to get covid than if that same person hadn’t been vaccinated,…
Hmmm. All seems pretty complicated to me.
I might speculate that people who get vaccinated might on average be more likely at baseline to be a more cautious person (at least with respect to covid infection). Maybe they’re also more likely to be older, and hence to have more of a reason to be cautious about getting infected. It would depend also on the timing with respect to variants and rates of community spread.
Joshua:
We could go on and on about this, but . . . I don’t see why you’d think that being vaccinated is about being cautious. From my perspective (and I think many others), being vaccinated is a freebie: it greatly reduces the risk of death or incapacitation from covid at essentially zero cost. I know that there are people out there who are afraid of getting the vaccine; I guess to them the cost is not essentially zero and they see some risk, so I guess they’re the cautious ones.
Name:
It’s your call if you don’t want to go out. I just went to the theater last night and it was full. There’s room for all sorts of people, and there’s enough on TV now that there’s no good reason to do out to public events if you don’t want to.
Andrew –
> I don’t see why you’d think that being vaccinated is about being cautious.
Mostly in the sense that there’s an association between people who think COVID’S basically a bad cold, and those who don’t want to get vaccinated. So those getting vaccinated are being more cautious in that they’re more concerned about the risk of ceroius illness.
> it greatly reduces the risk of death or incapacitation from covid at essentially zero cost.
But those who aren’t getting vaccinated disagee. They think that only those who are feeble are at serious risk from COVID and that the cost of vaccination is quite high.
> I know that there are people out there who are afraid of getting the vaccine; I guess to them the cost is not essentially zero and they see some risk, so I guess they’re the cautious ones.
Yes, that’s one of the ironies. There are many, many people put there who say those getting vaccinated are sheeple, who have been intimidated or hypnotized by Big Pharma and socialists trying to control everyone into getting the jab because they’re cowards (this is one of Joe Rogan’s themes). But seen from another angle, those same people could be considered as “scared” of a safe no-brainer treatment, because they’ve been brainwashed by malevolent grifters who seek to profit from fear-mongering about Bill Gates, the Great Reset, Big Pharma and vaccines.
Joshua:
Maybe the point of this particular sub-discussion is that “cautious” is not a particularly useful term to be using to describe people in this context. There are so many different dimensions of caution that it doesn’t really work to consider cautiousness as a general trait, even restricting to covid-related risks.
Andrew –
> Maybe the point of this particular sub-discussion is that “cautious” is not a particularly useful term to be using to describe people in this context.
Despite what I wrote above, I agree. Maybe there’s some underlying behavioral trait or temperament that could be applied in association with vaccination, but how you’d measure it and what the implications might be is so complicated, I doubt the return on investment would make an investigation worthwhile.
I checked the Pfizer RCT report and it seems like there were 21 deaths in the vaccine group (from 21,926 total participants in vaccine group) and 17 deaths in the placebo group (from 21,921). So your “15%” (presumably you’ve averaged in the equivalent data from moderna) seems highly misleading to me. It’s based on very small numbers. I’ll leave it to the statistical experts here to assess the statistical significance of this difference (I obtain a dreaded P value of around 0.5 using a relative risk calculation)!
These trials were massively underpowered to assess excess deaths. They use healthy populations with low comorbidities and an age profile skewed well towards the younger sectors. Their aim is to assess efficacy (effect of vaccine in suppressing infection) and safety.
Much more useful (in addition to the multiple analyses linked to on this thread) are the numerous studies that assess vaccine efficacy and vacine-preventable Covid deaths and excess deaths in the “wild” where vaccines are meant to be used. There are several of these now; an example is Xu et al. 2021 ( doi: 10.15585/mmwr.mm7043e2 ) which assessed non-Covid releated deaths in large cohorts (6.4 million vaccinated; 4.6 million matched non-vaccinated) and found much lower non-Covid-related deaths in the vaccinated cohort relative to the non-vaccinated (read paper for details). So not only does vaccination greatly reduce Covid-related deaths but it is associated with a “healthy vaccinee” effect. These and some of the other analyses link on this thread show that vaccines have a large favourable impact on excess deaths.
None of this is that surprising. We’ve known for 200 years that humans have a system for protecting against invading organisms that involves priming of the immune system. The default expectation was always that a Covid vaccine would be successful and so it turns out.
Chris:
+1
+1 to Chris. Also: Chris, Dr. Gelman pointed this same thing out to Anonyd00d last year. And many of us pointed out the fact that the vaccine trials were underpowered to assess all-cause mortality as an endpoint. We just have very noisy estimates.
Sadly, Anonyd00d doesn’t have the capacity to learn anything from thoughtful blog responses. He’ll likely repeat the same nonsense another year from now, and then drain more collective hours of time from blog readers.
It is interesting to note when RCTs are informative and when they’re the devil’s spawn.
Anoneuoid –
There’s a good chance you won’t see this, and I know that you seem to regularly think clarifying your views at my request isn’t worth your time, but I’m hoping that you’ll clarify your views as Phil requested here.
https://statmodeling.stat.columbia.edu/2022/08/23/i-agree-with-dean-eckles-that-this-study-effects-of-political-versus-expert-messaging-on-vaccination-intentions-of-trump-voters-is-too-noisy-to-be-useful-when-will-we-reach-the-point-where-res/#comment-2073936
Not responding might lead some people to think that there’s a reason you’re not clarifying, other than you just missed the request or forgot about it.
I responded to phil already, he didn’t respond to me: https://statmodeling.stat.columbia.edu/2022/08/23/i-agree-with-dean-eckles-that-this-study-effects-of-political-versus-expert-messaging-on-vaccination-intentions-of-trump-voters-is-too-noisy-to-be-useful-when-will-we-reach-the-point-where-res/#comment-2074115
I stopped responding to you because you were asking what the evidence for “healthy vaccinee effect” was when the topic of the comment you responded to was the paper Chris cited (and misunderstood). You were spamming the thread with worthless comments.
Anoneuoid –
> I stopped responding to you because you were asking what the evidence for “healthy vaccinee effect” was when the topic of the comment you responded to was the paper Chris cited (and misunderstood).
?
I asked you if you had evidence that the “healthy vaccinee effect” generalizes to the real world re the COVID vaccines, specifically, particularly given that there was a concerted effort to vaccinate those at higher risk/older people.
I asked you that (as I indicated explicitly) because you seemed to be generalizing that the “healthy vaccinee effect” could be applied to comparing excess mortality stats among those vaccinated and not vaccinated.
Do you have evidence that’s a valid application? Or are you saying that you wouldn’t try to apply the “healthy vaccinee effect” to COVID in the real world without such evidence (and don’t have any)?
Of course, I also touched on just some of the other reasons why I think it’s problematic to reason about the efficacy of vaccines by backwards engineering excess deaths during the pandemic.
Anoneuoid –
> I responded to phil already….
?
Sometimes your comments are too technical for me to follow. This must be one of those cases, as I see a comment directed to him subsequent to his comment directed at you but I don’t get how you clarified your view on the topic he requested:
do you agree or disagree with the following statements:
(a) The vaccines are safe in the sense that getting vaccinated is extremely unlikely to kill you.
(b) If you are vaccinated and get covid, you are much less likely to die than if you are unvaccinated and get covid.
Taking the percentages from Daniel’s comment, 50% (Trump video) is just barely significantly different from 44% (control), and 45% (Frieden video) is not significantly different from 44% (control); it’s thus highly unlikely that the Trump treatment differs significantly from the Frieden treatment, since the Frieden value is closer to the Trump value than the control. So isn’t (part) of the problem here that the two treatments should be compared directly to one another, rather than comparing them separately to the control?
Gregory:
Yes, that’s my “difference between significance and non-significance” comment.
Ok, gotcha! Just read your Am. Stat. paper with Stern now– thanks. Checking Dean Eckles Twitter, he also made the same point–sorry for repeating it!
“When will we reach the point….”
When researchers want to do legitimate science instead of politically biased science and resume puffing?
But I guess these days with Democrats outnumbering Republicans ~10:1 among tenure-track faculty with a party registration, politically biased science and resume puffing are pretty much the same thing. Fortunately, we have liberal faculty research that “casts doubt on the idea that the ideological tilt of faculty members is because of discrimination.” :) And I agree that in America we should always be tried by a jury of likely offenders. :)))
https://www.insidehighered.com/news/2017/02/27/research-confirms-professors-lean-left-questions-assumptions-about-what-means
What’s a good primer for a social science researcher to walk the labyrinth of potential statistical mistakes?
A couple of comments. While the criticisms of the four facts as being problematic were wrong on the facts, that those facts were badly written to the point of being arguable seems problematic.
Another point is that a lot of the folks interested in using facts to persuade pseudoscience believers, vaccine skeptics, and the like usually don’t have a clue as to what the mindset of the people they’re trying to persuade is.
Here’s someone who has thought about this a tad more deeply than most.
https://www.youtube.com/watch?v=Va0RCgbywGc&ab_channel=PhilosophyTube
> 3. When new studies are being designed, there is the expectation of large effect sizes, so no effort taken to design a precise study.
And when some researchers do design studies with high precision, they are likely to get effect estimates that are “small” (by distorted standards), often making for a not-so-enthusiastic reception to the work.
@Phil
I agree that site became confusing. You can download the mortality data here by clicking around:
https://gis.cdc.gov/grasp/fluview/mortality.html
Or directly download the week 32 data (the latest available) here:
https://www.cdc.gov/flu/weekly/weeklyarchives2021-2022/data/NCHSData32.csv
Sum for 2020 is 3,433,988 deaths. For 2021 it is 3,449,555 deaths. If you want to look at it starting in March (Week 9 2020 to Week 8 2021 etc), then we see 3,580,049 vs 3,457,261.
For comparison 2017-2019 had 2.80 – 2.85 million deaths per year.
So for whatever reason the vaccination campaign had little impact on all-cause mortality. As mentioned above, this is also what we should expect based on the RCT results (where there were ~30 deaths per group and the rate was 15% higher in the vaccinated).
It is, however, not what most people expected based on what was claimed about these vaccines.
Anon,
Sorry, I missed the fact that you had responded.
You say “So for whatever reason the vaccination campaign had little impact on all-cause mortality” but we don’t know this. How many people would have died if not for the vaccination campaign? I don’t know. Many people would have behaved very differently, and would be behaving very differently. Perhaps more people would have died due to not having the protection provided by the vaccines, but perhaps fewer people would have died because they would have been more careful not to get covid.
I note you didn’t respond to my questions about whether you agree the vaccine is safe (in the sense that it is very very rare for someone to die from the vaccine) and whether you agree that it reduces the chance of dying of covid among people who are infected with covid.
Phil –
> but perhaps fewer people would have died because they would have been more careful not to get covid.
There are a couple of other factors to consider there.
One is that the vaccines likely reduced the rate of spread for a while (particularly during Delta) and the other is that they likely slowed the rate of deaths (both as a result of and independently from the reduced rate of spread) until more effective treatments were developed.
So if it’s true that more people would have been more cautious and thus there would have been fewer infections for that reason, that effect would likely have been cancelled out, at least to some degree, by more infections because of a lack of protection against infection provided by the vaccines (particularly during Delta). That plus the benefit of more time for therapeutics to be developed, it seems to me, pretty much stacks the deck against the “fewer deaths without vaccines” counterfactual conjecture.
Not to take away from your larger point that there’s a lot of uncertainties.
Counterfactuals are hard. That last point, it seems to me, is one that Anoneuoid has some difficulty integrating.
Yes, I try to first respond to the main point rather than letting the conversation expand. Now it seems we agree that all cause mortality has not meaningfully changed.
So how do we reconcile that with the fact that so many things believed to affect it *did change*? The simplest explanation is that mask mandates, lockdowns, vaccinations, etc actually had negligible net effect to begin with. Of course, this is in conflict with all the highly confounded (by different testing rates, healthy vaccinee effect*, etc) observational studies that were published claiming those interventions had a large effect.
Otherwise we have the surprising coincidence that all these things cancelled out.
As said before, the vaccines should protect against viremia (at first, until waning/mutations first negate this and then we get antigenic imprinting and ADE). This should prevent some people from getting severe covid for a time. So the fact that overall mortality remained about the same at the time of max effectiveness indicates there is some harm being done.
Unless you have a theory for why these things all canceled out, then we are stuck with the surprising coincidence explanation. As we know from Lakatos, that is a sign of a degenerating research programme. Ie, it is something to be avoided.
* Joshua, please just follow the conversation with Chris about this upthread rather than asking again.
Anoneuoid –
From where I sit, you go to great lengths to avoid actually addressing points or answering questions clearly. I wouldn’t know why you do that, but I think it’s unfortunate (and it makes quality exchange impossible). I allowed room for the chance that maybe you’d answered Phil’s question but I just didn’t understand how you had done so. Well, it’s clear that he didn’t think they were answered either.
First you said that you had answered it. Well, neither I or Phil saw that you had. Then you said that you didn’t answer it because you wanted to answer something else first. Which is contradictory if not odd. But even still, at least as far as I can tell, you still haven’t actually addressed his questions straight on, but instead just kind of dance around them.
> So the fact that overall mortality remained about the same at the time of max effectiveness indicates there is some harm being done.
Again, you seem to me to be overly certain – as you leave out possibilities that would have to imply you’re making assumptions about counterfactuals, without sufficient evidence.
Ignoring all the noise about calculating excess deaths (which I think shouldn’t be done, at all), your logic is still seems insufficient to me. The reason being – even if is true that there’s no signal of vaccine effectiveness directly evident in excess deaths, that could be the case because many people who would have died absent vaccines, didn’t die because the vaccines protected them. I’ve asked you about this multiple times yet as far as I can tell, you haven’t addressed that point and just repeatedly reverse engineer as if you have evidence to support your conclusion that eliminates that possibility. Perhaps you do have the needed evidence, but you haven’t as of yet presented it?
> * Joshua, please just follow the conversation with Chris about this upthread rather than asking again.
Again, you say that you’ve addressed something where I just don’t see how you’ve done so. I don’t see anywhere in your discussion with Chris where you indicated that you have evidence that there is a “healthy vaccinee effect” in the real world, with the COVID vaccines. In fact, you haven’t even shown evidence that on average, those who have gotten vaccinated haven’r been at higher risk (in poorer health, with more comorbidities, on average older), than those who haven’t. Please, if you’re going to say that you’ve addressed that point (let alone the one about your counterfactual assumptions), provide a link for the evidence you’re using.
Ok, I won’t ask again. If you want to continue not actually answering questions, that’s your prerogative. But you should at least not say you’ve addressed points that you haven’t addressed.
It is not worth answering your questions, because you won’t put in the minimal effort to scroll up and find the paper Chris cited. The one being discussed in the comments you have now written dozens of paragraphs about. The one that would answer your question.
It is just a waste of time.
Anoneuoid;
So in that study, they say:
The findings in this report are subject to at least four limitations. First, the study was observational, and individual-level confounders that were not adjusted for might affect mortality risk, including baseline health status, underlying conditions, health care utilization, and socioeconomic status.
Then they say:
Second, healthy vaccinee effects were found in all but the youngest age group.
So they didn’t control for confounders regarding health status, but there was a “healthy vacinee effect?”
‘
Hmmm.
Then,
Lower rates of non–COVID-19 mortality in vaccinated groups suggest that COVID-19 vaccinees are inherently healthier or engage in fewer risk behaviors
So that study says that behaviors might explain the “healthy vaccinee effect” they found – which correct me if I’m wrong, wouldn’t be a very good basis for reverse engineering about the efficacy of the vaccines.
And then, look at their references….
Lower rates of non–COVID-19 mortality in vaccinated groups suggest that COVID-19 vaccinees are inherently healthier or engage in fewer risk behaviors (7,8);
7 and 8?
Jackson LA, Nelson JC, Benson P, et al. Functional status is a confounder of the association of influenza vaccine and risk of all cause mortality in seniors. Int J Epidemiol 2006;35:345–52. https://doi.org/10.1093/ije/dyi275external icon PMID:16368724
Simonsen L, Taylor RJ, Viboud C, Miller MA, Jackson LA. Mortality benefits of influenza vaccination in elderly people: an ongoing controversy. Lancet Infect Dis 2007;7:658–66. https://doi.org/10.1016/S1473-3099(07)70236-0external icon PMID:17897608external icon
Did you notice that the references were related to the flu vaccine, and not real world data on COVID vaccination?
So you want to reverse engineer about the efficacy of the vaccines using a noisy metric, because a study that didn’t control for health status that referenced a “healthy vaccinee effect” that could even be explained by BEHAVIORS, and because of references to the use of vaccines in a completely different context.
So, once again:
I don’t see anywhere in your discussion with Chris where you indicated that you have evidence that there is a “healthy vaccinee effect” in the real world, with the COVID vaccines.
So maybe it wouldn’t be a waste of your time if you actually addressed my question, instead of just hand-waving at some reference as if it does, when in fact it doesn’t.
Lower baseline mortality in the vaccinated group (for whatever reason) *is* the “healthy vaccinee effect”. Your critique of the paper makes no sense at all.
The actual problems with the paper are they don’t tell us the covid-related mortality for some reason, even though they had that data. Then that they mixed together pre-vaccinated and unvaccinated, and from their tables it looks like the vast majority of “unvaccinated” were actually “prevaccinated”. But they don’t give us the required information to be sure.
It is yet another very strange production by the CDC.
Anoneuoid –
> Your critique of the paper makes no sense at all.
Lol. First, I love how you ignore the issue of the confounds. So they speak of a “healthy vaccinee effect” without controlling for pre-existing conditions (baseline health, comorbidities, etc.). That doesn’t seem particularly valid to me at a foundational level. Particular since we know that the vaccinated population is significantly older than the unvaccinated. Even other factors like racy/ethnicity demographics, which initially would put the vaccinated cohort at lower risk, at this point would seem to be inverted – where those who are vaccinated are of a racial/ethnic demographic that would put them at higher risk.
Apparently you think it’s not important to (at least try to) control for extremely significant predictors for outcomes when assessing a “healthy vaccinee effect,” and well, that is your prerogative – but I have to say I don’t get how that would work.
But I guess the explanation, in the end, must just be that you know better about this than I.
Then, you say it isn’t relevant whether it might be behaviors that would predispose vaccinated to lower risk than their unvaccinated peers, as compared to bio-mechanics or physiology.
Yeah, I don’t get that either. For the simple fact that (1) behaviors change over the course of the pandemic and would presumably then differentially affect outcomes at different time periods (theoretically changing the magnitude or even direction of this “effect”), whereas purely physiological cause/effect factors for taking a vaccine and looking at outcomes would presumably remain pretty much a constant. And (2) obviously the behavioral aspects affecting cause/effect for COVID outcomes would be quite different than the behavioral components for cause/effect outcomes with the flu or other vaccines which have grounded the the very “healthy vaccinee” concept. I guess you think you can just mix and match these kinds of things across any environment, across any demographic or baseline health aspects, across any behavioral patterns, and just reverse engineer to your hearts content without taking these issues into account. That isn’t how I’d think about it, but then it’s clear that what I think “doesn’t make any sense.”
But here’s what I really find most interesting – is how you pick and choose which types of research you find as complete non-starters and reasons to bang that NHST due to confounds and statistical practice problems and other issues, and which you think are just fine for reverse-engineering your pet theories (to the extent that you’re ever actually explicit about what you think rather than just vague hand-waving). I’m looking for a logical framework for how you do it, but haven’t as yet been able to figure out what it is. I know that you’ve explained how your bias-free and just seeking out “truth” – but I tend to be skeptical about that with everyone (including myself) and that’s why I think it’s important to interrogate the theoretical causal mechanisms. I guess I’m not just inclined to take your word for it.
At any rate, I won’t again ask you to provide it – but if you do have any actual real world evidence for a “healthy vaccinee effect” related to the COVID vaccine, as in actual data to explain WHY those getting vaccinated would be a lower risk for higher mortality, in contrast to their relatively greater age, perhaps more comorbidities, etc., feel free to provide some – as I’ve asked and presumably either you don’t know of any or are just holding out for some reason.
Joshua, your continued persistence to “figure out” Anonyd00d is admirable, and it is–at the end of the day–your time to waste, but I just wonder after over two years of this, whether you (or any of us) will really have any breakthrough. Some people are just broken, ill, overconfident sociopaths. He thinks colonoscopies can cause colorectal cancer, low vitamin C was to blame for COVID deaths at the start of the pandemic, badly lost a COVID-related bet with another blog reader due to overconfidence and poor scientific understanding, and once stated that in the US, more people died bowling than in car accidents.
https://statmodeling.stat.columbia.edu/2022/02/13/driving-about-85-m-p-h-in-a-45-m-p-h-zone-on-the-winding-southern-california-road-when-he-lost-control-of-his-suv-he-was-not-charged-with-any-legal-violation/
“Healthy vaccine effect” *is* the confound. This is pretty simple stuff, slightly above finding the paper being discussed in the comments. Go ask on stack exchange or elsewhere what it means.
To all in this thread:
Enough! Once the discussion devolves to recommendations to go to Stack Exchange, we’ve reached true “garbage time” levels of discussion. I appreciate that everyone is polite to each other, but this is still going nowhere and it’s time to stop. Next time please anticipate this and stop much sooner. Thank you.
Unanon:
I’d forgotten the bowling thing. That was indeed ridiculous.
Sorry Andrew,
But apparently no one else on this blog is willing to figure out that “healthy vaccinee effect” simply means lower baseline mortality and speak up (you could almost ask anyone willing to put 10 minutes into looking it up). That is not a good sign there is difficulty with establishing such a simple concept.
Anyway, I am gone for awhile.
If I might be indulged an additional post on the “healthy vaccinee” effect story. The effect is actually a hypothesis arising from studies of flu vaccine efficacy including effects on all cause mortality and it’s defined as – “Vaccinated study participants have a lower proportion of comorbidities than unvaccinated study participants, as indicated by baseline characteristics.”
Of course “healthy vaccinee” could be used to describe a real observation; e.g. a study might include careful analysis of preexisiting health allowing a defined conclusion: “we determined healthiness/levels of comorbidities in our population samples and determined that vaccinees tended to have better baseline health than non-vaccinees”.
Baseline health status in the Xu et al study discussed here wasn’t determined and so it seems to me that the “healthy vaccinee” effect is used in its hypothesis meaning (which is consistent with the statements that future analyses will be required to address their observations). So while observation of an apparent healthy vaccination effect on non-Covid-related mortality as in the Xu et al study may indicate conforming to the “healthy vaccinee” hypothesis it seems quite likely that some of this effect could indicate an opposite causality – i.e. that being vaccinated has some real element of protection against non-Covid mortality. This could arise for example because non-vaccinated individuals who catch Covid have compromised health which hastens their demise from non-Covid morbidities beyond the period for which Covid-contributions apply. This is clearly a real effect especially in vulnerable populations (e.g. https://www.nature.com/articles/s41586-021-03553-9 ) There may also be psychological reasons that feed into health associated with vaccination – e.g. being vaccinated is associated with reduction of fear, worry and stress and this affords some protect against non-Covid mortality.
Yes, thank you for actually looking into it. In the past I used to be sure to share links but it triggers the spam filter and most people do not bother clicking them anyway.
The reported deaths kept coming in after the original document was published (but dated before the cutoff). You need to check the later FDA documents. I’ll see if I can find them.
The above was a reply to Chris
I was trying to find my old posts on this blog but that wasn’t working. Here are the most recent numbers I am finding right now:
Moderna:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8482810/
Pfizer:
https://www.fda.gov/media/151733/download
So your 21 vs 17 number does appear to be the updated one. It was originally reported as 15 vs 14 I believe.
The sample sizes were about the same so 37 deaths vs 33 deaths. Then 4/33 is ~12%. Of course these numbers are small, but from that we should expect there is little effect. And indeed, we observed all cause mortality stayed about the same.
As for the other study you mentioned:
You misunderstood this paper (it has been discussed in the comments here before). The “healthy vaccinee effect” means the vaccinated people were less likely to die to begin with. In this case that would translate to mean they are ~66% less likely to die whether or not they got the vaccine.
The paper is a great example why these observational studies lead to the wrong conclusions. I am surprised you got two “+1 comments”. Andrew knows better at least.
OK, we’ve established that the Pfizer/Moderna trials can say nothing about the effects of these vaccines on excess deaths. So your assertion that “….the moderna/pfizer RCTs reported ~15% higher all-cause death rate in the vaccinated groups.That is the best evidence out there that the vaccines had little impact on excess deaths.” is wrong.
However, there is lots of evidence that the vaccines have a positive effect on reducing excess deaths. Several posters have linked to data that indicate this. Obviously vaccines greatly reduce Covid-related deaths (which are excess deaths). The paper I linked to indicates in very large cohort studies that vaccinations are also associated with reduced non-Covid deaths. I said in my post (and here) that they are “associated with”. That doesn’t mean that vaccination is the cause of reduced non-Covid mortality although there is no doubt some causality in that direction. As stated in the paper it’s likely that some (maybe most) of the causality is in the other direction – e.g. vaccinated individuals are inherently healthier or engage in fewer risk behaviors. As the paper also states: ” future analyses will address these issues.”
It doesn’t say nothing. It says whatever effect there is, is likely small.
Yes, the latter is the “healthy vaccinee effect”. I’m not sure you are understanding what this means.
It means (if we accept the results of that paper) when we do observational studies comparing vaccinated vs not, the baseline mortality rate of those who got vaccinated is ~33% that of the unvaccinated. So the vaccines would appear to be ~66% effective vs death even if they did nothing.
That paper indicates the other observational studies you cited are likely overestimating the effectiveness by a large margin. It does not support your argument.
“It doesn’t say nothing. It says whatever effect there is, is likely small.”
It doesn’t even say that. This is the point you are not getting. The observed outcome in the sample is consistent with a large vaccine effect in reducing all cause mortality, a small vaccine effect in reducing all cause mortality, a small vaccine effect in increasing all cause mortality, and a large vaccine effect in increasing all cause mortality.
I get frequentist reasoning (that every value in the arbitrary confidence interval is equally likely) just fine.
It is just a computationally efficient approximation of the credible interval, which is a convenient summary of the posterior. The likelihood/posterior tells us any benefit in terms of all-cause mortality is most likely to be small, if it exists. And when we look at overall all-cause mortality following the vaccination campaign, that is indeed what we see.
However, we can also look at studies confounded by healthy vaccinee effect, different rates of testing, reasons for the tests, etc. Then we see something very different.
Anoneuoid –
> . And when we look at overall all-cause mortality following the vaccination campaign, that is indeed what we see.
You repeatedly say things like this – but I don’t see that you have sufficient information to make such a statement given the myriad relevant interacting and confounding variables.
> However, we can also look at studies confounded by healthy vaccinee effect,
Do you have evidence of a “healthy vaccinee effect” in the real world with the COVID vaccine?
The studies selected for healthy individuals. In the real world, vaccination was perhaps significantly skewed towards older people, or people with more comorbidities (within limits as people who were already extremely ill might have avoided vaccination to some extent).
What is the evidence you’re using to assume a “healthy vacinee effect” in the real world with the COVID vaccine?
“The likelihood/posterior tells us any benefit in terms of all-cause mortality is most likely to be small, if it exists.”
You’re just hand-waving now.
Run a Monte Carlo simulation sampling the study’s age distribution, total N, and time horizon, and use life tables to represent the hazard. Then create 2 other trial arms by increasing and decreasing the hazard by 20%. Tabulate outcomes. How often does *decreasing* the hazard result in *more* deaths than not? How often does *increasing* the hazard result in *less deaths* than not? I’ll wait for your answer. You might learn something!
Then account for slight imbalances between arms from potential geographic variability; even within the US, average life expectancy can differ as much as 4-6 years on a state-by-state basis (doesn’t have much to do with business involving infant death rates–the effect holds up for life expectancy of 65-year-olds).
https://www.cdc.gov/nchs/data/nvsr/nvsr70/nvsr70-18.pdf
Then account for the idea that the **vaccinated** group had a slightly greater proportion of individuals with comorbidity at the start of the trial. https://www.nejm.org/doi/suppl/10.1056/NEJMoa2034577/suppl_file/nejmoa2034577_appendix.pdf
Andrew already told you last year that the all-cause mortality estimates were too noisy to make any conclusions. Most of the blog readers told you that too. Get a clue, Anonyd00d!
It is obvious you haven’t done that for yourself, otherwise you wouldn’t be trying to apply the life tables to this dataset.
Anyway, your advice is to p-hack my way to the wrong answer for a question I didn’t ask.
I’ll stick with knowing that RCT data indicates about 90% chance the mortality benefit is less than 10%, and about 85% that its under 5%. We can fudge that to 70% and 65%, respectively, to account for most sources of systematic error.
Yes, its hard to account for the subjects dying at around half the expected rate despite it being in the middle of a pandemic (because they excluded high risk subjects). But the observational studies are even worse, so this is the best we have.
Then it turned out all cause mortality stayed about the same after mass vaccination… Looks like I’m doing “impossible” things over here by applying basic logic and probability concepts to a problem, rather than using NHST.
Anoneuoid –
> It says whatever effect there is, is likely small…
What is your rule for determining when a sample is big enough to “say” something meaningful about fhe effect of an intervention with respect to a particular outcome?
Apparently you think 38 outcomes of interest total between the @44k participants in the two groups is sufficient, so I’m curious to know what your cutoff point is.