The other day I criticized the Centers for Disease Control and Prevention for canceling the publication of a report on vaccine effectiveness. Apparently this move to unpublish was unusual; from a news report, “‘I’ve never seen a case where an article in the [Morbidity and Mortality Weekly Report] that got to that stage was not published,’ said Dr. Michael Iademarco, who led the center that included the publication’s operations from 2014 to 2022.”
But then today I was at a conference where Jay Bhattacharya, the acting director of the CDC, was asked about this unpublished report, and he said it was because it was using a really bad statistical method. It was only a brief exchange and there was no time for him to give a reference or to explain why he thinks the method is bad. I still haven’t myself seen a copy of the report so it’s hard for me to judge.
I think the best next step would be for the CDC to release the report in question, along with a critical response from a statistician explaining how the method is flawed. Bhattacharya said it was common knowledge that the method was terrible, so at the very least it would be a valuable educational opportunity to see this article that was on the verge of publication, and to understand its fatal problems. As a citizen, as well as in my role as statistician, I find it frustrating to hear about this dispute and not be able to see the controversial document and an explanation for why it’s not to be trusted.
It may be that the CDC is in the process of doing this. There could be a statistician right now writing that document explaining the problems with the almost-published paper.
Or maybe they sent it back to the original researchers to redo using a better analysis.
It’s kinda scary if the CDC was routinely using a terrible statistical method. Or maybe there’s more to the story. I just don’t know, which is why I’d like to see the study and also to see the criticism.
You can check JB’s X profile for answers. It’s a test negative design.
https://nitter.poast.org/DrJBhattacharya/status/2047142060495352302#m
Thanks for the link. That’s a start, but . . . I don’t want to see 140 characters. I want to see a report with references and an explanation! I’m not saying Bhattacharya is wrong, I just want to see the full story. I assume the decision to un-publish the report was made with the consultation of some statistical expert . . . there must be some report here, no?
Well, here is 2025 a paper about it:
https://www.thelancet.com/journals/langlo/article/PIIS2214-109X(25)00056-7/fulltext
Google Scholar lists a bunch when you search on ‘test negative design vaccines’
Best,
John
John:
Wait–but that link says that “test negative design” method is good. It’s the opposite of what Bhattacharya and his interviewer were saying.
I guess the point is that we have to step back and say that no method in itself is good or bad; it depends on how it is applied. Which is why I want to see the original report, along with the criticism explaining the problem with the method in that context.
“Nixon said that Bhattacharya had raised concerns about “the observational method used in the study to calculate vaccine effectiveness””
https://www.washingtonpost.com/health/2026/04/22/covid-vaccine-report-blocked-cdc-mmwr/
It seems that JB is your statistician.
“Dr. Bhattacharya had met with the study’s authors but that they had not wanted to change its design.”
https://www.nytimes.com/2026/04/22/us/politics/cdc-covid-vaccine-study.html
OK, fine, it was an immediate decision. But Bhattacharya is busy; he won’t have time to look into these statistical questions in detail. As acting CDC director, he should be able to find a statistician within the organization and task that person with evaluating the statistical claims in that paper. The statistician could then write the report. It doesn’t seem appropriate for the director to intervene unconditionally in this way.
JB is a political appointee (and I don’t need to mention his baggage). He absolutely is neither qualified nor would it be proper for him to make decisions about scientific publications.
This clearly was a political decision. There’s no reason to give this monstrous tool the benefit of the doubt.
The MMWR is not a peer-reviewed journal, and some of the things it publishes are pretty laughable. Mask studies, for example: there was a now-infamous mask paper that had a whopping sample size of … 2. I even found another MMWR paper that had a sample size of 1. And both of these papers were among the 90 papers cited by the CDC’s mask “science” page. The whole thing was beyond ridiculous.
Here is the “test negative design” for the recently released influenza study:
“Test-negative, case-control designs were used to estimate influenza VE among case-patients and control patients receiving outpatient or inpatient care for an acute respiratory illness (ARI) during the 2025–26 influenza season. ARI definitions varied by network. Case-patients were those with ARI who received a positive influenza virus molecular assay test result,† and control patients were those with ARI who received a negative influenza virus test result.
Data Analysis
To assess the association between influenza vaccination and influenza-associated outpatient visits or hospitalization, multivariable logistic regression was used. Odds ratios were calculated and adjusted for study site, patient age, date of illness, and other potential confounders.§ VE was estimated as (1 − adjusted odds ratio) × 100(%). Patients were considered vaccinated if they had received ≥1 dose of any 2025–26 influenza vaccine ≥14 days before the index date (ARI onset or clinical encounter).¶ Patients were excluded if they were vaccinated <14 days before the index date or had received a positive SARS-CoV-2 molecular test result at the time of testing for influenza** (2).
VE against an outpatient medical encounter and against hospitalization was calculated for any influenza, influenza A and B, and influenza A subtypes (A[H1N1]pdm09 and A[H3N2]) across networks and care settings, when possible. VE point estimates and 95% CIs are included in this report; CIs that exclude zero were considered statistically significant. VE estimates were not reported for strata with sparse data resulting in unstable model estimates, indicated by very wide CIs (range ≥100), even when case count thresholds were met (3)."
I can't imagine that Bhattacharya really gives a hoot about any of that. When he says he doesn't like the "observational" nature of the study, he means the numbers can't be trusted since the doctors decide who is and isn't scored as test negative, and the vaccine companies are giving them big payouts to make sure the vaccines look good!
https://www.cdc.gov/mmwr/volumes/75/wr/mm7509a3.htm to save someone a step in looking for the influenza paper. I agree it’s frustrating that we can’t see the cancelled paper, but based on this from NYT …
> The same method was also used in a study of the flu vaccine published last month. Had Dr. Bhattacharya been at the agency’s helm at that time, he would have raised objections to that report as well, the Health Department official said.
… it seems that we should be able to use the influenza paper as a proxy for whether the methods seem appropriate or not. (i.e. it is stated that the objection is to the method, not to its use in this particular context).
(Hmm, maybe not?) the paper seems to be a good match, but I can’t find the text described by @MattSkaggs therein …? Sophie Zhu, PhD, PhD Joshua Quint, Tomá, et al. 2026. “Interim Estimates of 2025–26 Seasonal Influenza Vaccine Effectiveness — California, October 2025–January 2026.” MMWR. Morbidity and Mortality Weekly Report 75. https://doi.org/10.15585/mmwr.mm7509a3.
OK, this is it: Patrick Maloney, PhD, M. P. H. Emily L. Reeves, M. P. H. Kristina Wielgosz, et al. 2026. “Interim Estimates of 2025–26 Seasonal Influenza Vaccine Effectiveness — United States, September 2025–February 2026.” MMWR. Morbidity and Mortality Weekly Report 75. https://doi.org/10.15585/mmwr.mm7509a2.
(The first was California, the second is all of US. But they appeared in the same issue of MMWR.)
(Previous comment seems to have been swallowed in moderation?)
The link I gave above is wrong (it’s a very similar paper in the same issue of MMWR, but for California rather than the whole US). The proper link:
Patrick Maloney, PhD, M. P. H. Emily L. Reeves, M. P. H. Kristina Wielgosz, et al. 2026. “Interim Estimates of 2025–26 Seasonal Influenza Vaccine Effectiveness — United States, September 2025–February 2026.” MMWR. Morbidity and Mortality Weekly Report 75. https://doi.org/10.15585/mmwr.mm7509a2.
In a short video clip I saw on Bluesky of Emily Oster talking to JB, it suggested that they both had long-standing issues with the test-negative design. I don’t know what these are, so I did a little bit of digging. There are obviously aspects of it that are tricky (as with any attempt to extract causal information from observational data …), but I couldn’t guess what EO/JB’s particular beef was – several of the examples below suggest *downward* bias in the estimate of efficacy due to various factors …
I’d welcome comments from someone who knows more about the field than I do …
Chua, Huiying, Shuo Feng, Joseph A. Lewnard, et al. 2020. “The Use of Test-Negative Controls to Monitor Vaccine Effectiveness: A Systematic Review of Methodology.” Epidemiology (Cambridge, Mass.) 31 (1): 43–64. https://doi.org/10.1097/EDE.0000000000001116.
Meester, Ronald, and Jan Bonte. 2024. “The Test-Negative Design: Opportunities, Limitations and Biases.” Journal of Evaluation in Clinical Practice 30 (1): 68–72. https://doi.org/10.1111/jep.13888.
Yu, Mengxin, Tom Hongyi Liu, Kendrick Qijun Li, et al. 2026. “Test-Negative Designs with Various Reasons for Testing: Statistical Bias and Solution.” Epidemiology 37 (2): 228. https://doi.org/10.1097/EDE.0000000000001940.
Doll, Margaret K, Stacy M Pettigrew, Julia Ma, and Aman Verma. 2022. “Effects of Confounding Bias in Coronavirus Disease 2019 (COVID-19) and Influenza Vaccine Effectiveness Test-Negative Designs Due to Correlated Influenza and COVID-19 Vaccination Behaviors.” Clinical Infectious Diseases 75 (1): e564–71. https://doi.org/10.1093/cid/ciac234.
The problem is simply replacing good/expensive methodology with dubious assumptions. We were already at this point with RCTs (vs deducing the expected result from your theory, comparing to alternate theories, and testing against new data). Every new dubious assumption makes it worse.
We really need another tier of medicine for people with low standards vs high, that is really the root of these problems. Most people with high standards have no problem with the low standard folks trying whatever they want. It should just be labeled as such and not forced on people.
For what it’s worth, there’s now a blog post
https://insidemedicine.substack.com/p/exclusive-heres-the-covid-19-vaccine
that discusses, and provides a link to, a draft of the paper:
https://insidemedicine.substack.com/api/v1/file/ab3d90bd-5bb8-4b2a-bf5e-0eccb93e0230.pdf
If memory serves, Bhattacharya was an author on an early paper claiming that Covid was not that serious, because a relatively high percentage of the people in Santa Clara County (near Stanford) who were tested for antibodies to Covid were positive, but little clinical disease had (by that time) been reported. However, the sample was people who answered an advertisement, and evidently people who thought they had had Covid, or been exposed to it, were more likely to respond. As I recall, there was a fair amount of discussion of that paper here. So, yes, I think Bhattacharya cares about results, not methods.
Memory served: https://academic.oup.com/ije/article/50/2/410/6146069?guestAccessKey=
With Jay Bhattacharya it is best to assume there is some combination of bad faith, political opportunism, and batshit crazy going on. He is the Graham Hancock of health economics.
Michael:
I’d never heard of Graham Hancock, so I looked him up, and, wow! Made-up ancient civilizations. A real throwback to the 1970s, kind of warms my nostalgic heart.
But then I saw that he’s been on Joe Rogan, so he must be the real thing! He just needs to go on Freakonomics and Sean Carroll to complete the trifecta of credulity.
John:
Yeah, also he misrepresented what I wrote. He wrote that I “incorrectly thought we had not accounted for the possibility of false positives. He later recanted that harsh criticism.” That’s just false. I never thought or claimed that they’d not accounted for the false positives. And no, I did not later recant a criticism that I never made, nor did I recant any criticism of that work that I did make.
Unfortunately, Bhattacharya has a government job and an audience of millions, while I just have the few thousand readers of this blog. And “government official misstates the position of a critic” is the sort of dog-bites-man story that will never gain traction.
Grrrrr.
Yes, we really need transparency from Bhattacharya on the specific aspects of this design he feels invalidates the study’s findings. Because right now it looks like a political appointee arbitrarily implementing scientific censorship under the guise of rigor.
The test-negative design is a variant of a standard case-control study that has been used hundreds of times throughout the past few decades to estimate vaccine effectiveness for influenza, COVID, pneumococcus, rotavirus, etc (https://pubmed.ncbi.nlm.nih.gov/31609860/). There is good characterization of the underlying theory, assumptions, and advantages involved with this type of study (https://academic.oup.com/aje/article-abstract/184/5/345/2389013; https://www.sciencedirect.com/science/article/abs/pii/S0264410X13004659), and it helps partially control for some of the biases from differential healthcare seeking behavior. Recent COVID-19 vaccine estimates from test-negative design studies align with other approaches including target trial emulation (https://journals.lww.com/epidem/fulltext/2024/03000/comparison_of_the_test_negative_design_and_cohort.3.aspx) and randomized control trials (https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2834520). See also slides 28/29 here from back when CDC scientists were still allowed to present data to ACIP https://www.cdc.gov/acip/downloads/slides-2025-09-18-19/04-Srinivasan-covid-508.pdf.
Is it perfect? No, no observational method can guarantee causality, of course. But the decades of experience with this method, the current understanding of underlying theory and assumptions, and the empirical concordance of estimates with this and other approaches contribute to why it continues to be widely used. Why single out this study design then? No one knows! All we have to go off of is soundbites and short tweets claiming ‘fundamental flaws’; it’s the opposite of scientific transparency and the latest example of this administration’s policy-based evidence making.
Lets use a test negative design to test for ESP effectiveness.
We live in a crazy world where vaccine studies are held to a lower standard than psionic studies. And lets be honest, the results of both kinds of studies are pretty irrelevant, people choose to believe what they want anyway.
It’s important to monitor the effectiveness of vaccines post-licensure, i.e. after clinical trials have been completed and the vaccine is in use in the population. Epidemiologists at public health agencies want to make sure the vaccine retains effectiveness against any new strains of the pathogen, and to see how well the vaccine is working in real-world use. Post-licensure effectiveness studies also help to compare vaccine effectiveness across specific populations, such as people with immune suppression, as there is often not enough data from clinical trials to do this.
There are ethical issues with withholding vaccines from people who are at risk of a disease once the vaccine has been recommended. This means that randomized designs for post-licensure effectiveness studies are less viable, though options exist in specific circumstances. Generally though, for ongoing monitoring of vaccine effectiveness, the only realistic option is an observational design.
Among observational designs for vaccine effectiveness evaluation, a test-negative case-control (TNCC) design is actually considered one of the more robust designs. It helps to mitigate biases due to differences in health-care seeking behaviour, access to testing, and baseline health between vaccinated and unvaccinated people that can be more severe in other designs.
Naturally, there has been a huge amount of work done on validating the TNCC design and trying to measure its biases. Vaccine epidemiologists at public health agencies, who use this and other designs regularly, want to measure vaccine effectiveness as accurately as possible. People in the field do not think there is a perfect design that removes all biases, but TNCC generally works quite well.
Some further reading that may be of interest:
A good general overview including discussion of strengths and weaknesses for TNCC: https://pubmed.ncbi.nlm.nih.gov/23499601/
A bias assessment for TNCC in COVID-19 vaccine effectiveness: https://pubmed.ncbi.nlm.nih.gov/37414791/
Exactly. We already have good large scale clinical trials. But no one study should be taken as definitive, science is the accusation of many studies.
Jay Bhattacharya is one of the signatories of the Great Barrington Declaration .
This tweet gives an interesting glimpse of his attitude towards the COVID pandemic.
Bhattacharya Tweet
I think what’s striking here isn’t just the specific dispute about the CDC paper or the statistical critique. It’s the broader pattern. Jay’s rise to the very top of our public‑health infrastructure was built on his incessant complaints that he was the victim of government actors “politicizing science,” suppressing dissenting views, and putting a thumb on the scale. The framing was that he and others were victims of a system that punished inconvenient findings.
But when you look at this episode, you see that exact dynamic playing out. The issue doesn’t appear to be the statistical methodology per se – Jay could have allowed the publication and then let the debate about the methodology play out in the public sphere.
What does seem to be at stake is that the findings were inconvenient for Jay’s preferred narrative. The evidence for that is that instead of engaging the substance, he shut the publication down.
This is the part that stands out to me: the rhetoric of “free speech” and “scientific openness” is often deployed asymmetrically. Many self‑described free‑speech advocates are very loud about what they see as “censorship” when it affects them, only to reveal that their advocacy wasn’t a principled one when they have the power to suppress work they find inconvenient.
Joshua:
I agree with you. It’s super frustrating. Especially frustrating is that, to the extent that Bhattacharya has legitimate concerns about the statistical method, he can work with a statistician to express these concerns openly and scientifically.
Andrew —
Indeed.
I highly doubt he will ever engage on this at a sophisticated level.
The CDC has a very stringent process for publication review. So his use of his authority as a government appointee not only suppresses the authors of this article, but, in overriding the publication‑review structure, he’s effectively suppressing everyone involved in that process.
I just looked at his Twitter. He has many recent posts complaining about what he sees as the Biden administration suppressing debate on scientific topics. I don’t know why anyone would take his free‑speech advocacy seriously.
When I was spending quite a bit of time reading Covid papers, I’ve come across test negative designs. In fact, I’d call it the primary observational study method the CDC people use for all of the flu/Covid vaccine studies (So from that perspective, one can say the refusal to publish is unusual.
I have doubts about the test negative design, but haven’t found the time to investigate it fully.
This claims to be the paper itself
https://insidemedicine.substack.com/p/exclusive-heres-the-covid-19-vaccine