Ryan Cassidy writes:
I have read several of your blog entries, along with your AmStat article. I subsequently found that the Santa Clara Sero-Prevalence authors appear to be more-or-less doubling down on their >2.x% “population-adjusted” prevalence estimate, which has recently been published.
I’m not a journalist, but rather an alumni member of the Stanford community. I don’t know any of the authors of the study. I’m simply a curious observer here about how estimates can seemingly get inflated depending on assumptions, that are apparently sufficient for press in an international (though I suppose not particularly celebrated?) journal.
Do you maintain that the authors’ described method of “weighting for population demographics” is seriously flawed?
My reply: Yes, I think their statistical analysis is seriously flawed; however, as I’ve said before, their estimate of the prevalence may be correct. The main change that my analysis (and others such as the non-Bayesian analysis from Will Fithian) did was to increase the uncertainty of their estimates; also we called into question their weighting analysis. It’s possible to get some of that uncertainty back down if you make stronger assumptions about the specificity of the tests used in the Stanford study, and I can’t really comment on the substance of that; basically, what I did was to use the same data used in that publication and more carefully account for the uncertainties.
Cassidy continued:
If nothing else, this is an interesting case study in medical statistical academic politics!
Taking a step back, however, even if the sero-prevalence was more like ~1.0% as your analysis indicates if more likely than the ~2.5% number that Bhattacharya et al. went racing to Fox News with, the Barrington-Declaration-esque political point they may be accused of trying to make goes something like “lockdowns are unnecessary, because the infection-fatality-rate is no worse than the flu”, right?
If this is the point they’re trying to make, then doesn’t both your and Bhattacharya et al.’s analysis/debate miss the point, because even if the infection-fatality-rate of both Covid-19 and Influenza were identical, the major difference is that Covid-19 is much more infectious than the flu? In other words, if 5 times more people in a given year contract Covid-19 than The Flu because of the greater infectiousness of the former, 5 times more die in that year with no lockdowns, even if the infection-fatality-rates are matched?
My short answer to that is that any given study provides only one piece of the puzzle. Bhattacharya et al. had lots of reasons for believing in their policy prescriptions, and my perspective on these issues is pretty narrow and technical, so really all I was saying is that they were understating their uncertainty and making questionable assumptions in their adjustments, and this annoyed me because they had lots of opportunity to do better, both before and after the initial preprint came out. The uncertainties in the analyses remain large, so you can’t rule out their estimates either.
Andrew –
FYI, Bhattacharya recently explained to me in a tweet that your criticism has been “answered.”
At any rate, aside from the statistical issues….
Bhattacharya et al. took the Santa Clara study (and I’ll take your word for it that they understated the uncertainties) and went on a national TV campaign to extrapolate from that study to proclaim COVID to be basically like the seasonal flu. Bhattacharya did that, and Ioannides did it as well.
I STILL can’t get over how in doing so, they broke (what I consider to be) a fundamental principle; They extrapolated to broadly promote an estimate based on an non-representative, convenience sampling. And I’m even more confused as to why you don’t seem to see that as being a big issue. And I wish someone would explain to me why relatively few people seem to think that was much of an issue. (a notable exception are the scientists in this group: https://www.youtube.com/watch?v=NTXgbN6uB1I)
Of course, that’s on top of other issues like recruitment through an email (or was it facebook?) by the wife of one of the authors, and for me what’s even worse, recruitment through promising the potential of an “immunity passport” if a potential participant tested positive without making sure to account for false positives – in other words, basically telling potential participants if they tested positive it was fine for them to just go home and hangout with grandma.
In this blog there have been plenty of Andrew’s posts talking about using post-stratification weighting to extrapolate from terribly non-representative convenience samples. Mainly survey data, largely political opinion polling and often in the service of predictive models for elections. But the principle is still the same. Many people feel with sufficient statistical technology applied after the fact, we can infer reality from convenience samples. It’s done every day and generally accepted as useful.
I say this not to defend the Bhattacharya, Ioannides, et. al. irresponsible and (if it were not such a serious matter) laughable flights of fancy. But it’s kind of like the old joke with the punchline, “We’ve established what you are, now we’re just haggling over price”. This blog would seem the last place you’d see running convenience sample data through a model and declaring the results generalizable.
This blog would seem the last place you’d see running convenience sample data through a model and declaring the results generalizable.
Should have been
This blog would seem the last place you’d see running convenience sample data through a model and declaring the results generalizable dismissed as invalid.
My criticism is with with respect to “external validity” and the identification of a “target population” when they went from the Santa Clara study (which as far as I can tell didn’t even adjust for SES and which possibly didn’t adequately adjust for age – see the video) to a widespread and long-lasting TV/media campaign to extrapolate and state implications at the national (or even global?!) level.
The internal validity is one issue. But the problem with the external validity was made obvious by their accompanying statements projecting morbidity and mortality outcomes that were off by orders of magnitude. As to whether the magnitude of miscalculation was more a function of underestimating infectiousness (as suggested by Cassidy) or underestimating virulence (IFR) or overestimating “herd immunity”… well who can really say? But clearly, they went from the Santa Clara study to vastly underestimate the impact of COVID, and on that basis aggressively advocate policy implications that they themselves said wouldn’t hold up if they were vastly underestimating (although later they didn’t adjust their policy advocacy after it became obvious they vastly underestimated).
But hey – even there, everyone makes mistakes! And statistics (and epidemiology) is hard! But then they go on to point the “politicization of science finger” at others (which in some cases is justifiable) while claiming saintly victimhood regarding the politicization of their own science.
Name:
In our discussions of the Stanford study, both in the blog and in our paper with Bob Carpenter, we considered three issues:
1. Uncertainty about specificity and sensitivity of the test, and propagating this into uncertainty about prevalence;
2. Adjustment for demographic and geographic differences between sample and population, which we recommend doing using MRP;
3. Unknown differences between sample and population such as arise from sicker people in some places being more likely to get tested.
Our technical points were mostly on items 1 and 2, where it seems that the Stanford team was overconfident about what could be learned from the data, even without considering item 3. I do not think that in their revision they answered the criticisms of Fithian or me, except to the extent that they are strongly asserting that the false positive rate of their tests was less than 0.5% or whatever. You can’t really get that from their data without additional assumptions. As a Bayesian, I’m fine with the Stanford team making such assumptions; I’d just like to see these assumptions made explicitly, rather than implicitly via careful selection of which calibration data to use. Also I don’t think their revision addresses my concerns with their extrapolation from sample to population, but it’s hard to say as they were not able to share their data, and I didn’t see that they tried to analyze their data using our code.
I agree with the comment by Name that I am not one to criticize a study just because it uses a convenience sample! I also agree with Joshua that the use of a convenience sample is a concern (see item 3 above). It’s just one more concern, and it does put more of a burden on the adjustment (item 2 above).
Does this crew ever discuss their interpretations of the indisputably increased death rate in the US that began with and persisted through the pandemic? Is it consistent with their beliefs about mortality rate? Do they think the deaths are induced by “lockdowns” or some other policy rather than COVID? To be clear, my question isn’t rhetorical. I don’t know what the view of these folks is about the issue.
Jacob:
I’m not sure, but my guess is that they would say that there’s no evidence that policies regarding masking, distancing, remote work, etc., had any effect, and that this death rate, or something like it, would’ve happened in any case as the disease spread through the population. I guess if any of the authors of that paper are reading this thread, they could reply right here!
After two years, do we know what percent of cases end up with detectable antibodies in their blood? And how long these last absent repeated exposures?
I’d bet there are lots of mild cases out there that go undetected in these seroprevelance studies, but would like to see some data on this.
> After two years, do we know what percent of cases end up with detectable antibodies in their blood?
Even independent of that, also at issue is uncertainty as to the correlation between antibodies and immunity – even against re-infection (infection of vaccinated) let alone against severe disease or death. And then there’d be questions of variability across variants.
All of which of course doesn’t stop policy prescriptions based on projecting morbidity and mortality numbers based on seroprevalence studies, with their accompanying sampling uncertainty.
Wlall of which explains why the pandemic has been over at least 5 X since the first time it was over in May 2020.
Two great points:
1- “any given study provides only one piece of the puzzle”
2 – “more carefully account for the uncertainties”
For those interested in the COVID IFR, we tried to obtain an appropriate estimate specifically with these two points in mind:
https://doi.org/10.1017/S0950268821002405
I’d look at it like this:
https://covid.cdc.gov/covid-data-tracker/#trends_totaldeaths
2020: 364k (~10 months)
2021: 461k (12 months)
2022: 155k (~4 months)
That is 36k, 38k, and 39k “covid deaths” per month. So covid deaths have been accumulating at a pretty much constant rate of ~450k per year in a population of about 331 million. If we assume each person gets infected once each year then IFR ~ 0.14%.
So that is our starting point. Now we need to consider:
A) What percent of people actually get infected each year?
B) What percent of these deaths were “from covid” rather than “with covid” or due to inept medical intervention?
C) What percent of “covid deaths” were missed?
I’ll guess something like: A = 0.5, B = 0.5, C = 0.25
That gives: B*(1+C)*IFR/A = 0.175%
Plug in your own plausible values. Perhaps you think only 25% of the population gets infected each year, and 100% of the “with covid” deaths were “from covid”. Then you would get IFR ~ 0.7%.
If 100% get infected, only 25% were “from covid”, and no deaths were missed then IFR ~ .035%.
“B) What percent of these deaths were “from covid” rather than “with covid” or due to inept medical intervention?”
Well, just to add another complication, at least early in an outbreak of a new disease, I don’t think it is appropriate to exclude deaths due to “inept medical intervention.” This was a new illness, and some of the things that were done early on turned out to be harmful. But they weren’t inherently inept: they were what happens when we have a new disease and we don’t know what works and what causes harm and what is just without effect. It takes time to sort that all out.
I’m old enough to remember when Legionnaire’s disease first appeared. Many people died. The disease is still around, but has something like a zero infection fatality rate. Today, it is easily treated with antibiotics. But most of those antibiotics didn’t exist back then, and as for the ones that did, it took some trial and error to find out that they did work. If you’re among those unlucky enough to be among the first to get a new disease, dying from complications of harmful treatments really is attributable, ultimately, to the disease itself. Only after there is widespread established knowledge of effective treatment practices should that become a separate category.
The same thing happens all the time outside of a pandemic. Sunscreen that only blocks UVB is a good example. Low fat (ie, high carb) diet is another. It really is almost everything that public health research recommends.
Essentially we have a very rudimentary understanding of the human body, so messing with something is more likely to break it than help.
also, the with/from thing is kind of a meaningless distinction in a lot of cases. if someone is fighting a moderate-to-bad early-stage case of covid and their weakened immune system catches strep throat and nobody notices because covid also causes sore throat so they get sepsis and the combination of the covid clotting issues and the sepsis causes organ failure and death, is it really meaningful to attribute that death to strep/sepsis and not to covid? very unlikely they would have gotten sepsis without getting covid, and the covid probably directly contributed to the death anyway. (real example of someone i know btw.)
Of course there are always multiple contributing factors. But, eg, last year my ma took her car to a new mechanic for an oil change and they somehow loosened the brake rotor in the process. Luckily this resulted in a squealing sound after a few weeks so a different mechanic was able to easily diagnose and fix the problem caused by the first one.
This could have been a disaster that would have never happened if the car didn’t need an oil change.
Anoneuoid –
> and they somehow loosened the brake rotor in the process
If you don’t know how they did it, then how do you know they caused the rotor to be loosened?
If they took a wheel off to change the oil, I’d suggest a new mechanic. And I’d say it’s nuts to blame all mechanics because you mom took her car to a mechanic who knows nothing about cars.
Anoneuoid –
> It really is almost everything that public health research recommends.
If you know it’s “almost everything” you must have engaged in a comprehensive process of quantification.
Please show your work.
That is according to the second mechanic. Apparently, when loosening the oil filter, it is a relatively common thing you need to look out for if you bang your wrench into this or that.
Wow. So you just trusted the other mechanic because of his expertise?
> Apparently, when loosening the oil filter, it is a relatively common thing you need to look out for if you bang your wrench into this or that.
So you have any idea what it works take to loosen a brake rotor? Banging “this or that ” with a wrench, without taking the wheel off?
Really?
A relatively ommin thing to do?
Please ask the second expert to explain how to loosen a brake rotor without taking off the wheel. How much did you pay him for his expert advice?
Do you have any idea how to loosen an oil filter or what it takes to loosen a brake rotor?
The only correct numbers are for CFRs, but those are useless. Nobody knows the true IFR, but it is likely that many more people have had the virus which would bring the IFR down, It’s hard to demonstrate that though.
It was about a year ago when it was realized that at least one in four are infected. The issue now is multiple infections over time. This has been going on for a while, so time component is becoming more and more important.
This is is a totally stupid way to define IFR.
The purpose of IFR in this study was to argue what the projected deaths would be in a policy alternative where about 80% of the population was allowed to get infected without vaccination in early-mid 2020. For this, deaths due to “medical ineptitude” should be absolutely included. Lives saved due to vaccination should be excluded. Even doing this would underestimate deaths because of health system breakdown.
This is a contradiction.
The health system breakdown was *because* of medical ineptitude. They were filling up the ICU’s with people kept alive on ventilators for weeks with basically no chance of recovery. They also sent infected people into nursing homes.
I’ve yet to see any evidence that all cause or excess mortality is decreased after vaccination. In both the RCTs and observational data mortality appears to be slightly higher after vaccination.
Here. This might help.
https://youtu.be/6RQ9UabOIPg
Maybe looking at the video you can figure out the “this or that” you could accidentally bang with a wrench while changing the oil filter that will loosen the brake rotor?
I really have no idea, but it sounds like you are calling my mom a liar. I hope not.
But most likely there was other stuff going on, which is the point of the story. It makes no sense to ignore that in trying to fix something you can make it worse or cause other issues.
If I try to fix your car you will most likely get it back in worse shape, because I’d have no idea what I was doing.
> but it sounds like you are calling my mom a liar.
Classic logic fail.
No. I’m not calling your mom a liar.
I’m saying that likely either she was confused, or you’re confused.
Or it could be that you’re a liar (it is kind of odd how the story went from “somehow” to a more specific “common” problem of banging a wrench into one thing or another).
But I would guess that’s unlikely also.
Or the 2nd mechanic was a shyster, or knows nothing about cars and you should be more skeptical about people who claim to have expertise.
> If I try to fix your car…
And based on all this I can assure you I won’t be asking you to fix my car!
> It makes no sense to ignore that in trying to fix something you can make it worse or cause other issues.
I don’t think anyone is “ignoring” the possibility unintended consequences (are you mind-probing again?)
I think that maybe other people, but certainly I, notice what seems to be a selective and skewed manner with which you leverage the possibility of unintended consequences to advance an agenda or confirm biases.
Wooosh
Exactly. That is the analogy being made.
This is the greatest discussion that’s ever been had on this website
I enjoyed it.
How many millions of times do people get an oil change without a brake rotor being loosened, or any other unintended consequence for matter that would actually be far more likely?
And yet, Anoneuoid sees a one in a million? hundred thousand? ten thousand? occurrence and says “See, unintended consequences. Why do you “ignore” unintended consequences?”
Better you should drive your car until the engine seizes than get your oil changed?
Joshua, you are one big woosh.
In the analogy, “you” (where “you” represents lots of people who get their info from places like the NYT) are me applying my ignorant car logic to covid.
I mean the regular acceptance of totally implausible default premises (herd immunity for a respiratory virus, “no evidence antibodies provide immunity”, cloth masks stopping a tiny virus, etc) that are obviously wrong to anyone with knowledge of the topic. Then the defensive posturing in response to obviously correct claims (“are you calling my mom a liar”).
Anoneuoid –
> Joshua, you are one big woosh.
Well, I don’t know what being a “big woosh” means, but I think I’m safe in assuming it isn’t complimentary. I guess maybe it means that you think that things that are obvious pass over my head?
Anyway,
> I mean the regular acceptance of totally implausible default premises (herd immunity for a respiratory virus,…
I’m certainly not expert on the topic (unlike yourself, of course) but to me, depending on what you mean by “herd immunity,” the “implausibility” isn’t so clear cut. My understanding is that generally it means that there have been enough infections in a given community that the rate of spread is meaningfully diminished – such as that a vulnerable person’s chances of getting infected are diminished because of the number of people in their community who have already been infected (or vaccinated). So it effectively means progress along a continuum. I see plenty of evidence that is a realistic concept, with an ebb and flow as conditions develop. Certainly with measles, we can se how high rates of vaccination help to protect even those who aren’t vaccinated?
Perhaps you’re referring to dichotomous measure, where you’ve passed a threshold and spread just stops, or is over, in some permanent state? Yes, it would seem that’s not plausible for COVID, but I’m not sure who this “you” is that said that could happen.
> “no evidence antibodies provide immunity”,
Again, I think that the “plausibility” there depends on exactly how you interpret the syntax. As I understand it, that wasn’t a statement to say that there isn’t existing evidence that would suggest that infection and recovery would confer any degree of immunity (you weren’t specific as to exactly what statement you were referring to, by whom), but that there wasn’t at that time direct evidence that getting infected with COVID would eliminate all risk of re-infection, or for how long it might do so. So depending on what it was supposed to mean, it doesn’t seem to me to have been all that “implausible” a statement early on in the pandemic.
> cloth masks stopping a tiny virus, etc)…
Here too, it seems that the exact meaning is important. I don’t think it’s “implausible” now, let alone early on in the pandemic, that a cloth mask might reduce the rate at which infectious particles spread from a source. Of course, there’s a lot of uncertainty as to how much the rate might be reduced, and how that would translate into the level of infectious spread across all contexts. And sure, if someone said that cloth masks would completely prevent spread of any sort in all contexts, then it would have been foreseeably “implausible.”
> that are obviously wrong to anyone with knowledge of the topic.
I think here we have to consider that your standard of “knowledg[able] of the topic” is a standard that practically no one else could live up to – as your insights are clearly above pretty much everyone else on any of these topics.
> Then the defensive posturing in response to obviously correct claims (“are you calling my mom a liar”).
Do you mean defensive posturing here when a given individual deflected from saying something that was obviously wrong?:
https://statmodeling.stat.columbia.edu/2021/09/16/wanna-bet-a-covid-19-example/#:~:text=October%203%2C%202021%2010%3A23%20PM%20at%2010%3A23%20pm
Viruses like SARS and influenza are completely different from measles/polio/etc. They primarily replicate in the mucosa rather than in the blood and other tissues.
There has never been (durable) herd immunity to such viruses. The mucosal immunity only lasts a few months to maybe a couple years in a severe case.
Anoneuoid –
> (durable)
Ah. Nice to see you walk your statement back.
So now, I guess you can point me to the statements by experts that we’d reach a “herd immunity” that comprises a 100% elimination of any spread, permanently?
Because indeed, it does seem to me that should have been implausible to an expert.
Obviously you get a temporary immunity to colds and the flu…
The problem is many people are so misinformed and defensive about their ignorance it is near impossible to talk to them.
Exactly like if I tried to feign knowledge about cars and argue with people who knew what they are talking, which I would guess is something many people do as well.
Anyway, what I learned from this is that you *are* capable of rational thought but simply do not apply it to health related issues. Probably due to lack of knowledge on the topic.
Andrew is getting mad so I will stop.
Hmm. I’m an okay mechanic for little jobs. I’ve changed oil many times and I’ve done brake jobs. I do find it hard to imagine how an even modestly competent mechanic could loosen a rotor while trying to change the oil.
I will agree that one sometimes inadvertently hits something with a wrench when doing an oil job. Although, in my experience it’s more likely that one will hit something with one’s knuckles–wince. If the ignition wiring or the windshield washer system were damaged, I would be willing to consider that a consequence of a poorly-done oil change.
Out of curiosity, do you know the make and model of the car involved?
Bob76
I’m sure something got mangled in the description of what happened via people who don’t know what they are talking about (me and my mom).
Re: “…totally implausible default premises (… cloth masks stopping a tiny virus …)
I have seem (on news reports) that various types of masks have been tested and provide some protection, and from my own anecdotal experience with cloth masks (before I got surgical masks), a) they make it a bit harder to breath, so they restrict some air molecules which are much smaller than a virus; and b) I saw my breath in winter without them and not with them, so they restrict water droplets, where the majority of viruses are concentrated, as far as I know. (I know of no personal virus test which uses an air sample instead of a liquid sample.)
So I personally find the claim that cloth masks can’t restrict virus dispersal implausible.
There is a another use for cloth masks that seems valid: Using a cloth mask over a surgical mask as a “mask shield” — the point being that for many people (depending on the specific shape of their face), surgical masks may leave gaps between mask and cheek, and these gaps undermine the utility of the mask. But for some people, a cloth mask over a surgical mask can press down the gap between surgical mask and face. (For example, for my face, a surgical mask leaves gaps at the side, but a typical cloth mask fits tightly there, while it usually slips down over my nose if I wear it alone. So wearing a cloth mask over a surgical mask gives the benefits of a good fit all around, plus the finer mesh of the surgical mask.)
The virus spreads as an aerosol, not in large droplets that will be cause by the mask. In fact some of the masks act like a screen to aerosolize large droplets that increase the time they stay in the air.
The main thing these masks do is prevent the bulk of the aerosol from getting sprayed down and out towards the ground, instead it concentrates in a could around the wearers head.
In some cases you can imagine this is beneficial. Eg, when talking to a cashier. But then the next person in line steps directly into your cloud.
And look at south korea: https://www.worldometers.info/coronavirus/country/south-korea/
Afaik they are still wearing masks there and there is high compliance. Perhaps someone can correct me on that.
> Afaik they are still wearing masks there and there is high compliance. Perhaps someone can correct me on that.
They were also wearing masks with high compliance when their rate of spread was significantly lower than many other communities were there was less wearing of masks. Also note that they don’t wear cloth masks much there.
So your comment is odd at a number of levels. First, clearly there are other factors that are relevant to spread other than the simple dichotomous variable of wearing a cloth mask/not wearing a cloth mask. Obviously, you’d have to hold things like level of community immunity, or the infectiousness of a given variant at a particular time, to measure the impact of differential prevalence of wearing (cloth) masks. Second, citing the rate of spread in S. Korea currently would be largely irrelevant to your statement about wearing cloth masks anyway.
Do you really need to have it explained that the existence of any particular level of spread doesn’t mean that the wearing of (cloth) masks didn’t affect that level of spread. The question is what would the level of spread be in S. Korea if masks weren’t being worn at all, under any conditions. That’s a difficult question to answer, and no, just linking to some stats at Worldometers doesn’t really add much to any reasonable analysis.
Uncertainty should be respected not ignored. I heard this quote from Feynman today – I thought you’d enjoy it: “Religion is a culture of faith, science is a culture of doubt.”
Anoneuoid –
> The virus spreads as an aerosol, not in large droplets that will be cause by the mask.
The very small size of COVID virus is certainly relevant to the efficacy of wearing cloth masks to prevent transmission from a source, but to my knowledge there is uncertainty as to the size of all aerosol particles which transmit enough of the virus to cause an infection.
And it seems to me that there is uncertainty as to other relevant factors – like the effect of even cloth masks on air flow (under different conditions) and on humidity (behind the mask).
Of course, since you’re so certain that cloth masks can’t have any benefit whatsoever, you can provide some evidence I’m wrong and that all uncertainty is eliminated – and I would appreciate you providing links.
I suggest maybe she was getting her summer put on in place of the snow tires, or perhaps getting the tires rotated while getting an oil change done as well.
Even still, it would take a pretty incompetent mechanic.