Thurs 7 Mar 18 Apr 2024, 12:30pm at 19 West 4th St., room 517:
How large is that treatment effect, really?
“Unbiased estimates” aren’t really unbiased, for a bunch of reasons, including aggregation, selection, extrapolation, and variation over time. Econometrics typically focus on causal identification, with this goal of estimating “the” effect. But we typically care about individual effects (not “Does the treatment work?” but “Where and when does it work?” and “Where and when does it hurt?”). Estimating individual effects is relevant not only for individuals but also for generalizing to the population. For example, how do you generalize from an A/B test performed on a sample right now to possible effects on a different population in the future? Thinking about variation and generalization can change how we design and analyze experiments and observational studies. We demonstrate with examples in social science and public health.
Which paper is this talk referring to?
Some of it is here.
Thanks a lot.
> average effect without considering what that means in the context of variation
From the OFFICIAL US FDA doc ethnic vaccine efficacy data for mRNA vaccine,
https://www.fda.gov/media/144245/download
Page 27 Table 8: Subgroup Analyses of Second Primary Endpoint
Subgroup | Vaccine Efficacy % (95% CI)
White | 95.4 ( 90.3, 98.2 )
Asian | 74.4 ( -158.7, 99.5 )
dx:
It sounds like a bit of hierarchical modeling could be helpful here!
US Asians consist of SAsian, EAsian and SEAsian. SAsian and White genetically share ancient IndoEuropean ancestry, so the -ve could be for EAsian and SEAsian. But Qatari results with mostly SAsian migrant workers did show -v results after 8 months.
Note the way efficacy is defined rel to non vac control, -ve means vacinated could be MORE prone to infection. Most results were more or less self censored for period > 6 months after vac but Qatari results from the Cornell researchers there did show total -ve efficacy CI after 8 and 10 months rather than just approaching zero. The booster results were less than 4 months after vac.
https://doi.org/10.1101/2022.02.07.22270568
Table 1
Dose | Week | Grp | NVacPos | NUnvacPos | NVacNeg | NUnvacNeg | VEff
2 | 32 | 8th month after Dose 2 and no Dose 3 | 3625 | 13026 | 2003 | 7915 | -7.5 ( -15.3, -0.2 ) <–
2 | 36 | 9th month after Dose 2 and no Dose 3 | 2989 | 13134 | 1779 | 7901 | 1.5 ( -6.2, 8.7 )
2 | 40 | 10th month after Dose 2 and no Dose 3 | 6257 | 12946 | 3340 | 7974 | -17.7 ( -25.6, -10.3 ) <–
I recommend focusing on “test-negative design”, do your best to find out what that means (what assumptions are made, how well they apply to doctors seeing median age 12 year olds in Qatar with flu-like symptoms, etc) then return with more questions. It really matters exactly how these numbers are arrived at.
Sorry, but its really a PITA to figure out what exactly was done in these studies.
I will say the only people I know still testing positive for covid work at universities where for some reason they are still getting vaccinated and tested. Everyone else has moved on.
To counter one’s anecdotal evidence with another, I know a few a people who’ve recently been diagnosed with COVID, and only one of those works in a university. That one person claims he never got any boosters.
Interesting, can you give more info on why they got tested?
I go read https://old.reddit.com/r/Coronavirus/ and its like stepping into a different reality. I’d estimate I have (at least) weekly interactions with a few dozen people from 20s to 70s, from “communists” to “MAGA” and everyone stopped caring about this years ago now.
What other subset of the population is still getting tested that I don’t see? Maybe they are less social personalities so keep to themselves?
These are exactly the types of issues that need to be addressed by these “test-negative design” studies, but never are (beyond naybe some mention in the discussion).
Anon:
I can’t say for sure, but I live in a country with free healthcare. So I imagine all that happened was that they started feeling sick, then went to their GP to get checked out. I wouldn’t say any of them are non-social; one particular subset I assume you’re missing are those not living in the US, based on you mentioning MAGA.
Yes, I was thinking of the US. Afaict most people here even stopped using the free tests like 2 years ago, since they didn’t seem to serve any purpose. Now they also cost money I assume.
From the UK challenge study we saw regular testing (2x per day for 2 weeks) lead to ~50% of people *without covid* testing positive at least once.*
If tests are still free I bet your country us seeing a lot of false positives (in the infection sense, perhaps theres some viral RNA trapped in a booger or something).
* Ill go find the ref if you are interested
Anon
As usual, your comments are declaratory and definitive -overly so, in my view. There is some data about COVID test use in the US:
https://www.cdc.gov/mmwr/volumes/72/wr/mm7216a6.htm
Your statement that most people (I’ll grant you a “majority”) are not using the tests any more I likely true. I’m not so sure that going back 2 years is quite as true, and I think your characterization dismissing the use of the tests portrays the reality. From many of your comments, I gather than you view the tests as a bad joke (I have plenty of my own reservations), but I think you are letting your view color our comments excessively. According to the link, plenty of people have ordered and used the tests. Certainly, that will decline now that that the tests are no longer free. But they don’t appear to be as widely ignored as you suggest.
In my personal social circle (which afaict is of typical, or even above avg, size) its far beyond the majority stopped testing/caring, its ~100%. The few I do hear about are related to university employment.
But read that subreddit and its like a whole other universe. Who is this population? Because this seems like key info required to interpret epidemiological data.
Your ref’s data goes to May 2022, which is essentially two years ago. It doesn’t really help either way.
I’ve also noted before that I don’t think I know anyone who would answer a political phone survey. Its gotta be like the least tech savvy 1% answering these things.
Anon
Well at least that explains a lot. Your social circle is large – I’m sure it is larger than mine. But your confidence in its representativeness seems related to how sure you are about your observations. I live in a different world – perhaps that is why I am constantly shocked by what “the public” is willing to do (e.g., voting behavior, consumer decisions, etc.). We indeed live in different worlds.
Anoneuoid: I have heard from parts of the USA like that. People keep coming down with ‘the flu’ and having to cancel social plans, and people they know keep dropping out of sight with unspecified health problems. And its interesting how across the rich world death rates peaked in 2022 as governments removed airborne infection control measures, or how gatherings for the very rich often have lots of infection-control measures placed just out of the camera.
Anon:
Yeah, it’s a big country with various barely-overlapping social groups. My mom got covid last month. It happens. And she’s never worked at a university. It’s not a political thing; she just wants to stay alive and healthy.
Andrew:
I am not so concerned about who is “getting” covid, which is obviously a small subset of the total at this point.
It is about the reporting step. Most people are not getting tested, but some subset of the population is still doing doing so. We need to know who those people are to correctly interpret the epidemiological data. Same as it appears the unvaccinated who died of covid were largely those who were already mortally ill so didn’t get vaccinated for reasons related to that. The so-called “healthy-vaccinee effect”.
Seems there is intense selection bias at work.
Sean:
What data are you using for US mortality? I have been looking at this:
https://www.usmortality.com/explorer/?c=USA&t=deaths&ct=weekly&v=2
If you are constantly shocked that indicates you should search for a different model of what is going on.
More importantly, you are actually informally testing *your* hypothesis, rather than some default null hypothesis. This is the correct thing to be doing. This is why I like your discussions here.
I don’t care about the political stuff really, I care about using science to prevent/cure diseases and such. Join me in advocating for testing *your* hypothesis (royal “you”) rather than a default null hypothesis no one believes.
But before that, lets join forces to encourage independent replications so these hypotheses are actually being compared to reliable data. That is my entire point of posting on this blog.
Anoneuoid: I am most familiar with the province-by-province data published by Statistics Canada around May of the following year (their site is down today), but my understanding is that the US and most of Europe have been similar. The study of veterans in the USA and the study of Koreans and Japanese by Kim et al. in Annals of Internal Medicine give some hints why people die more often after surviving COVID.
It looks like your US source shows deaths in the USA at the end of 2022 about 17% higher than in most winters before COVID, and deaths at the end of 2023 up about 8%. Norway counts its death rate in 2022 as about 7% faster than before COVID https://sciencenorway.no/covid19-mortality-ntb-english/norway-7-per-cent-more-deaths-than-expected-this-year-due-to-covid/2128261
Anon,
My family had some mild respiratory illness and we tested after symptoms started, and it wasn’t COVID according to the lateral flow immuno-tests. It resolved in about 6 days, so it was probably a normal “cold”. Other people we know have gotten COVID and tested positive in the last few months. Data from sewage treatment suggests that late Dec through most of Jan was a boom in COVID in the LA area that was nontrivial. The advantage of the sewage data is that it’s not really biased by people having to choose to go get tested. I think if anyone around here has “serious” respiratory illness they’re likely to get some kind of test, either at their doctor’s or OTC.
https://www.cdph.ca.gov/Programs/CID/DCDC/Pages/COVID-19/CalSuWers-Dashboard.aspx
COVID is still around, and people are still getting it. It really sucks to get, but most people have had the disease at least once and in this area most people are vaccinated with between 2 and 5 doses of vaccine. My whole family got the Flu and COVID booster back in Oct or so. My impression is that most people have much more mild symptoms after both vaccine + at least one natural infection than they were having back in 2020-2021. That also is reflected in death and hospitalization rates. However the post-COVID syndromes are still of concern, as anyone who has followed the story of “Physics Girl” https://www.youtube.com/watch?v=xbcjf-hrOAs who has been severely debilitated since she got COVID at least a year ago now.
Anoneuoid –
> What other subset of the population is still getting tested that I don’t see? Maybe they are less social personalities so keep to themselves?
It’s kind of remarkable how fluidly you traverse across the boundary between careful analysis and strangely unqualified opinions and silly anecdotal reasoning.
I know quite a few (mostly older) people who still test for covid, and who have tested both positive and negative in recent months.
But what I do or don’t know about who has done or hasn’t done what is mostly irrelevant – for obvious reasons. One problem, obviously, is a sampling bias. A second would he observer bias and a third would he a reporting, or social desirability bias.
I think it’s just weird how you so energetically handwave away such clearly established patterns of bias to of confirm the conclusions you’ve already drawn. It’s just a naked corruption of the scientific process. I dare say it’s even worse than your dreaded NHST (may the lord not strike me down for such blasphemy.)
What I can’t figure out, exactly, is why you so weirdly mix together such starkly contrasting approaches to analysis.
Joshua
You are forgetting that Anon’s “personal social circle” is larger than average – and, dare I say, larger than yours. I suppose it is large enough that there is no selection bias.
Dale g
My bad. Certainly the fact that it’s larger would mean it’s representativd and his own fanaticism related to covid wouldn’t likely suggest any biases.
BTW, this site indicates (ballparking) an average of 1.5 million tests daily.
https://covidtracking.com/data/national
It’s remarkable how 100% of anoneuoid’s “representative” sampling would completely miss such numbers of people. Perhaps someone with good math skills could calculate the associated probabilities given that he talks to dozens of people weekly, has extensively questioned each and every one about their personal health habits, and confirmed the reporting of each of them, and determined that “100%” of them have “moved on.”