More coronavirus testing results, this time from Los Angeles

In comments, Joshua Ellinger points to this news article headlined, “Hundreds of thousands in L.A. County may have been infected with coronavirus, study finds,” reporting:

The initial results from the first large-scale study tracking the spread of the coronavirus in [Los Angeles] county found that 2.8% to 5.6% of adults have antibodies to the virus in their blood, an indication of past exposure.

That translates to roughly 221,000 to 442,000 adults who have recovered from an infection, according to the researchers conducting the study, even though the county had reported fewer than 8,000 cases at that time.

“We haven’t known the true extent of COVID-19 infections in our community because we have only tested people with symptoms, and the availability of tests has been limited,” study leader Neeraj Sood, a professor at USC‘s Price School for Public Policy, said in a statement. “The estimates also suggest that we might have to recalibrate disease prediction models and rethink public health strategies.” . . .

The early results from L.A. County come three days after Stanford researchers [including Sood] estimated that 2.5% to 4.2% of Santa Clara County residents had antibodies to the coronavirus in their blood by early April. . . .

The Santa Clara study recruited around 3,300 participants from social media . . . The study was composed differently in Los Angeles; 863 adults were selected through a market research firm to represent the makeup of the county. The county and USC researchers intend to repeat the study every two to three weeks for several months, in order to track the trajectory of the virus’ spread.

It’s good to see this sort of study. When discussing the Stanford study yesterday, we expressed concern that the estimate could be a statistical artifact. But it’s hard to say: much depends on the error rate of the assay, as well as how the statistical analysis was done.

The “good news” is that, as prevalence rates rise, inferences will be less sensitive to testing errors and other statistical issues. As data quality improves, inferences should be more robust to choices in statistical analysis.
To get back to the Los Angeles study: the news article refers to “a report released Monday”—that’s today!—but unfortunately I don’t see a link to the report. I hope the researchers share their data and code. This is an important topic, so best to engage the hivemind.

As discussed in yesterday’s post, it would be good to have data, on demographics and locations of the people tested, on the test results, and also on what symptoms the people reported. If confidentiality makes it difficult to share all this, just forget the location information and give us age, test results, and reported symptoms and co-morbidity.

If someone can find the report, data, and code, please share the links in the comments.

P.S. Shiva Kaul in comments pointed to a link to a version of the report, but I removed the link here because I’ve been told that it’s not an official document from the research project. No data and code yet. The unofficial report stated that 35 out of 863 people (4%) of the sample tested positive, more than twice as high as the 1.5% observed in the Santa Clara county study. This time they poststratified based on sex, age, ethnicity, and income, but not on geography. No data were reported on comorbidity or symptoms. They assume the test has a specificity of 100%.

The unofficial report concluded, “Further population representative serological testing is warranted to track the progress of the epidemic throughout the country and the world.” I agree. As noted above, when tests are done in populations with higher exposure rates, statistical concerns regarding specificity will not be such a concern.

P.P.S. Joseph Candelora points out that the numbers in the above-linked report are not quite consistent with the press release, so I’m not sure what to think. The authors should just post their report, data, and code somewhere. Too bad the L.A. Times reporters didn’t ask for all that as a condition for running their story.

P.P.P.S. Thomas Lumley has some discussion of the tests and Joseph Delaney has some discussion of the implications of the population estimates.

165 thoughts on “More coronavirus testing results, this time from Los Angeles

    • Actually thinking about it, depending on how the selection of the test worked, the CIs people have calculated might actually *underestimate* the false positive rate of the study.

      This is because a number of manufacturers are available offering tests with manufacturers’ test results, which are known to the authors ahead of time. Hence if the authors of the study went through and selected the test with the lowest specificity (which seems a natural thing to do), then this selection issue would actually mean the specificity would be even worse underestimated.

      Explaining the reasoning behind the selection of Premier Biotech would be very important.

      • Good point! I didn’t consider that.

        Also, specificity estimates will probably not have great generalizability from the sample that was used to evaluate test performance to the population the test is actually applied to. Lots of small biases can have major ramifications when trying to estimate prevalence for a rare disease with an imperfect test.

    • Zhou:

      See the P.S. The rate was higher in this new study. The only reason the Santa Clara study gave such a high rate is from the poststratification, but I don’t fully trust the poststratification because I haven’t seen those numbers. In raw rates, it was 1.5% in Santa Clara county and 4% in Los Angeles county.

      • Okay, thinking about it some more, I suppose for this USC study, a smart idea would be to use the Santa Clara study to bound the false positive rate on this new study. This avoids any complaints about selection bias, and dodgy manufacturers testing. Agresti-Coull gives us about 2% there, so the findings here in the LA study do seem to avoid the specificity issue.

        Still concerned that their supposedly representative sampling method generated a non representative sample though. Would be good if they published participation rate, as well as that data on symptoms/contact they keep collecting and then mysteriously not showing.

      • tl;dr Another problem with the LA and Santa Clara studies is that the estimates are effectively of disease prevalence at 18-64, but the large majority of Covid-19 deaths are at 65+. Looking at some plausible corrections for this mismatch resulted in IFRs of about 0.3% to 1.4%, which is more similar to previous reports and worse than seasonal flu. A suggestion for future research is to oversample older adults if your goal is to use prevalence data to estimate Covid-19’s IFR.

        Zhou, Andrew, and Peter — As your comments help to explain, the results of the linked study tend to support high Covid-19 prevalence in Los Angeles county, despite test quality problems. However, there are big logical gaps between this finding and the conclusion that Covid-19 has about the same risks as seasonal flu, which seems to be the main message that is being attached to this research.

        A major problem with the LA and Santa Clara studies seems undiscussed, yet is covered elsewhere on this blog in an entirely different context: the dangers of uncareful age analyses when fatality rates increase with age (eg, in the Case and Deaton posts).

        First, note that only 5% of the Santa Clara study was 65+ (and an unknown percentage of the LA study, but also small because the adult LA population is only 17% 65+). Therefore, the Santa Clara and LA studies only really tell us about Covid-19 prevalence at 18-64.

        Second, note that the large majority of Covid-19 deaths are at 65+. I don’t have age data for Santa Clara and LA specifically, but for the US as a whole about 80% of Covid-19 deaths have been at 65+, and that seems like a common percentage across areas.

        So, when the authors estimate infection fatality rates by taking a ratio of Covid-19 death counts to prevalence estimates, they are essentially taking a ratio of death counts at 65+ to disease prevalence at 18-64. The numerators and denominators do not really match. Overall, I’d expect the older age group to have lower prevalence because there is a far greater incentive to social distance at older ages, since the risk of death if one gets Covid-19 is so much higher.

        On the other hand, if governments decide to “unleash” Covid-19 and treat it like seasonal flu, then most of the US would be infected (>90%?) because of the disease’s very high R0. This would likely remove the difference in prevalence between ages 18-64 and 65+, simply as a product of almost everyone getting the disease.

        To see how much of a problem this could be, I tried giving the authors’ the benefit of the doubt about test specificty and selection bias, but assumed that Covid-19 prevalence at 65+ was currently half that at 18-64. I did some quick estimates of IFR for the scenario where Covid-19 was treated like seasonal flu, so differences in prevalence between age groups disappeared. IFRs ranged from 0.3% to 1.4%, which is higher than seasonal flu and more consistent with other Covid-19 estimates. Also, if 90% of the US adult population were infected, an IFR of 0.3% to 1.4% would give 690,000 to 3.2 million deaths — far worse than seasonal flu. And this does not count any deaths from exceeding hospital capacities.

        One should really consider many age groups, not only 18-64 vs. 65, but this comment is already far too long, and many age groups only makes the problems worse for the Santa Clara and LA studies.

        Some hopefully constructive recommendations do result: If you are performing prevalence testing for the purpose of estimating Covid-19’s IFR, then you may want to oversample older people or (better yet) collect enough data to get age-specific estimates! Similarly, it may be worth oversampling other groups at increased risk of death in Covid-19 cases, such as people with comorbidities.

        • More:

          I’m kinda thinking that when Ioannidis etc. talk about IFR or CFR of 1/1000, they’re talking about the rate in the under-65 or under-75 population. But I can’t be sure I agree that the mixing of age groups results in all sorts of confusion.

        • I took their comments to mean a population average of 1/1000, which population of course is mostly under 65. So more or less the same thing. Any statements that “mix age groups” are nigh uninterpretable, as you say.

        • Andrew:

          Yeah, it seems important to keep age groups clearly in mind when estimating IFR.

          To take two age groups and popular IFR estimates from Verity et al.

          For 30-39, IFR was estimated as 0.08%
          For 70-79, IFR was estimated as 4.3%

          The LA and Santa Clara studies (with Ioannidis) suggest IFR is overestimated by about an order of magnitude.

          If the 0.08% IFR of the younger people is overestimated by an order of magnitude, that doesn’t change our the public health understanding or plans for Covid-19 much.
          If the 4.3% IFR of the older people is overestimated by an order of magnitude, that’s huge! Society is much less at risk and control measures are overkill.

          The LA and Santa Clara studies try to reach conclusions about overestimation of IFR in general (all ages together), but they include mostly people in younger age groups where overestimation of IFR doesn’t matter and few people in older age groups where overestimation of IFR would matter. Seems like a bad idea.

          ::shrugs::

        • Andrew, do you mean 1/1000 or 1/10,000? Because Bhattacharya clearly believes the all-age death rate is between 1 in a thousand and 2 in a thousand. See his interview here, post-release of the Santa Clara study: https://youtu.be/k7v2F3usNVA?t=795 (queued to the relevant time).

          Further, Bhattacharya and Bendavid are already on record with a 1 in _ten-thousand_ estimate, in their WSJ editorial from 3/24 (https://www.wsj.com/articles/is-the-coronavirus-as-deadly-as-they-say-11585088464). Here are some excerpts:

          “…The first confirmed U.S. cases included a person who traveled from Wuhan on Jan. 15, and it is likely that the virus entered before that…

          …An epidemic seed on Jan. 1 implies that by March 9 about six million people in the U.S. would have been infected…

          As of March 23, according to the Centers for Disease Control and Prevention, there were 499 Covid-19 deaths in the U.S.

          If our surmise of six million cases is accurate, that’s a mortality rate of 0.01%, assuming a two week lag between infection and death. This is one-tenth of the flu mortality rate of 0.1%. Such a low death rate would be cause for optimism.”

          In the context of the whole article, it’s apparent that this estimate was not their idea of a best-case scenario or a lower bound for the IFR — they were legitimately positing this as a reasonable estimate for the disease IFR. Clearly they’ve changed their position based on their study results, but only to where they believe 1 or 2 in a thousand is the right number.

          Ioannidis is being a little less precise — his video interview said “in the same ballpark as seasonal influenza”. And there was a lot of handwaving in his Stat article from last month, but he eventually landed on 3 in a thousand for illustration purposes. Of course, he used that in concert with an absurd estimate of only 1% of the US population being infected by this highly transmissible, novel virus for an illustrative death count of 10,000 — an illustration that feels like scientific malpractice.

          But I’m pretty sure they all legitimately want to believe, and want the public to believe, that the IFR is on the order of 1 in 1000.

        • Joseph:

          NYC has more than 10,000 coronavirus deaths and fewer than 10 million people, so obviously the IFR has to be more than 1/1000. I’m guessing that at some level they are thinking about the general non-institutionalized and under-75 population, or something like that.

        • I think their arguments would be:
          a) We estimate as high as 2 in a thousand for Santa Clara
          b) IFR in NYC will be higher than SC because it has an slightly older population and higher incidence of comorbidity
          c) NYC is overcounting deaths
          d) NYC is approaching herd immunity levels of infection

          I don’t see how you watch that Bhattacharya interview and come away with their estimates being for anything other than the full population of SC County, barring none. Same for Ioannidis, although it’s a little less clear what actual number he means when he says “in the same ballpark as seasonal influenza”.

          But all that said, let me look back at what they actually say in their paper:
          “We can use our prevalence estimates to approximate the infection fatality rate from COVID-19 in Santa
          Clara County. … we estimate about 100 deaths in the county. A hundred deaths out of 48,000-81,000 infections
          corresponds to an infection fatality rate of 0.12-0.2%. If antibodies take longer than 3 days to appear, if
          the average duration from case identification to death is less than 3 weeks, or if the epidemic wave has
          peaked and growth in deaths is less than 6% daily, then the infection fatality rate would be lower. These
          straightforward estimations of infection fatality rate fail to account for age structure and changing
          treatment approaches to COVID-19. Nevertheless, our prevalence estimates can be used to update
          existing fatality rates given the large upwards revision of under-ascertainment.”

          I originally read “these straightforward estimations of infection fatality rate fail to account for age structure” to be a caution against generalizing these numbers outside of Santa Clara County, to other parts of the country with different age prevalence. But I suppose it might be acknowledging some groups somehow not properly accounted for in their estimate for Santa Clara itself. Hard to say.

        • Joseph and Andrew:

          Thanks for the links and excerpts Joseph. Wow.

          It’s hard to understand why Ioannidis is *confident* in the Covid-19 IFR estimate of less than 0.2% or so. The guy usually sounds so damn reasonable, then this!? Andrew, maybe you have a similar reaction, and that’s why you suggest Ioannidis is discussing IFR for under-65s, but it seems to me he is discussing IFR for all ages.

          My best guess of what’s happening is some combination of the below:

          * Ioannidis and coauthors have still-private data from serveral in-progress studies, which mostly indicate prevalence is high. That boosts their confidence in the low IFR.

          * The authors don’t realize quite how bad the test specificity problem can be, especially if it involves differences in specificity between different shipments of the test (bad batch quality consistency from the manufacturer) or a similar issue.

          * The author’s data shows many test recipients reporting mild Covid-19-like symptoms. The authors hesitate to share this, which they view as a white lie because people will scream “selection bias!” but the authors know selection bias isn’t the main contributor, since they see similar Covid-19-like symptom rates from in-progress prevalence studies with *good* random selection practices. They conclude, “If anything, it is test sensitivity that is underestimated, Covid-19 is probably more common than even we estimate, so the IFR is lower still.”

          What they don’t get is that people just generally report a lot of Covid-19/flu-like symptoms when asked, probably especially if symptoms are in the news. For the 2009 H1N1 pandemic, the CDC asked 216,000 US adults if they had flu-like symptoms in the past 30 days. 8% percent said they did, even in March when the pandemic was long over! (paper PMID: 21248680).

          * The authors don’t get the age standardization problem I mention above.

          A lot of speculation from me here, so take it with salt, but I’m baffled. And as always there’s that worry I could be the one who’s wrong about Covid-19…

  1. Sounds like the same test being used and maybe the same methods:
    https://pressroom.usc.edu/what-a-usc-la-county-antibody-study-can-teach-us-about-covid-19/

    “Premier Biotech, the manufacturer of the test that USC and L.A. County are using, tested blood from COVID-19-positive patients with a 90 to 95% accuracy rate. The company also tested 371 COVID-19-negative patients, with only two false positives. We also validated these tests in a small sample at a lab at Stanford University. When we do our analysis, we will also adjust for false positives and false negatives.”

  2. If they used the same test and made the same specificity assumptions then it is plausible that they could be vulnerable to the same sources of error. If the “true” specificity is 99.0% and they used 99.5% they would get a similar over-estimate of prevalence.

  3. I’m expecting a carbon copy of the Santa Clara County paper. Bhattacharya mentioned both Bendavid and Sood and when he announced that he had raised the funds for serology testing, and Sood (the lead on the LA County study) was a co-author on Bendavid’s paper.

    Presumably they were able to get the LA County one out quicker (10 days after samples were gathered rather than 14) because they had worked out the bugs first time around.

    Will be interesting to see. But it rankles a bit that the pre-print seems to be unavailable, despite the press release having gone out.

  4. It looks like this study provides much stronger evidence of a decent prevalence rate than the Santa Clara study. The non-linear effects of false positives mean that the higher prevalence seen in this study does a lot to reduce the variance of their estimate.

    https://colab.research.google.com/drive/15oTjyjls4Ca6PO_bCp18ySxnVwJntVQ- applies my simple model to the data in this study. Note that a prevalence of 0 is distinctly excluded by this analysis (the density at 0% prevalence is very close to 0). There is still a big amount of variance (2% to 6%) in the center 95% interval, but that’s a relatively small amount of variance relative to the orders of magnitudes within the Santa Clara study.

  5. Release shenanigans aside, its not warranted for such high confidence to be placed in the specificity of this test.

    This is the one they are using:

    https://www.nbcnews.com/health/health-news/unapproved-chinese-coronavirus-antibody-tests-being-used-least-2-states-n1185131

    The test has not only not been validated for FDA, it is from an unapproved Chinese manufacturer in that country and is now banned from additional export. There is no indication of the reason.

    In any case, 30/30 test at Stanford for this use only shows they were not shipped a box of rocks. For attempting something that is considered extremely difficult, finding a low % through all the noise, and extremely important (advising government on epidemic control), much higher caution is needed. Either in validating the test for 99%+ thenselves or in having a CI reflecting the possibility the test is less specific.

    Your prior piece noted the problem in the blanket scale up assumption end of the CI with what is seen in New York, so no comment on that besides “I agree.”

  6. Meanwhile, on the economic front, the price of oil has gone negative … big time (negative $37.63 per barrel).

    I’m not kidding.

    The Wall Street Journal reports:

    The price of a barrel of West Texas Intermediate crude to be delivered in May, which closed at $18.27 a barrel on Friday, ended Monday at negative $37.63. That effectively means that sellers must pay buyers to take barrels off their hands.

  7. “Conducting serologic tests in representative samples is the best available approach”

    “Residents within a 15-mile radius of the testing site were eligible for participation in the study.”

    “Weights were calculated to match the…distribution of our sample to the Los Angeles county 2018 census estimates. ”

    Perhaps they should make inference to a population prevalence in that 15-mile radius and not to the county of Los Angeles. It isn’t like infectious disease exposure is randomly assigned across space – the infection rate is literally a function of proximity to other sick people.

    • Oh my bad I re-read it – I think it was 15 miles from EACH testing center, thus presumably covering most of LA county.

      One thing I’d like to see in the next write-up is a discussion of selection-into-showing up. My current read is that this is an encouragement sort of design, so that the offer to get tested is randomly assigned. But take-up is not random among those randomly selected to be made an offer, and I don’t see a lot here about who did and didn’t show up to be tested.

      But I think I was wrong on my reading the first time: they didn’t draw a radius and set up six test sites, they set up six test sites and then drew a radius around each of them. calc

  8. It is getting ridiculous to watch this. Yes, all these tests are flawed! Most people here only started paying attention when it didn’t fit with their politics. The vast majority of the difference in cases/deaths between different regions are due to rate of testing. It is so obvious if you look at that data.

    Are the tests a useful measure of the thing making people have the severe HAPE-like symptoms that were dying on the ventilators before the ER and critical care doctors figured out that was the wrong treatment? We have no idea.

    • In terms of population, North Dakota and New Jersey are close in testing but the latter has ten times the reported cases and thirty times the reported deaths. In terms of population, Veneto has made twice the tests of Lombardia but it has reported half the number of cases and one sixth of the number of deaths. There may be something else in play.

      • The reason why Veneto had much less deaths than Lombardia is related to the higher number of tests, but mostly to the different containment strategy (much more focus on contact tracing and aggressive testing than on testing only severe cases and hospitalization). Above all, however, it was due to the governor of Veneto asking for help to Andrea Crisanti, brilliant parasitologist recently returned to Italy after spending most of his career abroad. See https://hbr.org/2020/03/lessons-from-italys-response-to-coronavirus

        • I’ve not follow the article, but I read a few weeks ago an interview with Giorgio Palù where I imagine he presented the same points.

          The thing is that the person I replied to thinks that when you do more tests the consequence is that you will have more cases and more deaths, not less.

        • Uh, I didn’t see your reply, it looks like it’s not easy to follow threads on this blog (the “search” button in the upper right corner only searches through posts, not comments). Then ok, the person you were replying to (Anoneuoid, right?) is sorely mistaken: more tests done soon (together with contact tracing and isolation at home) provably led to less deaths.

    • The vast majority of difference in deaths is due to testing? What?

      You’re claiming that NYC and have Santa Clara County have the same infection prevalence?

    • So I would love the new studies to be correct. The problem is how they don’t fit with other data.

      https://observationalepidemiology.blogspot.com/2020/04/more-on-ca-infection-rates.html

      New York is looking at a population fatality rate of around 0.16% or so. So that implies an infection rate of ~100% using the actual logic in the papers (they back calculate a IFR based on the death in the county and their prevalence rate). Spain and Italy are around 0.04% for the entire country, meaning an infection rate of about 25% for the entire country, a rate of infection that is stunning.

      The two things that could really help are:

      Making the data available in the pre-print phase, with lots of details and let some statisticians see what they can do with it. Some selection effects may be intractable (https://twitter.com/asymmetricinfo/status/1252572855168106496) but more could be done

      Doing at least one of these studies in a hard hit area. CA is the biggest state in the union and has 31 death per million. NY, just the definite covid deaths, has 989 deaths per million. The place with 30 times more deaths is obviously where you should validate these extreme results.

      • Joseph:

        I appreciate your first remark, “I would love the new studies to be correct.” Part of the problem in these discussions is that they can be come politicized, not just in terms of regular politics but in the way that people take sides and become invested in one scientific position or another. I think it’s important in these discussions for us to be able to make criticisms without taking sides regarding what might be learned in the future.

        Regarding your second point: Yes, I too am skeptical that the rate of exposure in NYC is 100% or 50% or anything close to that. I guess it all depends on how many people are asymptomatic. That’s one reason I’m annoyed that the Stanford/USC team did not report their data on symptoms.

        • > Part of the problem in these discussions is that they can be come politicized, not just in terms of regular politics but in the way that people take sides and become invested in one scientific position or another. I think it’s important in these discussions for us to be able to make criticisms without taking sides regarding what might be learned in the future.

          Absolutely. But the picture is complicated because the authors have deliberately placed themselves into the political context before a thorough vetting of their work and before even conducting a fully comprehensive analysis (such as examining variables that affect response rates).

          There is a tricky need to balance time pressures to get this information out, but is hard to have a non-politicized discussion when the authors have injected themselves into the political frame.

          Maybe there are good reasons for them to do so – but this isn’t simply a problem with the discussions.

        • “But the picture is complicated because the authors have deliberately placed themselves into the political context before a thorough vetting of their work”

          Three of the authors (at least?) have placed themselves into the political context before even doing the study …

      • I would not characterize a nation-wide average of 25% infected as “stunning”. Virtually every country in the world will be at or beyond that range eventually. No reason it can’t have happened somewhere like Spain, Italy or NYC.

        The 100%+ rate estimate for NYC is obviously wrong. But a 25% rate is entirely plausible.

        • 25% would be stunning given that both the Stanford and the LA county data show that prevalence in those two areas is guaranteed to be less than about 5-6%

          That’s about all they say, but they do provide that upper bound estimate.

        • Why would an infection rate that’s a factor of 4 to 5 higher be stunning? At the unchecked infection rate, that could be as little as a week difference in first infection date.

        • 4-5 higher than the maximum possible rate in LA. The real rate in LA is probably quite a bit below the estimate here because realistically it’s known that false positives will be higher than 0% as apparently assumed in this study.

          there are several issues: 1) cross reactivity with other coronaviruses, and 2) large scale tests of this *type* of antibody test showing more like several percent false positives, up to 5% is typical.

          the 100% specificity is based on *thirty* control tests… so with 5% false positive rate there’s a 21% chance you’d get 30 negatives in 30 controls.

          We already know from confirmed cases, that LA county has a prevalence of more than 13823/9.8M ~ .0014 and from this kind of study it’s clear it has less than say 0.06

          between those two it’s hard to say much. So our estimate for LA county should be something like 2% perhaps.

          Now, 25/2 = 12x as high.

          Let’s look at confirmed cases themselves…

          Googling I see they’re at 141k cases in NYC this morning. Population of NYC is 8.2M, so that’s a confirmed rate of: 141/8200 = 17%

          So I guess it isn’t what you’d call stunning, unless like me you’d been not following NYC too carefully. I’m west coast based, and I knew NYC was bad and didn’t follow their numbers too closely.

          So, yeah. Maybe NYC has 25% infections.

        • I don’t doubt we’re heavily undercounting the number of actual cases. A factor of 15 (25%/1.75%) is an awful lot, yes, but I wouldn’t be stunned — it’s in the range of what I’ve had in my head for a while.

          Assuming about 17,000 deaths from current infections in NYC, that’s an IFR of 0.85%. All those numbers feel reasonable to me, based on everything else I’ve looked at.

        • Once they got testing ramped up, my prior was that people with symptoms were probably being undercounted by a factor of 2 or 4 but not 10, with 50% asymptomatic that’d be 4 to 8 x undercount. Not 15… but I’m ok with the idea that it could be 25% in NYC, just a little rattled by doing the math honestly. I still think it’s more likely to be 5-10% than 25% but 25% is not out of the question.

          I still think it’s insane that we never got community survey type PCR tests. The false positive rate there is low, and VERY good prevalence estimates could have been done using less than 1/10 of the test capacity for 1 week.

        • >>> Spain and Italy are around 0.04% for the entire country, meaning an infection rate of about 25% for the entire country, a rate of infection that is stunning.

          >> I would not characterize a nation-wide average of 25% infected as “stunning”. […] No reason it can’t have happened somewhere like Spain, Italy or NYC.

          > 25% would be stunning given that both the Stanford and the LA county data show that prevalence in those two areas is guaranteed to be less than about 5-6%

          Santa Clara and Los Angeles are quite far from Italy and Spain. Why would that matter? :-)

        • I was thinking NYC specifically, but see above. I now agree that 17% is the lower bound for NYC since they’ve confirmed with PCR at least that many, so 25% is definitely plausible given not all cases are ascertained.

        • 141/8200 = 1.7%

          And I don’t see how 5% in LA may pose a problem for 25% in NYC. Deaths are 61 per million in one place and 1700 per million in the other.

        • Also, we know that death rates shoot up when hospitals are overtaxed. Here in LA we’ve never even filled ICUs as I understand. So we expect higher, though maybe not ~ 20x as high death rates in NY.

          It’s just stunning to me the idea that 25% of NYC could have gotten the virus already. I think that’s more a statement about my psychological prior, which is that symptomatic case ascertainment is probably off by a factor of 2, or maybe 4 or something but not more. So that’d take the 1.7*4 = 6.8% and then with say 50% asymptomatic, we multiply by 2 = 13.6% would be kind of an upper end of what I expect.

          it’s just a personal expectation at this point.

          Could it be 25%? Yes I suppose it could, but given that you corrected my calculator error, I now think that’s pretty far into the tail.

        • I haven’t seen anything supporting the idea that NYC has had excess Covid deaths due overtaxed hospitals. What I have seen is anecdotes from doctors that 80% of Covid patients who undergo invasive intubation end up dying anyway (vs 50% from other respiratory diseases). If anything, that suggests to me that the number of deaths is (sadly) not particularly sensitive to availability of treatment.

        • I still can’t follow your logic: “the Stanford and the LA county data show that prevalence in those two areas is guaranteed to be less than about 5-6% => 25% in NYC would be stunning”.

          You later say that “the estimate for LA county should be something like 2% perhaps.” Then your argument becomes: “prevalence in LA may be 2% => 25% in NYC would be stunning”.

          Why would be 12.5x prevalence be so stunning given that reported cases are 12.5x and reported deaths are 29x?

          (I personally think it may be lower than 25%, but I will be more surprised if it happens to be 10%)

        • I actually think it IS sensitive to treatment, and the deaths in NYC were due to the wrong treatment.

          In LA they are putting people on oxygen masks, and placing them in a prone position, which opens up more lung capacity supposedly. They’re doing this earlier, and getting much lower death rates according to what I’ve heard from someone who has connections to Cedars Sinai. It seems likely that NYC treatment protocols killed a lot of patients.

        • Daniel said,

          “I actually think it IS sensitive to treatment, and the deaths in NYC were due to the wrong treatment.

          In LA they are putting people on oxygen masks, and placing them in a prone position, which opens up more lung capacity supposedly. They’re doing this earlier, and getting much lower death rates according to what I’ve heard from someone who has connections to Cedars Sinai. It seems likely that NYC treatment protocols killed a lot of patients.”

          Sadly, I’m inclined to agree that the ventilators may cause more harm than good. Using ventilators for coronavirus seems like an instance of “That’s the way we’ve always done it,” based on superficial symptoms rather than considering the details of the new disease.

        • Anonymous…

          look above, where I agree that it’s not stunning if you’ve been following NY closely, which I haven’t been. My main interest has been national and international level and local to my state (CA).

        • Daniel, the anonymous above was me (I forgot to sign and the comment was put in moderation).

          I imagined that you followed the evolution in NY with more interest, given how confident you were about the reason why the peak was lower in LA.

          It’s good to we aware of what has been happening this month in NYC, like it was a good idea to follow what was happening in Lombardy last month, or in Wuhan two months ago.

      • New York is looking at a population fatality rate of around 0.16% or so. So that implies an infection rate of ~100% using the actual logic in the papers (they back calculate a IFR based on the death in the county and their prevalence rate).

        Well about 3 weeks ago ~15% of the NYPD were out sick: https://www.cnbc.com/2020/04/01/more-than-1000-new-york-city-police-officers-are-infected-with-coronavirus.html

        Once again, around 15% of pregnant women tested positive:
        https://www.nejm.org/doi/full/10.1056/NEJMc2009316

        And keep in mind those tests are not antibody tests, they only (supposedly) measure current infection. So 15% at once is huge.

        • However, around 15% of women testing positive means that it’s important to monitor the fetuses they are carrying and the children they give birth to. We may, in a few months, be talking about something like “prenatal corona virus syndrome” – or there might be something that is not detectable in the prenatally exposed children for a couple of years or more.

        • A rate of “out sick” is hard to know what to make of it. But when I saw that 15% rate among women coming in for having babies, that struck me as something I wouldn’t expect to be much higher than the actual population rate. Because I can’t think of reason pregnant women would be out there getting more exposed than the general population, if anything perhaps the opposite.

        • They are late term pregnant women-who are making hospital visits/visits to OBGYN visits very often-and as we know hospitals and doctors offices probably were already hot infection spots in late March in NYC. If one is to guess-there have probably been 18K NYC deaths from COVID, meaning that, as others have said, by the numbers of the Stanford study 100% of the city has had the virus. If, instead, it is “only” 20-25%-that is a huge difference in lethality.

        • A factor of 4 a pretty big difference in lethality, but not enormous.

          For me, take the SC County estimate (the high one, 0.2%) and adjust for differing population characteristics (% elderly and % obese) and you can easily get that up to 0.3%. Double that for the various problems in their methodology (particularly self-selection) and you’re at 0.6%.

          The actual number of deaths to date in NYC is about 14,000. I’m going to rough out that there are about 3,000 people intubated in NYC, let’s add them in as deaths. So 17,000 deaths, at 0.6% implies 2.8m infections or 35% of the NYC population has caught it. Seems not unreasonable.

          Without my factor of 2 adjusting their death rate, we’d be at 70% which of course is not impossible, but seems unlikely.

          Not sure where I’m really going with this, other than I think I’m going to put 0.6% as the NYC fatality rate in my head. Not that far off from Imperial College London which assumed 0.83% would die with 81% of the population infected if zero mitigation measures were taken. Presumably the measures we’re taking are selectively shielding the most likely to die at least a little bit, and NYC is a touch younger and less obese than rest of country so 0.6% in NYC wouldn’t be misaligned with their number. Would also mean we could potentially get to herd to immunity with just another round of unchecked infection, that if we’re lucky wouldn’t be too much worse than this one. I’m not going to make that call though — I could easily be off by a factor of two and an unchecked round of infection would mean way way worse than what we are currently coming out of. But it does make me want to try Imperial College London’s idea of alternating suppression and unrestriction, keeping a close watch on the ICU case count when you’re in the unrestricted time.

        • There are thousands of uncounted dead in NYC-even with the recent adjustments-so my 18K was a very modest estimate of where we stand. We would need to get something like 80% for safe immunity (if lasting immunity is even conferred-which is not clear). So being off by factor of 4 means something like 50-75K dead with no mitigation, right? Sounds like a lot to me. Definitely a policy-making difference in my mind.

        • The issue I have in mind is that pregnant women that don’t suspect they have covid would be avoiding hospital births at this time. The report also observes that these women typically have multiple interactions with the healthcare system before birth, which could I think expose them to catching the virus in and around hospitals.

        • I don’t know what to make of the police data, but if the maternity ward data is representative then it means that ~14% of New York was infected during the end of March into the beginning of April. The mean time from infection to death is ~3 weeks, so it’s not unreasonable to count deaths from April 1 to the present as resulting from that rate of infection.

          In an NYC population of 8.5M, a 1% Infection Fatality Rate and a 14% prevalence would mean ~12,000 deaths, and an 0.5% IFR would mean ~6k deaths. Since April 1 there have been something like ~8500 deaths in NYC (https://www1.nyc.gov/site/doh/covid/covid-19-data-archive.page).

          There’s an awful lot of uncertainty in a calculation like that but the maternity ward data is consistent with an IFR that’s closer to 1% than it is to 0.1%.

        • You may be “zeroing in” on 50% asymptomatic but I’ve seen nothing, ever, to convince me that anywhere remotely near 50% of infected people get sick. The evidence is so scattered and fraught that people can “zero in” on ‘most any number they like.

          For me it seems to suggest about 80-90% asymptomatic, which has the additional advantage of helping explain why the darned thing is so hard to stop from spreading!

        • In Vo (Veneto) they tested every one and found 81 positives. 57% were symptomatic, defined as cough or fever. It should be noted that there are also cases without cough and fever, so some of the “asymptomatic” 43% could have had in fact other symptoms.

          What is the evidence that suggest to you that about 80-90% are asymptomatic?

    • He made a mistake, alltest and biotest are two different companies. Alltest kits were tested by a lab in Oxford and shown to be unreliable, so the UK government is sitting on a large amount of them unused. The biotest kits are now banned from export out of China, because the Chinese government claims they are untested/unreliable.

  9. The Premier Biotech LFIA test separates out IgM and IgG bindings on the test strip. It would be useful to see the data for each result reported separately (ideally read by equipment which reported optical densities). That would allow inferring how long ago infections took place, helping to adjust infection rate models.

    https://academic.oup.com/cid/article/doi/10.1093/cid/ciaa310/5810754
    https://premierbiotech.com/innovation/covid-19/

  10. I’m sorry if this has already has been asked but if the specificity is crucial to getting accurate numbers with such a low percentage, why didn’t they try to use a larger pool of COVID negative participants for their reference? It seems like 30 people is woefully too small if you want to verify the specificity is near 99.5 to 100 percent as they are claiming.

    • What they should do is to make the test sample larger so that there are more potential people with antibodies. I’m just doubtful that asserting a test have 100% specificity is a good strategy. Nothing is for sure. Based on the Tom Lumley article that Andrew linked to, antibody tests are actually less accurate than PCR diagnostic tests. That’s because each person’s antibodies are “slightly different”. So the test must be able to ignore diversity between people but pick up the diversity between antigens. That doesn’t sound like something one can be 100% sure about.

  11. FWIW, the testing was apparently with a lateral flow immunoassay (LFIA) device. MedRxiv paper from the National COVID Testing Scientific Advisory Panel: https://www.medrxiv.org/content/10.1101/2020.04.15.20066407v1.full.pdf
    “Within the limits of the study size, the performance of most LFIA devices was similar…The LFIA devices achieved sensitivity ranging from 55% (95%CI 36-72%) to 70% (51-84%) and specificity from 95% (95%CI 86-99%) to 100% (94-100%). There was no evidence of differences between the devices in sensitivity (p≥0.015, cf. BenjaminiHochberg p=0.0014 threshold) or specificity (p≥0.19 for all devices with at least one false positive test). The main study limitation is that numbers tested were too small to provide tight confidence intervals around performance estimates for any specific LFIA device. Expanding testing across diverse populations would increase certainty, but given the broadly comparable performance of different assays, the cost and manpower to test large numbers may not be justifiable. Demonstrating high specificity is particularly challenging; for example, if the true underlying value was 98%, 1000 negative controls would be required to estimate the specificity of an assay to +/-1% with approximately 90% power. Full assessment should also include a range of geographical locations and ethnic groups, children, and those with immunological disease including autoimmune conditions and immunosuppression.”

    I would expect that testing of a range of groups would be especially important given that there are a large number of different coronaviruses circulating in human populations (potentially causing serological false positives, as was found with SARS given antibody cross-reactivity), and I would expect exposure to different coronaviruses to vary widely across different human populations.

  12. How much could a specificity of a test be population dependent due to false positives from antibodies to other milder coronaviruses which may not be uniformly distributed in the population?

    I also find it interesting the authors make no attempt to address the strong age and health status dependence of any fatality estimate nor the possibility that the lack of testing led to an undercount of COVID deaths.

    • Potentially an enormous problem. Based on what is known with SARS-CoV-1, there may be some cross-immunity between the established human coronaviruses and SARS-CoV-2.

      https://science.sciencemag.org/content/early/2020/04/14/science.abb5793

      HCoV-OC43 and HCoV-HKU1 are endemic, widespread and seasonal with ~ annual immunity so the potential to find antibodies of them in any sampling conducted in April is high. Another reason full confidence in serology tests you haven’t confirmed don’t do this is not warranted.

      The general problem I have with the Stanford group (both studies are tied to them with author overlap, same test and similar methods) ia that they don’t seem to be considering there is a possibility they are wrong, even though they are trying to do something that is both novel and VERY statistically fraught.

    • Jon said, “How much could a specificity of a test be population dependent due to false positives from antibodies to other milder coronaviruses which may not be uniformly distributed in the population?”

      My understanding is that in general, sensitivity and specificity of a test can be highly population dependent, for a variety of reasons.

      • Somewhat related –

        These studies that extrapolate from data from particular locations to estimate national or global fatality rates rest on an assumption that there is no there are no different strains and/or thee is no significant difference in fatality among different strains.

        Maybe that’s viable. I don’t have the technical k knowledge to know. But my guess is that these researchwrs are basically ignoring a huge uncertainty.

        • “my guess is that these researchers are basically ignoring a huge uncertainty.”

          Sounds plausible to me.

        • > Sounds plausible to me.

          That there are no significant differences or that they’re ignoring an uncertainty? I assume the latter?

  13. Another interesting thing:

    https://twitter.com/teh_galen/status/1252460309173940224

    Quote from the LA Times story:
    “Instead, she owes her test to an eagle-eyed friend, who showed her an online survey from the county health department and USC. The survey was seeking people to participate in a randomized trial to collect blood samples to test for antibodies.

    Since she had exposure to two people who had later been diagnosed with COVID-19, but had no sign of symptoms herself, her friend said she may be a good fit for the study.”

    This woman tested negative.

  14. From the LA County study:

    > Quotas for enrollment in the study for population sub groups were set based on age, gender, race, and ethnicity distribution of Los Angeles County residents.

    Strange that they didn’t control for SES. While I suppose that SES might not be a predictor for outcomes of antibody testing, if they have a sample skewed for SES, it would bring into question their extrapolations w/r/t mortality rates:

    As they write:

    > Quantifying the extent of infection is crucial for estimating the infected fatality rate of COVID-19.2-4

    Once again, it’s one thing to calculate infection rates but just leaping from infection rates to inferring mortality rates seems questionable to me. But they’re the pros and I’m just an Internet schlub, so maybe someone can explain to me why what they’re doing is justifiable?

    As for their discussion of limitations:

    > On the other hand, our results would be biased downward if those who had symptoms consistent with
    COVID-19 islolated themselves and did not participate.

    This seems rather like a stretch to me. I would guess that if people are symptomatic, they’d be *more* likely to want to get a test!

    I am curious why they haven’t done more analysis to investigate what they factors were which might help to explain who did and didn’t choose to participate. If you’re going to go on national TV to talk about these studies with the specific intent of influencing policies that have a massive impact on people’s lives, then I think there is a basic responsibility to do a more comprehensive analysis.

    • The group who does a comprehensive and thoroughly vetted analysis will definitely NOT be getting any air time on national TV. That takes an order of magnitude longer to produce than the “model hot takes” approach. By the time the careful group is on their second draft being circulated to reviewers the hot takes group has already been on TV half a dozen times.

      • Yah. Unfortunately. Some people might blame the media for that. I don’t.

        I suppose at some level there’s a legitimate position that there’s a balancing act..there are reasons to get this kind of analysis out there as quickly as possible.

        In particular, while I wouldn’t have any problem with them going out and discussing their findings about infection rates in particular locations – for them to jump to national extrapolations of mortality rates doesn’t seem justifiable to me.

        Andrew didn’t agree with the characterization of “supremely” irresponsible maybe “highly” irresponsible would be more appropriate.

        • Deciding whether to blame the media or the media whores for this issue is like trying to decide whether to blame the politician or the voters who voted him into office. It’s a distinction without a difference.

          Or put another way, it is what it is.

    • Furthermore, they haven’t considered that there could be uncounted deaths from pneumonia, kidney failure, and heart attacks that were caused by the COVID virus, but no test was ever run. NYC revised its death toll to include people who died with evidence of COVID, but no test was run.

      One observation on the uncertainty in the numerator: A quick search will find an article where CDC researchers estimated the US death toll from the 2019 H1N1 epidemic as being 12,500 with an uncertainty range of 9,000 – 18,000 ( https://www.cdc.gov/h1n1flu/estimates_2009_h1n1.htm ) . The estimate was made in 2011.

      • I watched a video interview of one of the authors of the Santa Clara study, Ioannidis.

        He spoke of the possibility of overcount of deaths due to the “died with” verses “died of” Covid classification, but neglected to discuss the uncertainty in the other direction as you mention.

        Same thing in that he explicitly discussed the policy implications of death rates but didn’t explicitly discuss the policy implications of morbidity rates.

        • I am happy to finally see this brought up here. The death counts are also wrong just like the case counts. It has now been widely acknowledged that most countries only count deaths in hospitals with a positive Covid test. Word has finally leaked that lots of people have died in care homes and at home (a passive “herd immunity” type tactic). The whole movement to slow the influx of patients to protect hospital capacity has the obvious result of some people not getting the right care. Those in care homes are probably “sacrificed” since they are likely to have died of something else soon. Also, the vital statistics are starting to come out which makes it impossible for countries to hide those deaths. My blog post here links to an Economist tracker that shows that in many countries, deaths are undercounted by 50-80%. That’s a huge degree of undercounting. There are indeed some over-counting because people in care homes who have no access to testing can’t tell for sure the death is due to Covid-19. However, the excess mortality is too much to explain away.

        • > I am happy to finally see this brought up here.

          If “here” is the blog we have been discussing that for weeks.

        • I mean brought up in this thread! If you scroll up, there were comments about how updating the prevalence changes the IFR using the official death counts without adjustment.

      • We studied precisely this effect for Italy in https://www.medrxiv.org/content/10.1101/2020.04.15.20067074v2 using raw mortality data from any cause and we see a large excess in the data of March 2020 against a counterfactual prediction (we use Gaussian process conditional mean method and Synthetic control method). We see no excess against official COVID-19 below age of 70, but a large excess above 80: people were dying at home and did not get tested. Our total number of deaths in Italy is more than double the official number.

        So this contradicts Ioannidis claims that official numbers are inflated, it is the opposite, they are underestimated, at least in Italy.

  15. Controlling of geography and pre-setting quota are essential practices but would not be sufficient to adjust selection bias. The weighting makes little difference (unweighted 5.13% vs. weighted 5.23%). Using market research database for recruiting is ok, as it is reasonable to assume no credible bias regarding COVID-19 infection. One key information can be considered is the survey screening and reporting of that part of the data. One can ask recent health, potential exposure, believe of potential infection, desire to get free test etc. before qualify a respondent. That part of the data should also be revealing and used for adjustment, likely downward of the current estimate.

    • I think they should just make super clear that **study participants do not get told what their test results were**. I don’t know the specific bioethics here, but that’s what I would push for.

  16. Wouldnt a better design for these studies, have all positive test results followed up by a secondary non-antibody based test. That seems like a low burden given the low prevalence rates and would obviate our concerns about specificity.

      • The LFA can be confirmed by a more expensive test, such as a neutralization test. Such confirmation might be the most cost-effective way to tighten the CIs, even though errors may not actually be independent.

        • As far as I know, no.

          Source: Podcasts with Christian Drosten and Hendrick Streeck.

          But I might have misunderstood.

        • So, when I look at viral neutralization tests, I understand it to mean they take the blood, expose viruses to it, and then try to use the viruses to infect cells in a dish.

          If there is cross reaction of antibodies to other coronaviruses with COVID-19 viruses then in fact you will neutralize some of the COVID viruses, and could get false positives,it would depend on various parameters of how you define the test.

          It may in fact be that asymptomatic people are largely people who have had colds caused by human coronaviruses and have antibodies to that… If that’s true, it could be a huge huge leap forward in the epidemic… if we can identify a human coronavirus to which people produce antibodies that are effective, we could simply infect people with a virus that causes a 10 day minor upper respiratory infection and produce herd immunity… I hope some people are studying this question very intensely.

    • In the British survey of lateral flow antibody assays they used an ELISA test as validation.
      The problem is, from the photos with the press release I got the impression that they took just enough blood for this one cartridge and perform the test right there in the drive-through. They might not have any logistics in place for storing other samples to take back to the lab.

  17. In a new paper https://www.medrxiv.org/content/10.1101/2020.04.15.20067074v2 we use mortality data from Italy to estimate IFR. In Bergamo province 0.56% of total population died, so that is a lower limit on IFR. We argue about 1/3 of total population mortality has been missed by official numbers, because they were not tested. We improve a bit on that with Lombardy data to get lower bound on IFR of 0.84%. We predict NYC and Santa Clara lower bound of 0.5%, because their populations are younger. The mean prediction calibrated on Diamond Princess data is only 20% higher than the lower bound, but 95% upper limit extends to 1.6% for Lombardy.

    So if one takes 0.5% IFR and a mortality twice the current number (83) in Santa Clara one gets infection rate of 1.7%, which is above the crude number they measured in Stanford study. Doubling the current mortality seems reasonable because mortality is still increasing rapidly (it was 66 on friday, 83 today) and because some may have been missed (in NYC they started accounting for that).

    One should not assume that infection rate of Santa Clara equals LA: we see higher number of deaths per population in LA (617 out of 8M), so it is reasonable that infection rate is higher, and if we assume the same scaling as above one gets 2.5%. So numbers of order a few percent are a reasonable expectation based on the current estimates of IFR and current mortalities. What is unreasonable is to claim IFR 0.1-0.2%, this is violated by NYC, Bergamo, Lombardy, Madrid…

    • I found your paper today and posted a link to it a couple of hours ago in the comments to the next entry of the blog. It’s really interesting, I had done some superficial analysis of excess mortality using the Istat data but your detailed analysis, and the integration of info from different sources, is great. And it is also great that the Italian Statistical Office has made the data available.

    • Thanks Professor Seljak. Very interesting study.

      My caution would be that it may be too reliant on assuming equivalent populations within age bands. My impression from following reported deaths is that death rates are highly dependent on both age and a few comorbidities such as diabetes. Given the much higher rates of obesity in US than Italy, I think your lower bounds for NY death rates may unfortunately be inappropriately low.

      Also, the estimated death count for Santa Clara that you’ve used when estimated IFR from the Stanford study seems a touch high. I don’t have a good sense for how long it takes for the infected to eventually pass away, but doubling the death count from April 19 when comparing to samples taken on April 3 & 4, when the doubling time implied from your death counts is 9 or days, seems like you’re likely pulling in too many deaths. Ultimately I think it’s that their sampling techniques are overstating prevalence, not that they’re underestimating deaths.

      • You are making an excellent observation, but one we actually did try to account for: our proposed model is NOT that age dependent IFR is location independent, but that it tracks the overall Yearly Mortality Rate (YMR). This model gave us the result that 26% of all COVID-19 deaths are below age of 65 in NYC (data say 23%), in a stark contrast to Lombary where it is below 10%. So it is not just that there is a difference in age distribution that we account for but also that younger people are dying more relative to older people in NYC (when compared to Lombardia) as tracked by the age dependence of YMR. Note that these “younger people” are probably not that young, but they are below 65.

        But when it comes to overall YMR the numbers are 0.98% for Lombardy and 0.62% for NYC, this is not controversial, even if it goes against your intuition that people should be dying more in NYC because of co-morbidities. I think this number has a lot to do with birth rates, immigration etc. which have nothing to do with co-morbidities.

        I certainly agree with your comment on Santa Clara, but it is an interesting observation that in Stanford paper they assume 100 total deaths at the end of pandemic and we are already at 83, with number still increasing rapidly. Note that I was not assuming official numbers will go to 160 but I also include
        a 30% underestimate, which is a modest correction that NYC already applied. The smallest correction factor in our Italy analysis was 30% (Bergamo and Lombardy), every other region had a lot larger corrections in terms of estimated total fatality vs official fatality. So I don’t see 160 total fatality number as so unreasonable for Santa Clara for infection rate defined as of early april, but I agree that even 1.5% may be an overestimate (and 2.5-4.2% even more so).

        • Ah, thanks for that. Re-reading your study’s discussion, I see wasn’t paying close enough attention and assumed you were doing something different from what you actually did to get the NY lower bound estimate.
          I see now how my concern about a focus on age was ill-posed. It does feel a bit odd to assume that Covid would scale ALL causes of death equally (e.g. a place with an outsized number of car accident deaths isn’t going to see Covid deaths in relative proportion), but since the vast majority of mortality in western countries comes from chronic disease (including diseases of old age), I guess your study’s approach makes sense.

          And I think I see what you’re saying on the Santa Clara death estimate. Do I understand correctly that I could decompose your study’s estimate of double the 4/19 death count into: a) a 30% increase due to methodological undercounting and b) roughly a 55% increase to account for the fact that all the deaths from cases that were active on the study date have yet to be counted? (130% * 155% ~= 200%). That seems reasonable. Although I think the places that have yet to be overwhelmed may not be undercounting to the extent that NY and Italy have done — my sense from news broadcasts is that a number of the NYC deaths at home have occurred because people are scared to go the hospital because of stories of the hospitals being so overwhelmed.

          Last thing – I don’t think the Stanford paper is estimating 100 _total_ SC County deaths at end of pandemic, but rather that the cohort of infections that occurred on/before the 4/3 & 4/4 sample dates will end up resulting in 100 deaths. But perhaps I’m misreading your comment above.

    • What is unreasonable is to claim IFR 0.1-0.2%, this is violated by NYC, Bergamo, Lombardy, Madrid…

      IFR is also determined by treatment, etc. It is not a property of the virus:

      What he doesn’t do, he says, is rush to put patients on a ventilator. In fact, only one of his patients so far has been given this form of treatment. It is best avoided for as long as possible, he says, because the machines exert too much pressure on the lungs, and the air supplied is too rich in oxygen. That can lead to patients dying of collapsed lung. He says he was astounded at the extent to which ventilators have been used to treat patients in Italy.

      https://www.spectator.co.uk/article/is-germany-treating-its-coronavirus-patients-differently-

      https://twitter.com/MRamzyDO/status/1252214077725736965

    • At a glance, his math looks correct but overcomplicated. With small prevalence (like this study) and no good way to estimate the “likelihood ratio”, you can get basically get an equally-accurate answer by simply dividing the study’s prevalence value by your estimate of the likelihood ratio.

      Think that Covid-positive were twice as likelihood as Covid-negative to enroll? Then divide the study’s reported by prevalence by two (since the authors made no adjustment for self-selection). Think Covid-positive folks were 10 times as likely? Divide reported prevalence by ten.

      Nothing magic there. The hard part is coming up with an estimate of the likelihood ratio, and the link offers no guidance on that. For reasons known only to me, I pick 3 for the SC study and 2 for the LA County study. Unfortunately, there’s no good reason to think I’m right.

    • Thanks for pointing out that article. It includes a first-hand account of what some might think of as an abstract issue: “I live in Santa Clara County, and I was aware that the study was taking place. Many of my family’s friends were also aware that it was taking place, and we know of at least one person who tried very hard (unsuccessfully) to find the ad so that she could participate, because as she said, she had been sick and wanted to know if she had had COVID-19. Hence, based on my own experiences, my personal guess is L=5. But there is nothing magic about my guess, yours may differ.”

  18. Thx. Yes, I thought he rather picked his “5” out of a hat, but that the basic criticism – that the degree of potential influence of self-selection bias was not reasonably addressed with in the Santa Clara methodology – was compelling.

      • I think it depends on the participation rate. If participation rate is something like 50% then okay. If the marketting firm just gave them a million webscraped emails and they spammed those and 0.1% replied then we’re back to the original problem. It’s very frustrating that the authors don’t provide important details like this that we know they have.

      • Why so? How this market research firm formed this database is a black box and, having visited their website, it doesn’t appear to me they know what the word representative means. Would you trust using data from such a databases as a researcher? I wouldn’t

        • Maybe you’re right, I don’t know anything about this market research firm, but I know that there were MANY problems with the FB data. Including that FB basically decides who to show the ad to by looking at people who know people who clicked the ad etc. So it’s like a concentrator/attractor for people who are super interested in getting tested.

        • No disagreement about facebook. Nobody should be using it to collect samples for his/her studies. But I don’t also think anybody should trust sales managers of market research firms when they claim their proprietary database is representative of the county population.

  19. The Dutch blood-bank tested about 7000 samples from donated blood. Unfortunately no article is yet available. Only media reports of a first estimate at 3%.

    They also arrive at approximately 3% antibody prevalence from the period of first 4 weeks of outbreak. They state specifically that there is about a 10% false positive rate on their test which they – i must assume – (attempt) to correct for.

    I suppose the main benefit is that they will repeat this measurement every 4 weeks from a steady flow of blood donors. And simultaneously a randomised measurement is ongoing which should hopefully give some insight into the selection-biases.

  20. I’m not sure if anyone pointed this out specifically, but the cost of being wrong in studies of this magnitude are Incredibly High. I have seen many Academics being overly charitable to outright downplaying the impact of these results on the public with statements such as “it was a good try”, and “I appreciate this study”, while being extremely critical of the study methodology.

    Do these researchers deserve this level of praise?? (Eran Bendavid, John Ioannidis, and Jay Bhattacharya). They are actively promoting their flawed methodology non-peer reviewed study findings directly to the Mainstream Media. There are news articles touting these results across the board (left and right wing outlets) from CNN to Fox news. Multiple Interviews of the study Authors extrapolating their findings to country wide statistics that are playing on Prime-time Mainstream television, all the while their data isn’t even released to other scientists to scrutinize.

    The public and our government leaders (cough… Trump), are ill-equipped to even understand any criticisms of these two studies and most have already taken at face value the claims of 50-85x positive rates. We have Shutdown Protests beginning to sweep the country, and these antibody results are being directly used as justification “We shut down out country for the FLU”.

    An IFR of <.1% vs an IFR of 1% is the difference between 3,282,000 Million lives lost. I can’t even think of any other scientific study in History that has the potential for so much harm…

    • Things being used as cover for political machinations are fungible. The politics is going to do what the politics is going to do, it is neither enabled or caused by one or two particular “studies”. You have your causality backwards. If there weren’t this study to cite, they would simply make up non-existent studies if necessary and cite them instead. The politics would not change no matter what some research group publishes or doesn’t publish.

      • I agree with your description of the causal chain, but these authors are directly injecting themselves directly into discussion of policy development, as advocates for specific policy outcomes.

        I would have to beleive that they see themselves as addressing a critical need in a very fine sensitive framework.

        But where I see a big problem is when scientists specifically reinforce uncertainty that support their conclusions and downplay uncertainties that work against their conclusions. When taking on TV about this study, they should highlight the questions such as about impact of self-selection bias in their recruitment methodology.

        We could argue about the level od differential impact their participation in the public sphere has on the formation of public opinion. There’s plenty of evidence that people will filter pretty much whatever experts have to say so as to reinforce their preexisting beliefs.

        But that isn’t an explanation for scientists selectively focusing on uncertainties so as to support their work.

        Please look at Ioannidis talking about uncertainties w/r/t fatality rates caused by the “with COVID 19” categorization vs. the “of COVID 19” designation (a valid issue) – without even mentioning the uncertainties w/r/t fatality rates caused by people dying without having been tested.

        The questions about causality shouldn’t lessen the responsibility for these scientists to be more accountable.

        • “But where I see a big problem is when scientists specifically reinforce uncertainty that support their conclusions and downplay uncertainties that work against their conclusions. When taking on TV about this study, they should highlight the questions such as about impact of self-selection bias in their recruitment methodology.”

          +1

        • Martha:

          I wasn’t so happy about this quote from Ioannidis in the New York Times: “It’s not perfect, but it’s the best science can do.”

          That ain’t cool. They made mistakes in their analysis: that’s understandable given that (a) they were in a hurry, (b) they didn’t have any experts in sample surveys or statistics on their team. Nobody’s perfect. But it’s not “the best science can do.” It’s the best that a bunch of doctors and medical students can do when they don’t have time to talk with statistics experts.

          What’s the point of describing hasty work as “the best science can do”? How hard would it be for him to say, “Yes, we made some mistakes, but what we found is consistent with our existing understanding, and we hope to do better in the future”?

          But as long as news reporters will take statements such as “it’s the best science can do” at face value, I guess some scientists will say this sort of thing.

        • Andrew –

          I would argue that their mistakes were not only statistical in nature.

          I’m not an epidemiologist or even a scientist, and I may well be wrong, but in my view they made fundamental methodological and epidemiological errors, and their treatment of significant uncertainties contrasts with basics tenents of the scientific method.

          It’s all understandable. Scientists are people too, and people aren’t always (or barely ever) objective when they feel they’re advocating on important issues in which they’re heavily invested. It happens. But they should be asked to be accountable.

        • Joshua:

          I disagree. The three biggest concerns were false positives, nonrepresentative sampling, and selection bias. They screwed up on their analyses of all three, but they tried to account for false positives (they just used too crude and approach) and they tried to account for nonrepresentative sampling (but poststratification is hard, and it’s not covered in textbooks). They punted on selection bias, so there’s that. I feel like the biggest malpractices in the paper were: (a) not consulting with sampling or statistics experts, (b) not addressing selection bias (for example, by looking at the responses to questions on symptoms and comorbidity), and (c) overstating their certainty in their claims (which they’re still doing).

          But, still, a big problem is that:

          1. Statistics is hard.

          2. Scientists are taught that statistics is easy.

          3. M.D.’s are treated like gods.

          Put this together, and it’s a recipe for wrong statistical analyses taken too seriously.

          But, that all said, their substantive conclusions could be correct. It’s just hard to tell from the data.

        • Andrew –

          Thanks for the response. I respect your opinion and appreciate you providing a venue for me to express mine.

          As far as I’m concerned, they should have described this as an observational study and said it shoukd not be used directly to infer broader mortality rates.

          If they had done so, then fine, it is what it is.

          But they have done more than that. They went on to engage in a public information campaign to actually encourage the public and lawmakers to use their observational study, with a convenience sampling, as to extrapolate mortality rates very broadly – as basis for making life and death decisions. Literally.

          I’m not condemning them for that. It’s complicated. But I think it deserves explicit criticism.

        • Andrew said, “I wasn’t so happy about this quote from Ioannidis in the New York Times: “It’s not perfect, but it’s the best science can do.” ”

          Yeah, that’s one to wince about. Maybe it’s the best that they could do, but it seems arrogant to exaggerate that to say it’s the best that science can do.

        • And it opens up the potential for their flawed methodology to be viewed as some kind of gold standard and being repeated unquestioningly by others.

      • This is absolutely on point! Politicians are using these scientists as pawns. That’s not to say scientists can gain clout by offering politicians what they need to hear. Or, the scientists who happen to deliver the right goods get featured in these daily briefings. Here in NYC, Cuomo has been jumping at whatever statistic is improving the previous day to point to green shoots for days now; eventually, he will be right, and he will say he’s been right all along. The curious thing about how this will end is that the scientists find themselves in a safe space whether they are right or not. That’s because once the politicians have chosen a path, they will defend that at all costs. (This happens all the time in businesses. Once the execs spent the money, all they care about is to prove that the money is well spent.) Of course, any program evaluation needs the counterfactual so it’s basically impossible to know for sure that the course of action was effective so that helps protect these advisors also.

  21. About all I’ve learned so far is that public health data should be developed more. MA doesn’t report mortality figures on a monthly basis. Heck, their website’s last ‘Death Report’ is 2017. MI does but they showed a decrease over the last 4 years for March – and that was considering fewer accidents/murders, etc. So that’s worth what? NYC provides data about Covid-19 with bands from 0-17, 18-44, etc. Way too big. And that’s not the only problem. NY state’s data sucks. Most state data sucks.

    It’s nice to see all these models built, but we don’t know enough now to make those models more than toys. I can take the daily MA reports, which are good – with one exception – and see the ‘instant’ readings of risks of death and hospitalization by age decade by comparing to confirmed cases, but I don’t know what that means because I have no idea how many actual cases there are. And they report stuff like 98% have pre-existing conditions: is that super important or kind of important? Does it mean we’ve been doing a good job of infecting the vulnerable? That was a partial joke: the state totally failed to quarantine nursing/elder care facilities and that’s where the deaths are pouring out. In fact, they sent in the National Guard to test, which scared off a lot of staff, and then decided to empty some homes to use them as extra hospital space, only to then find the people had become infected! So if we estimate the vulnerable population (and stratify that), do we get a materially different model? Beats me because the numbers aren’t reliable, if they exist. And that exception: they compressed the decades of the 80’s, 90’s and 100’s though that’s where all the action has been. As in 25 centarians had died before they hid that data, but only 1 person under age 30 has died total (and she was ill).

    What about density issues? If you look at MI’s cases, they’re all around Detroit. That’s why out-state is protesting: they believe they can open businesses and distance differently than the metro area. Is that sensible? Can’t tell because the data sucks. Is it that NYC’s hospital system broke while other areas’ did not? Beats me.

    My wife was extremely ill with Covid-19. I seem to have had a very mild case. Is that because I take an ARB? (Does it outcompete the virus for ACE2 receptors? Maybe in a few years, we’ll know that answer.) She developed hypoxia like altitude sickness. And she has astma, develops pneumonias, and is on a chemo drug! It may be her history saved her because the merest hint of opacity on her chest x-ray put her on doxycilin and that stuffed any nascent opportunistic bacterial infection. All I know is we wanted to keep her off a ventilator, so she was watched carefully and made to move around a lot because low oxygen at altitude kills you if you stop. Avoid edema! Huge problem: mental involvement was intense, which you’d expect with low oxygen. I keep reading of people making terrible decisions while ill: they can’t think straight. Friend of mine’s dad passed out, broke his back (but he’s really hard to kill). I keep telling older people you need to be watched because if you get sick you won’t be able to think.

  22. Hi Andrew,

    Just wanted to point you to this study of COVID-Sars2 assays over at Medrxiv (not peer reviewed yet) and they found the Hangzou test to have only 87% specificity. If that is anywhere near accurate the test is not suited to background infection studies and my guess is they’ll have to retract results from both studies.

    https://www.medrxiv.org/content/10.1101/2020.04.09.20056325v1.full.pdf

    Thanks,
    Devin

    (Associate Professor, USC)

    • 87% specificity = 13% false positive rate, yet they only got ~ 4% in this study, so something isn’t quite right.

      Still, the *uncertainty* in the FPR as well as the recruitment, is so high that it’s impossible to understand more than just “there aren’t dramatically more than 6% in LA” out of this study.

  23. Is this ethical: I see a posting of one of the Stanford recruitment ads on Reddit (See thread https://old.reddit.com/r/CoronavirusUS/comments/ftxwl7/treatment_news_5_critically_ill_covid_patients_on/fma36su/ ) .

    Among things it says:
    —test is “FDA approved antibody test for Coronavirus”
    —” In China and U.K. they are asking for proof of immunity before returning to work. If you know any small business owners or employees that have been laid off, let them know– they no longer need to quarantine and can return to work without fear ”

    Is it considered ethical to mislead people to participate in your study? How would any review board allow this? This would seem to me like a no-brainer. The Reddit thread is 19 days old, so it doesn’t look like it was a recent fake to distort the discussion.

  24. Thanks for the fascinating discussion!

    Relevant to this discussion is perhaps the technical briefing today for parliament in the Netherlands. It contains some info on two serology studies.

    https://www.tweedekamer.nl/downloads/document?id=df1fab72-9b13-41d2-9c5e-adf65eedd1b5&title=Presentatie%20de%20heer%20Van%20Dissel%20RIVM.pdf

    For those not fluent in Dutch, some highlights:
    – Pg8: Excess fatality statistics. Some data-lag but clearly more than seasonal flu.
    – Pg 22: Randomly sampled serology test. Apparently 99% spec, 85% sensititity. First 2.000 analysed. Of those: 3,6% positive for covid. With a relevant age pattern.
    – Pg 23: A test of bloodbank donors is quite similar. ~3% positive. The geographic pattern for that study fits with the covid hot-zones in NL.In the hot-zones the fraction of donors which are positive climbs to about 8-9%.

    This would mean roughly half a million infected. There is an official 4.000 counted death toll today. Although it is known to be substantially under reported (only counted when diagnosed, lack of testing). So more likely to be up to double that.

    p.s.
    There is also a lot in there about the extent to which kids are infected and transmit the virus or not. Partial School reopenings are thankfully now planned for early may.

Leave a Reply to Joseph Candeora Cancel reply

Your email address will not be published. Required fields are marked *