“I don’t want ‘crowd peer review’ or whatever you want to call it,” he said. “It’s just too burdensome and I’d rather have a more formal peer review process.”

I understand the above quote completely. Life would be so much simpler if my work was just reviewed by my personal friends and by people whose careers are tied to mine. Sure, they’d point out problems, but they’d do it in a nice way, quietly. They’d understand that any mistakes I made would never have any major impact on our conclusions.

OK, not really. Actually I want my work reviewed by as many people as possibles. Friends and colleagues, yes. But strangers also. The sooner the better.

But I understand that lots of people want the review process restricted. That’s the whole principle of journals like Perspectives on Psychological Science or the Proceedings of the National Academy of Sciences: if you’re well connected, you can publish pretty much whatever you want. And some of these folks get pretty hot under the collar if you dare to question the work they have published and promoted.

It’s a pretty sweet gig. Before publication, you say the work is a fragile butterfly that can’t be exposed to the world or it might get crushed! It’s only safe to be seen by friends peers, who can in secret give it the guild seal of approval. The data need to be kept secret too. Everything: secret secret secret, like the deliberations before choosing the next pope. And, after publication, the paper is Published! so it can’t be questioned—if you find any flaws, it’s your responsibility to Prove that these flaws materially affect the conclusions, and you can’t ever be so mean as to suggest any lack of competence or care on the part of the researcher or the journal. Also, once it’s published, no need to share the data—who are you, annoying second-stringer, to keep bugging us about our data?? We’re important people, and we’re busy on our next project, when we’re not promoting the current work on NPR.

In this case, there’s no PNAS, but there is the willing news media. But not all the media are willing to play the game anymore.

The source of the above quote

It comes from the Buzzfeed article by Stephanie Lee that I mentioned last night.

Lee’s article recounted the controversies involved in with the Santa Clara and Los Angeles coronavirus prevalence studies that we’ve been discussing lately.

My favorite part was this quote from Neeraj Sood, a professor of public policy at the University of Southern California and one of the authors of the two controversial studies:

Sood said he plans to eventually post a paper online, but only once it has been peer-reviewed and approved for publication.

“I don’t want ‘crowd peer review’ or whatever you want to call it,” he said. “It’s just too burdensome and I’d rather have a more formal peer review process.”

When I say this quote is my favorite part of Lee’s article, I don’t mean that I agree with Sood! Rather, it’s a great line because it reveals in distilled form a big problem with modern competitive science.

“It’s the best science can do.”

Before going on, let me emphasize that I’m not trying to paint Sood as a bad guy. I’m using his name because it’s his quote, and he can take responsibility for what he says (as can I for my own public statements), but my point here is that the attitude in his quote is, unfortunately, all too common in science, or in some corners of science. I’m amused because of how it was stated, but really it bothers me, hence this post.

Now for the background.

Sood and about 15 other people did two serious studies in a short time, sampling and covid-testing over 4000 people in two California counties. They’ve released none of their raw data or code. For one of the studies, they released a report with some summary data; for the others, bupkis. They did, however, make a bunch of claims in the news media. For example, Sood’s coauthor John Ioannidis was quoted in the New York Times as saying of their study, “It’s not perfect, but it’s the best science can do.”

That ain’t cool. They made mistakes in their analysis: that’s understandable given that (a) they were in a hurry, (b) they didn’t have any experts in sample surveys or statistics on their team. Nobody’s perfect. But it’s not “the best science can do.” It’s the best that a bunch of doctors and medical students can do when they don’t have time to talk with statistics experts.

What’s the point of describing hasty work as “the best science can do”? How hard would it be for him to say, “Yes, we made some mistakes, but what we found is consistent with our existing understanding, and we hope to do better in the future”?

But as long as news reporters will take statements such as “it’s the best science can do” at face value, I guess some scientists will say this sort of thing.

I have no problem with them going straight to the news media

I have no problem with Ioannidis et al. going to the news media. They have done work that they and many others have good reason to believe is influential. It’s policy relevant, and it’s relevant right now. Go on NPR, go on Fox News, go on Joe Rogan, go on Statistical Modeling, Causal Inference, and Social Science—hit all the major news media. I’m not one of those people who says, “What do we want? Evidence-based science. When do we want it? After peer review.” If you really think it’s important, get it out there right away.

But I am annoyed at them hyping it.

If that Santa Clara study was really “the best science can do,” then what would you call a version of the Santa Clara study that did the statistics right? The really best science can do? The really really best science can do?

It’s like in the Olympics: if the first gymnast to go out on the mat does some great moves but also trips a few times but you give her a 10 out of 10 just because, then what do you do when Simone Biles comes out? Give her a 12?

I’m also annoyed that they didn’t share their data. I can see there might be confidentiality restrictions, but they could do something here. For example, in the Times article, Ioannidis says, “We asked for symptoms recently and in the last few months, and were very careful with our adjustments. We did a very lengthy set of analyses.” But none of that is in their report! He adds, “That data will be published in a forthcoming appendix.” That’s good. But why wait? In the Los Angeles study, they not only didn’t share their data, they didn’t even share their report!

“Crowd peer review” is too “burdensome” and they couldn’t put in the effort to share their data or a report of their methods, but they were able to supply “B-roll and photos from the April 10-11, 2020 antibody testing.” Good they have their priorities in order!

Here’s what I wrote earlier today, in response to a commenter who wrote that those researchers’ “treatment of significant uncertainties contrasts with basics tenets of the scientific method”:

I disagree. The three biggest concerns were false positives, nonrepresentative sampling, and selection bias. They screwed up on their analyses of all three, but they tried to account for false positives (they just used too crude and approach) and they tried to account for nonrepresentative sampling (but poststratification is hard, and it’s not covered in textbooks). They punted on selection bias, so there’s that. I feel like the biggest malpractices in the paper were: (a) not consulting with sampling or statistics experts, (b) not addressing selection bias (for example, by looking at the responses to questions on symptoms and comorbidity), and (c) overstating their certainty in their claims (which they’re still doing).

But, still, a big problem is that:

1. Statistics is hard.

2. Scientists are taught that statistics is easy.

3. M.D.’s are treated like gods.

Put this together, and it’s a recipe for wrong statistical analyses taken too seriously.

But, that all said, their substantive conclusions could be correct. It’s just hard to tell from the data.

Sood’s a professor of public policy, not an M.D., so point #3 does not apply, but I think my other comments hold.

Back to the peer-review thing

OK, now to that quote:

“I don’t want ‘crowd peer review’ or whatever you want to call it,” he said. “It’s just too burdensome and I’d rather have a more formal peer review process.”

What an odd thing to say. “Crowd peer review” isn’t burdensome at all! Just put your data and code up on Github and walk away. You’re done!

Why would you want only three people reviewing your paper (that’s the formal peer review process), if you could get the free services of hundreds of people? This is important stuff; you want to get it right.

As Jane Jacobs says, you want more eyes on the street. Productive science is a bustling city neighborhood, not a sterile gated community.

Conclusion

I’ll end it with this quote of mine from the Buzzfeed article: “The fact that they made mistakes in their data analysis does not mean that their substantive conclusions are wrong (or that they’re right). It just means that there is more uncertainty than was conveyed in the reports and public statements.”

It’s about the science, not about good guys and bad guys. “Crowd peer review” should help. I have no desire for these researchers to be proved right or wrong; I’d just like us all to move forward as fast as possible.

203 thoughts on ““I don’t want ‘crowd peer review’ or whatever you want to call it,” he said. “It’s just too burdensome and I’d rather have a more formal peer review process.”

  1. I could not agree more.

    If they said “these numbers seem a bit higher than we expected but there is a lot of noise in the data — these studies are hard” then I would be celebrating the work. It does have some real contributions, as the plausible direction of the error lets us greatly decrease the likelihood that CA has 20% seroprevalence.

    In fact, if they had just not talked about infection fatality rate then they’d have been a lot better off. It’s the weak IFR calculation (in the discussion!!) that caused the most concern, especially when compared to New York. And IFR can be different in different places and contexts, it is just the strength of the comments. You can’t say “IFR varies” and then say “about on par with the flu” with such confidence, unless the quotes are taken wildly out of context.

    The other interesting thing, on twitter, is how many statisticians have shown corrected confidence intervals that still preserve a lot of the information while more precisely characterizing the sources of variance. I bet one of them would like to be a co-author on an important paper and could do even better with the raw data (and, to be clear, I am not one of those people — the Bayesians did some remarkable things that are out of my reach).

    • > In fact, if they had just not talked about infection fatality rate then they’d have been a lot better off. It’s the weak IFR calculation (in the discussion!!) that caused the most concern,

      Yah. At least for me.

      I frequently see epidemiologists being very careful to make it clear that non-random and non-representative sampling should *not* be use for broad national- or even worse global-level extrapolations…leading to something like conducting interviews where Ioannidis asserted that this virus has a fatality rate more or less equal to the seasonal flu. The fact that they didn’t explicitly caveat with the potential self-selection bias in their study (again, I’m talking about during the publicity campaign) only adds to the problem.

      As I’ve said repeatedly, Ioannidis is a famous epidemiologist and I’m an Internet schlub. My baseline belief is that I must be in error here…but I have yet to see an explanation for *why* it’s OK to use a convenience sampling with significant problems to make extrapolate so broadly on such a critical issue.

      Perhaps it will turn out that Ioannidis’ extrapolation is correct. Perhaps he has more data that he hasn’t put out there – that he’s relying on justify his extrapolations.

      Would that then mean that it’s OK to do what he has done?

      • “Would that then mean that it’s OK to do what he has done?”

        No. John Ioannidis has a checklist for why “most research is false”. Hyping one specific study is dangerous. Here is a totally different quote to the “on par with the flu” one:

        “Our research suggests a higher seroprevalence in Santa Clara county than we expected, especially given the small number of deaths so far. It’s unclear how this fits into higher rates of clinical outcomes in Lombardy and New York City, but we plan to follow up to try and understand how rates differ from place to place. We already see interesting variations in national covid-19 mortality rates between France and Germany, two countries both with high quality health care and a substantial number of infections, and better understanding this could help manage the epidemic”.

        Conveys the questions raised in context and makes it clear that there is a lot of unknowns here. If, later on, the test was shown to be less precise then they imagined then that is one of the reasons that they would be checking.

        • > What is the point of doing a study if you have to add such a caveat?

          This confuses me. I’d ask what is the point of doing such as study if you *don’t* add such a caveat? Without such a caveat, IMO, the study is pretty worthless. The study is useful within a particular range of application, and pretty useless outside that range. That’s why you need to make that range of application explicit.

        • Let me offer an argument ad absurdum – with appropriate caveats about fallacies :-)

          Consider conducting a study with convenience sampling from a nursing home in NY and used it to justify extrapolating fatality rates for Nebraska? Certainly, such a study should be strongly caveated, yes?

          So then the question for me is where do you draw the line?

        • They literally cannot provide the quote you suggested. Bhattacharya and Bendavid in particular have already gone on record with much _higher_ estimates of prevalence than they found in the Santa Clara study. So it was in fact lower than they expected.

        • Joseph C.:

          I think they could turn around and say something like, “Yeah, well we weren’t talking about deaths in nursing homes. That’s a different story.”

        • This time, I’m not talking about death rates. Purely prevalence. In their March 24 WSJ article, they said:

          “The epidemic started in China sometime in November or December. The first confirmed U.S. cases included a person who traveled from Wuhan on Jan. 15, and it is likely that the virus entered before that: Tens of thousands of people traveled from Wuhan to the U.S. in December.

          Existing evidence suggests that the virus is highly transmissible and that the number of infections doubles roughly every three days. An epidemic seed on Jan. 1 implies that by March 9 about six million people in the U.S. would have been infected.”

          So it’s sloppy of me to say that they’d already gone on record with higher prevalence rates than what their study determined for SC County, as six million is only 1.8% of the total US population. But that was an estimate for a point in time 25 days before they collected their data, and 7 days before SC County went on lockdown, with a virus they said has a doubling time of every 3 days. So I don’t see how to get from their prior statements to “2.5% to 4.2% on April 4 is a _higher_ seroprevalence than we expected”.

          If you haven’t read that WSJ article, I’d suggest it. It’s a nightmare.

        • Ioannidis said that forecasts of *the potential* of million deaths in the US without mitigation were “science fiction.”

          (I’ll note that he accepted the rhetorical framework of the question posed, which is thst 2 million deaths were “forecast.” Again, how can that approach to uncertainty be defended as scientific?)

          In other words, not improbable, or very unlikely, but not in any way consistent with reality.

          And at the same time he’s suggesting that we have a very, very high infection rate already.

          I’ll let smart people do the math of a mortality rate of @ .8% from the testing in NY, and consider the implications of that rate absent any chance of undecount and higher mortality rate had hospitals not been overwhelmed under the condition of no mitigation.

      • Howdy,

        John Ioannidis is only one of the authors of the Santa Clara study. I don’t think it is fair to single him out b/c he is famous also and conveyed a couple of dissenting talking points to the IFR being bandied about back in February and March. I think he mentioned at the time that he was concerned over the conflicts of interests that might be influencing the approaches under consideration. That is a fair reminder. It is more difficult to suggest that one should do less than more with respect to COVID19. Overdiagnosis and overtreatment are major concerns to address, more generally as well.

        • Hi Sameera –

          > I don’t think it is fair to single him out b/c he is famous also and conveyed a couple of dissenting talking points to the IFR being bandied about back in February and March.

          Maybe. But I kind of think that because he is famous and in particular, because he is famous for raising *important* questions about research standards, he in particular should be held to a high standard, particularly in such a critical context.

          > It is more difficult to suggest that one should do less than more with respect to COVID19. Overdiagnosis and overtreatment are major concerns to address, more generally as well.

          I agree – and I recognize that the critical nature of the context might compel a scientist to take action she might not do otherwise. I’m not condemning, but I am criticizing.

          Nontheless, I still am asking why it’s OK to extrapolate broadly from non-representative and non-random convenience sampling…

          I will note that above, Joseph offers a quote (I assume from Ioannidis) that seems *entirely* appropriate to me:

          > “Our research suggests a higher seroprevalence in Santa Clara county than we expected, especially given the small number of deaths so far. It’s unclear how this fits into higher rates of clinical outcomes in Lombardy and New York City, but we plan to follow up to try and understand how rates differ from place to place. We already see interesting variations in national covid-19 mortality rates between France and Germany, two countries both with high quality health care and a substantial number of infections, and better understanding this could help manage the epidemic”.

          I think the standard displayed in that quote should be displayed uniformly.

        • I don’t know about everyone else, but I thought Andrew made a great point about author 16 of 17 in an earlier post. You will notice we all got quiet about John Ioannidis until he made some rather strong statements to the NYT about the research project. Those comments are directly from him and the source of the current questions. If he was misquoted then I am happy to withdraw these thought-lets. But it is a key policy issue and so it is fair to ask hard questions if a study makes strong recommendations.

          The quote above was constructed by me, trying to sound like John Ioannidis the time I met him in person, when we were on a (sadly never funded) grant on meta-analysis together. Smart people disagree about many parts of the epidemic. Understanding it is hard.

          But that doesn’t mean we shouldn’t try to. Any global IFR statement needs to consider the places with high death rates already seen; it’s totally possible they vary and it is possible that ending the lockdown in CA could work out super-well. But we need to be honest about the level of evidence here. And I am not bringing up these points — the authors are making them in the media. In https://www.nytimes.com/2020/04/21/health/coronavirus-antibodies-california.html?smid=tw-nytimes&smtyp=cur, the lead author of the LA study said:

          With better estimates of the virus’s prevalence and lethality, it may be easier to reopen society in a manner supported by data, said Neeraj Sood, vice dean of research at the University of Southern California Price School of Public Policy and an author of the Los Angeles County report.

          “We can model the scenarios,” he said. “We should not make decisions just based on I.C.U. mortality.”

          He is correct, but that means that outlying data (like his study) gets subjected to stricter scrutiny than normal as it is a key input into a major policy decision. If the authors wanted to not make this jump then we’d also be having a very technical discussion about variance estimates, but it wouldn’t be in the context of actual policy recommendations and so be a lot lower stakes.

        • I’m not sure what point you’re making re:Ioannidis. His coauthors Bendavid, Bhattacharya and Sood _also_ conveyed dissenting talking points on the IFR in March, in their case in the WSJ in an editorial attributed to Bendavid and Bhattacharya (with contribution from Sood). So I don’t think Ioannidis is really getting singled out here.

    • I really do not get why this is the paper that got such a strong response. In January all these tests came out of nowhere with no discussion of how accurate they were at all. Everyone was pretty much silent on that:

      Preliminary assessment of analytical sensitivity for RdRp assay.
      We tested purified cell culture supernatant containing SARS-CoV strain Frankfurt-1
      virions grown on Vero cells, and quantified by real-time RT-PCR assay as described in
      Drosten et al. (2) using a specific in-vitro transcribed RNA quantification standard. The
      results are shown in Figure 3. All assays are highly sensitive.

      […]

      Specificity testing
      1. Chemical stability
      To exclude non-specific reactivity of oligonucleotides among each other, both assays were
      tested 40 times in parallel with water and no other nucleic acid except the provided
      oligonucleotides. In none of these reactions was any positive signal detected.

      2. Cross-reactivity with other coronaviruses
      Cell culture supernatants containing human coronaviruses (HCoV)-229E, -NL63, -OC43, and
      -HKU1 as well as MERS-CoV were tested in all three assays (Table 2). For the non-
      cultivable HCoV-HKU1, supernatant from human airway culture was used. Virus RNA
      concentration in all samples was determined by specific real-time RT-PCRs and in-vitro
      transcribed RNA standards designed for absolute viral load quantification.

      [Table 2 shows no reactivity in three tests each of 5 different coronavirus-infected culture supernatants]

      3. Tests of human clinical samples previously tested to contain respiratory viruses
      Both assays were applied on human clinical samples from our own diagnostic services,
      previously tested positive for the viruses listed in Table 3. All tests returned negative results.

      [Table 3 shows 75 clinical samples that tested positive for a variety of viruses returned negative results]

      https://www.who.int/docs/default-source/coronaviruse/protocol-v2-1.pdf

      So it looks like these tests were adopted with no data on real world specificity/sensitivity.

      • This test uses a well-established method whose real-world performance is established.
        Its sensitivity and specificity, as far as the lab work is concerned, rest solely on genetic matches and mismatches, and these can be shown by simply running the sequences you have selected for your tests against a database of sequences or potential cross-reactants to get an accurate prediction of the performance.
        You then run a few clinical samples to make sure you didn’t make any mistakes along the way, and off you go.
        This test was available less than a week after the genome of the virus was released, and it’s proven to work ever since: we do not have clinical Covid cases that test negative (if you test in the right place), and a positive test result is a fairly good predictor for getting sick. It works miles better than having nothing, and we had it early. Virologists don’t criticize it because they understand what the method can do, and what the verification that has been done actually means.

        The lateral flow antibody assay is also a well-established method whose performance problems are also established, and genetics don’t allow us to predict what works and what doesn’t. It’s a completely different challenge.
        Look to ELISA tests for reliable antibody testing, but don’t expect to be done with a drive-through weekend of data gathering.

        • For one thing the accuracy depends on swabbing the correct part of the throat. These lab results ignore all sorts of real world sources of error like bad swabs, contamination, coinfection, etc.

        • Yes, but no other test you can use does better in this regard.
          Well, actually, a lung CT with “ground glass shadow” in both lungs is pretty conclusive now for most clinicians, but that wasn’t the case back in January.

        • Yes, but no other test you can use does better in this regard.

          A test that was informed by real world performance data would do better… That is why that is the usual method to come up with a good method.

      • See: https://www.statschat.org.nz/2020/04/19/counting-rare-things-is-hard/

        It isn’t that the tests are bad. They are super useful. It is that it is hard to measure rare rates of disease using any real world test, a problem that haunts this part of epidemiology (cancer screeners have dealt with it for decades). In a higher prevalence population the error rate would be a lot less of an issue. It is just that when the event rate is very rare then the innate properties of this tests can induce very large errors, which could have been handled with the variance estimate but wasn’t. This was then used to estimate a very precise estimate of IFR, which is unlikely to be as accurate as it is precise.

        • a) The primary purpose of the test is diagnostics, not epidemiology. Let the epidemiologists figure out the real-world variance in getting the virus from the patient to the lab.
          b) It’s theoretically 100% specific, no false positives.
          Quoting from your source: “As I said above, the exact value of the false negative rate doesn’t matter that much when you’re estimating population prevalence.”
          There you go.

        • I am not sure what you are trying to say? My critique is the high confidence extrapolation of a low IFR rate, in the range of the seasonal flu. There is a lot of information here, even with the false positives. I am not sure what you are rebutting but my concern was the use of this data to estimate a high confidence IFR with low variance, done directly in the preprint

          What that has to do with population prevalence being informative, if perhaps too precise, is unclear???

          And a test done on a random sample is about epidemiology — if they were diagnosing people as having antibodies using it that would be a different discussion.

        • Anoneuoid was criticising the rt-PCR tests. You seem to have misunderstood that as talking about the antibody test?

  2. “We’re important people, and we’re busy on our next project, when we’re not promoting the current work on NPR.”

    That’s a really good point. People are much happier to spend time promoting their work than debating it.

  3. I wish, along with the quote, you had included some of my thoughts about *why* I felt their actions (not necessarily the study in itself) contrasts with basic tenets of the scientific method (i.e., not appropriately accounting for poor recruitment methodology and more importantly conducting a publicity campaign where they leverage this study to support extrapolating broadly from non-random and non-representative sampling to infer basically national or global fatality rates).

    A thought – I’ve been reading a bit about the advantages of randomly approving study proposals for funding… It seems to me that “crowd peer review” has a kind of random element compared to more traditional peer review. And maybe that “random” element conveys some advantages.

    Maybe part of the way forward would include some formal element of randomizarion in the process of research review. Obviously, that wouldn’t mean just picking a bunch of people out of the phone book – but maybe diminishing the “gatekeeping” element of peer review doesn’t have to be an all or nothing, throw the baby out with the bathwater kind of thing?

  4. Has anyone else seen this? the Chinese cdc checked the test they used and found 4/150 false positives.

    https://imgcdn.mckesson.com/CumulusWeb/Click_and_learn/COVID19_CDC_Evaluation_Report.pdf

    My confidence in the Santa Clara and USC findings keeps converging closer to zero everyday.

    And who needs to release your data for scrutiny when you can just relay your unchecked results directly to everyone in Los Angeles, https://youtu.be/S6mSpIeBS18 , during the daily department of health briefings.

    50x, yes Venice Beach here I come.

    • So if we pool all of the validation test together, assuming equal quality all the way around, then we get 2/401 + 4/150 = 6/551. That moves the false positive rate from 0.5% to 1.1%. That makes the expected number of false positive in 3330 rise from 17 to 37. When the report is based on 50 positives that could easily halve the expected prevalence, all by itself, thus at least doubling the IFR (and making NYC awkward instead of impossible).

      • note the numbers you calculate are correct but the algebra isn’t… correct would be (2+4)/(401+150) = 6/551

        If we assume that we know there’s “around a 95% specificity” for this *type* of test, and therefore use a beta(95,5) prior and add your data…

        The posterior would be beta(646,11) which has 95% interval [0,.062] so the data from all their studies is perfectly consistent with zero true positives

        • Dang it, in my calculation I typed “pbeta” when I meant to type “qbeta”… the 95% false positive rate is:

          [0.0083,.028] with median .016

          so since they had a final estimate of about 3% positive in Santa Clara, it’s totally consistent with the idea that the real positive rate is less than half that (half the positives coming from false and half from true).

        • Sorry about the sloppy algebra. Making a point and shouldn’t have turned it into an equation when it was wrong.

          Yeah, the big issue is the discussion based on some very rough death rate extrapolation:

          “As of April 10, 2020, 50 people have died of COVID-19 in the County, with an average increase of 6% daily in the number of deaths. If our estimates of 48,000-81,000 infections represent the cumulative total on April 1, and we project deaths to April 22 (a 3 week lag from time of infection to death), we estimate about 100 deaths in the county.”

          and an underestimation of variance.

          The findings, themselves, have some real value in understanding plausible infection rates.

        • Looks like 21% positive antibody tests in NYC:

          https://pbs.twimg.com/media/EWTOsGkWoAA-3X-?format=jpg&name=900×900
          https://www.youtube.com/watch?time_continue=12&v=HFY_P6yR0BY&feature=emb_logo

          So 1.7 million cases in NYC when there were 250k positive tests in NY state as of yesterday. NYC accounts for about 50% of the total for the state, so 125k positives in NYC. That would be a 13x under-ascertainment.

          But really we should compare antibodies to positive tests about two weeks ago which was 140k for NYC (assume 70k for NYC). So that would be 24x more cases than detected.

          Of course that is naively assuming the sampling strategy was good, etc.

        • With a larger number like this, I trust the point estimate more. Specificity errors of +/-2% still give broadly similar estimates of prevalence.

          This is also compatible with a rough population IFR in the 0.5 to 1% range (given death rate in NYC to date), which is showing up an awful lot in places with careful testing. Obviously, the age, sex, and health specific estimates are much more valuable. And it will vary for many reasons from place to place, including how many nursing homes got infected.

          But it suggests more mortality than the flu, an older talking point that I wish would be retired. There is a smart conversation to be had about which pieces of the lock-down are most necessary. High quality evidence helps this discussion and there may be ways to improve both disease reduction and the economy, at the same time. But you need good estimates of costs to do a cost-benefit analysis.

        • I think the idea is if you test positive for the antibodies it means you contracted the virus more than 2 weeks ago (it takes that kind of timeline to develop them). On the other hand it takes 5-10 days to develop sufficient symptoms to seek a test. so It might make sense to say a week ago rather than 2 weeks ago.

        • But you still probably also have people testing anti-body positive who are testing positive now and will only become a confirmed case a week from now when their symptoms peak and they head reluctantly to the hospital.

        • Zhou, I don’t think immunologically naive (ie. people who don’t have antibodies at time of exposure) people will test antibody positive until they develop significant antibodies, which will be ~ 15 days after initial infection, but symptoms are occurring around 5-10 days.

          At least that’s typical for viruses in general. There is a definite possibility that people have immunity to SARS-Cov-2 because of cross reaction with antibodies for other coronaviruses, but I don’t know anything about cross-reactivity of the *test*

        • Let’s call it 13x under-ascertainment. That’s especially reasonable considering the rapid rate of rise in NYC and the fact that a considerable false-negative rate is being considered for the PCR test.

          Still, let’s think about given that 21% have had the disease, how fast did it have to grow in order to get there?

          1.7 M cases. Imagine starting from some point in the past where there were 100 cases.

          100*2^(t/n)= 1.7e6 so log(1.7e6/100)=t/n * log(2) or n = log(2)*t / log(1.7e6/100)

          I’m not sure when they did this survey but I’ll say April 15 or so, and let’s put the 100th case back around the time of the first death from Santa Clara, say Feb 6, so that’s 69 days, the doubling time was

          So basically we’re pretty sure the doubling time was 5 days or so, but the truth is *less* than that, because lockdowns were put in place Mar 20. So I’m still going with the idea that the ~ 3 day observed doubling time was fairly accurate.

          The fit to the cases curve was fine as an estimate of growth even though testing was variable. Why? because although there was under-ascertainment, the rate of under-ascertainment stayed more or less constant throughout the period. The logarithmic derivative eliminates any concern about the *overall* level and just looks at how fast things are changing relative to the current level:

          let N(t) = n*exp(r*t)/c with c the ascertainment and n the observed

          then the logarithmic derivative:

          1/N(t) dN/dt = r*(n*exp(r*t)/c) / (n*exp(r*t)/c) = r

          as long as dc/dt ~ 0

        • I think I put a less than sign… sigh…

          the doubling time was log(2)*69/log(1.7e6/100) ~ 5 days

          but really less than 5 days because we didn’t have uncontrolled exponential growth until April 15, as lockdown happened Mar 20.

    • But why do you have so much confidence in “excluded cases”? That is 150 “suspicious patient(s) with the infections contact history” who had samples collected Feb 4-8 that were excluded “according to Diagnosis and Treatment Plan for Pneumonia Caused by [2019 novel coronavirus infection] issued by the National Health Commission of P.R. China.”

      Why couldn’t 4/150 of the excluded cases have developed antibodies at some point?

      • I want less confidence in everything, when the base rate is so low. :-)

        It was an illustration of how small updates to the underlying data make the results shift meaningfully. It is a problem with very rare exposures and these tests will be much less vulnerable to issues of precisely how specific they are as the event rate rises. Probably already good enough for (as an example) New York State.

    • That is kind of the point. Apparently there is an appendix that will be released. But, at this point, we have two studies that are being discussed in the media with major points that are not yet out to be scrutinized. Like the entire LA study, and the careful work in the Santa Clara study on baseline characteristics.

      This is kind of like “trust us” science.

  5. >Just put your data and code up on Github and walk away. You’re done!

    I would be very surprised if the median researcher publishing these kinds of things has code that they can just put on GitHub and it’ll be meaningful and useful. Many researchers don’t have good (or any) format version control practices, aren’t documenting well, and are creating deliverables manually as opposed to within their code, and so there’s no code that they have that they can just post and other people can run to generate their results. This is a problem.

      • I totally agree. Use an open source language, give me a .txt file I can run to install your dependencies in the right versions, write clean documented code that someone can actually tell what it’s doing, generate your tables and charts programatically. Good coding practices are not some kind of optional special thing that it’s acceptable to argue you didn’t need to do because you were in a hurry or you didn’t have the money, they are a basic part of contributing to a body of research literature, and Excel is not an acceptable place for your model to live. I’ve had this fight, I’ve lost this fight, I will keep having and losing this fight.

        • ” they are a basic part of contributing to a body of research literature”

          They’re more than that: they’re critical for generating accurate results. If you’re code isn’t properly documented, you probably don’t know what it’s doing.

        • We are, very gradually, winning this fight. 20 years ago nobody was saying these things. 10 years ago when I wrote this https://www.nature.com/articles/467753a it was a bit of a fringe view. Now it’s fairly mainstream. In another couple of decades we will have won.

          I would emphasize that publishing *whatever you’ve got* is better than nothing. If it’s a shoddy piece of crap, then *everybody needs to know* that it’s a shoddy piece of crap. Everyone writes a shoddy piece of crap sometimes, and they can produce good results (although it’s less likely).

          If you used an Excel spreadsheet, publish the damn spreadsheet. If you spent half an hour in a shell tweaking parameters to your figure-generating tool until the chart looked nice, *publish the transcript of the shell session*. I don’t care. Your code (or spreadsheet, or shell session) is your method. If you don’t publish it, you’re not doing science.

  6. Andrew,

    Thank you for another excellent post!

    I 100% agree with you ire regarding peer review. The goal is accuracy and rigor, not reputation.

    Anecdotally, I have seen and experienced myself instances of lowly PhD students find legitimate errors (e.g., in calculations or analysis) and being “hushed” by their pre-tenure advisors to avoid angering more senior or “celebrity” scientists.

    Now regarding the Stanford COVID-19 Study: We should hold its authors accountable to their errors and to improve their work. But we should not scapegoat this study, either. Many others warrant attention and criticism, too.

    The IHME Model (from U. of Washintong) and SEIR Model (from Imperial College of London) fail to clearly mention limits of their work within the paper and when publicizing it. (Nice paper by Daniel J. Bernstein evaluating the Imperial College London model: ).

    States and scientist continue to push Social Distancing, but where is the rigorous analysis supporting it? Saying “look the curve of cases is going down as we social distance” is trite and trivial, since other factors exclusive to Social Distancing — hygiene, herd-immunity, wearing of masks, weather, natural course of the disease, etc. — could be causing some or all of the affect. How can we be sure we are not fooling ourselves? I am all for continuing Social Distancing if its benefits outweigh its tremendous social, economic, and other costs.

    What about the recent article in Science () claiming that digital Contact Tracing could dramatically aid in monitoring the virus? They fail to mention clearly that their method is highly dependent on having an accurate understanding of how a highly-transmissible pathogen spreads, which is very difficult to determine. For example, there are many nuances to how influenza can spread (an important one being if the virus resides in mucus or not) that are difficult to measure. What about those for SARS-CoV-2?

    What about the purported mortality of COVID-19 relative to say, influenza? In Italy, 96.3% of deaths have had 1-or-more comorbidity; in NYC, 99.2% of deaths have had 1-or-more underlying condition. If you are are a healthy person, your likelihood of dying from COVID-19 is minimal. So is it acceptable for experts to still report crude infection-fatality-rates when SARS-CoV-2 only seems to cause severe outcomes in a specific sub-population?

    Not trying to rant. I just think it is important that we avoid scapegoating one or a few studies.

    • There are “severe outcomes” in other groups of patients. Unexplained strokes reported in people 30-40 years old. Lasting lung damage in previously fit divers who managed their infection at home and were clear of the virus 6 weeks ago (Innsbruck). A pneumonia by itself is a severe experience you wouldn’t wish on anyone even if you caan survive it with the proper care.

      It’s not just the mortality that is the problem, it’s the hospitalisation rate, and that has been well established ever since Italy broke out over a month ago, and mortality data shows conclusively we’re heading for a much higher excess mortality than even a severe flu season already.

      But your were talking about mortality. It is strongly correlated with age; 95% of deaths in Germany so far are of people at least 60 years old. Good luck finding anyone without a comorbidity in that sample! They’re practically a given!
      I agree that a crude IFR is something we should have moved past ever since January when we had the age data from Wuhan, but simplicity prevails.

      • Severe outcomes occur for any disease. The key is their frequency. For example, the very common adenovirus can cause severe outcomes in healthy patients, including pneumonia and myocarditis. Further complicating things are nosocomial infections — what if SARS-CoV-2 ravages their immune system leaving it bare for other opportunity pathogens to invade? (This is especially likely with invasive ventilation.) If these are occurring, I agree would should investigate and address them ASAP; but again, the key in frequency. To date, I have only seen anecdotal and case-by-case evidence, which is not reliable enough to draw conclusions yet.

        And regarding hospitalization-rate, that likely varies with region and country for various reasons. For example, parts of NYC and NJ are over-run, but not others (caution: this is anecdotal evidence, but it makes some qualitative sense). Perhaps a better measure would be hospitalization-rate adjusted for population-density.

        The fact that deaths from COVID-19 correlate strongly with age is precisely why IFR an other measures can be misleading. A pathogen that kills only at-risk patients (like SARS-CoV-2, strep pneumoniae, strep pyogenes, etc.) and only causes mild to minor symptoms for healthy patients is not polio, smallpox, tuberculosis, etc. Still a problem, certainly, but it likely won’t kill millions of healthy children and adults, either.

        Simplicity does prevail. But that does not mean its right or the best path forward.

        • a) straw man. Polio, Smallpox, tuberculosis have lethality >10%, what we had out of Wuhan for this was 3.4%.
          b) “only causes mild symptoms for healthy patients” — I gave you anecdotal evidence why this can’t be true (healthy patients with strokes/lasting lung damage). Many doctors on youtube related this as well. Why do you say this with confidence and expect everyone to accept it?
          The data you base this on is mortality data. A severe case that leaves you with a lasting “precondition” is something else. Where is your data that matches your confidence?

        • (a) My point is the sensationalizing of COVID-19 as a “once in a century” pandemic would, by definition, place it in the category of polio, tuberculosis and smallpox (if you assume “century” means the past 100 years here) since they have each had a major outbreak during the past century. I don’t think its a fair comparison to make, and has strawman elements of its own.

          (b) Claiming that COVID-19 leaves lasting pre-conditions itself needs data. Do you have any supporting this conclusion that isn’t anecdotal?

          And I have seen many of these videos on YouTube and Facebook. If you look closely, many patients in the ware are obese; so they are not healthy. I would not rely on these videos, personally, regardless of what they show. There are too many unknowns about the source, their agenda and/or mental state, etc., to make any conclusion from them.

          Only causes mild symptoms for healthy patients stems from the data in Italy, NYC, and NJ. Most of the positive cases have not seen hospitalization. Take Italy’s data: Only 20% of cases have been severe (18%) or critical (2%). 80% of cases have been mild or less.

          And I don’t expect anyone to except my claims; feedback is welcome.

          My rationale for SARS-CoV-2 likely only causing mild symptoms in healthy patients arises from its underlying biology (at least what we know from the past 4 months). The renin-angiotensin-system (RAS) is key here.

          In healthy people, this system works to increase cardiovascular, renal, and pulmonary function to meet demand. In hypertensive, obese, diabetic, etc., — i.e. “unhealthy” patients — RAS is chronically overactive. Therefore, its opposing physiological pathway — which uses ACE2 as its main affector lower RAS activity — is also over active. As a result, unhealthy people have more basal ACE2 available for SARS-CoV-2 to use at the start of infection and beyond — thereby giving the virus a “head start” against the immune system. Couple this with the known fact that unhealthy people, especially those who are obese/diabetic, have chronic inflammation (being obese, to the immune system, is an infection) and a higher propensity for severe inflammation — I hypothesize that they would have a much more severe outcome.

          In contrast, healthy people should, over time, have minimal basal ACE2 since their RAS is function at healthy levels and needs no major adjusting (it can self adjust to a degree; its opposing axis is more for drastic overshoot).

          Since 80% of cases in Italy were mild (and the same seems consistent in the US and Germany) I would conjecture healthy people likely have minimal short-term and long-term consequences from SARS-CoV-2. But it is just conjecture, not gospel. We need more data. Hence why I wish data for COVID-19 worldwide was more stratified by age, underlying condition, etc.

        • (a) The epidemic this gets most compared to is the Spanish Flu, which had 0.8% mortality in Nashville as pretty much the national maximum.

          (b) I am not confident in saying there will or won’t be serious long-term outcomes for previously healthy people, because we can’t have data since we’re not in the long-term yet. I’m at the “somebody needs to look at that, and meanwhile let’s be cautious” stage, lest we all insulate our homes with asbestos again.

          You have a plausible mechanic, but no data and no source, and “80% of cases were mild” with zero data on the health status of mild vs severe cases that survived makes that point moot.

          We have consensus on “need more data”, but that is why we can’t have consensus on “only causes mild to minor symptoms in healthy people” without data, and without describing how big the “sub-population” is. I think “cardiovascular” alone describes 20% of the population?

        • (a) Fair. Regardless of comparison, my issue is the sensationlizing of COVID-19 as “world changing” and “apocolyptical” and other adjectives use by media, politicians, and some scientists, to describe the virus. Data does not and has not supported such a claim. (At least in terms of mortality; data does not indicate we millions will die like during Spanish Flu.)

          (b) I wish China reported data better. We would have some “long-term” data (if you count ~2-3 months as long-term, which really depends) of outcomes from patients there. Hopefully Italy and the US do so in a few months and over the year.

          And yes, with any new disease we cannot know the long-term risks. Should we enforce draconian measures — i.e., mass Social Distancing for months —long-term for that reason alone, though? For such measures also have many negative consequences long-term. Granted, neither are easy to predict accurately — if at all. Tricky problem.

        • Anonymous –

          > (a) Fair. Regardless of comparison, my issue is the sensationlizing of COVID-19 as “world changing” and “apocolyptical” and other adjectives use by media, politicians, and some scientists, to describe the virus. Data does not and has not supported such a claim. (At least in terms of mortality; data does not indicate we millions will die like during Spanish Flu.)

          Let’s assume no mitigation – in another words the mass Social Distancing for months that you describe as “draconian.” Lets assume a .5% fatality rate. Let’s assume that 20% of the cases are serious enough to require hospitalization. Let’s assume that fatality rate is significantly more concentrated among obese people, people with hypertension and diabetes, etc.

          What do you estimate at the total # of infections in this country, and as a result the number of deaths and the number of hospitalizations? How about the lost work hours? How about the businesses who fail because people won’t use their services? How about the businesses that would fail because they couldn’t maintain sufficient staffing? How about the essential businesses that wouldn’t be able to maintain operation because they couldn’t maintain staffing? How about all the lost work hours from people who spent a couple of weeks at home but never got bad enough to go to the hospital?

          How do you calculate the cost to society of having all those infected people running around (given that we don’t have a functional system for testing and contact tracing on a large scale) and infecting people who are uninfected?

          How do you calculate the impact to communities that would likely have much higher mortality and morbidity rates? Do you think that we should just consider disparate impact in different communities as something we should just accept as the lottery of life?

          What about the impact to our healthcare system, to our nurses and doctors?

          What adjectival phrases other than “world changing” and “apocalyptic” do you think would be more apt?

          Granted, you stake that it’s not easy to predict, if at all. But it also seems to me that to state that “world changing” if not necessarily “apocalyptic” is sensationalization, you must have some predictions in those areas. And “draconian” also, it seems to me, suggest that you have a clear picture that the “draconian” measures wouldn’t be necessitated either way down the line. Is that right? If so, what gives you that confidence?

        • Now I know that’s a lot of questions – and listing them looks like “just asking questions” as a rhetorical device.

          But I really an curious as to how you resolve those questions as it seems to me that to write what you’ve written then you must have done so.

          So obviously, no responsibility to answer them all. But I am actually curious about your answers to all of them – but primarily, I guess, to perhaps one I didn’t ask – what might be an infection rate that would seem to you to divide between better outcomes in the one direction as opposed to the other?

        • Joshua,

          Your hypothetical scenario raises valid concerns. We cannot ignore a brand new pathogen that is clearly able to kill people en masse. I would not want to see anyone suffer or die. That is why I have pursued a career in engineer/science.

          Personally, I think there is a compromise to the draconian Social Distancing we have now and the darwinian “let the virus run its course” approach some are calling for. Allow healthy people to continue working and socializing per normal if they choose. We know that their risk of severe short-term (and hopefully long-term) outcomes is minimal; by doing this, we build herd-immunity. Our economy and livelihood will suffer some — things will still be different. But life can still proceed for many.

          Simultaneously, protect those who are at-risk. Do not force them to work unless they choose to do so. Require that stores have specific hours for at-risk people only to minimize transmission. Increase safety at long-term care facilities. There will still be some infection of this population, but we can do so without halting most economic/social function.

          All this sounds simple in principle, but is difficult in practice. There will be transmission to those who are at-risk. We will have to learn “on the go” and adjust.

          Now consider the alternative, which is the draconian Social Distancing now commonplace.

          How many have lost their jobs? How many have had their depression, anxiety, and overall mental health spiral to substance-abuse, spousal-abuse, and even suicide? How many will develop poor diets due to stress and last for years? How many small business will not recover? What happens if this results in a greater recession (wouldn’t be surprised, given the economy is a powder-key of leverage presently)? What happens to people then? Economic and physical well-being are inextricable. Lower one and you lower the other.

          And most important, how sure are we that Social Distancing is really working? In practical terms, strategies like Social Distancing are “all or nothing” in a sense if the underlying disease is high transmissible, like SARS-CoV-2. Even minor deviations from 100% Social Distancing — e.g., avoid any human contact for weeks — deteriorate very quickly if even a few people violate it. One person could infect tens to hundreds.

          So is Social Distancing really working if we are still able to visit parks, stores, supermarkets, and other locations en masse? What about the many essential workers? All it takes is a few mistakes to result in high transmission to continue.

          To re-iterate: I am all for Social Distancing if it works as advertised and its positive outcomes exceed its negative ones. But I have seen such data presented yet, even though its clearly crucial.

          Finally, what if another hypothetical situation is the care: What if the fatality-rate is 0.05%. What if its 0.01%? Hence the importance of randomized serological testing. We could discuss hypothetical scenarios. Without any reliable data on fatality, however, such discussing will invariably turn philosophical.

        • Forgot to answer your main question:

          > what might be an infection rate that would seem to you to divide between better outcomes in the one direction as opposed to the other?

          I do not have any specific value.

          Having such a cutoff over-simplifies a complex situation, in my view. Even if the fatality-rate was very low, say 0.05% or 0.01% and it infects 50 million people. That is still 25,000 and 5,000 deaths, respectively. Should we enact mass Social Distancing to save these lives? They are still people. Is it right to forgo them to death?

          But what about the consequences of Social Distancing on everyone else, though? How many “lives” will it cost in terms of social, economic, and mental damage? Trying to place economic value on lives and livelihood (ethical concerns aside) is not a tractable problem. So how can one even use these values as the basis for comparison?

          Clearly, its difficult situation with not clear optimal solution. All we can do is pick a strategy, acquire as much data as possible, adjust quickly based on data to better mitigate risk and negative affects, and repeat.

        • Anonymous –

          Thanks.

          > Allow healthy people to continue working and socializing per normal if they choose. We know that their risk of severe short-term (and hopefully long-term) outcomes is minimal; by doing this, we build herd-immunity. Our economy and livelihood will suffer some — things will still be different. But life can still proceed for many.

          So I question that at a lot of levels. First, we have somewhat limited knowledge about immunity. Betting a lot on it existing and lasting would really be a bummer if it doesn’t pan out. So I gather that most people with expertise think the probability of a reasonable robust immunity are relatively high. But from where I sit, a lot of this about examining the high damage risk even if it is low probability. And that is something that, IMO, people in general are not very good at. A bet on establishing a lasting immunity is a very highly leveraged bet. Not that there are any real low leveraged bets here, but some may be higher than others.

          Second, I don’t like the scenario of let low people wander around and protect the highly vulnerable people because it seems rather unrealistic to me. I don’t see how our society has any real means for protecting highly vulnerable people and communities at relatively high risk. As such, a bunch of low vulnerable people wandering around, IMO, means pretty much the same amount of highly vulnerable people getting infected as taking virtually no mitigation steps at all (excluding the steps of a lot of testing and a lot of contact tracing).

          I’ll paint a little scenario. I have a friend who lives in CA and who basically takes care of an aunt who lives in NJ. His aunt is on dialysis and just recently got a positive test. His aunt has a live-in healthcare provider. My fried made it clear that that healthcare provider should be tested, but the agency she works for had absolutely no mechanism by which to make sure that happens. That healthcare provider lives in a community that is at relatively high risk. It is easy, when you don’t live in one of those communities, to not realize just how far-fetched it is to imagine that people in such communities would actually be protected at the level at which they would need to be protected. The idea of contact tracing doesn’t even really exist for that woman. She finally got a phone number to call for getting a test and she was told that because she didn’t have symptoms she couldn’t get a test. This is someone who was exposed to an infected person and will continue to work with a community of vulnerable people and who lives in a community where the risk from infection is relatively high.

          Maybe, just maybe, if we really did have the infrastructure to enable comprehensive testing and thorough contact tracing, then it would be possible for us to really sustain protections for vulnerable communities by my view is that if that’s every going to happen it’s only going to happen if we buy a lot of time to set it up. But if you’re talking about lifting these restrictions that you call draconian in the near future, then I think that you should consider that you’re essentially rolling the dice with a whole lot of lives. Maybe in the end that’s justifiable. Maybe in the end all those people are going to get infected anyway even with these measures you call draconian. I’m willing to say that I don’t know. But my own personal belief is that there are a lot of reasons to hedge against very high damage risk.

          > Simultaneously, protect those who are at-risk. Do not force them to work unless they choose to do so. Require that stores have specific hours for at-risk people only to minimize transmission. Increase safety at long-term care facilities. There will still be some infection of this population, but we can do so without halting most economic/social function.

          See above.

          All this sounds simple in principle, but is difficult in practice. There will be transmission to those who are at-risk. We will have to learn “on the go” and adjust.

          > How many have lost their jobs? How many have had their depression, anxiety, and overall mental health spiral to substance-abuse, spousal-abuse, and even suicide? How many will develop poor diets due to stress and last for years? How many small business will not recover? What happens if this results in a greater recession (wouldn’t be surprised, given the economy is a powder-key of leverage presently)? What happens to people then? Economic and physical well-being are inextricable. Lower one and you lower the other.

          My answer to all of those questions is that this is not a binary forking path. None of those harms just disappear with the lifting of the measures you call draconian. And possibly, without those measures, those harms will even be more sustained in the long run. Think of the argument that the sooner you institute those measures the more harms you mitigate. Is that wrong? I don’t know exactly, but my guess is that in this country where the other mitigation measures are so lacking, the worse upside risk are those which are associated with doing nothing, implementing the measures too late, or lifting them too early.

          > And most important, how sure are we that Social Distancing is really working? In practical terms, strategies like Social Distancing are “all or nothing” in a sense if the underlying disease is high transmissible, like SARS-CoV-2. Even minor deviations from 100% Social Distancing — e.g., avoid any human contact for weeks — deteriorate very quickly if even a few people violate it. One person could infect tens to hundreds.

          So I think this is a key issue, because I think there is a lot of uncertainty how whether, or more relevantly, how much social distancing is working, but that they are most definitely not all or nothing. Further, I think that not implementing those measures is not a binary forking path. Many, many of the harms that come from social distancing will also be present with no social distancing. The existence of those harms is not a function of the existence of the social distancing measures. That is why I don’t like the term “draconian” – because to me it implies an either/or scenario that doesn’t exist. We have a choice between bad options. Neither, that I can tell, are either draconian or non-draconian in any meaningful or realistic sense.

          > So is Social Distancing really working if we are still able to visit parks, stores, supermarkets, and other locations en masse? What about the many essential workers? All it takes is a few mistakes to result in high transmission to continue.

          So I guess this is where we meet at a middle ground, where some measures are carefully relaxes along with a very careful system of measurement and surveillance. The main point I’d make here about that is that we should be focusing first and foremost on setting up the proper infrastructure to enable that to happen. I would like to see a healthcare work force, paid well, to get this to happen.

          > To re-iterate: I am all for Social Distancing if it works as advertised and its positive outcomes exceed its negative ones. But I have seen such data presented yet, even though its clearly crucial.

          Yes, the hard data to draw firm conclusions are lacking. Thus, for me, the best option is to hedge against the highest downside risk.

          > Finally, what if another hypothetical situation is the care: What if the fatality-rate is 0.05%. What if its 0.01%? Hence the importance of randomized serological testing. We could discuss hypothetical scenarios. Without any reliable data on fatality, however, such discussing will invariably turn philosophical.

          So there we agree.

        • Anonymous –

          Sorry for being long-winded.

          Part II

          > Having such a cutoff over-simplifies a complex situation, in my view. Even if the fatality-rate was very low, say 0.05% or 0.01% and it infects 50 million people. That is still 25,000 and 5,000 deaths, respectively. Should we enact mass Social Distancing to save these lives? They are still people. Is it right to forgo them to death?

          Ok. I get that maybe it simplifies it to a point that’s useless. But I do think that at some point we need to construct a range by which we can frame decisions about what kind of infection rate and what kind of fatality rate exclude some kinds of decisions. I don’t know if there’s a way around that, as difficult as it is. Otherwise, I guess we are relegated to feelings about what to do which might be very hard to negotiate. Maybe in the end that is inevitable, but before assuming it is, I think that it’s important to establish some kind of range.

          > But what about the consequences of Social Distancing on everyone else, though? How many “lives” will it cost in terms of social, economic, and mental damage? Trying to place economic value on lives and livelihood (ethical concerns aside) is not a tractable problem. So how can one even use these values as the basis for comparison?

          Again, I don’t think these are binary questions that hinge on the existence or lack thereof of social distancing measures.

          > Clearly, its difficult situation with not clear optimal solution. All we can do is pick a strategy, acquire as much data as possible, adjust quickly based on data to better mitigate risk and negative affects, and repeat.

          Sure. But then we are relegated to just entrusting others to make the decisions for us based on whatever reasons they choose to base their decisions on. Unfortunately, in today’s environment that takes place without reasonable exchange to discuss what the different decision parameters might be. Maybe finding a range for those parameters won’t make any difference anyway in the end. But I at least like to think it makes something of a difference, even while acknowledging that might just be a fantasy.

        • Joshua,

          No need to apologize for being long-winded. I appreciate your taking the time to discuss in detail.

          > A bet on establishing a lasting immunity is a very highly leveraged bet. Not that there are any real low leveraged bets here, but some may be higher than others.

          Yes and no. If the fatality-rate is similar to season influenza, like 0.05%, 0.01%, or lower? (Now we don’t know this; but it is still possible; hence why serological testing is so crucial.) The leverage of the bet changes.

          And with regard to immunity — sure, it may not last long. But waning immunity and loss of immunity are different. We know for other coronaviruses that full immunity usually lasts for a few years before waning. In terms of our best data on the biochemistry and genetics of SARS-CoV-2, we don’t have reason yet to think it will differ much, if at all, from its siblings. (If you have data/papers claiming otherwise, please share!) Sure it could be different; but that is very unlikely and, honestly, it would be apparent by now with the virus having been present for 5 months worldwide. So the risk here may not be as high as it seems.

          > I’ll paint a little scenario…But my own personal belief is that there are a lot of reasons to hedge against very high damage risk

          Foremost, my best wishes to your friends aunt. That sounds like a very frustrating situation and hopefully it ends well.

          Addressing the text. Would the live-in worker not be considered essential? And therefore would current Social Distancing not apply? Second, I would argue her inability to receive a test is a failure of government more than anything else. So I’m not sure if this scenario applies.

          Regardless, I understanding your point. “Protect and isolate” seem simple as nous, but are very complex and trick in reality. I concede that. To be honest, I would have to give all this more thought to provide a more thorough solution. And frankly, such a solution would likely need to be specific to location, demographic, etc., such as states and/or metropolitan regions.

          > My answer to all of those questions is that this is not a binary forking path. None of those harms just disappear with the lifting of the measures you call draconian. And possibly, without those measures, those harms will even be more sustained in the long run.

          Fair. We really cannot know. Human action is not something anyone can predict. My point was rather to illustrate that long-term draconian measures can have their own severe consequences; and these could outweigh the disease if my mis-gauge its severity. So we should consider them with great care.

          > So I think this is a key issue, because I think there is a lot of uncertainty how whether, or more relevantly, how much social distancing is working, but that they are most definitely not all or nothing. Further, I think that not implementing those measures is not a binary forking path. Many, many of the harms that come from social distancing will also be present with no social distancing. The existence of those harms is not a function of the existence of the social distancing measures. That is why I don’t like the term “draconian” – because to me it implies an either/or scenario that doesn’t exist. We have a choice between bad options. Neither, that I can tell, are either draconian or non-draconian in any meaningful or realistic sense.

          All fair. As I said above, we have no way of knowing. Perhaps “draconian” is not the right adjective; you raise valid points about it not being the right adjective; so maybe another is better.

          > So I guess this is where we meet at a middle ground, where some measures are carefully relaxes along with a very careful system of measurement and surveillance. The main point I’d make here about that is that we should be focusing first and foremost on setting up the proper infrastructure to enable that to happen. I would like to see a healthcare work force, paid well, to get this to happen.

          I agree. We need to be VERY careful about the surveillance we allow, though (cough cough PATRIOT Act). Otherwise — yes, let us bolster the healthcare system as much as we can. Such change is a long-time-coming, anyways; for various reasons, our healthcare system (especially hospitals) is broken and needs fixing. Bolstering it for COVID-19 is a positively leveraged bet if we do it well — since it will be useful for future pandemics and otherwise.

          > Yes, the hard data to draw firm conclusions are lacking. Thus, for me, the best option is to hedge against the highest downside risk.

          I agree, But again, we have to be very careful about what we allow. And we want to constantly be evaluating the strategy with data and informing the public; not just proclaiming it every day in a press-conference.

          > Ok. I get that maybe it simplifies it to a point that’s useless. But I do think that at some point we need to construct a range by which we can frame decisions about what kind of infection rate and what kind of fatality rate exclude some kinds of decisions. I don’t know if there’s a way around that, as difficult as it is. Otherwise, I guess we are relegated to feelings about what to do which might be very hard to negotiate. Maybe in the end that is inevitable, but before assuming it is, I think that it’s important to establish some kind of range.

          I think what you propose here may be very useful for COVID-19 and future pandemics. We want do it carefully, though. And I worry that if done now, amid the chaos, we may make a poor decision; but perhaps we won’t. Whatever we do and choose, we just want to make sure we evaluate it rigorously and carefully.

          > Again, I don’t think these are binary questions that hinge on the existence or lack thereof of social distancing measures.

          They do, to a degree. As I said above, if we are wrong and COVID-19 is much less fatal that we think, then our measures will seem like a “cried wolf” scenario to much of the public (as we are seeing now in the US). This may impede and slow implementing future Social Distancing (or other measures), which may be absolutely necessary for future pandemics.

          > Sure. But then we are relegated to just entrusting others to make the decisions for us based on whatever reasons they choose to base their decisions on. Unfortunately, in today’s environment that takes place without reasonable exchange to discuss what the different decision parameters might be. Maybe finding a range for those parameters won’t make any difference anyway in the end. But I at least like to think it makes something of a difference, even while acknowledging that might just be a fantasy.

          I was more postulating what my choices would be if I were making decisions. But I’m not in reality (and that is probably a good thing). I do wish there was a much more “scientific” discourse preset among all leaders (regardless of side) presently. Seems like dogma is more important than science; and that is scary. Ergo why we should hold our politicians, experts, and pundits accountable to their decisions, whatever the outcome.

          To summarize: I think we can agree that we need better data on infections via randomized serological testing. That is key. Why we don’t have this data — almost 2 months since SARS-CoV-2 first appeared on US soil and 4 months since it first struck Italy — is baffling and frustrating. Imagine if we had this data now or a month ago?

        • You guys are having a good discussion here and I’m not sure how much I’ll contribute by butting in, but I do want to make a few points:

          1. Voluntary actions like those ‘Anonymous’ suggests can be effective, as I noted in my post about Sweden a couple of days ago. But they are still killing a lot of people: they just overtook Switzerland in terms of deaths per million population, and odds look good for them passing Netherlands in the next week or so. As far as fraction of population killed by the virus, they’re already in the top ten and climbing the ladder pretty fast.

          2 …but then, most of the people dying in Sweden are old, and many of the rest have chronic health problems. That’s true everywhere in the world as far as I know. I think that for many old people (including my mom), a day of her life is as precious as a day of hers. But my expected days of life remaining is probably 7 times hers or so. In terms of life-years lost, losing a few thousand old people is not nearly as bad as losing a few thousand young people. It seems pretty harsh to me — lots of these people would surely have years ahead of them if not for the virus. But you do have to weigh it against the alternative.

          3. But also, the other countries are going to have to come out of lockdown sometime, so their fatality rate will also increase. Right now Sweden looks really bad compared to, say, Norway, but Norway may end up catching up, at least somewhat. Or not.

          4. Although most social distancing in Sweden is voluntary, they’re doing it. I don’t trust voluntary actions like that in the U.S., not at all. We have had (and still have) politicians denouncing the idea; ministers and others who actively encourage dangerous behavior; vocal people who think it’s all a hoax. As the Swedes themselves say, they have a population that values their government and generally trusts it and is willing to follow government advice. We don’t have that. Also, I think our population is much less healthy (more obesity, more diabetes). A policy of voluntary action that is working OK in Sweden might be a disaster here.

          Putting it all together, my current thinking probably puts me a bit on the ‘cautious’ side of the median, when it comes to how and when to back off the lockdown. I think we need the ability to do a lot more rapid testing; probably need regulations to protect vulnerable populations (such as requiring businesses to offer hours when only seniors and people with chronic health issues can be present); regulations to slow transmission in general (such as no more than 1 person per 20 square meters in a store, or something; and gloves and masks for all employees who deal with the public, and so on). In a better world, or maybe a better country, we could just tell employers what is necessary to keep employees and customers reasonably safe, but in the world and country we live in I just don’t think that would work. But I am not sure.

        • > Yes and no. If the fatality-rate is similar to season influenza, like 0.05%, 0.01%, or lower? (Now we don’t know this; but it is still possible; hence why serological testing is so crucial.) The leverage of the bet changes.

          Fair point. Agreed.

          > (If you have data/papers claiming otherwise, please share!)

          Nope. And I certainly hope that it turns out that way.

          > Foremost, my best wishes to your friends aunt. That sounds like a very frustrating situation and hopefully it ends well.

          Thanks. We’ll see. So far the symptoms are mild but she’s at an incredibly high risk.

          > Addressing the text. Would the live-in worker not be considered essential? And therefore would current Social Distancing not apply? Second, I would argue her inability to receive a test is a failure of government more than anything else. So I’m not sure if this scenario applies.

          Sure. It’s a failure of government. But what I was trying to illustrate is that we’re talking in the context of a largely failed government, at least thus far. I think that there’s an inverse relationship between the need for mandated social distancing and the existence of effective contact tracing and testing. The best scenario, by far, would be optimal tracing and testing and minimal mandated social distancing.

          Aside from government functioning, I think that there’s another factor that doesn’t work well in our favor. I lived for Korea and my impression from that experience is that their orientation much further along the collective side of the individualist —> collectivist scale plays a big role in their successful contact tracing and testing. From living there, I suspect that I’m more inclined in that direction myself than you might be – but again my hope would be that the tradeoffs could be considered openly (with data) and calmly as a society, although I’m not particularly optimistic in that regard. More likely, IMO, is that people will spend energy digging in and pointing fingers and “otherizing” those who tend towards the opposite pole.

          > My point was rather to illustrate that long-term draconian measures can have their own severe consequences; and these could outweigh the disease if my mis-gauge its severity. So we should consider them with great care.

          Sure. Agreed. It doesn’t help when people moralize the need to create a balance.

          > Such change is a long-time-coming, anyways; for various reasons, our healthcare system (especially hospitals) is broken and needs fixing. Bolstering it for COVID-19 is a positively leveraged bet if we do it well — since it will be useful for future pandemics and otherwise.

          Yes. Good point. Something that was in the back of my mind but not really in focus. It’s another aspect of the risk analysis that I think is important – the carry-on benefits from whatever the short term measures are.

          > They do, to a degree. As I said above, if we are wrong and COVID-19 is much less fatal that we think, then our measures will seem like a “cried wolf” scenario to much of the public (as we are seeing now in the US). This may impede and slow implementing future Social Distancing (or other measures), which may be absolutely necessary for future pandemics.

          Sure. Agreed.

          > I was more postulating what my choices would be if I were making decisions. But I’m not in reality (and that is probably a good thing). I do wish there was a much more “scientific” discourse preset among all leaders (regardless of side) presently. Seems like dogma is more important than science; and that is scary. Ergo why we should hold our politicians, experts, and pundits accountable to their decisions, whatever the outcome.

          Totally agree.

          To summarize: I think we can agree that we need better data on infections via randomized serological testing. That is key. Why we don’t have this data — almost 2 months since SARS-CoV-2 first appeared on US soil and 4 months since it first struck Italy — is baffling and frustrating. Imagine if we had this data now or a month ago?

          It’s incredibly disturbing because not only is it such a high stakes issue, but because the lack of ability has absolutely no viable or logical justification that I can see. It’s really just so hard to fathom.

          I live in a rural area of NY (in the Hudson Valley). There are about 24 cases in my “town” (I don’t really live in a town but that’s how it’s divided up). I have no way of ascertaining where those cases have occurred. Have they occurred from community spread? I have no idea. Are they all in a cluster of a couple of families not near me? I have no idea. Have any of the workers in the local stores had COVID? I have no idea. We have essentially completely given up on contact tracing as soon as they decided that because of not enough testing they would just tell people with symptoms “no testing for you” and that they should just go home and self-isolate. It seems insane to me that this is where we are.

          My understanding is that in Korea, people get a message on their cell phone any time someone near them tests positive, giving them the location where that person lives or works…and very thorough contact tracing follows. Scary in a Big Brother way? Sure. But I happen to have an underlying condition that puts me in the high risk category (hypertophic cardiomyopathy), and I personally find that kind of government surveillance less concerning than having no idea whether someone shopping in my local supermarket has tested positive. I can certainly understand why others would feel the opposite way. But can’t we at least have a national discussion?

          Anyway, thanks for the convo.

        • Joshua,

          > But I happen to have an underlying condition that puts me in the high risk category (hypertophic cardiomyopathy)

          Yikes. I’m sorry. Stay safe and healthy!

          > Sure. It’s a failure of government. But what I was trying to illustrate is that we’re talking in the context of a largely failed government, at least thus far. I think that there’s an inverse relationship between the need for mandated social distancing and the existence of effective contact tracing and testing. The best scenario, by far, would be optimal tracing and testing and minimal mandated social distancing.

          While I agree that digital Contact Tracing could be a useful tool, it is complicated.

          Proximity gives you an *idea* of who was infected. But it misses a lot. What did they touch in their proximity? Before touching these things, did they happen to cough, sneeze, tough their face, tough their mask, etc., that could have transmitted virus? Who then touched said surface that was also in this proximity during the next hour? 2 hours? 6 hours? 1 day?

          Imagine the above scenarios in Penn Station, an apartment complex, a park, etc. A nightmare! In terms of Network/Graph Theory, you could not even solve the problem because the key factor — true physical contact — is not possible. And depending on your timescale, the problem may soon become not tractable to compute for the population of the US or even a small city. (This would be an interesting exercise in Graph Theory, actually. I wonder if computational cost grows linearly, geometrically, or exponentially with time.)

          So while I agree digital Contact Tracing could be another useful tool, it’s certainly not a panacea. South Korea’s solution seems useful, but I wonder how effective it has been? Hard to know yes/no without detailed data. I’d be more comfortable, honestly, if a private business did this than the government. As backwards as it sounds, its easier to re-obtain rights from a company than the government (although they may be getting the data either way), should it be necessary. (But…there are caveats and tradeoffs to this. I understand those who feel otherwise.)

          > It’s incredibly disturbing because not only is it such a high stakes issue, but because the lack of ability has absolutely no viable or logical justification that I can see. It’s really just so hard to fathom.

          100% agree. I wish we knew what happened behind-closed-doors. How did “experts” advising officials not realize this earlier? Given just how crucial true infection counts are, it really is inane and unacceptable. (Unless there are good reasons for the situation that we are not privy too.)

          Chris,

          Sweden is taking an interesting approach. What’s difficult is that without knowing true infection numbers for each nation, we will never know whose strategy was the most sensible.

        • Anonymous –

          Joshua,

          > While I agree that digital Contact Tracing could be a useful tool, it is complicated.

          Proximity gives you an *idea* of who was infected. But it misses a lot. What did they touch in their proximity? Before touching these things, did they happen to cough, sneeze, tough their face, tough their mask, etc., that could have transmitted virus? Who then touched said surface that was also in this proximity during the next hour? 2 hours? 6 hours? 1 day?

          Well, the proximity info. is only part of the process. Perhaps it isn’t even the most important part in terms of protective outcomes. But in terms of how people can live their lives (and economic impact) it might be more important. For example, right now I’m not going to go shopping because I have no idea if there has been ANY community spread in my community. In theory, proximity info attached to some other results from contact tracing investigation (which I think happens in Korea), then I might now that there’s more security in my going shopping as long as I limit my contact with individuals. I might be able to resume more normal economic activity.

          It certainly seems to me that there’s a consensus that washing my hands, not touching my face, etc. are the aspects that will bring the biggest return…but cell phone data, it seems to me, could be a fairly important part of a comprehensive picture.

          > Imagine the above scenarios in Penn Station, an apartment complex, a park, etc. A nightmare! In terms of Network/Graph Theory, you could not even solve the problem because the key factor — true physical contact — is not possible. And depending on your timescale, the problem may soon become not tractable to compute for the population of the US or even a small city. (This would be an interesting exercise in Graph Theory, actually. I wonder if computational cost grows linearly, geometrically, or exponentially with time.)

          Yes, but my understanding is that they use cell phone data to pretty closely track where people traveled, and then follow that with an associated contact tracing. No, it won’t solve the problem, but seems to me it would contribute to incremental or marginal improvement. Many pieces of incremental improvement can lead to significant improvement.

          I think of it like this. Any one behavior of mine might seem of infinitesimal value. If I do or don’t touch that doorknob, or wash my hands after touching the mail, I might have a tiny chance of reducing my risk. So sometimes I feel silly taking precautions against that infinitesimal risk. But then I think of adding together all those behavioral changes and while I still won’t completely alter the role of simple chance, I may be able to tip the scales a bit.

          > So while I agree digital Contact Tracing could be another useful tool, it’s certainly not a panacea.

          Sure.

          > South Korea’s solution seems useful, but I wonder how effective it has been? Hard to know yes/no without detailed data. I’d be more comfortable, honestly, if a private business did this than the government. As backwards as it sounds, its easier to re-obtain rights from a company than the government (although they may be getting the data either way), should it be necessary. (But…there are caveats and tradeoffs to this. I understand those who feel otherwise.)

          In NY State it looks like Bloomberg is going to get many millions to organize a team for contact tracing. Perhaps you have more faith in the advantages of the private sector over the public sector than I (in this situation in particular, I think that public health officials probably have indispensable expertise and information), but in the least I would agree that there can be an additive or multiplier effect from combining resources.

          > 100% agree. I wish we knew what happened behind-closed-doors. How did “experts” advising officials not realize this earlier? Given just how crucial true infection counts are, it really is inane and unacceptable. (Unless there are good reasons for the situation that we are not privy too.)

          Yah. Makes no sense.

          Another thought –

          I have been thinking about the “purpose” of modeling and thinking that part of this picture is that people can have different priorities in that regard. More specifically, a focus on the welfare of doctors and nurses may in some sense be balanced against deaths of citizens and a hit to the economy. Personally, I think that the welfare of people who are putting their lives on the line at great sacrifice deserve a kind of “post-stratification” weighting that might, in a sense, place an extra value on their welfare. Just another complicating factor.

        • Anonymous –

          Here’s what sent me down that thought path: I heard a modeler discussing the various model types for COVID 19, and he said (paraphrasing), if our most important goal is reducing the strain on the healthcare system, then you can often get away with lower levels of fidelity in the model.

          I think I agree with that, and that is an important factor in the risk assessment process. We all necessarily have subjective prioritizations in how we assess the “fit for purpose” of the modeling.

          Perhaps one of the most important parts of the process is to try to make explicit what our priorities are so we can find common interests there, or negotiate different interests.

        • Joshua,

          > Yes, but my understanding is that they use cell phone data to pretty closely track where people traveled, and then follow that with an associated contact tracing.

          I was not aware of this. In principle, this should work well. In reality, it may have lots hiccups. Whatever we do, we just want to be sure the time/funds we spend are worth it by constantly evaluating the setup; not just implementing it and not changing it for the better. (But I think we already agree on that, anyways.)

          > In NY State it looks like Bloomberg is going to get many millions to organize a team for contact tracing. Perhaps you have more faith in the advantages of the private sector over the public sector than I (in this situation in particular, I think that public health officials probably have indispensable expertise and information), but in the least I would agree that there can be an additive or multiplier effect from combining resources.

          Time will tell. I just hope Contact Tracing doesn’t become the new hot topic; that must be serological testing to determine true count of infections. (Not to say we ignore Contact Tracing; rather, just make serological testing our priority.) We definitely agree on this; let us hope our leaders and experts do as well…

          > I have been thinking about the “purpose” of modeling and thinking that part of this picture is that people can have different priorities in that regard. More specifically, a focus on the welfare of doctors and nurses may in some sense be balanced against deaths of citizens and a hit to the economy. Personally, I think that the welfare of people who are putting their lives on the line at great sacrifice deserve a kind of “post-stratification” weighting that might, in a sense, place an extra value on their welfare. Just another complicating factor.

          Agreed. Our healthcare system is broken. And it urgently needs fixing. As COVID-19 shows. It is unable to elastically adjust its demand of physicians, nurses, units, etc. It is unable to protect those who are protecting us — and that is not acceptable. Hopefully COVID-19 was the “wakeup call” necessary to promote change; although, I wish it did not have to come at the cost of lives and livelihoods.

        • polio mortality is far less than 10%
          70% are asymptomatic
          24% have mild illness
          .1 to .5% have some paralysis, but in about half of those there is a complete recovery.

          The distinction though is most polio victims were children, so the years of life affected were much greater for each severe case. It also is far more dangerous to adults, as are many other childhood diseases.

        • Thanks for this information. It is consistent with the following: My brother is believed to have had polio when he was a small child (a bit under 2). It was not diagnosed as such then, but he did have an illness that occurred after he had learned to walk. He lost the ability to walk, but was able to relearn to walk after he got better. The diagnosis of polio came when he was much older (perhaps in his fifties?) and developed what appeared to be early-onset arthritis — but the physicians diagnosed it as post-polio syndrome.

    • Andrew –

      Ok. I’m going to try again, you “agreed” with this:

      > So is it acceptable for experts to still report crude infection-fatality-rates when SARS-CoV-2 only seems to cause severe outcomes in a specific sub-population?
      +++++++++

      No. But that’s pretty much *exactly* what Ioannidis et al. did with their widespread publicity campaign to leverage their studies to weigh in on mass media to assert, pretty aggressively, that the IFR is around .1%

      Again, I’m not talking about the study, per se.

      So what is the line between scapegoating and holding the authors of this study accountable for their publicity campaign?

      • Joshua,

        I agree. We should hold Ioannidis and colleagues accountable for their study. We should do the same for anyone who has went to media to popularize their work, thereby influencing policy and those it affects.

        For example, Marc Lipsitch and colleagues at Harvard have popularized Social Distancing as an effective measure using their SEIR Model. Yet this model has crucial flaws that Daniel J. Bernstein addresses in his recent paper: “Further analysis of the impact of distancing upon the COVID-19 pandemic” https://www.medrxiv.org/content/10.1101/2020.04.14.20048025v1. It seems, at least as of March 30, Lipsitch and colleagues did not respond to Bernstein’s inquiry for code and data.

        So let us hold everyone accountable, not just a few. It is easy (and all too human) to make a single person or group Public Enemy #1, which would not be productive to fixing the overall problem — i.e., scientists popularizing and sensationalizing their data without any rigorous evaluation by peers.

        • Also Joshua,

          To clarify: I am not claiming IFR may be as low as Ioannidis and colleagues suggest. Rather, I am claiming it is not responsible to claim its 1-3% for everyone — as many scientists, politicians, and pundits have — when it clearly presents severe outcomes in specific sub-population. Neither are rigorous claims to make without the right data and mislead the public.

          Sameera,

          Thank you!

        • I’ve not seen 3% publicized for IFR, just CFR myself. Numbers I’ve been seeing have been more like 0.6%-1% …

        • In my locale, I have seen them used interchangeably by pundits/politicians — which I think has added confusion to the discourse.

          Whether CFR or IFR, my point is comparing the two for an entire population and not accounting for risk-factors has been irresponsible. Especially with the known uncertainty of each measure.

          For IFR: It is not a rigorous measure without knowing the true number of those infected. As we are learning, the value is likely much higher (which makes sense for a highly contagious airborne respiratory virus). So we really have no idea what it should be.

          For CFR: Positive cases these stem from positive PCR testing. But the PCR test has limits and flaws: the primary one being that “detected” viral RNA does not equate to infection. PCR will amplify anything if the primer works, even inactive/fragmented virus that is not alive/infectious. Further, we don’t know “how much” virus constitutes an infection; given how mild infections from SARS-CoV-2 are hard to distinguish from other common respiratory illnesses, it is hard to tell which mild cases are *with* SARS-CoV-2 versus *from* SARS-CoV-2. So for CFR, we don’t know the true number of actual infections, so we cannot know its true value.

        • The IFR has nothing to do with how contagious the disease may be (or if there is any effect is indirect, everything else being equal a more contagious disease will be worse if no-one wants to get near you to provide adequate care).

        • Maybe when you say infection fatality rate you mean population fatality rate?

          Or maybe you mean that being highly contagious health systems can’t cope with it increasing the probability of dying once infected? But that would be a problem for the case fatality ratio as well.

        • > The IFR has nothing to do with how contagious the disease may be

          Correct. My mistake.

          My conclusion happens to apply to some highly contagious respiratory viruses (rhinovirus, adenovirus, influenza, SARS-CoV-2) but does not apply to all; e.g., you would expect many undetected infections if the virus causes mild to no symptoms in many patients. But this is not always true. I should not have generalized.

        • Carlos,

          When we don’t know who is and isn’t infected, the communicability of the disease will have a great influence on our ESTIMATES of the IFR.

          We don’t know the IFR because we don’t know the two numbers needed to compute it. So we use the best available (with time lag cause-ascertainment issues) death counts for the numerator and we use proxies for the denominator.

          The relationship between the available proxies for “number infected at a given point in time” and the true denominator depends on how many unknown cases are out there.

          A highly communicable disease and/or one with lots of asymptomatic spreaders will tend to drive our number infected proxy lower and thereby drive our IFR estimate higher. For any given set of assumptions on our part, if the true communicability is higher that will make our IFR estimate too high.

          Conversely, a less communicable disease and/or one with fewer asymptomatic spreaders will tend to make any given estimate of IFR too low.

          It’s not that being more contagious makes it more likely to kill an infected person. It’s that being more contagious THAN WE ASSUME IT IS will make it more like to kill an infected person THAN WE ESTIMATE IT IS. And vice versa.

        • I apologize, I misread “the value is likely much higher” as “the IFR is much higher, which makes sense for a highly contagious etc.” rather than “the number of infected is much higher, etc.” (which means that the IFR is lower than what we calculate when we underestimate the number of infected people).

        • Carlos,

          I think some of us have talked/typed about all this stuff (“IFR!” “Fatalities!!!” “Asymptomatic??”) so much the last few days it’s hard to keep one conversation straight from the next.

          For my part I’m going to read 1/2 as much and comment 1/10 as much for a while and see how it goes…

        • I remember it. We already knew for weeks there was a massive number of asymptomatic/mild cases from the age distribution of the Chinese 70k patient data and Diamond Princess. We knew that those people werent getting tested so the real IFR looked more like 0.1-1%.

          The WHO said there was “no evidence” for that because they looked at some spreadsheets of the chinese data and saw it said like 1% asymptomatic or something:

          Evidence from China is that only 1% of reported cases do not have symptoms, and most of those cases develop symptoms within 2 days…Globally, about 3.4% of reported COVID-19 cases have died. By comparison, seasonal flu generally kills far fewer than 1% of those infected.

          https://www.who.int/dg/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19—3-march-2020

          This evidence based medicine stuff is so inept…

        • I thought that it was obvious, but my point was that the WHO never said that. They talk explicitely about reported cases in your quote.

          A couple of weeks before that they said in their daily update:

          “Based on these available analyses, current IFR estimates range from 0.3% to 1%. Without population-based serologic studies, it is not yet possible to know what proportion of the population has been infected with COVID-19.”

          https://www.who.int/docs/default-source/coronaviruse/situation-reports/20200219-sitrep-30-covid-19.pdf

        • I thought that it was obvious, but my point was that the WHO never said that. They talk explicitely about reported cases in your quote.

          They compare CFR of covid to IFR of flu, which requires assuming CFR ~ IFR for covid.

        • Or assuming that IFR~CFR for flu.

          Or assuming that it means “By comparison, seasonal flu generally kills far fewer than 1% of those [reported] infected.”

          That speech may be wrong in many aspects, but saying that 3.8% of the people who get infected will die is not among them.

        • Or assuming that IFR~CFR for flu.

          No one would assume this. It is well known 99% of flu cases go unreported. The guy specifically assumed only 1% went unrecorded a few sentences earlier. Everyone knows what he meant who isn’t being purposefully dense.

        • To make sure it is clear: “The guy specifically assumed only 1% of covid-19 cases went unrecorded a few sentences earlier.”

        • Anonymous –

          > We should do the same for anyone who has went to media to popularize their work, thereby influencing policy and those it affects.

          I agree. I can’t speak to the situation with Lipsitch – but I take a look and try to follow.

          And I want to reiterate:

          > It is easy (and all too human) to make a single person or group Public Enemy #1,..

          So here’s a problem. I’m trying to be specific to a criticism not of Ioannidis, but of the his practice of science in this particular instance. It’s hard to draw a line as it inevitably begins to look personal.

        • Joshua,

          > I’m trying to be specific to a criticism not of Ioannidis, but of the his practice of science in this particular instance.

          Fair. I was not insinuating a witch-hunt or scapegoating of Ioannidis by yourself or anyone else. More trying to caution against things turning into that. Apologies if it seemed otherwise.

        • Anonymous –

          > Fair. I was not insinuating a witch-hunt or scapegoating of Ioannidis by yourself or anyone else. More trying to caution against things turning into that. Apologies if it seemed otherwise.

          Now I should apologize for writing my comment in a way that made it seem as I thought that were implying that. :-)

          I didn’t. I’m just acknowledging the structural problem, that I struggle with it, and sometimes don’t meet the mark.

        • Joshua,

          No need to apologize! I supposed we have entered true “threadception” now (ha ha!)

          I’m just glad there are places where one can have a thoughtful discussion — despite the enormity of the Internet and Social Media…such places are hard to find.

    • In this one comment you’ve addressed most of the big picture concerns I have, personally, been obsessing and fretting over these past two weeks. It’s hardly a rant, rather a welcome survey of the current state of (non) knowledge about this disease. Well stated.

      • Brent,

        Thank you! And glad to know I others have these concerns.

        My biggest worry is we are setting precedents based on more dogmatic science than rigorous science. That is scary and care have terrible short-term and long-term consequences.

        Two examples of such decisions are digital Contact Tracing (which, despite what papers say, is not a very tractable or numerically stable solution) and the “hype” around ventilators. The ventilators concerns me most.

        If SARS is occurring in COVID-19 patients like those who had MERS and SARS in the past, invasive ventilating will do little to help; in fact, it may do more harm — it can damage pressure-sensitive organs like the brain and kidneys while affording nasty nosocomial bacteria a direct path to the ravaged and weak lungs. We have spent so much time, energy, and money on procuring these ventilators when we could have spent it investigating other treatments or doing randomized serological testing.

      • Jon,

        Yes, the US has a high prevalence of comorbidities. Overall, it would be better to stratify by specific comorbidities patient-by-patient. Given technology in 2020, I’m not sure why this isn’t possible or occurring — but I am not a clinician and do not know how HIPAA affects reporting.

        My point was more addressing fervor around COVID-19 being a “once in a lifetime” pathogen, when the data does not indicate such, especially considering it preferentially attacks those with comorbidities.

        Adding some further caution, though: evaluating how comorbidities affect outcome, at least in terms of physiology, is nuanced and tricky.

        What was the probability of those with said comorbidities dying the next 2 months, 6 months, 1 year, 2 years, etc.? If they have a small expected life-span pre-COVID-19, then it was “in the right place at the right time” as a pathogen; another pathogen could have cause death, too.

        Also, not all comorbidities are equal and all exist with their own distribution of severity and amount. For example, I would imagine someone with hypertension and otherwise healthy (mainly, not obese) would be likely to survive COVID-19 — at least given our current understanding of how SARS-CoV-2 attacks the respiratory tract. Also, there are multiple kinds of hypertension. Someone who has been hypertensive for a year or two, versus decades, will experience different pathophysiological consequences. What is the distribution of these sub-sub-populations throughout the US?

        In contrast, someone with obesity but not hypertension (which is possible) would likely have a much worse outcome in terms of physiology: there lungs have to do more baseline work to respirate enough O2 and CO2; their immune system likely have a higher basal level of inflammation (being obese, to the immune system, is like a chronic infection!) AND likely to have a more aggressive inflammatory response locally/systematically. Like hypertension, obesity exists as a distribution. Someone with BMI 25-30 (overweight but not obese) versus BMI 35+ will experience different pathophysiological consequences. The same goes for how long someone has been obese for. Like hypertension, I’m not sure if we know exactly how this sub-sub-population distributes across the US.

        • “In contrast, someone with obesity but not hypertension (which is possible) would likely have a much worse outcome in terms of physiology”

          This made me curious as to whether low weight patients have a worse outcome (than normal weight patients) with coronavirus. I didn’t find anything exclussive to CV in a quick search, but did find https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6304312/, which says that being underweight puts patients with various viral infections at greater risk of hospitalization than patients with normal weight.

        • Martha,

          Good point. I too wonder how COVID-19 affects underweight people. I imagine not well, like you posit. Being underweight, like being obese, can have drastic negative affects on your immune system. The primary one being your immune system does not really function well.

          Andrew,

          We really need to keep an eye on them. High on the list of troublemakers, like Lizard People.

    • “I have seen and experienced myself instances of lowly PhD students find legitimate errors…”

      Any competent person with an undergrad degree can dive into the literature in their subject area and find errors and/or significant problems in published papers and text books.

      • Jim,

        Shocked you think I am competent (ha ha!).

        My issue was less with errors being present — those happen as part of science. (I’m sure someone will review my work in a few years and find it rife with errors.) Rather, my concern was the attitude my advisor (very understandably) had toward not reporting the error because the ire of senior/celebrity faculty could have negative impacts on our careers. That should not happen.

  7. “I don’t want ‘crowd peer review’ or whatever you want to call it,” he said. “It’s just too burdensome and I’d rather have a more formal peer review process.”

    But they do want public policy to be influenced by these results before they’ve gone through a formal peer review process, therefore before they reveal their data and methodology.

  8. I’ve looked through the medarxiv comments again today, and crowd “peer” review isn’t all that. If that’s an indication of what their email inbox looks like, I can understand the desire for restriction.

    I have seen the first 5G conspiracy theory commenter there today. Plenty of people citing “the Heinsberg study” when we know about as much about Heinsberg as we do about LA County: a few numbers with little context shared on a press conference that have since made the rounds with even less context as if they were gospel.

    And that is the true problem here: if you play to the crowd, you need to let the crowd review your findings! A pre-print or a press-conference is a publication, but it isn’t a scientific publication if it has no transparency attached to it. If you go public with something with the implicit assumption that “this is science”, be prepared to be transparent!

    There is a lot of misreporting of science in the media. As a consumer, I have usually been able to dig down to at least the abstract of the findings being reported before I hit the paywall; it is usually quite instructive. With these two studies (LA County and Heinsberg), I can’t, because all I end up with is a scientifically bare-boned press release.

    This isn’t just about peer review. I’m not a peer to any virologist, epidemiologist, or statistician. But I want to be able to see what I’m getting told about, and if you are telling people things but aren’t transparent about them, you’re not conducting science, you are engaging in politics under the mantle of science, and that’s misleading.

  9. “New York City had a higher rate of antibodies (21.2 percent) than anywhere else in the state and accounted for 43 percent of the total tested. Long Island had a 16.7 percent positivity rate, while Westchester and Rockland counties saw 11.7 percent of their samples come up with the antibody. The rest of the state, which accounted for about a third of those studied, had a 3.6 percent positivity rate. There were early variations by race/ethnicity and age as well.”

    So LA and SC counties are much like the rest of NY state — the parts that haven’t been hit hard. 3-4 percent.

    Geneva around 13% per another anti-body based study.

    Study quoted yesterday here was guessing at 23% infection rates in Lombardy and 65% in Bergamo province. Not based on antibody testing.

    • “So LA and SC counties are much like the rest of NY state — the parts that haven’t been hit hard. 3-4 percent.

      Geneva around 13% per another anti-body based study.”

      Look at what those numbers from NYC and Geneva imply for IFR, though …

      • Cuomo has a slide on that. He says 2.7 million infected and 15.5k deaths statewide for IFR of 0.5%. But actually the antibodies represent people who would have tested positive about two weeks ago. If we compare to the deaths as of April 7th (~5.5k) you get 0.2% IFR for that state.

        • Ideally, your “case” or “infection” variable and the “fatality” variable are each ascertained for specific people. Then it’s simple division.

          Nothing simple about doing it from aggregated data with unknown and variable lags between infection, testing and death or recovery.

        • That’s not how that works. Antibodies develop earlier in the course than death would occur. We haven’t yet seen the full extent of deaths from the people infected as of the test date, heck we still may not have seen all the deaths from everyone infected on the diamond princess.

          If we were to use the methodology from the Santa Clara study, we would be looking at total NY deaths through early May as the numerator, with the 2.7m as the denominator. Also, your death count is excluding likely Covid deaths.

        • Ideally, your “case” or “infection” variable and the “fatality” variable are each ascertained for specific people. Then it’s simple division.

          Nothing simple about doing it from aggregated data with unknown and variable lags between infection, testing and death or recovery.

          That’s not how that works. Antibodies develop earlier in the course than death would occur. We haven’t yet seen the full extent of deaths from the people infected as of the test date, heck we still may not have seen all the deaths from everyone infected on the diamond princess.

          If we were to use the methodology from the Santa Clara study, we would be looking at total NY deaths through early May as the numerator, with the 2.7m as the denominator. Also, your death count is excluding likely Covid deaths.

          I agree, but the CFR and IFR values everyone is throwing around are simply current deaths divided by current cases/infections. When I looked it up it sounded like standard practice with the understanding that would not be the “final” value:

          In epidemiology, a case fatality rate (CFR) — sometimes called case fatality risk — is the proportion of deaths from a certain disease compared to the total number of people diagnosed with the disease for a certain period of time. A CFR is conventionally expressed as a percentage and represents a measure of disease severity.[1] CFRs are most often used for diseases with discrete, limited time courses, such as outbreaks of acute infections. A CFR can only be considered final when all the cases have been resolved (either died or recovered). The preliminary CFR, for example, during the course of an outbreak with a high daily increase and long resolution time would be substantially lower than the final CFR.

          https://en.wikipedia.org/wiki/Case_fatality_rate

          To be consistent with this definition, we do have to account for the lag in antibodies though. The antibody results are a proxy for cases a few weeks before.

          But yes, I agree with all the problems this number has. And obviously it is an average that also depends on the population and treatments used, so you can’t just extrapolate from one place to the other.

          Further issues are that they only sampled adults who were not working but going to grocery stores so probably a less at risk population, and the deaths only include people who died in nursing homes or at the hospital.

        • Wikipedia is wikipedia. The relevant definition of CFR isn’t a ratio, it is “the probability that a case dies from the infection”. The ratio of cases to infections is a raw measure that might be used to try to estimate the CFR, but the actually relevant underlying figure is not that and preferably includes people yet to die. It makes little sense therefore to adjust the CFR or the IFR to re-introduce the death-delay bias.

          https://journals.plos.org/plosntds/article?id=10.1371/journal.pntd.0003846

        • Again, no. Case fatality rate is not the same thing as infection fatality rate. It’s right there in your wikpedia link — read it again.

          Most people here are seem to be careful to talk about IFR. I have yet to see a reference to CFR prior to your conflation of the two.

        • Case fatality rate is not the same thing as infection fatality rate.

          What gave you the idea I thought they were the same thing?

          I have yet to see a reference to CFR prior to your conflation of the two.

          You have yet to see a reference to case fatality rate in discussions of covid-19?

          I must be misinterpreting that entire post.

        • For better or worse this is the definition everyone has been using for case fatality rate:

          The fatality rate was defined as number of deaths in persons who tested positive for SARS-CoV-2 divided by number of SARS-CoV-2 cases. The overall fatality rate of persons with confirmed COVID-19 in the Italian population, based on data up to March 17, was 7.2% (1625 deaths/22 512 cases).3 This rate is higher than that observed in other countries2 and may be related to 3 factors.

          https://jamanetwork.com/journals/jama/fullarticle/2763667

          Infection fatality rate is the same thing except you attempt to account for all the unrecorded infections.

        • I must have misread you. And yes, I haven’t seen anyone on this site discussing case fatality rate, because it’s varies only by the number of undetected cases and is thus of no real use.

          In any case, any estimates are attempting to ascertain the final IFR, not the preliminary IFR that’s missing all future deaths from known infections. As I said above, antibodies develop earlier in the course of disease than death would occur — you’re adjusting in the wrong direction.

          The Santa Clara County study took measurements on April 3/4, assumed that was 3 days after it would be measurable by swab (used April 1 as swab infection comparison date), and assumed deaths would take 3 weeks from the April 1 date. The equivalent here would say to capture deaths through early May. I think they’re overestimating the time to death, and thus are incorporating too large a death count in numerator, but it looks directionally correct.

        • I see, here is the antibody paper:

          The under-ascertainment of infections is central for better estimation of the fatality rate from COVID-19.
          Many estimates of fatality rate use a ratio of deaths to lagged cases (because of duration from case
          confirmation to death), with an infections-to-cases ratio in the 1-5-fold range as an estimate of under-
          ascertainment. 3,4,21
          Our study suggests that adjustments for under-ascertainment may need to be much
          higher.

          https://www.medrxiv.org/content/10.1101/2020.04.14.20062463v1

          The sources they cite don’t really support the claim that is standard:

          Ref 3 no lag:

          In summary, the CFR calculated per total cases seems to remain the best tool to express the fatality of the disease, even though it might underestimate this figure in the initial phase of an outbreak.

          https://www.thelancet.com/journals/laninf/article/PIIS1473-3099(20)30246-2/fulltext

          Ref 4 no lag:

          The fatality rate was defined as number of deaths in persons who tested positive for SARS-CoV-2 divided by number of SARS-CoV-2 cases. The overall fatality rate of persons with confirmed COVID-19 in the Italian population, based on data up to March 17, was 7.2% (1625 deaths/22 512 cases).3

          https://jamanetwork.com/journals/jama/fullarticle/2763667

          Ref 21 does use lag:

          We re-estimated mortality rates by dividing the number of deaths on a given day by the number of patients with confirmed COVID-19 infection 14 days before.

          https://www.sciencedirect.com/science/article/pii/S147330992030195X

          So only 1/3 does the adjustment and one of the papers says the adjustment biases the value too. I’m not arguing for what is right or wrong but adjusting for the lag when determining CFR has *not* been standard practice for the covid19 data.

        • This is where you’re conflating CFR and IFR.

          See for example your Lancet link. It notes that CFR has two failings — undercounting the numerator (which lagging the denominator can adjust for) and undercounting the denominator due to undertesting. It seems to be making a case for unlagged numerator on the wish and prayer that the two effects will somewhat cancel each other out.

          No one makes the same claim with respect to IFR, which doesn’t have a typical undercount due to undertesting in the denominator. With IFR, it’s incumbent to lag the denominator because you’re never going to luck into the two errors canceling.

          And what you’re calculating is the IFR. The CFR is, of course, completely unaffected by this new estimate of total infections and ultimately a pretty worthless number.

          In any case, could you explain what you’re driving at here? The only number of any use to anyone is the percentage of people who end up dead as result of catching the virus. If you know it, or have a good estimate of it, it allows you to rough out the number who would with a given number of infections. It’s been the key number that the studies’ authors (Ioannidis, Bhattachary, Bendavid, Sood) have been using to contextualize the disease. What exactly are you trying to do with your number?

        • This is where you’re conflating CFR and IFR.

          One thing at a time.

          I only claim that the custom has been to calculate CFR without adjusting for the lag when it comes to covid-19? I don’t claim that was the right thing to do, but that is how the number everyone has been hearing was arrived at.

          Do you agree or disagree?

        • No, your claim was “you get 0.2% IFR for that state.” That’s all I care about. I have no interest in a bad number (CFR) or the prevalent custom for calculating it.

        • No, your claim was “you get 0.2% IFR for that state.” That’s all I care about. I have no interest in a bad number (CFR) or the prevalent custom for calculating it.

          Yes, if you calculate it the same way the CFR has been calculated and reported to world up until now. Apparently you missed all of that somehow but consistency with what the world has been hearing is why I did the calculation the way I did. I do not claim that is the best way to calculate it, but if you wish to compare to the previous CFR calculations that is what needs to be done.

          You and the authors of that santa clara paper seem to want to change the meaning of the number completely then compare apples to oranges.

        • What value is there to calculating an IFR the way that you have seen CFR calcs done in order to compare it to CFRs? All you are doing is going the long way around to getting an estimate of the undercount ratio. That ratio is much more plainly understood directly — by comparing the serology estimate of infections to the number of confirmed cases.

          You claimed to get a 0.2% IFR for NY. Your method for calculating the IFR is wrong. It also doesn’t comport with, and therefore is not comparable to, the IFRs presented by the authors discussed in this post, in any of their publications. I don’t know what else to tell you.

        • What value is there to calculating an IFR the way that you have seen CFR calcs done in order to compare it to CFRs?

          Because people are saying “it is 50x less deadly than we thought”, etc.

          You seem to be a special case because you said you never heard of anyone mentioning the CFR, so you don’t understand the relevance this has had to the other 99.99% of humanity. But I do agree the original way everyone collectively agreed to refer to as the CFR was flawed.

        • Not what I said. I said most people _here_ seem to be careful to talk about IFR. I have yet to see a reference [here] to CFR prior your conflation of the two.

          Who here is saying it is 50x less deadly than we thought?

          You seem to be anchored on poorly defined mortality rates based on truncated deaths and case counts, which rests on the hope that two errors cancel each other out. Fixing one of the errors doesn’t give useful information about the disease fatality rate.

          The authors of the studies under discussion here have used neither of the (bad) approaches you’re focused on, so I’m not sure why they’re of such importance to you.

          A true estimate of the fatality rate would look at the number of deaths among the cohort infected. We have a cohort estimate (of to-be-determined quality) from the NY study. If you want to talk useful fatality rate estimates, you’re going to need to get an estimate of the deaths from that cohort. Simple as that.

        • Who here is saying it is 50x less deadly than we thought?

          This is all about the paper that claimed that:

          These prevalence estimates represent a range between 48,000 and 81,000 people infected in Santa Clara County by early April, 50-85-fold more than the number of confirmed cases.

          Sorry, but you are on ignore now.

  10. I am not a statistician, just a quantum mechanic by training. I only read their original research note ( not peer reviewed ) and I admit that at first I was biased by John Ioannides’s name on the report, I’ve read some of his work and agree about the lack of discipline in presenting research and statistical validation. However, I will ask the question. Santa Clara is largely white and has a fairly high income per capita, their sampling channels were FB and emails and they selected the right size for each population. I get that. The issue I have is what questions did they ask in selecting the candidates? I have seen questionnaires that are not coherent with each other. So how is that research applicable to NYC or Chicago given the fact that many non white/poor people show morbidity. Yes, I know a little bit about sampling but not enough in the human/medical domain. In my domain particles get along much better than than humans.
    Best Regards

  11. “It’s the best that a bunch of doctors and medical students can do”.

    Yeah, plus others including a public health policy guy (Sood), a financier, an enigmatic tech consultant/operator of a dormant public health non-profit, and a clinical psychiatrist whose last reasearch project was as a public health graduate student (yes, she’s now an MD, but not a researcher). And no indication of author contributions.

    Like other followers of this blog, I saw flashing red lights around selection bias and poststratification, but I think any reasonable non-scientific reader of the study would be skeptical based upon authorship alone.

    So my point is:

    Shouldn’t peer review prevent this sort of agenda-driven “research” from reaching the light of day? Or at the very least ensure that it be held to the rigorous standards that host institutions purport to espouse? I agree 100% about the current state of the peer review process and the open science/open data movement, but IMO there have to be formal checks on what gets out in both written and verbal (i.e. interviews, press conferences, etc.) form prior to release. Both within institutions and the expert communities within and between them. I don’t know what that system looks like, but IMO we are on a very dangerous slippery slope of politicization of research at a time when we desperately need impartiality.

    • Peer review is required by law. Nothing can prevent someone from writing a paper, having it reviewed by whoever they see fit and then releasing it to the media.

      Peer review is required in order to receive the imprimatur of certain institutions, organizations, publishers as being legitimate. But the media at times like this is looking for anybody with some decent credentials after their name who has something provocative to “publish”.

      • Yes, peer review is required of publications presented as evidence in legal proceedings, but is the process itself required by law? I’m dubious.

        And yes, what I’m saying that academic peer review should prevent researchers from going public with something provocative until they have their theoretical and methodological ducks in a row such that, at the very least, armchair statisticians like myself can’t see the holes in their work.

        • luca,

          As typos go, leaving the “not” out of “Peer review is NOT required by law” is a spectacularly bad one. Thanks for the catch. Dang.

        • But these are strange times, it’s true!

          Didn’t Hunter S. Thompson say, “When the going gets weird, the weird turn pro.”? That phrase has been stuck in my mind lately like an ear-worm tune you can’t quite hearing in your head.

  12. All this carping aside, and cross-referencing with other data, the conclusion of this study is probably broadly correct – the incidence of subclinical infection is very high and the IFR is about 0.1%.

  13. One thing I haven’t seen mentioned here is this: why didn’t the authors plug in a Bayesian posterior distribution for the test’s false-positive rate when trying to work out the implications for the percentage of population already infected? Taking point estimates is known to “leak” uncertainty in a multi-stage inferential process, yet I’ve repeatedly seen the same pattern of using point estimates for highly uncertain parameters in other studies related to the pandemic. Most notable was the Imperial College study giving forecasts for the course of the pandemic, which relied on point estimates for both R0 and the IFR when it would have been trivial to modify their Monte Carlo procedure to draw these parameters from an appropriate Bayesian distribution.

    • “Most notable was the Imperial College study giving forecasts for the course of the pandemic, which relied on point estimates for both R0 and the IFR when it would have been trivial to modify their Monte Carlo procedure to draw these parameters from an appropriate Bayesian distribution.”

      The Imperial College London study looked at 4 different scenarios for R0. I don’t see how that’s compatible with parameterizing R0 in a Monte Carlo simulation.

      And did they even use a Monte Carlo procedure? All the outputs in their paper look like multi-scenario deterministic projection to me.

      • > The Imperial College London study looked at 4 different scenarios for R0. I don’t see how that’s compatible with parameterizing R0 in a Monte Carlo simulation.

        That’s just a poor man’s posterior: R0 has these 4 values each with 1/4 probability.

        All you do is say R0 comes from x distribution, and then draw those and do it a few hundred times instead of 4.

        • Have you read the study?

          I mean sure, you could change their approach entirely. But they were looking at 4 more distancing policy scenarios, under each of the 4 R0 scenarios, under 5 different distancing timing on-trigger scenarios and I think 2 different off-trigger scenarios for a total of at least 160 scenarios. For all I know, their population simulation model takes an hour to run. Multiplying the run time by a few 25’s (“a few hundreds” divided by 4) may have been impracticable.

          Further, they would have to find a way to convey those results that is readable by their audience, which I believe was the UK government. Their display approach was heavily geared toward R0 scenarios, and that’s the part I don’t see as compatible with parameterizing R0 in a Monte Carlo simulation.

          From my reading of the study, their main goal was mainly to show the feasibility of the on-trigger/off-trigger methodology, and that its feasibility was not substantially compromised by R0 over a range of reasonable values.

          In any case, the comment I replied to referenced “their Monte Carlo procedure”, which from what I can tell is not a thing that exists.

        • Sure I’ve read it. they needed some script or other to run all those scenarios. and I think they could have published what they had and then run a big weekend long Monte Carlo using very simple priors and published that a few days later.

          the role of uncertainty is very large here. doing point estimates in any of these studies is killing the credibility of any of them.

        • You don’t know how long it takes to run their model, which simulates the behaviors of everyone in the country over the course of 18 months. Generating their results could’ve taken days. Doing it another 50 times over could’ve taken months.

          What they did is fit for purpose. Illustrates over a range of R0s that the on/off cycling can work to keep the resource demands in the range of what’s available. Their policy is by nature robust to uncertainty in the R0 value because it uses observed data to steer the ship, and if the underlying R0 is not as expected the observations keep them on course.

        • One of the models the Imperial College study used was an agent-based model. I would expect the differential-equation-based model to run much faster, which suggests they could have run it repeatedly with different parameter draws.

          And to reiterate, why didn’t the paper being critiqued in this blog post — the one Ioannidis lent his name and credibility to — make the effort to properly propagate uncertainty about the false positive rate of the antibody test? I suspect it has something to do with the reluctance in some quarters to put probability distributions on anything other than future events, or to treat such distributions (e.g. Bayesian priors) as just a regularization trick with no epistemic significance.

        • Nothing you’ve said here supports your assertion that your change to ICL’s methodology would’ve been “trivial”.

        • An agent-based model is evaluated using a Monte Carlo procedure. It’s a series of simulations. To incorporate the parameter uncertainty, all you have to do is draw the uncertain parameters anew at the beginning of each new simulation. That’s a pretty trivial change.

  14. On the all-important specificity issue, if they are able to find a few hundred samples of pre-covid blood instead of 30 samples, then they have a much better argument for the specificity being close to 100%, should the observed false positive rate is still at 0%.

    On the NY antibody test result, they are following the same strategy – the PR comes before any paper or data are published

    • Gotta love this line:

      > The author and biotech investor Peter Kolchinsky tweeted that the “flaws with this study could trick you into thinking that getting shot in the head has a low chance of killing you”.

    • The testing results also may be artificially high because “these are people who were out and about shopping,” Cuomo added. “They were not people who were in their home, they were not people isolated, they were not people who were quarantined who you could argue probably had a lower rate of infection because they wouldn’t come out of the house.”

  15. Something is amiss here. NY state is stay at home. Who is out “shopping”? These studies should release much more details about the protocols used to recruit volunteers. Clearly, high-risk groups are less likely to be found outside. How many sites were there? How were the blood samples stored and where were they taken? So many questions. Are they unleashing all this PR before they release any information to get in front of us “crowd peer reviewers”?

  16. I suggest letting Joe Rogan decide whether the research is up to par or not. Joe is not an academic statistician but he is well-endowed with common man sense. Along with Eddie Bravo, he can explain what it means to the average no-nonsense American MMA fan.

  17. This is responding to “anonymous” above. Specifically, there’s this line about how X really high % of mortality (95+%) is in people with 1 or more underlying conditions. With the implication that you – dear reader – do not fall into this category as long as you are an “average person”. And probably, neither would most people you love or care about.

    Well I have news for you!

    What are the underlying conditions that are associated with severe outcomes in Covid-19? Age, obesity, hypertension, hyperglycemia (diabetes), general heart disease and immune compromise. Basically, the metabolic syndrome + a few obvious other things.

    Guess what? Most adult Americans are on the ‘metabolic syndrome spectrum’. The large majority of American adults are overweight and obesity prevalence is around 40%. Think about that – 40%. The large majority have at least some degree of insulin resistance and hence are walking around with sub-optimal blood sugars (hyperglycemia) much of the time, in addition to all the other biochemical havoc that insulin resistance wreaks. Downstream of insulin resistance, elevated uric acid, and chronic inflammation we see lots of hypertension and dyslipidemia. Most American adults over 50 have some degree of atherosclerotic disease, even if they are asymptomatic.

    Like some other folks have been pointing out too- the hospitalization rate is pretty high, even for young people, and there are plenty of stories of apparently very healthy young people having bad outcomes with this thing. Going into respiratory distress is a traumatic experience even if you don’t die.

    In short, this narrative about ‘underlying conditions’ elides the fact that the conditions in question are actually quite common in our population, and almost surely impact many people you love and care about. If you are over the age of 30 and have not taken great care of yourself (or been genetically blessed), you probably have some degree of metabolic syndrome too. Don’t kid yourself. Even if you don’t – or you have things well-managed- you don’t have to travel far and wide outside the Ivory Tower to realize that most places in this country aren’t doing so well, and are chock full of vulnerable people.

    Sermon over :)

    • Thanks Chris, now I don’t have to go look up those numbers, I have been thinking a similar thing recently… Basically, the risk factors we know of are not that specific. p(Risk Factor | Bad Outcome) is clearly pretty high (most of the people with bad outcomes are people with risk factors) but p(Bad Outcome) is low, so p(Bad Outcome | Risk Factor) = p(Bad Outcome) p(Risk Factor | Bad Outcome) / P(Risk Factor)

      if P(Risk Factor) ~ 0.5 and p(Bad Outcome) ~ .1 and p(Risk Factor | Bad Outcome) ~ 0.8 then p(Bad Outcome | Risk Factor) ~ .1*.8/.5 = .16

      since .16 *.5 + x *.5 = .1, x = p(Bad Outcome | No Risk factor) = .04

      so Risk Factor is not particularly predictive of anything

      If you want to “keep everyone with risk factors sequestered” it’s more or less 50% of the population and yet you’re doing hardly much to reduce the bad outcomes, particularly if you’re letting it rip through this “lower risk” group, which is only a bit lower risk. Sure you cut bad outcomes (I’m including deaths, hospitalizations, etc) down from 32M people to 8M. But then you’re going to have to isolate the crap out of those “high risk people” = 50% of the population, or they really don’t get much protection.

      Basically it’s a no-go

    • Been thinking this for a long time too.

      So what they should do is look at extreme measures of each of these risk factors. If they are genuine risk factors, then extreme obesity, diabetes, etc. should be very good predictors of a negative outcome, and would actually be useful information. We already know that extreme age is, in fact, a very good predictor.

    • Chris,

      Foremost, I don’t like in an Ivory Tower (nor would I want to, being afraid of heights); more the small and messy apartment of a PhD student. Nor do I want to deride those who are obese or otherwise; not my intent whatsoever.

      > Guess what? Most adult Americans are on the ‘metabolic syndrome spectrum’. The large majority of American adults are overweight and obesity prevalence is around 40%

      My point by highlighting risk-factors was to elucidate how using a bulk/gross value for fatality-rate absent of risk is not accurate. Even if those with risk-factors are 40% of the population, that is not 100%. It is a sub-population. You have to treat it as such statistically.

      > Like some other folks have been pointing out too- the hospitalization rate is pretty high, even for young people, and there are plenty of stories of apparently very healthy young people having bad outcomes with this thing. Going into respiratory distress is a traumatic experience even if you don’t die.

      No rigorous data available (that I can find) on hospitalization of young people (e.g. stratified by decade). Also, infection-hospitalization-rate, like infection-fatality-rate, is not a accurate measure unless we know the true value of those infected. Say we are underestimating those infected by 5, 10, or 25 fold? These would have dramatic effects on infection-hospitalization-rate.

      Some healthy people will have bad outcomes. That goes for any disease, including common ones like common cold (pneumonia, myocarditis). As I have stated, the key is frequency. And current anecdotal evidence suggests these young people are outliers. But without any detailed data on the topic, we really don’t know.

      > In short, this narrative about ‘underlying conditions’ elides the fact that the conditions in question are actually quite common in our population, and almost surely impact many people you love and care about. If you are over the age of 30 and have not taken great care of yourself (or been genetically blessed), you probably have some degree of metabolic syndrome too. Don’t kid yourself. Even if you don’t – or you have things well-managed- you don’t have to travel far and wide outside the Ivory Tower to realize that most places in this country aren’t doing so well, and are chock full of vulnerable people.

      Fair. The US has a epidemic of obesity. It has for years. It has cost us trillions and ruined the livelihood of millions. (As an aside: I wonder if we will see a call to action on this like we have for COVID-19. Mitigating the obesity in the US would be a very positively-leverage bet long-term.)

      And it makes ~40% of our nation more likely to experience severe outcomes from ANY respiratory disease, not just COVID-19. So we must consider this disease when evaluating COVID-19 and its fatality. How much is inherent to COVID-19 being an aggressive, nasty virus and how much is it being opportunistic and preying on those who are vulnerable to any illness?

      Daniel,

      One caution: Probability of a bad outcome depends on physiology (e.g., healthy or unhealthy). Therefore, shouldn’t p(Bad Outcome) for healthy people and p(Bad Outcome) for those with risk-factors being different values?

      • > One caution: Probability of a bad outcome depends on physiology (e.g., healthy or unhealthy). Therefore, shouldn’t p(Bad Outcome) for healthy people and p(Bad Outcome) for those with risk-factors being different values?

        That’s what the conditional probability I calculated were about… The point being that the difference wasn’t huge.

        P(Bad Outcome | Risk Factor) ~ 4x as big as P(Bad Outcome | No Risk Factor)

        which is a big deal. If you can take all the risk factor people and shoot them into space where they’re held in stasis, and then start back up the virus, then your bad outcomes will be 1/4 as large as they would have been…. They’re still a few percent of the population though.

        • *face palm* I misread you comment — apologies. It makes sense.

          Still, I caution against statements: “so Risk Factor is not particularly predictive of anything”.

          The statement does not hold in practical terms.

          Someone who is obese, hypertensive, and/or diabetic is much more likely to suffer severe outcome from any respiratory illness. Sure, the conditional probabilities workout that “the difference wasn’t huge”, but it makes little physical sense (given available data) that a healthy person (say age 25 who is athletic and eats/sleeps well) is only slightly less likely for an unhealthy person (say an age 60 who is obese, diabetic, and hypertensive). If the difference was small, we’d have seen many more young, healthy people suffer severe short-term outcomes; and that just has not occurred. So a factor or additional probability is likely missing.

        • There’s so much focusing on the deaths that maybe you didn’t read in the news how the hospitals filled up with younger people

          https://www.weforum.org/agenda/2020/03/coronavirus-young-people-hospitalized-covid-19-chart/

          I’m including hospital visits and etc in “bad outcomes”. My basic prior is that some fixed fraction of the population gets “really sick” across the board in adults, but that younger people survive it better.

        • Hospitalization is correlated to age.
          https://www.cdc.gov/coronavirus/2019-ncov/covid-data/covidview/index.html?CDC_AA_refVal=https%3A%2F%2Fwww.cdc.gov%2Fcoronavirus%2F2019-ncov%2Fcovid-data%2Fcovidview.html#hospitalizations

          I’m not really clear on their survey method, whether it’s cases (and each case only counted once), or if it’s cases on a certain day, in which case longer hospital stays could get could get counted multiple times and shorter stays not at all.

        • Yeah, that makes sense. But the differences are around a factor of 5 between say 18-49 group and 65+ which noticeable but not super dramatic (like say the death rate for cases, which is ~ 0.4% for 40 year olds in Italy, and 20.2 percent for 80+, a factor of 40-ish).

          I probably shouldn’t have said fixed fraction when I meant something more like a slower changing fraction than for CFR. Basically while bad outcomes increase with age, also the severity of those bad outcomes increases with age, so that the risk of the most severe outcomes increases at a much higher rate.

        • > I’m including hospital visits and etc in “bad outcomes”.

          Sure. But this prior is extremely sensitive to the total number of known infections. So I’d caution against placing too much faith in it without knowing the true number of infections. For example: In NYC as of April 22, 5179 people age 18-44 were ever hospitalized with COVID-19, with total cases being 52380. Now factor in the results of the serolgical study from NYC (assuming its somewhat accurate) that showed we are underestimating cases in NYC by ~12x. Therefore, the true p(Bad Outcome)/12 = 0.1/12 = 0.008. So all bad outcomes would reduce which makes current rates of hospitalization and death seem much less profound.

          And the 4x difference between P(Bad Outcome | Risk Factor) and P(Bad Outcome | No Risk Factor) is a direct function of your assuming P(Risk Factor) = 0.5. There are many reasons why it would be lower and why a single value does not represent it well. Again, we just don’t know — so applying such statistics is nothing beyond an exercise in thought.

        • > I’m including hospital visits and etc in “bad outcomes”.

          Sure. But this prior is extremely sensitive to the total number of known infections. So I’d caution against placing too much faith in it without knowing the true number of infections. For example: In NYC as of April 22, 5179 people age 18-44 were ever hospitalized with COVID-19, with total cases being 52380. Now factor in the results of the serolgical study from NYC (assuming its somewhat accurate) that showed we are underestimating cases in NYC by ~12x. Therefore, the true p(Bad Outcome)/12 = 0.1/12 = 0.008. So all bad outcomes would reduce which makes current rates of hospitalization and death seem much less profound.

          And the 4x difference between P(Bad Outcome | Risk Factor) and P(Bad Outcome | No Risk Factor) is a direct function of your assuming P(Risk Factor) = 0.5. There are many practical reasons why it would be lower and why a single value does not represent it well. Again, we just don’t know — so applying such statistics is nothing beyond an exercise in thought.

  18. Sood loves “the crowd” when we are willing to listen uncritically to him reporting his findings on the news, but he turns on us as soon as we, the crowd, start acting like scientists. Something tells me that when Sood coins the phrase “crowd peer review” he means it as an oxymoron.

    With apologies to Aaron Sorkin: “I have neither the time nor the inclination to explain myself to a man who rises and sleeps under the blanket of the very research that I provide, and then questions the manner in which I provide it! I would rather you just said “thank you” and went on your way.”

    • Lt. Kaffee : Did you screw up the analysis?

      Col. Jessup : I did the job I…

      Lt. Kaffee : [interupts him] *Did you screw up the analysis?*

      Col. Jessup : *You’re God damn right I did!*

  19. I guess a charitable reading of this is that crowd review leads to mounds of uninformed criticism, which can be time-consuming to rebut, and if not rebutted could persuade an uniformed audience that the study is flawed.

  20. I think the beauty of being a software engineer is that I’ve been told I’m wrong ten times before my first cup of coffee usually. (Pesky compilers, always picking those nits.) Code reviews are usually a several-times-a-day thing for most engineers these days unless you’re working on something larger. Even so, you should have someone to bounce things off of earlier on in the process.

    I genuinely can’t imagine a career where I go a whole day without hearing I’m wrong.

    • I spent about the first third of my career as a compiler writer. Your comment brings a maniacal look to my face as I gleefully rub my hands together, memories of implementing language standards with a totally anal thoroughness running through my mind … and in a couple of cases I helped write those nitpicky standards myself!

Leave a Reply to Garnett Cancel reply

Your email address will not be published. Required fields are marked *