Skip to content
 

“Sorry, there is no peer review to display for this article.” Huh? Whassup, BMJ?

OK, this is weird. Yesterday we reported on an article with questionable statistical analysis published in the British Medical Journal. This one’s different from some other examples we’ve discussed recently (Surgisphere and Stanford) in that the author list of this recent article includes several statisticians.

One way to get a handle on this situation is too look at the reviews of the article. Fortunately, as Keith O’Rourke points out, the journal has an open peer review policy:

For research papers The BMJ has fully open peer review. This means that accepted research papers submitted from September 2014 onwards usually have their prepublication history posted alongside them on thebmj.com.

This prepublication history comprises all previous versions of the manuscript, the study protocol (submitting the protocol is mandatory for all clinical trials and encouraged for all other studies at The BMJ), the report from the manuscript committee meeting, the reviewers’ comments, and the authors’ responses to all the comments from reviewers and editors.

That’s great! There are some conditions:

In rare instances we determine after careful consideration that we should not make certain portions of the prepublication record publicly available. For example, in cases of stigmatised illnesses we seek to protect the confidentiality of reviewers who have these illnesses. In other instances there may be legal or regulatory considerations that make it inadvisable or impermissible to make available certain parts of the prepublication record.

In this case, though, what we have are statistical analyses of public data, so there should be nothing stopping us from seeing the entire record.

But then there’s this:

Sorry, there is no peer review to display for this article

Whaaaaaa?

The closest thing to a review is this journal editorial, “Lockdown-type measures look effective against covid-19,” which reports some concerns about the data (“subject to variable quality, accuracy, and inconsistent testing practices”) but also makes the statement, “This study is as good as it could be given the data available,” which seems highly debatable given our discussion from yesterday.

Why is there no peer review for the original article?

As regular readers know, I think peer review is overrated. But if a journal is supposed to do open peer review, we should get to see it, no?

P.S. BMJ has posted the peer reviews! Here they are. The reviews have lots of questions about data quality:

We think the biggest limitation of the data is that it does not take into account the heterogeneity of the response within a country . . . we wondered about the data from some countries in particular. Most countries show a lot of variability with the dots all over the place. . . .

But I saw no serious concerns raised with the statistical modeling, though. I think this may have partly been a problem of trust, as one reviewer writes:

The background of the research team is strong, consisting of experts in epidemiology, public health and statistics from distinguished tertiary institutions.

And another reviewer writes:

I think the authors did such a good job with the flawed data they have to work with . . .

Also, none of the reviewers commented on that Canada graph.

Overall, this document shows the strengths and weaknesses of traditional peer review. The strength is that the peer reviewers raise lots of specific concerns and are interested in the reasoning in the article, conditional on its basic approach being valid. The weaknesses is that the reviewers trust their peers—the authors of the article—and don’t ever consider the possibility that all the modeling being done there is a hot steaming mess.

53 Comments

  1. Anoneuoid says:

    Everyone is still assuming slowing the spread is a good thing. Maybe, but that is an assumption.

    I could also see everyone young and healthy going to covid parties and taking a couple weeks off to get immunity at the same time as a way to interrupt the transmission. Instead, this is going to drag on and on and end up with something like the TSA everywhere checking people.

    • Adede says:

      Have you been following the news? That’s the herd immunity strategy the UK tried. It didn’t go well.

    • I agree it’s an assumption, but with fatality rates around on the order of 1% and serious consequences like stroke, kidney damage, lung damage, etc on the order of several percent, its a pretty good assumption. Particularly given that the low-symptom people don’t seem to produce lasting immunity, and can get it again and seem to have risk of antibody dependent enhancement.

      • Also, that the low-risk people (say 10 years old to 25 years old) also seem to be the most likely to be asymptomatic, and super-spreaders, so basically having a bunch of them go out and actively get this disease puts many many other older high-risk people at risk.

      • Anoneuoid says:

        I dont think weve seen much evidence for reinfection yet, it might not have the same symptoms and get missed though. But short lived immunity is a reason to *not* flatten the curve.

        • AllanC says:

          Can you walk me through the logic here? I do not follow how a lack of short term immunity means we should not take active steps to reduce the number of infected patients at any given time step.

          As far as I understand it, measures to reduce the number of infected individuals alleviates the burden on hospitals and other services; allowing them to cope with the influx of patients. This is true whether there is long term immunity or not.

          • Anoneuoid says:

            Its just the idea of herd immunity. If immunity is shortlived and everyone is gradually getting it, then there will be a constant flow of susceptibles as the immunity wanes and they can get reinfected again. If everyone gets it at once then everyone is immune at the same time and the virus will die out since there are no susceptibles.

            I’d guess 40% of the US pop is at pretty much no risk from covid, that 40% being immune at the same time for a few months may be sufficient to interrupt transmission.

            • By FAR the better way to make this happen is via vaccine. So it’s worth holding off on this strategy until at least 1.5 years from initial onset of pandemic to give the best chance of a vaccine enabling us to kill off the virus from at least first and second world countries via vaccine.

              Doing this now would be foolish IMHO. I honestly believe with all the effort going into it, one of the vaccines will be effective enough to control the pandemic before July next year, maybe even before Jan.

              • Anoneuoid says:

                How is it better let people continue to die for 1.5 years if you could just end it in a month or two for practically free?

                And I expect the vaccine is going to have all the same problems of not being very effective or safe for 60% of the population (elderly, obese, etc) and waning immunity. A vaccine basically mimics mild illness.

                I honestly believe with all the effort going into it, one of the vaccines will be effective enough to control the pandemic before July next year, maybe even before Jan.

                No one has even published on ADE where it was seen for SARS yet, they just keep saying “we didnt see ADE”. Yea, you only looked in young healthy animals/volunteers challenged with homologous virus before waning… and some of that just takes time (eg, waning). Parallelization does not speed it up.

              • The idea that we’re going to “end it for free” is pretty much exclusively the case if you can arrange for *all* of the following:

                1) You can target around 40% of people who are at low risk.
                2) Those people also turn out to be almost all of the superspreaders.
                3) Even though you get the super spreaders all sick at once, you somehow keep them from superspreading to all the older and higher risk people.

                Otherwise you’ll see overwhelming hospital use that make NYC look like a walk in the park, you’ll see 2M other people dead in 2 months as collateral damage, and it won’t even get the pandemic under control because you’ll still have enough residual virus to continue infecting people and 6 months out the immunity wears off and we are back where we started except with all the losses we could have avoided.

                It’s just kind of a fantasy in my opinion. What we need right now is to suppress the infection rate a lot more than we are, massive quantities of pooled testing and quarantine, and wait until we can arrange for most everyone between the ages of say 10 and 50 to be vaccinated. At that point we’ll get all the advantages you talked about and none of the death suffering and disability. Even if it’s not very safe for the elderly due to ADE, the herd immunity factor from having most younger people vaccinated would be dramatic.

                Intentionally infecting people is the opposite of a good idea IMHO.

              • Anoneuoid says:

                Pretty sure superspreading is more about environment, not innate. Eg coughing next to an air intake in a shared building or on public transportation. Maybe how much saliva is produced or something nfluences it somehow too though.

                I also havent heard of anywhere having hospital issues like in nyc, wuhan, and lombardy since they stopped putting everyone on ventilators right away. Which is like the most dangerous, expensive, and resource intensive medical intervention you can do. From talking to a respiratory therapist they are better at dealing with the patients now in other ways too (inhaled corticosteroids, etc). He said its just not a big deal at his hospital anymore. And of course you know my opinion on vitamin c and hbot.

                Finally, if people *know* they were just exposed they are much more likely to be careful for a week or two than just in general. Having an excuse to sit inside playing video games and watching movies isnt that bad.

              • confused says:

                I agree that intentional infection would be a very bad idea.

                But I really do not think 2M deaths in the US is plausible no matter what we do. Nursing homes were hard hit early; combined with better knowledge about the disease, death rates in the US as a whole seem like they must be lower than seen in NYC in March/April, so 2M deaths would require something like 100% infection – maybe more.

                (Yeah, if hospitals were really seriously overwhelmed large-scale, that would change. But that doesn’t seem likely. Houston and Phoenix looked scary 2 weeks ago but have stabilized & maybe even begun to drop. US hospital systems overall seem much more resilient/surge-capable than assumed in March.)

              • Anoneuoid says:

                But I really do not think 2M deaths in the US is plausible no matter what we do.

                I think it is guaranteed if you wait long enough just due to the “died with covid” definition.

              • confused, if 40% of America = 120M people all got sick intentionally tomorrow, you don’t think hospitals would be overwhelmed?

              • Specifically, imagine that we did an absolutely EXCELLENT job of identifying low risk people. Say we had a model that predicted correctly 90% of the time. We encourage everyone to put in their demographics and blood type and etc and we get a risk score, and then we tell everyone with a risk score of level 1 should go try to get infected this week… and somehow people trust this thing and do it. Suppose 40% of people have a risk score of 1

                so then, 302M * 0.4 * 0.1 of these people turn out to be at risk of adverse events… that’s 12.8 M people all sick in say a 2 week period needing hospitalization.

                With that level of demand, I can’t imagine that’d lead to less than say 10% of them dying, which leaves 1.3 M dead in a month or so.

                And that’s ignoring the potential for this group to spread to groups with higher risk.

                actively encouraging people to get COVID is EXTREMELY BAD, do NOT do that please Anoneuoid, even with EXCELLENT risk models it’d be a disaster.

              • Anoneuoid says:

                actively encouraging people to get COVID is EXTREMELY BAD, do NOT do that please Anoneuoid, even with EXCELLENT risk models it’d be a disaster.

                Look around you, it is already a disaster. Continuing that for another year and a half is also EXTREMELY BAD.

              • Superspreading is partly about environment but there are very important innate components.

                I have knowledge of a lab that is working on saliva based testing which said that they experienced a range of concentrations between about 1e4 and 1e9 virions per ml, with typical closer to say 1e5 or 1e6. That was in a sample of 7 to 10 people. If 10% of people produce 1000x as much virus as others in the upper respiratory and vocal tract then they are going to drive superspreading much more.

                Also there’s a reason that one commonly cited superspreading event occurred at a choir practice. Some people are just vocally different from others. My sister in law for example, you ALWAYS pull the phone away from your ear when she answers the phone. She’s just LOUD (she was a choir major in music school).

                So, some of it is environment, airflow, etc, but a big part is viral concentration and vocalization.

              • confused says:

                >>confused, if 40% of America = 120M people all got sick intentionally tomorrow, you don’t think hospitals would be overwhelmed?

                In a theoretical world where literally 120M people got COVID in one day, yes.

                But that wouldn’t happen even in a hypothetical where the US strategy was “protect the elderly and vulnerable, encourage young healthy people to throw big parties with COVID-positive people attending”.

                People don’t just shift their approach that quickly (“wait, we were supposed to stay home just last week?”), not all young healthy people would be OK with this (I’m 30 and no pre-existing conditions, and I wouldn’t be), and not everyone exposed becomes sick the first time they are exposed.

                >>With that level of demand, I can’t imagine that’d lead to less than say 10% of them dying, which leaves 1.3 M dead in a month or so.

                I think this is far too pessimistic. 1.3M deaths out of 120M infections for *specifically young healthy people*? That seems too high even if zero of them got medical care. (Not everyone who goes to the hospital for COVID would necessarily die otherwise, after all.)

                >>actively encouraging people to get COVID is EXTREMELY BAD

                Yes. Because with tons of younger people infected, some won’t be careful enough, and older more vulnerable people will get it. And because of the possibility of unknown long-term effects (which by definition we can’t judge severity of). And general ethics of it (I think we will have a vaccine relatively soon – more like 6 months than a year and a half.)

                But talking about 2M deaths in the US just seems beyond the range of reasonable possibility.

              • confused says:

                >>I think it is guaranteed if you wait long enough just due to the “died with covid” definition.

                Well, hypothetically if the disease continues to circulate at endemic levels after the end of the pandemic, and if we continued to test widely for COVID and to count any death with a COVID-positive test as a COVID death, yeah eventually we would.

                But after the pandemic is over we won’t do that.

                And we are 4 to 6 months in (first US case January, first US death February, really blew up in March) with 140k recorded deaths. The pandemic is not going to last 12 times that long (4 to 6 years). It would die down to endemic levels naturally well before that, even if we don’t get a vaccine, which IMO is almost guaranteed way before that.

        • Navigator says:

          What curve is it?

          If we are thinking hospitalizations or mortality curves, those have been flat all along (hosp.) or have gone down dramatically (mortality), if we look at weekly, not cumulative data:

          https://gis.cdc.gov/grasp/COVIDNet/COVID19_3.html

          • Anoneuoid says:

            The curve that matters for herd immunity is the infection curve, which is estimated from the number of people who have tested positive.

            • Navigator says:

              Nobody knows what that curve looks like, as testing was very sparse and selective in the beginning of all this. We can assume it had a huge ‘bump’ early before any measures were taken stateside, but in reality many were infected (symptomatic or not).

              Due to increased testing, the rates/cases we see now resemble some sort of random sampling more, although far from ideal.

              The duration of a ‘positive’ status in those who tested is something nobody is adjusting for, as we have no clue how long people stay infected and what part of that period they are infecting other people (too many individual differences). Let alone those who are asymptomatic and infect others vs. asymptomatic but not infectious (apparently both are the options).

              We keep hearing that herd immunity may not be long-lasting, but I haven’t seen any good evidence on how that was determined. Sounds like a lot of ‘hunches’ and guesses based on similar viral infections, but applied to covid (which may be correct, for all we know).

              • Anoneuoid says:

                we have no clue how long people stay infected and what part of that period they are infecting other people (too many individual differences).

                There were a few papers on this. Infectious virus is only cultural for about a week, even if PCR is still positive for another week or two. Also there is a big peak in infectivity for about two days right when symptoms start. Here is one: https://www.nature.com/articles/s41591-020-0869-5

                We keep hearing that herd immunity may not be long-lasting, but I haven’t seen any good evidence on how that was determined.

                Because in people who have asymptomatic/mild cases they saw antibodies start to wane after a few months. Same was seen for SARS. Eg: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7362084/

                T-cell mediated immunity lasted longer for SARS and that is probably the case here too. So reinfection will probably result in a modified course of illness.

              • confused says:

                T-cells are probably a major factor. And there is now a suggestion that neutralizing antibodies may not wane so quickly as total antibodies. So the immunity situation may be better than it seems.

                I don’t think “natural immunity” would eradicate the virus, but it would probably drop it from epidemic/pandemic levels to endemic levels. I hope we will have a vaccine well before we get there, though.

              • Anoneuoid says:

                As data start to accumulate on the detection and characterization of SARS-CoV-2 T cell responses in humans, a surprising finding has been reported: lymphocytes from 20–50% of unexposed donors display significant reactivity to SARS-CoV-2 antigen peptide pools1,2,3,4.

                https://www.nature.com/articles/s41577-020-0389-z

    • jim says:

      “Everyone is still assuming slowing the spread is a good thing. Maybe, but that is an assumption.”

      Wow.

      It **is** an assumption. And if you’re trying not to die or kill other people, it’s a pretty good one.

      Stopping the transmission could eliminate the virus altogether, aside from preventing thousands of people from dying even if the virus isn’t ultimately eliminated. The “get infected” strategy is just Russian Roulette with a few more chambers.

      • Anoneuoid says:

        If you would have read more carefully there are reasons to think more people will die by dragging it out. And that is not even considering the economic and social fallout.

        So it turns out you are the evil one who wishes death on others with your half-baked theories. /s

        • I don’t think anyone called you evil here. I believe your strategy is misguided but I don’t think you are intentionally trying to create more harm.

          The assumption I have is that much of the effect you are hoping for from the strategy of intentional infection can be had within 6 months using even less than fully effective vaccines. even at the current levels of infections we will see less death and disability using the wait for a vaccine strategy.

          IMHO the economic fallout may indirectly force a reckoning with the US underlying economic problems such as poverty traps and regulatory capture. also the lessons learned from work at home may well produce dramatically improved working conditions and reduced pressure on housing prices… so I don’t see the economics as a compelling reason to accept sudden enormous infection levels. also it seems like the current situation is at least partially due to people intentionally infecting. there are multiple reports of such things, and most people hospitalized in LA county are under age 40

          • Anoneuoid says:

            The economic fallout is going to be widespread closing of small business, bailouts of politically connected corporations, negative interest rates, no more cash to prevent bank runs due to the negative rates, mass unemployment and automation since its too risky to hire someone, an exodus from urban areas to more rural.

          • Anoneuoid says:

            Demolition man is probably the most accurate portrayal of what’s coming. A very rich corporate elite with an underclass in constant revolt. No one will touch each other, there’s no toilet paper, mass surveillance will be used to fine you digital credits for saying the wrong things, etc.

      • Anoneuoid says:

        And also. Not a single report of vitamin C levels in a covid patient has been published yet.

        Not a single one.

  2. Dale Lehman says:

    As terrible as this paper is, they did provide the data. Given the choice between the data and the peer reviews, I’ll take the data. Of course, there is no reason we should have to choose.

  3. confused says:

    I agree this would be hugely risky, so I’m not really disagreeing with your comment overall. But I do have some questions:

    >>serious consequences like stroke, kidney damage, lung damage, etc on the order of several percent

    Do you have a source for this? (Not challenging you, just hoping you have seen actual data, which I have been unable to find.)

    I have seen stroke as 1.6% of *hospitalized* COVID patients (not all positives).

    I would really love to see incidence numbers for other forms of damage.

    >>Particularly given that the low-symptom people don’t seem to produce lasting immunity, and can get it again

    Really? Given how many mild/asymptomatic cases there have been (probably something like 20 million in the US) I would expect lots of reinfections, not just one or two cases reported, if this was usual.

    I am sure there will be some (non-zero) reinfections because people’s immune systems differ.

    But if loss of immunity was common enough to matter in the near-term, I think we’d see more issues already. Since people differ, if the average duration of immunity was (say) 4 months we’d see plenty of people getting reinfected at 2-3 months.

    And we are probably 4 months past NYC’s peak infections. (Peak deaths were early April so peak infections probably mid-March).

    >> and seem to have risk of antibody dependent enhancement.

    Do we have evidence of this? I’ve seen it mentioned as *possible* but not any actual evidence seen for this specific virus.

    • confused says:

      Oops – this was supposed to be a response to Daniel Lakeland above.

    • Yes, the data is not quite so clear. I think the 1.6% of hospitalized patients is not right. I think it’s closer to 1.6% of infections. I have a friend who says he personally knows 3 people aged mid 40’s who’ve had strokes in the last few months. So I was inclined to believe that order of magnitude. It seems hard for me to believe that a disease that can kill about 5% of ascertained cases in the US and is estimated to be around 1% IFR would somehow only have say 10% of infected patients hospitalized and 1.5% of those stroke… that’d be a stroke rate of 0.15% for a disease with a ~1% mortality rate ? You could argue most of the strokes kill, but that may well be because most of the hospitalized so far have been older.

      I’d like to have better data, no question, but it’s not like I can point you to a reliable source of data. I think our data collection has been absolutely abysmal. I wonder if data is available in say Sweden where they must have much better data collection.

      >Given how many mild/asymptomatic cases there have been (probably something like 20 million in the US) I would expect lots of reinfections, not just one or two cases reported, if this was usual.

      My impression is they’re looking at immunity waning in the period around 3 to 5 months or so, so most people who were infected in say April are still probably somewhat immune, but by Sept wouldn’t necessarily be. This is consistent with info available about SARS1.

      The antibody enhancement is also based on experience with SARS 1 so it’s theoretical but with an underlying reason for that theory.

      • confused says:

        >>It seems hard for me to believe that a disease that can kill about 5% of ascertained cases in the US and is estimated to be around 1% IFR would somehow only have say 10% of infected patients hospitalized

        That would mean 1/10 of hospitalized cases are fatal – why does that seem like too few hospitalizations?

        Data from China in February was suggesting ~16-20% hospitalized, ~2-3% fatal – so that’s a comparable ratio, actually fewer hospitalizations per death (probably because care has improved somewhat).

        Source for the 1.6% of hospitalizations
        https://news.weill.cornell.edu/news/2020/07/strokes-occur-more-frequently-in-patients-with-covid-19-compared-to-flu-but-overall

        The article links to a JAMA paper, so probably reasonably reliable, at least by the standards of fast-turnaround science…

        >>that’d be a stroke rate of 0.15% for a disease with a ~1% mortality rate ? You could argue most of the strokes kill, but that may well be because most of the hospitalized so far have been older.

        I think it’s because most deaths are respiratory failure, with no stroke involved

        >>My impression is they’re looking at immunity waning in the period around 3 to 5 months or so, so most people who were infected in say April are still probably somewhat immune, but by Sept wouldn’t necessarily be.

        Sure but people’s immune systems vary. If the *usual* duration of immunity is 3-5 months, then there should be a significant fraction of people at 2-3 months… and there were tons of people infected more than 3 months ago. We should already be seeing a whole lot of reinfections (not just a couple of semi-anecdotal cases) if that were true.

        >>This is consistent with info available about SARS1.

        Maybe… but I’ve also seen a paper abstract saying T-cell immunity from SARS1 is still functional now (17 years later). Antibodies waning doesn’t necessarily mean protection goes away.

      • confused says:

        also, I really doubt IFR in the US *now* is ~1%. New York in March/April sure. But not the current places where infections are happening (different age distributions/fewer nursing home outbreaks, more has been learned about care).

        CDC estimates 0.65% (range 0.5%-0.8%). And I think that is based off serology.

        And if T-cells etc. mean some people beat the infection without developing serology-detectable antibodies (there is some tentative evidence for that) then IFR estimates are generally too high.

  4. James Watson says:

    Andrew: in defense of the BMJ, I think that there is always a delay between publishing of the paper and publishing of the peer review (I have no idea why). We had a paper in the BMJ recently and it was peer reviewed but the review is still not showing up on the website. If you look at papers that are a few months old the peer review shows up. Maybe they edit the reviews for clarity?

Leave a Reply to Andrew Althouse