Rapid prepublication peer review

The following came in the email last week from Gordon Shotwell:

You posted about an earlier pilot trial of calcifidiol, so I wanted to send you this larger study. The randomization is a bit funky and if you were interested it would be great to hear what sorts of inferences we can make about this data.

And here’s all the vitamin D evidence to date.

I had a few spare minutes so I clicked on the first link, and this is what I saw:

That was fast!

P.S. Shotwell adds:

The whole thing turned out to be a huge mess and created a lot of outcry. Here’s a copy of the paper if you’re interested. My [Shotwell’s] charitable read of what happened is this:

1) Patients were allocated to wards using a more or less random process (you just got the next available bed)
2) All the wards had more or less the same treatment
3) The authors gave 5/8 wards vitamin D
4) Since they thought of the initial patient allocation as being random, they thought they could analyze the data as though it were patient-randomized instead of ward randomized

I [Shotwell] think it’s a good example of why multilevel models are a good default. Like you might think that your patient allocation is random, but because they didn’t prove that with data it lost all kinds of credibility and potentially delayed adoption of a good therapy. A MLM here would have either alerted the authors that their assumptions were false, or proved to readers that they were true.

Oooh—I love the connection to multilevel modeling.

Shotwell continues:

Those notes are basically what I think happened and aren’t based on any first-hand knowledge. They may be overly charitable because I almost certainly have a bias in favour of vitamin D study authors. It’s probably worth including a link to their explanation as well as some criticism on pubpeer which are available here.

There are plenty of other commentators who basically think the study is fraudulent and unethically presented. I take the view that it’s probably some combination of unwarranted statistical assumptions and writing an academic paper in their second language. This is a good thread by Perry Wilson about this which I [Shotwell] thought was fair.

28 thoughts on “Rapid prepublication peer review

  1. I think you may have mentioned something on this before, but what do you think of removing papers altogether? I was a bit disappointed not to be able to see it. What about a warning label of some kind for papers rather than a retraction? That way, everyone could see what the error was and learn from it. If it was a small error, then it could have edited and corrected versions, like the preprint servers. If it was just completely a wash, then maybe everyone could learn from it and it could feature a ‘warning do not cite’ label. When it simply vanishes, it seems like lost information (not from the perspective of information in the study, but information about the study itself).

    • Jd:

      Yeah. I agree that it’s weird to take it down entirely. It seems like they could just keep it up and add a heavy RETRACTED watermark to every page. Kinda like what JPSP should’ve done with that ESP paper…

  2. But this paper is still apparently fine where they cut the planned sample size by 60% when vitamin C was showing the benefit they designed the study to detect, then conclude “not significant = no benefit”: https://pubmed.ncbi.nlm.nih.gov/33576820/

    Or this one where ~27% of the vitamin c group tested negative vs 0% of the control, which they ignore since the p-value they calculated was somehow > 0.9: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7877333/

    It is amazing to see which crappy papers get taken down.

    • Or this one, where they conclude:

      Findings: The first dose of the BNT162b2 vaccine was associated with a vaccine effect of 85% (95% confidence interval [CI] 76 to 91) for COVID-19 related hospitalisation at 28-34 days post-vaccination.

      https://www.ed.ac.uk/files/atoms/files/scotland_firstvaccinedata_preprint.pdf

      That is fine as far as it goes, but Table 2 shows this dropped to 61% for days 35-41 and 58% after 42 days. This goes unmentioned in the paper. So does the 36% *increase* in hospitalizations seen in the 18-64 year old group for days 7-13, something that should be monitored because these vaccines are known to induce lymphocytopenia (suppress the immune system) for the first week:

      The largest changes from baseline in laboratory values were transient decreases in lymphocyte counts, which resolved within 1 week after vaccination (Fig. S3) and which were not associated with clinical manifestations.

      https://pubmed.ncbi.nlm.nih.gov/33053279/

      The key day 1-7 data is not reported at all.

        • The CIs and adjusted model has its own problems, I am just ignoring those for now and treating the model output the same as the authors interpreted the cherrypicked results.

          I would be worried about the countries advocating one dose of these vaccines and not telling people to be careful for the first week because they are quite possibly *even more at risk than before the vaccination*. Also that they seem to be only ~ 60% effective after a month.

          People who have been in fear the last year are getting the vaccine then think they are safe, so they will like lick a table, etc as a funny display of freedom.

          Btw, I still know no one who has died from covid but now one person in a nursing home who died a couple weeks after the vaccine following trouble breathing and vomiting. No investigation, covid test, etc was performed.

        • Sounds like a potential heart attack any number of other causes of death–sorry for the loss of the person you know. It’s encouraging to hear you don’t know anyone who has died of COVID. But perhaps the circle of people you know is rather small? I know of people who have died or nearly died from it, including otherwise healthy 35-year-olds. I’m sure others do too. I’m not sure how your personal experience is very relevant to any sincere discussion of the trade-offs between the harms and benefits of vaccination.

        • Also spO2 in the low 80s I was told. So basically this person had symptoms of covid but was not tested for covid due to the prior vaccination.

          I’m not sure how your personal experience is very relevant to any sincere discussion of the trade-offs between the harms and benefits of vaccination.

          It is pretty obvious. Covid deaths are less likely to be recorded if the person was vaccinated since they don’t even run the test. If they had died of the same symptoms within a month of a positive pcr test, this would be recorded as a covid death.

        • It’s still not obvious to me. Is the likelihood of not being investigated after death (after being vaccinated) more or less common that those that have died without prior vaccination? Are you sure about the spO2 levels, or is it hearsay? I’m sure you know that low spO2 is associated with heart failure, yes? (Vomiting and difficulty breathing also associated with heart attacks).

          Is there any information coming from the nursing home about your acquaintance besides those you’ve chosen to share that don’t fit the narrative of possible COVID-related death? Do they not test anyone at that nursing home or all nursing homes?

          Have there been other cases at the nursing home that can be traced to/from your acquaintance? If not, this perhaps decreases the likelihood it was COVID-related. It seems it would be difficult to make any sort of conclusions about the relationship between prior vaccination and investigating cause of death based on a single example (Again, sorry for your loss).

        • Is the likelihood of not being investigated after death (after being vaccinated) more or less common that those that have died without prior vaccination?

          Reasoning, history, and now my anecodote tells us so. Is there any data being collected? Or will there just be “no evidence” about this because no one funds it?

        • Five residents of a skilled nursing facility received positive SARS-CoV-2 nucleic acid test results in two separate COVID-19 outbreaks separated by 3 months. Residents received at least four negative test results between the two outbreaks, suggesting the possibility of reinfection. Severity of disease in the five residents during the second outbreak was worse than that during the first outbreak and included one death.

          […]

          The finding that all five patients with recurrent COVID-19 had either asymptomatic or mildly symptomatic courses during their first infections is noteworthy, suggesting the possibility that asymptomatic or mildly symptomatic initial infections do not produce a sufficiently robust immune response to prevent reinfection (5).

          https://www.cdc.gov/mmwr/volumes/70/wr/mm7008a3.htm

          People with weak antibodies have more severe illness. Sounds just like what they have been ignoring since last winter: Antibody dependent enhancement. The perfect storm has always been nursing home patients after antibody waning, probably towards a new strain too.

        • Oh, the good ol’ conspiracy theory. Thought so.

          I prefer incompetence. But due to incompetence, greed, conspiracy, whatever data on this has not been published.

          I have been saying this will be a problem for a year now in an attempt to save lives. You choose to listen to those who ignore it, we will see.

        • Here is another one:

          This study reports 33 cases with recurrent COVID-19 episodes following recovery and confirmed by positive qRT-PCR in each of the first and second episodes. Recurrences tend to be more severe than the first episode, one individual dying.

          https://pubmed.ncbi.nlm.nih.gov/33589297/

          Would you like to see the many other papers showing this is going on?

        • Nah, you don’t need to spend time Googling cherry-picked studies to weakly support points (or support wrong points) that are aren’t even related to the original discussion. There are many months of discussions/interactions you’re involved in to inform my priors on how this would go, ranging from election fraud to the miracles of multi-vitamins as a prophylactic. Why risk being accused of not understanding grade-school math, as you have written in the past? Not worth anyone’s time. Every message board has its troll, including this one!

        • Nah, you don’t need to spend time Googling cherry-picked studies to weakly support points (or support wrong points) that are aren’t even related to the original discussion.

          Simply quoting the literature is now cherrypicked studies?

          You provide no sources, no evidence, no anything. Just snarky comments pushing an anti-science agenda. Lots of this type of commentator has joined Andrew’s blog last year. That is too bad.

        • Anoneuoid –

          You’re creating a bit of a moving topic here, but as for this…

          > Reasoning, history, and now my anecodote tells us so. Is there any data being collected? Or will there just be “no evidence” about this because no one funds it?

          So does that mean that you have no actual evidence beyond a single anecdote and arguing from personal incredulity?

        • the miracles of multi-vitamins as a prophylactic.

          And the “miracle” of vitamins is in treating vitamin deficiencies. This is based 100 year old consistent and successful science.

          It is no more a “miracle” than changing your oil or power steering fluid or keeping the tire pressure in the correct range keeps your car running properly.

          This anti-science attitude is really ridiculous and leading to great harm.

        • So does that mean that you have no actual evidence beyond a single anecdote and arguing from personal incredulity?

          It means when someone has symptoms of covid during a covid pandemic, then dies, this death should be investigated to see if it was due to covid.

          That is what I think. If you are against doing that, and collecting data on whether that is happening (as it should) at equal rates in vaccinated vs not, then you are simply anti-science. There is no other way to put it.

        • Anoneoid –

          > If you are against doing that, …

          I haven’t said I am, nor suggested it in any way.

          > …then you are simply anti-science.

          It’s anti-science for you to imply that I have an opinion which I never said nor suggested that I have.

        • If you are pro science you should be agreeing with me and responding to unanon rather than wasting everyones time twisting what I said into strawmen.

        • Anineuoid –

          I asked you questions – whether you had evidence beyond a single anecdote, and an argument beyond arguing from personal incredulity.

          You didn’t answer my question.

          And instead of an answer, you created a burden of proof for me to prove I don’t have an “anti-science” opinion that I don’t have, and never auggeded that I have.

          One might assume that you shifted the burden because you don’t have good answers for my questions. You can disprove possibility by answering the questions.

        • Another new report out today of more severe infection the second time:

          Here we describe a case of re-infection in an individual from South India characterized by
          whole genome sequencing of the virus isolated from both episodes. Analysis shows the
          presence of an immune escape variant N440K in the Spike protein in both episodes of
          infection. Incidentally, this variant was also found in a case of reinfection previously
          reported by us in a healthcare worker from North India ​[1]​.

          In the first episode of infection, the 47-year-old male public service employee from Andhra
          Pradesh, India, was identified to be positive for SARS-CoV-2 on July 25, 2020, in
          nasopharyngeal specimens analysed as a part of routine surveillance and was
          asymptomatic. The cycle threshold values (ct) was 22.3 and 19.1 for ORF1ab and N genes
          respectively (Labsystems Diagnostic Inc.). The individual tested negative on August 2,
          2020, and tested positive again on September 10, 2020, during routine surveillance, but
          during this episode, he was symptomatic with fever, cough and malaise.

          https://osf.io/7gk69/

        • I said now there is one anecdote. Personal incredulity was your strawman. As usual I am the only one providing evidence and sources then nothing is learned from the discussion.

  3. How would a MLM have “alerted the authors their assumptions were false, or proved to the readers that they were true” with regards to whether randomization occurred at the patient or ward level?

    • The idea would be to include ward assignment as level in a varying intercept model. If the assignment is random there won’t be any information at the ward level and it basically reduces to a patient-randomized trial. If it’s not random you’ll see that there is some information at the ward level which would alert you that ward effects were important. I think this is probably a better randomization check than checking whether patient characteristics are the same between the two arms, and there’s an argument for always doing it, but I really don’t have that much experience in this area so it’s also totally possible that this is a silly idea.

Leave a Reply to Gordon Cancel reply

Your email address will not be published. Required fields are marked *