They want open peer review for their paper, and they want it now. Any suggestions?

Someone writes:

We’re in the middle of what feels like a drawn out process of revise and resubmit with one of the big journals (though by pre-pandemic standards everything has moved quite quickly), and what’s most frustrating is that the helpful criticisms and comments from the reviewers, plus our extensive responses and new sensitivity analyses and model improvements, are all happening not in public. (We could post them online, but I think we’re not allowed to share the reviews! And we didn’t originally put our report on a preprint server, which I now regret, so a little hard to get an update disseminated.)

For our next report, I wonder if you know of any platforms that’d allow us to do the peer review out in the open. Medrxiv/arxiv.org are great for getting the preprints out there, but not collecting reviews. Something like OpenReview.net (used for machine learning conferences) might work. Maybe there’s something else out there you know about? Do any journals do public peer review?

My reply:

Can PubPeer include reviews of preprints? And there is a site called Researchers One that does open review.

Also, you could send me your paper, I’ll post it here and people can give open reviews in the comments section!

11 thoughts on “They want open peer review for their paper, and they want it now. Any suggestions?

    • After a bit more poking around, there are some examples already, linked here:
      https://www.reviewcommons.org/refereed-preprints/
      https://www.peerageofscience.org/transparent-review-in-preprints/
      Each preprint gets a small blue tab to the right that opens a panel of reviews. Being based on hypothes.is, this seems potentially very powerful; it should be possible to use links to point the reader to the sentence being discussed. I’m surprised it’s so low-profile.

        • Yes, there’s been great progress in the last few years. But I think this idea would be another excellent step forward. We’ve all given a blistering rejection recommendation, only to see the paper appear virtually unchanged elsewhere. Even if those final lenient reviews were open, the previous hard critiques with the gory details would be lost. A good example of this: Peter Duesberg’s HIV denial piece, retracted by Medical Hypotheses.
          https://www.nature.com/news/paper-denying-hiv-aids-link-secures-publication-1.9737
          “Duesberg admits submitting the revised paper to more than four other journals before it was accepted by the {Italian Journal of Anatomy and Embryology}”. If the (no doubt vitriolic) intervening reviews had been public, fewer people might have fallen for the published article.
          Conversely, if open preprint review had been done 30 years ago, we could see exactly how those opposing the amyloid hypothesis for Alzheimer’s, or statisticians who refused to use p values, got the cold shoulder. Perhaps this could guide reforms of review and grant application processes.

  1. I agree that the authors should have put their paper on a preprint server. It’s hard to see how they can complain that the present process isn’t open when the didn’t take the simple and effective path to make it open at the start! But presumably they regret this…

    More constructively:

    1. “We could post them online, but I think we’re not allowed to share the reviews!” Have you actually asked the editor? In any case, even if you couldn’t share the reviews, you could share your response to them, which is your own writing. Just post it somewhere (e.g. your own blog or research group web page).

    2. Some journals, like eLife, publish the referee reports and responses along with the paper. These are often interesting to see. You could encourage your journal to do this.

  2. Given that preprint servers normally allow versioning anyway, I don’t see what is stopping you publishing the original manuscript as a preprint now, summarising the peer reviews you received as comments on your own preprint, and then iterating to the next version, summarising peer review comments again etc.
    Not exactly open peer review, but better than nothing.

  3. This is a perfect application for AI, however most published works rely on the limitations of reviewers. They couldn’t pass competent full knowledge review.

  4. Who says you’re not allowed to post the reviews? As an author I’ve never agreed (or been asked to agree) to keep referee reports private. I looked up journal policies for a few that I publish in (in physics) and they general only impose confidentiality requirements on the referees, not the authors.

  5. Copernicus does open peer review in geosciences and it’s great!

    Disclaimer: I’m a minor editor on two journals there (one of which I co-founded a while back)

Leave a Reply to Thomas Munro Cancel reply

Your email address will not be published. Required fields are marked *