Someone pointed to this post from last year, “If only Arxiv required researchers to sign at the top rather than the bottom of the page, none of this would’ve happened,” and asked about this statement of mine: “Seriously, though, setting aside the junk references, I don’t know that I would’ve noticed any problems with the paper had it been sent to me cold.”
My corresponded noted that my post pointed out all sorts of statistical analysis problems in the paper, red flags all of the place, and he asked why I would not have right away suspected fraud.
So let me explain.
My remark, “I don’t know that I would’ve noticed any problems with the paper had it been sent to me cold,” reflects that, when I’m sent a paper to review, I don’t review it forensically. Once I see the problems, I can’t un-see them, and, as I wrote in my post, the problems in that paper are clear, but in a quick review I might not have looked into all those details. I don’t think it is a requirement of a reviewer for a journal to investigate a submission in detail. Ultimately the correctness of an article is the responsibility of the author.
To put it another way, I don’t see journal review as a gatekeeping process that will keep erroneous articles from being published; rather, I see it as providing some information to the author and editor.
I’m on record as saying that the problem with peer review is the peers. That doesn’t make peer review useless. It is what it is. Peers can be very helpful in pointing out connections to the literature. I just don’t think it makes sense to see reviewers as gatekeepers. Again, the correctness of the work is ultimately the responsibility of the author. And, in any case, there will be post-publication review of papers that are of interest to later readers.
I don’t think you should absolve reviewers or editors of responsibility. Ultimately an author is responsible for what they do – but when you agree to review a manuscript, even though you are not paid, you are assuming some sort of responsibility. And, the editor is responsible for what the journal chooses to publish. There is a problem in that this ‘shared responsibility’ can become no responsibility when it is unclear who to blame. The Surgisphere fiasco certainly involved the (one of the authors) author making up data that they bear responsibility for. Their coauthors should also bear some responsibility – though most escaped by claiming ignorance. Peer reviewers appear to have been spared. I guess retraction is the way editors bore responsibility, though I don’t think it hurt the reputation of the NEJM (I can’t remember of more than one journal was involved). I think this was a case where your statement that the author is “ultimately” responsible allowed a number of people to escape any responsibility.
Reviewers are gatekeepers to some extent and it depends on the journal. If I’m reviewing a paper for a top journal I would quite likely indicate to the editor that the paper might be sound but not of sufficient interest for that journal (there’s usually a category tick box to give one’s thoughts on this, although the editor should make the final decision on this criterion). For other journals (e.g. PLoS one or Scientific Reports) where the sole criterion is scientific validity I would review appropriately.
Obviously if the paper is clearly flawed then as a reviewer one would make that clear and why. So that is a gatekeeping role.
I do agree though it’s not the role of the reviewer to identify articles that might ultimately be erroneous. If a paper is submitted that is made up or has fabricated data or images then it’s not the role of the reviewer to identify this, although s/he might notice this. Likewise it’s not the role of the reviewer to identify plagiarism though a reviewer might do this (I’ve identified plagiarism in several papers reviewed just because something doesn’t seem right with some text).
Reviewing does improve a paper in a number of ways in my experience both as an author or as a reviewer.
Dale, Chris:
I agree that reviews add value. For reasons of efficiency, I prefer post-publication review rather than pre-publication review, but, in any case, yes, reviews can be super helpful. Reviewers can catch all sorts of mistakes and make all sorts of useful perspectives.
I totally agree Andrew, publications live or die according to post publication perusal whether this is replication, extension by advancing the work, critique or complete indifference
There are two aspects of the review that are at odds: timeliness vs. thoroughness. First of all, many reviews have a rather quick turn-around required. Often when I agree to do a review, I can’t drop everything. I’m planning to do that review later on as my schedule allows. When I do get to it, I have to trade off a bit of thoroughness to get the job done on time. That doesn’t mean I’m an easy reviewer by any means. But, you can’t dig into every detail. Plus, there are aspects where I’m better suited to intervene than others. Furthermore, the authors will generally get a chance to respond. Some of my critiques may be mistaken due to context, etc. I out-right reject relatively few articles since the AEs usually do a good job sorting those out. Post-publication review would be nice. However, I have never seen any of the articles that I have ever reviewed go through such a process. There are too many articles out there and the numbers are growing all the time.
Yes good points. When asked to do a review of a paper, which often comes with a request to submit the review within 10 days, I’ve got into the habit of responding that I can do the review but will need longer, often suggesting 20 days – assuming I’m interested enough to accept the review. That way I feel under less pressure, the editor gets the review pretty much within the period that s/he will get from the other reviewers anyway, and isn’t left stressing whether my review is going to appear. From the other side of the fence as an author it’s rare to get the editorial decision with reviews within 5-6 weeks so one wonders how many of those “please return your review within 10 days” are ever honoured! IMO it’s better to let the editor know from the outset that my review might be later than 10 days.
I’ve learned over the years to do a 3 x 2 hours schedule when preparing things like reviewer reports or other writing based on reading, interpreting and summarising. So I’ll find a relaxed 2 hours to read and annotate the paper I’m reviewing and jot down points. With at least an overnight sleep I’ll find another 2 hours to do any checking, exploring things I’m uncertain about, drafting bullet points for a response. After at least another nights sleep I’ll find another 2 hours to consolidate the report. That’s usually enough to feel like I’ve done a decent job. In my experience overnight sleeps are great for assembling something coherent out of what might be a mixed up jumble of thoughts especially those that arise from the first 2 hour session.
Post-publication review is mostly a conceptual notion that incorporates all the responses to a paper. This might be formalised (a formal critique in a journal – very rare – or a scary posting on PubPeer highlighting what someone thinks is rubbish about your paper!). Normally it refers to the impact the paper has in stimulating further work by others that might involve real or implicit reproduction, development of your methods and interpretations, use of your data and so on; some of this can constitute an implicit endorsement of the paper. There have been some suggestions for more formalised post-publication peer review – not sure how much traction these have managed.
Yeah I think that’s very reasonable. Authors are responsible for microscopic details and for making them in principle reproducible. Reviewer mostly responsible for macroscopic details, e.g., is this significant, is it well-designed, does it omit or contradict previous work, are the arguments clearly presented, are the conclusions justified by the analysis, etc. As you say, if you see a microscopic error you can’t unsee it, but referee reports are not ordinarily entire reproductions of an analysis.