PCI Statistics: A preprint review peer community in statistics

X informs me of a new effort, “Peer community in . . .”, which describes itself as “a free recommendation process of published and unpublished scientific papers.” So far this exists in only one field, Evolutionary Biology. But this looks like a great idea and I expect it will soon exist in statistics, political science, and many other areas.

Here’s how X describes it:

Authors of a preprint or of a published paper request a recommendation from the forum. If someone from the board finds the paper of interest, this person initiates a quick refereeing process with one or two referees and returns a review to the authors, with possible requests for modification, and if the lead reviewer is happy with the new version, the link to the paper and the reviews are published on PCI Evol Biol, which thus gives a stamp of validation to the contents in the paper. The paper can then be submitted for publication in any journal.

X thinks we should start small and have a PCI Computational Statistics. Seems like a good idea to me.

I suggested this idea for the Journal of the American Statistical Association, but they’re super-conservative, I don’t think they’re ever gonna do it. So I’m with X on this: I’d like to do PCI Computational Statistics (or PCI Stan?) unless anyone can explain to me why this is not a good idea.

It would be also good to have a sense of where all this is going. If PCI stays around, will there be a proliferation of PCI communities? Or will things shake down to just a few? Something should be done, that’s for sure: Arxiv is too unstructured, NBER is a restrictive club, traditional journals are too slow and waste most of their effort on the worst papers, twitter’s a joke, and blogs don’t scale.

So, again, any good reasons not to do this?

19 thoughts on “PCI Statistics: A preprint review peer community in statistics

  1. To me it looks like a great initiative. The prior probability for basically any project involving multiple people to be succesful is low, so eventhough the project looks sweet, the posterior for success is still not great. But it feels worth a shot. Would be happy to participate in something like this, although I don’t really have the scientific credit (even less so in Comp. statistics).

  2. This sounds great. What is the next step you (and/or X etc.) would need to take?
    Perhaps “contact a Managing board member and explain him/her your project”?
    https://peercommunityin.org/faq/#How%20can%20I%20start

    The only downside I can see is lock-in with less-than-optimal terminology. For example, maybe it would be useful to make a semantic distinction between “peer recommended” (by a Peer Community) vs. formally peer reviewed, to reduce concern about journal prior publication policies (or just the perception of double-publishing).
    Related: The phrase “can be cited as peer reviewed” in the diagram above makes it sound like non-peer-reviewed things cannot be cited.

  3. First, you should be clear on the goals.

    Do you think this will remove the frustration of getting papers through journals? You cite traditional journals as being “too slow”, but that’s a policy decision. TACL reviews in under a month and Biostats does too. Then they make a quick decision. And if they want revisions, they make quick decisions on those. It’s ridiculous to send someone a paper with a review due months out—we all just wait until the editor nags us about it again to even think about it.

    Do you think it’ll provide a prestigous journal for comp stats papers? That’s what JMLR turned into when the editorial board quit the “traditional” paywalled journal Machine Learning. I have no idea what their turnaround time is—their reviewers probably spend their entire lives responding to author replies in the ML conferences.

    When you say computational stats, are you thinking something more software oriented like JStatSoft or something more math oriented like every other journal?

    What structure’s missing from arXiv that you think we need?

    Do you have an editor in mind? Presumably you (Andrew) don’t want the admin work of being editor. I’d have done this myself ages ago if I wanted to do the work.

    P.S. PCI Stan is way too narrow for a journal. Although I have been thinking we should do something like generate DOIs or something and try to turn the Stan Case Studies into something more publishable. There’s not a good venue for careful computational case studies.

  4. Do the reviews transfer over to the classic journals? At the moment, one of the major bottlenecks in traditional publication is slow review times partially driven by reviewer burden. If this system isn’t transferring over reviews, then it just seems to be doubling the length of the review process. If every paper now needs twice as many reviews, I can’t imagine this will speed up the dissemination process as a whole.

  5. For the field of evolutionary biology, the idea took off very quickly: PCI Evol Biol currently counts 298 recommenders who review and recommend preprints and recommend postprints. (I have no idea how large this community is, though, presumably several orders of magnitude compared with computational stats in a broad sense…)

  6. Interesting idea, but I predict they’ll mostly end up recommending papers and preprints that would’ve come to the attention of readers anyway. The early recommendations mostly seem to be recent papers from high profile journals like PNAS and Plos Biology, and preprints from the labs of well-established, well-known authors.

    I see a lot more value in a service like that arXiv overlay journal Discrete Analysis, because it’s only reviewing preprints, and because anybody can submit a preprint for consideration. That seems like it has some potential to direct some attention to work that deserves attention but that wouldn’t otherwise receive it (or would otherwise take much longer to receive it, at much greater cost to the authors and/or journal subscribers).

    But we’ll see. I suppose it’s possible they’ll build a reputation for finding diamonds in the rough, or for telling you which high-profile papers are really worth reading as opposed to being shallow/sloppy.

    • My bad, ignore what I just said, I got a bunch of key details wrong as to how PCI Evol Biol works. They’re reviewing preprints and papers submitted to them by authors, not finding their own stuff to review. And they’re reviewing already-published papers mostly as a way to help get the word out; in the future they plan to focus on reviewing preprints. So it’s actually pretty close to arXiv overlay journals like Discrete Analysis, which I think are a promising idea.

  7. I don’t know if you know about this story or are interested in it, but this recent retraction has raised some new issues about post publication peer review and whether publicly posted reviews should cite relevant literature:
    http://retractionwatch.com/2017/05/24/authors-retract-much-debated-blockchain-paper-f1000/

    Let me try to quickly summarize this year-long ordeal:
    Over a year ago two physicians posted a paper about blockchains (nothing unusual about that).

    A grad student then pointed out that the paper was plagiarized almost word for word from a blog post he wrote two years prior.

    Instead of retracting the paper the journal allowed the authors to resubmit a new version which cited the blog and changed phrases which were word for word identical.

    The story got picked up by Retraction Watch, and Neuroskeptic and I blogged about it. I specifically pointed out some obvious issues with the methodology but I’m not familiar with bitcoin technology so I didn’t know how deep the problems went.

    –One of the most interesting things about this story was how vehemently the peer reviewers defended the paper in interviews even after the plagiarism was exposed (they even went so far as to disparage the graduate student).

    The case was sent to COPE to decide if it should be retracted. Cases are supposed to be anonymized, however the committee was made aware of the Retraction Watch article and thus the authors of the paper, one with a Cambridge affiliation.

    COPE decided despite the blatant plagiarism and despite the concerns about whether the paper is even correct, the correction which added a reference to the blog was sufficient and a retraction wasn’t warranted (I think them and Wansink would get along well).

    –There is 0 doubt in my mind that if the authors were from an Asian country this paper would have been immediately retracted. PLOS ONE retracted the #creatorgate paper in 1 day.

    And so the case seemed to be over.

    But then a couple months ago my friend who is more familiar with bitcoin technology blogged about how the paper’s method made absolutely no sense.

    He posted his analysis as a comment on the paper, and the authors responded by again altering their manuscript and basically admitting their method is useless as described.

    Despite this the paper still was not retracted by the journal, but apparently it did encourage them to find an official reviewer with expertise in this area. This reviewer came to the exact same conclusions as my friend.

    The authors then for some reason decided it was finally time to retract the paper.

    Retraction Watch then ran their story and the new reviewer seemed to get most of the credit for the retraction.

    The issue this brings up is if a review is posted publicly should it be held to the same expectations of other scientific literature and cite relevant sources? It’s hard to imagine this reviewer didn’t read my friend’s blog post and wasn’t influenced by it. Sure, maybe he would have come to exact same conclusions anyways, but I can’t help but feel that if the blog post was a paper, and the reviewer’s review was also a paper, then clearly it would have been cited.

    • Well at a minimum some kind of hat-tip acknowledgement is warranted, no? That’s standard in blog discourse. But why do you think “t’s hard to imagine this reviewer didn’t read my friend’s blog post and wasn’t influenced by it”? Is the community that small, or is your friend’s blog major in this area? (I’m asking honestly–I’m not being sarcastic.)

    • Jordan:

      Wow. This sounds like a Wegman or Hesse situation, and it illustrates why copying without attribution—whether or not it’s plagiarism—is a big deal. When people copy without attribution, it often seems to be associated with the copyists not processing what they’re copying (as with Weggy mangling a formula or Chrissy Reply

        • After reading the link, I wondered what/who COPE was, so looked up its website. The information there seems pretty fuzzy; I can’t help but wonder if COPE is focused on “how to deal with complaints involving ethical concerns” more than on “how to behave ethically”.

        • I definitely agree with you. COPE recently suggested removing retractions entirely, which would further absolve journals of any responsibility:
          http://biorxiv.org/content/early/2017/03/21/118356

          On the one hand I agree it should be much easier to correct errors in articles, however the solution to me is simple, just don’t submit work to journals and only post as preprints, which allows you to fix any problems with a new version.

          Even preprints, under certain circumstances, should be retracted, and indeed they are. This is a touchy subject that I’m not supposed to talk about, but as an indexer of preprints I know which preprints have been retracted.

          Some articles just need to be vanquished to the shadow realm, for example if they are plagiarized or contain fraudulent data. Leonid has some thoughts on this:
          https://forbetterscience.com/2017/03/29/cope-the-publishers-trojan-horse-calls-to-abolish-retractions/

        • Thanks for the interesting links.

          One comment re your sentence, “Even preprints, under certain circumstances, should be retracted, and indeed they are.”. I think that for preprints there needs to be a category “withdrawn” for a preprint that the author(s) have decided is incorrect or needs more work (as contrasted with something retracted on the initiative of a preprint server, or retracted after being published by a journal). This would continue/extend what was a sensible tradition before electronic communication of preprints: that an author might reasonably decide to withdraw a paper from publication between submission to a journal and publication in the journal.

        • I agree there needs to be a withdrawn category. Unfortunately, how retractions of preprints are currently handled is far from optimal.

          What is currently done is when a preprint is taken down the link to the preprint just leads to “page not found”. Obviously there should be some sort of explanation for why it was taken down instead of just a broken link.

          However, withdrawing preprints is a complicated subject, just as complicated or even more complicated than journal retractions, so it isn’t always clear what should be listed as the explanation. It also isn’t clear if there should be any information about the preprint, so perhaps a broken link is the easiest and best solution.

          To see an example of how complicated this is imagine I am collaborating on a paper with someone. I then post the paper as a preprint. It turns out my collaborator didn’t want the preprint posted and he contacts the preprint server and demands it is taken down. What is the solution here? My collaborator is afraid of being scooped so he wants no record of the preprint ever being posted. (This is completely hypothetical by the way, if one of my collaborators did this I would just post the preprint elsewhere, including my own websites.)

Leave a Reply to Jon M Cancel reply

Your email address will not be published. Required fields are marked *