The devil really is in the details; or, You’ll be able to guess who I think are the good guys and who I think are the bad guys in this story, but I think it’s still worth telling because it provides some insight into how (some) scientists view statistics

I noticed this on Retraction Watch:

“Scientists clearly cannot rely on the traditional avenues for correcting problems in the literature.” PubPeer responds to an editorial slamming the site.

I’ve never actually read anything on PubPeer but I understand it’s a post-publication review site, and I like post-publication review.

So I’m heading into this one on the side of PubPeer, and let me deflate any suspense right here but telling you that, having followed the links and read the discussion, my position hasn’t changed.

So, no news and no expectation that this new story should change your beliefs, if you happen to be on the Evilicious side of this particular debate.

So, if I’m not trying to convince anybody, why am my writing this post? Actually, I’m usually not trying to convince anyone when I write; rather, I use writing as a way to explore my thoughts and to integrate the discordant information I see into coherent stories (with one sort of coherent story being of the form, “I don’t yet understand what’s going on, the evidence seems to be contradictory, and I can’t form a coherent story”).

In that sense, writing is a form of posterior predictive check, or perhaps I should just say posterior inference, a way of working out the implications of my implicit models of the world in the context of available data.

They say Code Never Lies and they’re right, but writing has its own logic that can be helpful to follow.

Hence, I blog.

Now back to the item at hand. The above link goes to a post on PubPeer that begins as follows:

In an editorial entitled “Vigilante science”, the editor-in-chief of Plant Physiology, Michael Blatt, makes the hyperbolic claim that anonymous post-publication peer review by the PubPeer community represents the most serious threat to the scientific process today.

We obviously disagree. We believe a greater problem, which PubPeer can help to address, is the flood of low-quality, overinterpreted and ultimately unreliable research being experienced in many scientific fields . . .

I then clicked to see what Michael Blatt had to say in the journal Plant Physiology.

Since its launch in October 2012, PubPeer has sought to facilitate community-wide, postpublication critique of scientific articles. The Web site has also attracted much controversy . . . .

PubPeer operates as a blog on which anyone can post comments, either to a published article or to comments posted by other participants, and authors may respond. It is a bit like an extended journal club; not a bad idea to promote communication among scientists, you might think, so why the controversy?

Why, indeed? Blatt explains:

The problems arising are twofold . . . First, most individuals posting on PubPeer—let’s use the euphemism commenters for now—take advantage of the anonymity afforded by the site in full knowledge that their posts will be available to the public at large.

I don’t understand why “commenters” is considered a euphemism. That’s the problem with entering a debate in the middle—sometimes you can’t figure out what people are talking about.

Anyway:

Second, the vast majority of comments that are posted focus on image data (gels, blots, and micrographs) that contribute to the development of scientific ideas but are not ideas in themselves. With few exceptions, commenters on PubPeer do no more than flag perceived faults and query the associated content.

But, wait, what’s wrong with commenting on image data? And “flagging perceived faults”—that’s really important, no? We should all be aware of faults in published papers.

Of course, I say this as someone who’s published a paper that was invalidated by a data error, so I personally would benefit from outsiders checking my work and letting me know when they see something fishy.

So what’s the problem, then? Blatt tells us:

My overriding concern with PubPeer is the lack of transparency that arises from concealing the identities of both commenters and moderators.

This is so wrong I hardly know where to start. No, actually, I do know where to start, which is to point out that articles are published based on anonymous peer review.

Who were the reviewers who made the mistake of recommending publication of those papers by Daryl Bem or Satoshi Kanazawa or those ovulation-and-voting people? We’ll never know. For the himmicanes and hurricanes people, we do know that Susan Fiske was the editor who recommended publication, and she can be rightly criticized for her poor judgment on this one (nothing personal, I make lots of poor judgments myself, feel free to call me out on them), but we don’t know who were the external referees who failed to set her straight. Or, to go back 20 years, we don’t know who were the statistical referees who made the foolish, foolish decision to recommend that Statistical Science publish that horrible Bible Code paper. I do know the journal’s editor at the time, but he was in a difficult position if he was faced with positive referee reports.

So, according to Blatt: Anonymous pre-publication review, good. Anonymous post-publication review, bad. Got it.

Indeed, Blatt is insistent on this point:

I accept that there is a case for anonymity as part of the peer-review process. However, the argument for anonymity in postpublication discussion fallaciously equates such discussion with prepublication peer review. . . . In short, anonymity makes sense when reviews are offered in confidence to be assessed and moderated by an editor, someone whose identity is known and who takes responsibility for the decision informed by the reviews. Obviously, this same situation does not apply postpublication, not when the commenters enter into a discussion anonymously and the moderators are also unknown.

Oh no god no no no no no. Here’s the difference between pre-publication reviews, as usually conducted, and post-publication reviews:

Pre-publication reviews are secret. Not just the author of the review, also the actual content. Only very rarely are pre-publication reviews published in any form. Post-publication reviews, by their very nature, are public.

As Stephen King says, it’s the tale, not he who tells it. Post-publication reviews don’t need to be signed; we actually have the damn review. Given the review, the identity of the reviewer supplies very little information.

The other difference is that pre-publication reviews tend to be much more negative than post-publication reviews. I find it laughable when Blatt writes that post-publication reviews are “one-sided,” “petty,” “missing . . . courtesy and common sense,” “negative and occasionally malicious,” and “about policing, not discussion.” All these descriptions apply even more for pre-publication reviews.

Why do I care?

At this point, you might be asking yourself why I post this at all. Neither you nor I have ever heard of the journal Plant Physiology before, and we’ll likely never hear of it again. So who cares that the editor of an obscure journal emits a last-gasp rant against PubPeer, a site with represents the future in the same way that editor-as-gatekeeper Michael Blatt represents the past.

Who indeed? I don’t care what the editor of Plant Physiology thinks about post-publication review. What I do care about is we’re not there yet. Any dramatic claim with “p less than .05” that appears in Science or Nature or PPNAS or Psychological Science still has a shot of getting massive publicity. That himmicanes-and-hurricanes study was just last year. And this year we’ve seen a few more.

P.S. Incidentally, it seems that journals vary greatly in the power they afford to their editors. I can’t imagine the editor of Biometrics or the Journal of the American Statistical Association being able to publish this sort of opinion piece in the journal like this. I don’t know the general pattern here, but I have the vague impression that biomedical journals feature more editorializing, compared to journals in the physical and social sciences.

P.P.S. Two commenters pointed out small mistakes in this post, which I’ve fixed. Another point in favor of post-publication review!

29 thoughts on “The devil really is in the details; or, You’ll be able to guess who I think are the good guys and who I think are the bad guys in this story, but I think it’s still worth telling because it provides some insight into how (some) scientists view statistics

  1. Great post Andrew. Off-topic point though – I think you might underestimate the breadth of readership of your blog .. “Neither you nor I have ever heard of the journal Plant Physiology before, and likely we’ll never hear of it again.” In the ‘impact factor world’, this journal is actually not that low – 8ish- ( don’t know where that sits in the distribution but it’s not in the long tail … and of course we know that impact factors are problematic from many standpoints, not least how they measure stuff – see for instance https://quantixed.wordpress.com/2015/05/05/wrong-number-a-closer-look-at-impact-factors/) .
    But my main point is that I know a load of biologists who read your blog!
    ( P.S. I am not a plant physiologist and I do not work for Elsevier!)

    • I second this—I read papers from Plant Phys (and this blog) almost every day. I try to ignore Journal Impact Factors, but Plant Phys is generally considered the #2 plant biology journal, after Plant Cell. Both are published by the American Society of Plant Biologists.

      One reason that some people find comments on image data uninteresting is that splicing of images (of gel lanes) together used to be considered acceptable but is now considered manipulation—it is easy to find examples in old papers. Another reason is that it can be difficult to distinguish “airbrushing” of flaws from images from image compression artifacts.

      For background on the case that brought PubPeer to the attention of plant biologists see http://retractionwatch.com/?s=voinnet

      PubPeer makes it hard to make it hard to register to comment nonanonymously. You are required to have a first first author paper first. (I have a co-first author paper [published in Plant Phys, incidentally], but that is insufficient). I posted an anonymous comment once—it took a long time to appear, so I think the moderation can be quite slow.

  2. Great post —

    I explain the discrepancies in the way people seem to contrast pre-pub peer review and post-pub peer review in two ways.

    First, people are used to pre-pub peer review. Change is bad, apparently.

    Second, in pre-pub peer review, the *editor* (who is generally supposed to be a respected scientist in the community etc etc.) knows who the reviewers are, and they are supposed to weigh the expertise of the reviewers in the balance. This, of course, punts the actual decision into the hands of the editor… this seems to me to be an appeal to authority (channeling that quote from Indiana Jones: “*top men* will be handling this from now on!) and is less than acceptable to me because it is not even remotely transparent.

    Anyway, thanks again for the post.

  3. “So, if I’m not trying to convince anybody, why am my writing this post? … I use writing as a way to explore my thoughts and to integrate the discordant information I see … a way of working out the implications of my implicit models of the world in the context of available data.”

    So a way to repeatedly practice the logic of discovery (induction). Makes sense!

  4. I haven’t looked at the PubPeer site, but I know that their are industry goons hired to attack scientific publications that threaten the financial well-being of certain industries. Is PubPeer not simply another forum for such attacks, calling into question the validity of the scientific work (rightly or not)? Sometimes, calling into question _any_ aspect of threatening work is sufficient to dismiss it completely (sadly).

  5. I don’t think it is true that “given the review, the identity of the reviewer supplies very little information”.

    The credibility of the poster affects my prior confidence in his facts and arguments. Not everything can be argued up from first principles and hard logic.

    It is possible to efficiently form opinions or digest information only because we use an informal network of imputed credibilities to judge different people. Priors matter. Reading a headline “Fermats Last Theorem Proved” in the Sun is a lot different than reading about it in Nature.

    But that in itself is not a good argument to abhor anonymous comments and hence I think Blatt is *still wrong* about that.

  6. Andrew, I do appreciate how you give disclaimers of your bases (i.e. not being a reader of PubPeer, and your assumptions about what it is). As someone who has read PubPeer, I think it is important to describe what they are in an aspirational sense, which is what you appear to assume they are in practice, and what they are in practice for much of the time. There are some absolutely excellent discussions on the site, but I would say the majority of comments are about detection or accusations of fraud. Some fraction highly likely to be accurate, others dubious- and made very casually.

  7. To quality the value of such post-publication review, I think we should distinguish between registered (named or anonymous) and unregistered reviewers. If we are talking about the former, I agree with you Prof. Gelman. But if we are talking about the latter, then I fear that trolling and low signal:noise comments will dwarf useful comments, and then I’d agree with the Editor of Plant Pathology.

    By registered, I mean that the identity is known to the owner of the webpage (whether a journal, working paper archive, or blog), but it needs not be disclosed to the rest of the world

    • PubPeer seems to be enforcing anonymity? e.g. I see comments attributed to “Peer1”, “Peer2” etc. Or

      That seems strange. I’m all for anonymity for those that want it but OTOH those that want to declare their affiliation ought to be free to do so?

      Isn’t that how it is on most of the Internet? PubPeer’s variant of anonymity seems strange. It seems almost counterproductive because it will encourage no sense of community sense. e.g. On most blogs there are long time posters who will, in general, care about their reputation. If not in real life at least in the narrow online blog sense.

      That encourages good and responsible commenting. OTOH if there was by design a randomization that prevented me from associating all my comments with a single identity, it sounds like an invitation for crappy comments.

        • But when nearly every paper is making basic errors like “the null hypothesis is rejected, therefore my research hypothesis is true”, no one is doing replication studies, and no one is making any numerical predictions, what credibility is there to lose by going anon? The base credibility is so low you need to assume by default that no one knows wtf they are doing, so there is nowhere to go but up.

  8. For an alternative approach to peer review, I recommend you take a look at the EGU journals such as

    http://www.clim-past-discuss.net/discussion_papers.html

    Here, the peer review is open to all to read and contribute (though only the reviewers nomimated by the editor are allowed anonymity). It’s been working well (not perfectly perhaps) for many years now. Disclaimer: I co-founded one of these journals and act as an editor for it.

    • I was going to mention the EGU model, but JA beat me to it. In my experience, as an author, reviewer, and commentor, I think this model represents an excellent combination of anonymity, disclosure, transparency, and both Pre- and Post-publication review. I wonder why the model has not caught on more widely. My favorite aspect of the EGU system is that the reviewers’ comments are subject to comment from others.

  9. Dear Andrew,
    the main problem of traditional pre-pub peer review is its secrecy and intransparency. While peer reviewing, scientists can either scratch each others backs, or destroy the work of competitors, or take personal revenge, without any consequence.
    I am not sure applying similar rules to post-publication peer review, where fully anonymous comments are intransparently moderated by website operator is the correct solution. Anonymity should be a tool of witness protection, i.e., where scientific misconduct or other inappropriate behaviour is reported. As Philip Moriarty, I do not understand why scientific discussion should be led one-side blinded.
    Here is my two relevant blogs on PPPR and its anonymity:
    https://forbetterscience.wordpress.com/2015/12/13/post-publication-peer-review-signed-or-anonymous/
    https://forbetterscience.wordpress.com/2015/12/07/open-science-open-scientists/

  10. FWIW, the *vast* majority of the comments on Pubpeer are verifiable scientific critiques on the articles, with only a surprisingly tiny number that feel “vendetta-y”.

  11. On this topic, an editorial just published in NEJM seems to be taking a strong anti-reanalysis/data sharing viewpoint in the name of turf wars: http://www.nejm.org/doi/full/10.1056/NEJMe1516564

    “However, many of us who have actually conducted clinical research, managed clinical studies and data collection and analysis, and curated data sets have concerns about the details [of data sharing]. The first concern is that someone not involved in the generation and collection of the data may not understand the choices made in defining the parameters. Special problems arise if data are to be combined from independent studies and considered comparable. How heterogeneous were the study populations? Were the eligibility criteria the same? Can it be assumed that the differences in study populations, data collection and analysis, and treatments, both protocol-specified and unspecified, can be ignored?

    A second concern held by some is that a new class of research person will emerge — people who had nothing to do with the design and execution of the study but use another group’s data for their own ends, possibly stealing from the research productivity planned by the data gatherers, or even use the data to try to disprove what the original investigators had posited. There is concern among some front-line researchers that the system will be taken over by what some researchers have characterized as “research parasites.”

    How would data sharing work best? We think it should happen symbiotically, not parasitically. Start with a novel idea, one that is not an obvious extension of the reported work.”

    • Thanks, R, for this gift that keeps on giving. Also from the Editor in Chief:

      “It is in everyone’s best interest to support independent review and publication of biomedical research while allowing wide public access to our pages…. Such an approach will maintain the veracity of the published work. This is in the best interest of the public, which will get all the published data and and [sic] will get it right. We will guarantee it.”

      http://www.nejm.org/doi/pdf/10.1056/NEJMe048260

      I am glad that, despite the Editor’s concerns, we will still get the data and and get it right.

    • Ouch! You beat me to it. I was just writing to Andrew, suggesting that this….editorial could be an interesting topic for the blog. I am at a loss for words…according to this point of view, Andrew’s great piece on the mortality rate of middle-aged non-Hispanic whites would be “parasitism”. Maybe of a minor sort, since he collected relevant data by himself, without the help of the original authors. Instead it was very interesting, very useful for science and also a great learning experience, for me and I think for many others. And that’s just one case: the episode, recently covered by Andrew, of the effect of noble-sounding German surnames on career, it’s another example of great “parasitism”. Personally I think that the two editors should be ashamed of what they wrote.

  12. I had a doubtful pleasure listen to Blaat’s talk during the Plant Biology meeting once…. it was a talk where he was giving some hints how to successfully publish in Plant Physiology/The Plant Cell journals… it was pathetic and so pretentious… this guy is so into himself and he is the ultimate evidence that medicine haven’t invented a youth pill….

Leave a Reply

Your email address will not be published. Required fields are marked *