In a story in Retraction Watch entitled “Rejection overruled, retraction ensues when annoyed reviewer does deep dive into data,” Adam Marcus reports:

As a prominent criminologist, Kim Rossmo often gets asked to review manuscripts. So it was that he found himself reviewing a meta-analysis by a pair of Dutch researchers — Wim Bernasco and Remco van Dijke . . . The paper, for Crime Science, analyzed 33 studies . . .

Rossmo found problems with the paper; in his words:

I [Rossmo] began by reading the 33 articles that comprised the basis of the systematic review. When I did so, however, I discovered different results from what was reported in the systematic review (e.g., simulated data, mixed findings, opposite results, violations of the posted selection criteria, etc.). . . . I wrote up a full replication report, concluded the article should be retracted, and sent it to the editor and the first author. This time the author said he would take another look at their work. A couple of months later, he responded that he found even greater discrepancies and agreed the paper should be retracted. Apparently, all the coding for the systematic review was done by the second author. The journal editors also agreed to the retraction.

This is a story with a happy ending because the author of the original article accepted the critique; he did not dodge it or fight it; his loyalty was to science and to society, not to his publication record.

From the retraction notice:

The authors attempted to replicate their own findings by re-assessing the 33 publications. Based on the results, they concluded that Professor Rossmo’s concern was fully justified. . . . The authors apologize to the readers and the Editors of Crime Science for any problems caused by drawing conclusions not sufficiently supported by evidence, and they thank Professor Rossmo for bringing the issue to their attention. . . . Both authors agree to the retraction.

When do you ever see that? This should be how researchers always behave when outsiders point out errors in their work, but unfortunately this clean admission of error is rare.

And note that I said “admission of error,” not “admission of guilt.” Everyone makes mistakes. We learn from our mistakes, but only when we are willing to learn. So I appreciate Bernasco and van Dijke for doing the right thing.

Just one thing

After the episode was over, the first author of the paper wrote:

Finding out and having to admit that we as authors had made errors was unpleasant, but the way it was handled left us room to make up our minds, and time to get back to the data and attempt to replicate our own work. The ‘whistleblower’ did not make his concerns public but contacted us and the editors of Crime Science about his inability to replicate our findings. This was painful, but becoming the target of a public debate would have been much worse.

This could also be read as an advice to those who identify errors (or worse) in the work of others: Do not immediately put the authors on the stand in public, but contact them and the journal editors first. The self-correcting power of science should not become a witch-hunt.

This could be good advice. I guess it depends on the context. The authors of this article responded quickly and appropriately to the criticism, and it all went well. In many other cases, authors simply ignore criticism, or they brush it aside, or they start coming up with alternative stories as a way of avoiding the issue. Sometimes I think it will be effective to contact the journal and the authors directly; other times this will just delay things. Recall that the pizzagate guy was being contacted by people for several years and managed to never address any of the concerns; it was only after public exposure that anything got done, and even then it took awhile.

Also, I think the author’s use of the term “witch hunt” is inappropriate. There is no such thing as a witch (as defined as someone who can actually do supernatural things); thus a “witch hunt” refers to a setting in which people are persecuted for something they did not do and could not have done.

In contrast, scientific errors are real. In addition, publicly writing about scientific errors is just the sharing of information. The original article with all its errors was out there in public, presumably because some readers would find it useful. These readers would be well served by finding out as soon as possible that the research is in error.

What I’m saying here is that, ultimately, this research is not being done for the benefit of author or the journal; it’s for the benefit of the readers of the journal, and ultimately for society. If you don’t want your work to be publicly discussed, you shouldn’t publish it. We make criticisms in public for the same reason that we write articles for publication, because we think this work is ultimately of relevance to people on the outside.