Red Team prepublication review update

A few months ago we wrote about the following project for prepublication review, as described by Ruben Arslan:

A colleague recently asked me to be a neutral arbiter on his Red Team challenge. He picked me because I was skeptical of his research plans at a conference and because I recently put out a bug bounty program for my blog, preprints, and publications (where people get paid if they find programming errors in my scientific code). . . . The Red Team approach is a bit different to my bounty program. Their challenge recruits five people who are given a $200 stipend to examine data, code, and manuscript. Each critical error they find yields a donation to charity, but it’s restricted to about a month of investigation. I [Arslan] have to arbitrate what is and isn’t critical (we set out some guidelines beforehand). . . . My own interest in this comes from my work as a reviewer and supervisor, where I often find errors, especially if people share their data cleaning scripts and not just their modelling scripts, but also from my own work. When I write software, I have some best practices to rely on and still make tons of mistakes. I’m trying to import these best practices to my scientific code. I’ve especially tried to come up with ways to improve after I recently corrected a published paper twice after someone found coding errors during a reanalysis . . .

They did the Red Team Review, and Arslan says:

It went well. 107 issues and some major problems among them (some nitpicking too).

Some details are here:

The Red Team Challenge (Part 1): Why I placed a bounty on my own research, by Nicholas Coles

The Red Team Challenge (Part 2): The Arbiter’s View, by Ruben Arslan

The Red Team Challenge (Part 3): Is it Feasible in Practice?, by Daniël Lakens and Leonid Tiokhin

This all looks good to me. I hope other people do this red team thing. I have sometimes paid people to find mistakes in my books, but it makes sense to think about these ideas more generally. If the authors are paying the reviewers, there’s a pleasant alignment of incentives, much different than the usual scientific publication process which has so many problems.

6 thoughts on “Red Team prepublication review update

    • It isn’t much for a day of work. The real reward is the privelege of reading the paper. Yes it is.

      There’s a big potential for reputational gains in the professional network, too.

Leave a Reply to Ruben Arslan Cancel reply

Your email address will not be published. Required fields are marked *