On fatally-flawed, valueless papers that journals refuse to retract

Commenter Carlos pointed us to this story (update here) of some scientists—Florin Moldoveanu, Richard Gill, and five others—all of whom seem to know what they’re talking about and who are indignant that the famous Royal Society of London published a paper that’s complete B.S. and then refused to retract it when the error was pointed out. I understand that indignant feeling. The Royal Society journal did publish an “expression of concern” about the fatally-flawed paper, and, that’s something, but I understand the frustration of Moldoveanu, Gill, et al., that:

1. The expression of concern does not clearly state that the paper is wrong, instead saying vaguely that “there was a divergence of opinion.” Yes, there’s a diversion of opinion: some people say the earth is flat, some do not.

2. The expression of concern expresses things conditionally: “a controversial paper may eventually be shown to contain flaws.” But this is misleading, because it does not clearly state that (a) the paper is “controversial” because it has a mathematical error that destroys its central argument, and (b) it’s not that the paper “may eventually be shown” to contain flaws; it’s that these flaws have already been publicly pointed out.

3. The flaws were also pointed out in the original review process and the editors simply disregarded the review that pointed out the fatal flaw in the paper.

4. Yes, the expression of concern is out there, but if you go to the original paper, you’ll find only a very subtle link to the expression of concern:

And the pdf of the article doesn’t mention the expression of concern at all!

Going medieval?

At this point, I’m ready to go medieval on the Royal Society, an organization which seems justly proud of its long history:

We published Isaac Newton’s Principia Mathematica, and Benjamin Franklin’s kite experiment demonstrating the electrical nature of lightning. We backed James Cook’s journey to Tahiti, reaching Australia and New Zealand, to track the Transit of Venus. We published the first report in English of inoculation against disease, approved Charles Babbage’s Difference Engine, documented the eruption of Krakatoa and published Chadwick’s detection of the neutron that would lead to the unleashing of the atom.

The Royal Society’s motto ‘Nullius in verba’ is taken to mean ‘take nobody’s word for it’. It is an expression of the determination of Fellows to withstand the domination of authority and to verify all statements by an appeal to facts determined by experiment.

Their leading journal, or at least the first one listed on their journals page, is called Royal Society Open Science. And that’s the journal that refuses to retract the offending paper.

Now, at this point, you could say, Hey, give them a break, this is theoretical physics we’re talking about, it’s super-complicated! To which I’d reply: sure, but there are some experts on theoretical physics out there, no? What’s the point of the Royal Society publishing anything at all on theoretical physics if they’re not gonna check it? If you want to publish papers with mathematical errors, we already have Arxiv, right? The Royal Society is supposed to be providing some value added, but here they seem to be just hiding behind the obscurity of theoretical physics. They’re refusing to make a judgment. Which, again, fine, you can refuse to make a judgment. But then why go through the referee process at all?

At this point, the Royal Society is looking almost as bad as Lancet. OK, not as bad as Lancet: the Royal Society didn’t let a fraudulent anti-vax paper sit in their journal for 12 years, and they didn’t go to the press and social media to defend a fraudulent coronavirus paper. So, OK, the Royal Society isn’t Lancet bad, but they’re still refusing to retract a paper that’s been shown to be wrong.

But, then again, everybody does it!

So, I was all ready to work myself into a righteous fury, but then I remembered . . .

Statistical Science published the Bible code paper in 1994 and never retracted it! Yes, they later published a demolition of that paper, but if you go back to the original Bible Code article, there’s no retraction, no correction, no mention of the refutation, and no expression of concern.

The Journal of Personality and Social Psychology published the ESP paper in 2011 and never retracted it. Many people have demolished that paper (hardly necessary, considering how bad it is), but if you to the article on the American Psychology Association’s website, there’s no retraction, no correction, no mention of the refutation, and no expression of concern.

The Bible code paper and the ESP paper are just as wrong, just as methodologically flawed, and just as bad as the recent Royal Society physics paper. The errors in all three of these papers are unambiguous—and they were unambiguous at the time. Indeed, I have the sense that the editorial boards of Statistical Science in 1994 and the Journal of Personality and Social Psychology in 2011 were pretty sure that these articles were B.S., while they were considering publishing them. And, of course, like the Royal Society paper, these articles made claims that would be hard to interpret as anything other than violations of the laws of physics. Why did those journals publish these terrible submissions? My guess is that they were bending over backward to be fair, to not play the role of censor. I don’t have any easy answers to that particular problem, except to spare a thought for poor Brian Wansink: He doesn’t seem to have any publications in Statistical Science or the Journal of Personality and Social Psychology (or, for that matter, Lancet or Royal Society Open Science), despite having written dozens of papers that are no worse than the ones discussed above. How fair is that?

Conclusion

So, yeah, I don’t really know what to say about all this. In the four examples above, the decision seems clear enough: the papers never should’ve been published. And, once the fatal flaws had been pointed out, they should’ve been retracted. But a general policy is not so clear. And, you know that problem when you repaint a dirty wall in your apartment, and then you realize you need to paint the rest of the walls in that room, then you need to paint the other rooms in the apartment? It’s the same thing: once we ask the Royal Society to start retracting papers that have fatal errors and no redeeming qualities, where do we stop? No easy answers.

I’d like to call this an example of the research incumbency rule, whereby flaws that would easily get an article to lose in the review process, are brushed aside once the article has been published. But that’s not quite right, given that all the above-discussed articles had flaws that must have been obvious during the review process as well.

P.S. More here from Retraction Watch.

12 thoughts on “On fatally-flawed, valueless papers that journals refuse to retract

    • So I started to read the abstract to figure out if his claims are at least plausible, and then I realized that this guy is actually somewhat well-known, see https://golem.ph.utexas.edu/category/2020/09/new_normed_division_algebra_fo.html

      Basically it’s a well-known classical result that there are only so many algebras of a certain type, and Joy Christian is convinced this result is wrong. I guess the abstract of the linked paper itself doesn’t directly ring any alarm bells unless one is looking for them; he doesn’t directly claim to have found a normed division algebra. I can kind of understand why a reviewer would give the paper a pass; not of the “I’m convinced this paper is right” but of the “I didn’t have time to check the proofs but what is being done seems interesting and there are no obvious red flags” type. If I had to review the paper, my first guess is that the guy is reinventing something (correct) that already exists somewhere, I mean he spends a lot of time introducing very basic concepts in convoluted notation which a reviewer won’t have patience for; and thus a reviewer can’t point to a specific error (can you spot it?) with which to reject. I started seeing red flags only on page 27 (“and this interpretation depends entirely on the argument put forward by Bell and his followers [5, 14]. This argument, however, is fatally flawed, as we now demonstrate”, and a few pages later are numerical experiments which any reviewer will ignore), but I was looking for them. Basically the paper doesn’t look like a crank paper, and since it isn’t a super-prestigious theoretical physics paper, is the postdoc reviewing it going to be a dick and reject it just because she doesn’t have the expertise?

  1. I’d like to play the devil’s advocate and say that this is less of a big deal than it is made out to be. I exaggerate (a little bit). First of all, this is in “Royal Society Open Science” which is not a theoretical physics journal. Secondly: the scientific literature is full of mistakes. This includes areas like theoretical physics (which I have no experience with) and mathematics (in which I have both written and reviewed journal papers), where it is not even clear whether reviewers “should” check proofs (see e.g. https://mathoverflow.net/questions/40729/does-a-referee-have-to-check-carefully-the-proof , especially the anecdote from the first reply with the quote “No one wants a fight. We publish them and then ignore them.”).

    Some poor graduate student may one day find Christiansen’s work and waste a few months until his advisor tells him that Bell’s inequality is still true. In this case, looking at articles that cite the work (which is the first thing a graduate student should do if she wants to build on it) already reveals the mistake. This is unfortunate, but mathematicians (and theoretical physicists) are to some extent expected to try to understand the papers they are citing unless they really are well-known in the community to be correct. All the incentives are stacked towards keeping incorrect results in the literature (so many things depend on having published papers, nobody wants to embarrass a colleague, most papers don’t get read at all), and some mistakes aren’t fatal, see http://alumni.media.mit.edu/~cahn/life/gian-carlo-rota-10-lessons.html#mistakes

  2. I’m a little embarrassed, but the link to the expression of concern, both in the image and in the link, is so subtle that I actually cannot find it at all.

    • Vampyricon:

      You link to a paper entitled, “Scientific Conclusions Need Not Be Accurate, Justified, or Believed by their Authors,” That’s fine with me. But in such cases I think authors should clearly state in the title or the abstract of the paper that they don’t believe the conclusions, and then in the body of the paper they could explain their reasoning.

    • I only glanced at the paper (philosophical writings are more dense than I can bear – probably true of any writing from a different discipline), but I think a fatal flaw is the distinction between writing for the public/decision makers and writing for the scientific community. If different standards apply to these two areas, then there must be ways of fencing them off from each other – but there are not. Scientists and decision makers way multiple hats and there is no way to enforce a distinction between the two. So, I don’t see how different standards can be applied to the two realms.

Leave a Reply to Andrew Cancel reply

Your email address will not be published. Required fields are marked *