Former editor of British Medical Journal says we should “assume that the research is fraudulent until there is some evidence to support it having happened and been honestly reported”

Chetan Chawla points us to this op-ed from Richard Smith, former editor of the British Medical Journal, who writes, “the time may have come to stop assuming that research actually happened and is honestly reported, and assume that the research is fraudulent until there is some evidence to support it having happened and been honestly reported.”

There’s also the problem with non-fraudulent articles that are just crap; see here for a discussion of an example from BMJ (long after Smith’s time as editor of the journal). Recall Clarke’s Law.

Anyway, it’s refreshing to see this sort of openness on Smith’s part but it’s also kind of scary. Bad science is annoying but it’s a relatively stationary target: we can point to the published mistakes, and, although this won’t deter the Gladwells of the world, it gives some of us a sort of paper trail that can help us make sense of these disputes. But bad claims in the news media and social media just twist and turn (see for example the discussion here). I guess we’ll continue to have a need for vetted scientific claims. It’ll just be harder on journalists who can’t just assume that “peer reviewed,” “PNAS,” “JAMA,” etc., are enuf.

17 thoughts on “Former editor of British Medical Journal says we should “assume that the research is fraudulent until there is some evidence to support it having happened and been honestly reported”

  1. Clarke’s Law is accurate, but what makes it scary is when you combine it with Sturgeon’s Law https://en.wikipedia.org/wiki/Sturgeon%27s_law you get the result that Richard Smith warns against.

    The journal editors were supposed to be gatekeepers that *only* let fraud slip through. They were supposed to stop mere crap. (Well, there were always journals that would let anything through, which is why some journals were more prestigious than others.) But they manifestly aren’t doing their job any more. I can think of a bunch of reasons:

    (1) More and more research is just too complicated to vet effectively. Even in the rare cases where there is transparency, a willingness to dig in ahs taken a back seat to reviewer carping around the edges of the methodology, which requires much less effort.
    (2) The incentives on editors to keep crap out has weakened. (For extra credit: opine why this might be.)
    (3) The rise of ArXiv and SSRN and various other nontraditional methods has made traditional publishing less important (except for tenure purposes) and has enabled claims to be made outside the traditional structure, which subtly shifts the status quo of editing.
    (4) Editors no longer take their reputations as seriously as they used to and therefore take the editing process less seriously. Or a similar effect for the reviewers on whim the editors rely.

  2. I’m not entirely sure that there are many reliable methods to verify that a study actually happened. Data can be easily simulated for many cases, so presence of data is no reliable sign. Ethics approval only means the study was designed, not that it happened. Heck, you don’t even *really* know for sure that qualtrics is getting you data from real humans! Maybe in some cases there are physical records or witnesses or videos, but that’s not something that could be vetted regularly.

    The truly scary thing is that the scientific method only works if people are honest and transparent. I don’t see what else we can really do except practice, teach, and reward these values.

    • I agree with what you say, but maybe we can also do something about the systems of incentives that drive authors and reviewers and editors. For example, my wife used to spend days writing careful and helpful reviews of papers, and got about zero reward for it from her university.

      • It’d be nice if we could, that’s for sure. I’ve gotten a bit more jaded over the years regarding incentives. We can reshape science into a different image by prioritizing prereg, data sharing, avoiding NHST and the like but ultimately, every set of incentives can be gamed. Even a lot of the psychology reform movement suggestions feel like it’s trading one set of heuristics for another. I think science needs to be pluralistic, with many paths to success.

        I think a lot of the most valuable scientific work (like your wife’s reviewing) happens out of the spotlight, done for intrinsic reasons. These days, when someone is very famous and eminent in a field and I don’t know them, my default position is to trust the accuracy of their research less until I have contrary evidence from reading their work or seeing their process.

        Maybe that’s TOO jaded though!

  3. About time.

    > Mol, like Roberts, has conducted systematic reviews only to realise that most of the trials included either were zombie trials that were fatally flawed or were untrustworthy.
    In Smith’s editorial there was a mention of a mix of fatally flawed or were untrustworthy – I do think most of the problems are of that sort but fraud gets more attention. Now the process of evaluating fraud could inform a studies flaws untrustworthyness – right.

    Also encouraging is Cochrane Collaboration stopping dragging their feet on this issue – it has been discussed for years. At one time they even banned the phrase “study quality” as seeming to be too judgmental of authors and wanted the phrase of “risk of bias” to replace it.

  4. The fundamental problems relate to the institutional structure of the material incentives for researchers. Until this aspect is remedied the problem will only get worse.

  5. Isn’t “fraudulent” a bit harsh? My thinking is influenced by Kuhn. The majority, the vast majority, of work in any discipline doesn’t advance things, and true breakthrough paradigm busters are rare. However, fraudulent means criminal, deliberately deceitful actions to me. Almost all of the articles published during my professional career were pretty boring, but true evil was rare. And despite the largely barren nature of it all, progress was made. As an example, this article https://pubmed.ncbi.nlm.nih.gov/24778131/
    probably seemed pretty unimpressive in 2013. And not many thought about the authors for long. We tend to extreme words in our discourse. Maybe we should reserve opprobrium for a few.

  6. The following really highlights the problem.
    Miyakawa (2020), the editor-in-chief of Molecular Brain, wrote to 41 authors asking them to provide further evidence (usually raw data) to back up some of the claims made in their papers before sending the papers out for peer review. The response was depressing.– Twenty-one (51%) of these papers were withdrawn.– Nineteen (46%) did not provide appropriate data to address Miyakawa’s concerns. – One (ONE! Just 2%) provided satisfactory data. I made a icon plot of stick figures to show this on page 2 of https://www.degruyter.com/document/doi/10.1515/edu-2020-0106/pdf. It is depressing.

  7. As a historian and philologist, I think its very important that data-collection and analysis are usually separate. That is, the person who publishes 543 Roman inscriptions is not the same person who uses three of them to argue that a particular person held specific offices under a particular emperor. This avoids many of the dangers in the statistical sciences where dishonest researchers can make up or selectively collect data.

Leave a Reply

Your email address will not be published. Required fields are marked *