What to do when you read a paper and it’s full of errors and the author won’t share the data or be open about the analysis?

Someone writes:

I would like to ask you for an advice regarding obtaining data for reanalysis purposes from an author who has multiple papers with statistical errors and doesn’t want to share the data.

Recently, I reviewed a paper that included numbers that had some of the reported statistics that were mathematically impossible. As the first author of that paper wrote another paper in the past with one of my collaborators, I have checked their paper and also found multiple errors (GRIM, DF, inappropriate statistical tests, etc.). I have enquired my collaborator about it and she followed up with the first author who has done the analysis and said that he agreed to write an erratum.

Independently, I have checked further 3 papers from that author and all of them had a number of errors, which sheer number is comparable to what was found in Wansink’s case. At that stage I have contacted the first author of these papers asking him about the data for reanalysis purposes. As the email was unanswered, after 2 weeks I have followed up mentioning this time that I have found a number of errors in these papers and included his lab’s contact email address. This time I received a response swiftly and was told that these papers were peer-reviewed so if there were any errors they would have been caught (sic!), that for privacy reasons the data cannot be shared with me and I was asked to send a list of errors that I found. In my response I sent the list of errors and emphasized the importance of independent reanalysis and pointed out that the data comes from lab experiments and any personally identifiable information can be removed as it is not needed for reanalysis. After 3 weeks of waiting, and another email sent in the meantime, the author wrote that he is busy, but had time to check the analysis of one of the papers. In his response, he said that some of the mathematically impossible DFs were wrongly copied numbers, while the inconsistent statistics were due to wrong cells in the excel file selected that supposedly don’t change much. Moreover, he blamed the reviewers for not catching these mistypes (sic!) and said that he found the errors only after I contacted him. The problem is that it is the same paper for which my collaborator said that they checked the results already, so he must have been aware of these problems even before my initial email (I didn’t mention that I know that collaborator).

So here is my dilemma how to proceed. Considering that there are multiple errors, of multiple types across multiple papers it is really hard to trust anything else reported in them. The author clearly does not intend to share the data with me so I cannot verify if the data exists at all. If it doesn’t, as I have sent him the list of errors, he could reverse engineer what tools I have used and come up with numbers that will pass the tests that can be done based solely on the reported statistics.

As you may have more experience dealing with such situations, I thought that I may ask you for an advice how to proceed. Would you suggest contacting the involved publishers, going public or something else?

My reply:

I hate to say it, but your best option here might be to give up. The kind of people who lie and cheat about their published work may also play dirty in other ways. So is it really worth it to tangle with these people? I have no idea about your particular case and am just speaking on general principles here.

You could try contacting the journal editor. Some journal editors really don’t like to find out that they’ve published erroneous work; others would prefer to sweep any such problems under the rug, either because they have personal connections to the offenders or just because they don’t want to deal with cheaters, as this is unpleasant.

Remember: journal editing is a volunteer job, and people sign up for it because they want to publish exciting new work, or maybe because they enjoy the power trip, or maybe out of a sense of duty—but, in any case, they typically aren’t in it for the controversy. So, if you do get a journal editor who can help on this, great, but don’t be surprised if the editors slink away from the problem, for example by putting the burden in your lap by saying that your only option is to submit your critique in the form of an article for the journal, which can then be sent to the author of the original paper for review, and then rejected on the grounds that it’s not important enough to publish.

Maybe you could get Retraction Watch to write something on this dude?

Also is the paper listed on PubPeer? If so, you could comment there.

13 thoughts on “What to do when you read a paper and it’s full of errors and the author won’t share the data or be open about the analysis?

  1. Rather than contact the journal editor, contact the journal! Any correspondence about the paper should go through the journal anyway, so it’s better to start there. The journal may well have a professional managing editor, who should know how to handle this. It’s also worth checking to see if the journal is a member of COPE: if so they should have practices in place to handle complaints.

  2. About 7 years ago I came across a very similar set of papers and I had largely similar experiences. The author first denied that there were problems, then went the “this went through peer review” route, refused to share data, argued that the presentation of mathematically impossible results are common practice in my field (turns out they are right), and ultimately threatened me with lawsuits via his lawyer. When I contacted editors I had very mixed responses. Some acted and retracted papers or issued expressions of concern, some refused to act until I went to the publisher, some told me that they could not act because they feared getting sued, and some editors refused to do anything at all. The paper with the biggest influence on the field and the most obvious and numerous errors never went corrected but is described here (there are even more errors than documented on pubpeer):
    https://pubpeer.com/publications/1E79BA4AA94EB722491B14AE871B0F

    My advice would be to contact editors anonymously. COPE guidelines are that editors need to investigate even if contacted anonymously and the cost to your career if you identify yourself is not worth it in my opinion and my experience.

  3. Dear someone,

    1. prepare an exhaustive report with all errors and all anomolies of all papers and upload this report to the public part of the internet, for example as project at OSF.
    2. ensure that all errors and all anomalies of all papers are as well posted on Pubpeer.
    3. contact the editors and the publishers of all papers after you have prepared and uploaded the report and posted the comments at Pubpeer.
    4. contact as well all co-authors of the papers and ask them for comments on the report with the findings.
    5. don’t expect that all editors and/or all publishers and/or COPE and/or OASPA and/or all co-authors and/or their affiliations are willing to communicate with you about any of the findings of your report.
    6. stay polite.
    7. don’t hesitate to file formal complaints with serious accusations of research misconduct against individuals and/or institutes when you have serious evidence that this is indeed the case.
    8. don’t expect that all institutes are willing to process such complaints and/or are willing to communicate with you about such complaints.
    9. publish papers in peer-reviewed journals about your experiences. An example is https://riviste.unimi.it/index.php/roars/article/view/9073

    Disclaimer: I am working together with others, and already for over 3 years, to get retracted a fraudulent study on the breeding biology of the Basra Reed Warbler in a Taylor & Francis journal. See https://osf.io/5pnk7/ and https://www.researchgate.net/project/Retracting-fraudulent-articles-on-the-breeding-biology-of-the-Basra-Reed-Warbler-Acrocephalus-griseldis for backgrounds. See also https://osf.io/ajsvw/ (in Arab).

  4. I don’t think “give up” can be an option here.
    Nor it should be an option to choose the “best” option _for us_.
    If there is a “best” option to choose here this must be the best option for the science we vaunt to contribute and to represent.

    I think that lies and cheating cannot be tolerated on scientific journals.
    I think that whenever we find out any kind of mistake on published work, it must be submitted to the authors, submitted to the journal editor and, if anything works, made public.

    I have been in the very same situation multiple times, and yes, I can attest that each of those three steps has unpleasant consequences.
    But to me this is a matter of sense of responsability.
    Published papers are the bricks of future developments of science.
    It is unacceptable to me to believe that we must accept we are potentially building faulty science on faulty bricks just because in cases like this “it is better to give up”.

  5. I’ve been dealing with the same kind of problem with the cover story for the 8 Dec 2017 issue of Science, except that I am able to show just how the authors got their data wrong. The editors eventually asked for a technical comment, although it is past the normal time limit for comments, but seem to be waiting for the authors to respond.

    I agree with Annalisa that giving up is not a good option. Rather, be polite but persistent.

  6. I think it depends a bit on considerations like this:

    1. Will anyone die? I’d feel much differently about a social psychology paper than one in, say Anesthesiology.
    2. Are these papers important in the field? The citation count might be one way to gauge this.

  7. What can be done in such cases depends strongly on which journal(s) is (are) involved, I think. It depends also a lot on the research field.

    Please let me explain why the situation is a bit better in my field, chemical crystallography: most journals in this field are published by the International Union of Crystallography (IUCr), a strong, global, and well-respected authority on the matter. Publication workup is based on a pyramidal organization including one Editor-in-Chief (1), Managing editors (1 per journal), Section editors (2-5 per journal), Co-editors (10-40 per journal). In some cases, the Editorial board is completed with a “Review board” (20-60 per journal). More importantly, every published item mentions who edited the paper (should be a Section editor or a Co-editor). With such a crystal-clear organization, you do know whom to contact for any issue regarding data availability, unresponsive authors, etc. This does not mean that the issue will be solved, but at least, this prevents any waste of time. In contrast, my own experience with questionable papers not published by the IUCr is very poor, particularly if authors are from China or India. For these countries, my success rate over 20 years for obtaining raw data (i.e. diffraction intensities collected with a diffractometer, in mot cases) is exactly 0%.

    As commented by Andrew in the post, PubPeer is a rather efficient alternative [see for example https://pubpeer.com/publications/CAFFF08EEAC3FEABFE7BBCD8A5D956%5D. There is a drawback, however: too many authors still consider PubPeer as being a finger-pointing machinery. Finally, correcting the literature is not impossible, although the process is time-consuming for something which is regarded as a “non-positive” contribution [for a recent example, see: 10.1039/C8RA05162C]. Moreover, it is well known that frequently, the original paper will sum more citations than the relevant paper correcting the original one…

  8. Don’t worry, I hear that new editors have taken over as of today and they will make it a lot harder for him to get away with this sort of thing. And after November 2020 he may well be out of a job.

  9. I taught a course on computational methods for social scientists as a postdoc. For the final project, each student had to attempt to reproduce a paper of their choice as preparation for their dissertation research. Needless to say, this had an interesting outcome because so many “experts” in my field of Economics don’t care about correctly computing anything…. When you look at the literature on the reproducibility of research, the results are terrifying. See the many papers by McCullough et al on the subject.

  10. Dear all,

    I am the person who contacted Andrew about this issue and I would like to thank all of you for your suggestions and different points of view. Since my initial communication with Andrew, I have contacted the respective editors, program committee chairs and publishers (some of the work was published in conference proceedings which in Engineering can be comparable with journals, which is not necessarily the case in other fields). As some of you pointed out, a single email was not enough so I had to be persistent with it, but ultimately at least some editors responded. The response from the publisher (IEEE) was not helpful at all as they don’t feel responsible for the work that they publish at all. The two editors who responded, contacted the author, but their requests for sharing the data were also rejected. They requested further explanations from the author so at least there is some progress, even if extremely slow.

    As for the PubPeer, the papers are not there as it is not used in my field (there are only 2 papers with field relevant keywords and both were published in journals from other fields).

    With all that being said, the biggest weakness that I find in the approach that I chose, is that each paper is dealt with one by one. In the best case, the papers will be corrected/retracted, but the big question here is how is it possible that the author published series of erroneous papers. I.e. it is bad to publish a paper with errors, but everyone sometimes makes mistakes. On the other hand, consistently publishing erroneous work cannot be simply explained by having an off day and speaks more about the author’s general standards. I would be curious if there is another approach to this issue that is not focused on single papers, but rather look at the big picture of the author’s work.

    • I suppose you could look at Wansink as a model. By attacking enough individual pieces, the pattern became clear, and ultimately he did have to step down. My memory is dim (it has been a long two years) but it feels like that “worked” in a crowdfunding sort of way – multiple people developed an interest in exposing him, and so the work of taking apart his bad papers became light.

      I think if you a directly at eg geting this person fired/expelled, your project will look like the vendetta of a crazy person. I think building your case the old plodding way will ultimately have a better chance of success.

      I do wonder what the base rate of fatal errors is in science.

      On another note, does IEEE have some relevant ethical standards you could point to? Is the author a member?

Leave a Reply to Annalisa Cancel reply

Your email address will not be published. Required fields are marked *