David Allison points us to this letter he wrote with Cynthia Kroeger and Andrew Brown:

Unsubstantiated conclusions in randomized controlled trial of binge eating program due to Differences in Nominal Significance (DINS) Error

Cachelin et al. tested the effects of a culturally adapted, Cognitive Behavioral Therapy-based, guided self-help (CBTgsh) intervention on binge eating reduction . . . The authors report finding a causal effect in their conclusion by stating,

Treatment with the CBTgsh program resulted in significant reductions in frequency of binge eating, depression, and psychological distress and 47.6% of the intention-to-treat CBTgsh group were abstinent from binge eating at follow-up. In contrast, no significant changes were found from pre- to 12-week follow-up assessments for the waitlisted group. Results indicate that CBTgsh can be effective in addressing the needs of Latinas who binge eat and can lead to improvements in symptoms.

This study is well-designed to test for causal effects between these groups; however, the authors did not conduct the statistical test needed to draw causal inference. Specifically, the authors base their conclusions from a parallel groups RCT on within-group analyses. Such analyses have been well-documented as invalid as tests for between-group treatment effects; instead, between-group tests should be utilized to inform conclusions (Bland & Altman, 2011; Gelman & Stern, 2006; Huck & McLean, 1975).

The Differences in Nominal Significance (DINS) error is a term used to describe this error of basing between-group conclusions on comparisons of the statistical significance of two (or more) separate tests . . . DINS errors are common within peer-reviewed obesity literature . . .

The difference between “significant” and “not significant” is not itself statistically significant.

Allison adds:

You might be interested in the Appendix titled “Why we are publishing this here instead of as a letter to the editor in the journal.”

Here’s the story:

Appendix: Why we are publishing this here instead of as a letter to the editor in the journal

We first contacted a Peer Review Manager of Psychological Services on May 12, 2018 to inquire as to how one should submit a Letter to the Editor to their journal, regarding an article published in their journal, because this article type was not an option in the author center of their online submission system.

On June 2, 2018, we received a reply from the Peer Review Manager stating the journal does not usually receive submissions like this, but that the editor confirmed we could submit it as a regular article and just explain that it is a Letter to the Editor in the cover letter.

On July 3, 2018, we followed these instructions and submitted the letter above.

On August 16, 2018, we wrote to inquire as to the status of our submitted letter. A reply from the Peer Review Manager was received on August 23, 2018 stating that the handling editor confirmed she is working on it and consulting with a reviewer.

On October 4, 2018, we received a decision from an Editor, stating that the editorial team was contacted and that they do not publish letters to the editor in this journal. Permission was requested to send a blinded version of our letter to the authors.

On October 13, 2018, the Peer Review Manager followed up to request permission to share a masked version of our letter with the authors of the original manuscript, so that they can potentially address the concerns.

On October 16, 2018, I replied to the Peer Review Manager indicating that I was unsure how to respond, because I was confused about the decision. I attached our previous email correspondence that mentioned how the Editor confirmed that we could submit the article as a regular article and explain that it was a Letter to the Editor in the cover letter – even though the journal normally does not have submissions like this. I mentioned that other journals often have original authors address concerns in a formal response and asked whether this person knew the process by which authors would address our concerns otherwise.

On October 17, 2018, I received an email from the Editor, indicating that our letter was read through carefully and our concerns were taken seriously. Multiple statisticians were consulted to see how to best address the concerns. The editorial team also was consulted with. The editorial team agreed that given the nature of our letter was statistical procedures related to a study, that publishing it in the journal was not the best step. It was explained that they reject papers that are not in the area of typical foci of their journal or readership, and that this is why the letter could not be accepted for publication. It was explained that they agreed the best step was to reach out to the author that needed corrections and wanted permission to send our blinded letter to them, so they can make the corrections.

On October 23, 2018, we responded to the Editor and provided consent to share our letter with the original authors. We also shared with them our plan to publish our submitted letter as a comment in PubPeer and update our comment as progress continues.

The Editor responded on October 23, 2018, asking whether we still wanted our letter to be blinded when they share it with original authors.

We responded on October 24, 2018, giving permission to send our letter unblinded. We said to please let us know if we can help further in any way and to please feel free to share with the original authors that we are happy to help if they think that might be useful. The Editor responded affirmatively and thanked us.

In sum, we decided to post our letter here. Posting our concerns here is in line with the COPE ideal of quickly making the scientific community aware of an issue. We also find the editor’s decision to not allow for the publication of LTEs discussing errors in their papers to be counter to the ideals of rigor, reproducibility, and transparency.

This is very similar to what happened to me when I tried to publish a letter pointing out a problem in a paper published in the American Sociological Review. I eventually gave up and just published the story in Chance. That was a few years ago. Had it happened more recently, I would’ve submitted it to Sociological Science.

Just today, someone sent me another story of this sort: A paper with serious statistical errors appeared in a medical journal, my correspondent found some problems with it but the journal editor refused to do anything about it. (In this case, the data were not made available, making it difficult to figure out what exactly was going on.) My correspondent was stuck, didn’t know what to do. I suggested publishing the criticism as a short article in a different journal in the same medical subfield. We’ll see what happens.

P.S. Since we’re on the topic of publication and conflict of interest, here’s an unrelated story.