She sent a letter pointing out problems with a published article, the reviewers agreed that her comments were valid, but the journal didn’t publish her letter because “the policy among editors is not to accept comments.”

The journal in question is called The Economic Journal. To add insult to injury, the editor wrote the following when announcing they wouldn’t publish the letter:

My [the editor’s] assessment is that this paper is a better fit for a field journal in education.

OK, let me get this straight. The original paper, which was seriously flawed, was ok for Mister Big Shot Journal. But a letter pointing out those flaws . . . that’s just good enough for a Little Baby Field Journal.

That doesn’t make sense to me. I mean, sure, when it comes to the motivations of the people involved, it makes perfect sense: their job as journal editors is to give out gold stars to academics who write the sorts of big-impact papers they want to publish; to publish critical letters would devalue these stars. But from a scientific standpoint, it doesn’t make sense. If the statement, “Claim X is supported by evidence Y,” was considered publishable in a general-interest journal, then the statement “Claim X is not supported by evidence Y” should also be publishable.

It’s tricky, though. It only works if the initial, flawed, claim was actually published. Consider this example:

Scenario A:
– A photographer disseminates a blurry picture and says, “Hey—evidence of Bigfoot!”
– The Journal of the Royal Society of Zoology publishes the picture under the title, “Evidence of Bigfoot.”
– An investigator shows that this could well just be a blurry photograph of some dude in a Chewbacca suit.
– The investigator submits her report to the Journal of the Royal Society of Zoology.

Scenario B:
– A photographer disseminates a blurry picture and says, “Hey—evidence of Bigfoot!”
– An investigator shows that this could well just be a blurry photograph of some dude in a Chewbacca suit.
– The investigator submits her report to the Journal of the Royal Society of Zoology.

What should JRSZ do? The answer seems pretty clear to me. In scenario A, JRSZ should publish the investigator’s report. In scenario B, they shouldn’t bother.

Similarly, the Journal of Personality and Social Psychology decided in its finite wisdom to publish that silly ESP article in 2011. As far as I’m concerned, this puts them on the hook to publish a few dozen articles showing no evidence for ESP. They made their choice, now they should live with it.

Background

Here’s the story, sent to me by an economist who would like anonymity:

I thought that you may find the following story quite revealing and perhaps you may want to talk about it in your blog.

In a nutshell, a PhD student, Claudia Troccoli, replicated a paper published in a top Economics journal and she found a major statistical mistake that invalidates the main results. She wrote a comment and sent it to the journal. Six months later she heard from the journal. She received two very positive referee reports supporting her critique, but the editor decided to reject the comment because he had just learned that the journal has an (unwritten) policy of not accepting comments. Another depressing element of the story is that the original paper was a classical example where a combination of lack of statistical power and multiple testing leads to implausible large effects (probably one order of magnitude of what one would have expected based on the literature). It is quite worrying that some editors in top economic journals are still unable to detect the pattern.

The student explained yesterday this story in twitter here and she has posted the comment, the editor letter, and referee reports here.

This story reminds me of my experience with the American Sociological Review a few years ago. They did not want to publish a letter of mine pointing out flaws in a paper they’d published, and their reason was that my letter was not important enough. I don’t buy that reasoning. Assuming the originally published paper was itself important (if not, the journal wouldn’t have published it), I’d say that pointing out the lack of empirical support for a claim in that paper was also important. Not as important as the original paper, which made many points that were not invalidated by my criticism—but, then again, my letter was much shorter than that paper! I think it had about the same amount of importance per page.

Beyond this, I think journals have the obligation to correct errors in the papers they’ve published, once those errors have been pointed out to them. Unfortunately, most journals seem to have a pretty strong policy not to do that.

As Trocolli wrote of the Economics Journal:

The behavior of the journal reflects an incentive problem. No journal likes to admit mistakes. However, as a profession, it is crucial that we have mechanisms to correct errors in published papers and encourage replication.

I agree. And “the profession” is all of science, not just economics.

Not too late for a royal intervention?

I googled *Economic Journal* and found this page, which says that it’s “the Royal Economic Society’s flagship title.” Kind of horrible of the Royal Economic Society to not correct its errors, no?

Perhaps the queen or Meghan Markle or someone like that could step in and fix this mess. Maybe Prince Andrew, as he’s somewhat of a scientific expert—didn’t he write something for the Edge Foundation once? I mean, what’s the point of having a royal society if you can’t get some royal input when needed? It’s a constitutional monarchy, right?

P.S. Werner sends in this picture of a cat that came up to him on a park bench at the lake of Konstanz in Germany and who doesn’t act like a gatekeeper at all.

45 thoughts on “She sent a letter pointing out problems with a published article, the reviewers agreed that her comments were valid, but the journal didn’t publish her letter because “the policy among editors is not to accept comments.”

  1. This seems to be a pretty generic problem, with a few honorable exceptions such as Physical Review Letters. I think this idea has been proposed before, but we need a broad-ranging journal of Comments to make up the gap. Otherwise there’s no way to intervene when a field goes off the rails, barring Royal or even better Divine intervention. The journal should be more rigorous than the ones being criticized, to avoid a diverging sequence. Statisticians, biologists, physical scientists and engineers should be involved.

    • Wonks:

      We do make fun of Tol here, but I don’t enjoy it. I laugh as a substitute for crying. It makes me so sad when researchers just make the same mistakes over and over again, as if their goal is to amass a publication record rather than to help understand the world.

      • I confess: I wanted to understand the world; but I wanted to send reprints to my grandmother who kept them proudly in that copper kettle by the TV. It was the copper kettle that mattered!

  2. An identical thing happened to me with Plos Medicine. A paper published there claimed to find a link between droughts and intimate-partner violence using flawed methods. I had been working for awhile using the same data to address the same topic, and found no link. So we submitted an article showing that the original was flawed, and they rejected our article – for reasons that also applied to the flawed paper theyd already published! Comparing the uncritical reviewer comments for the flawed published study (which found a significant effect) to the comments our rejected study (which found no effect) is a case study in confirmation bias and motivated reasoning.

    It really makes me more suspicious of a lot of published work, and I’m strongly considering leaving academia after this postdoc.

    • Matthew:

      Yes, here’s the policy from the Plos Medicine webpage:

      The editors make decisions on submissions based upon their potential to directly and substantially inform clinical practice or health policy, and their relevance to our international audience.

      If the original published article had the “potential to directly and substantially inform clinical practice or health policy,” then presumably a paper finding a serious flaw in the article would also have this potential.

      • Yes, they claimed that because our analysis was on the effects of drought, there study had no relevance to a non-medical audience because drought is non-modifiable. Of course this also applied to the original, flawed paper.

  3. Andrew –

    Are there any reasons you can think of, why journals would have a policy of not publishing comments, other than your speculation here?:

    > I mean, sure, when it comes to the motivations of the people involved, it makes perfect sense: their job as journal editors is to give out gold stars to academics who write the sorts of big-impact papers they want to publish; to publish critical letters would devalue these stars. But from a scientific standpoint, it doesn’t make sense.

    I get why, if you thought a particular article was important enough to publish in the first place, it pretty much demands that you publish an article that shows how the first article was flawed.

    But I would imagine there might also be reasons, consistent with a goal of advancing science, to have a policy of not publishing comments – perhaps along the lines of slippery slope reasoning (I hate slippery slope reasoning, FWIW).

    My point being, I think it’s best to avoid motive-impugning unless you really have passed a high bar of supporting causal evidence, unless you have mind-reading skills – in which case you should publish a paper where you prove their efficacy.

    • I’m just speculating here, but I remember reading once (I think it was in the Chronicle of Higher Ed, but I don’t remember now) a journal editor who said they hated replication studies, because in the ideal world no one would cite the original, and (they claimed) no one cites the article that (failed) to replicate the original. I understood him to be saying that journals are ranked by metrics that count citations, so the incentives are structured so that you don’t publish things that won’t get cited(especially if the effect is to decrease citations of the original that would have occurred . Obviously, this is part of the larger problem with the incentives in academia nowadays. This was a while ago (probably over a decade) so things might be different now or I might be misremembering.

      • Joe –

        Thanks. I don’t doubt that perverse incentives exist. If the incentive you describe were the only one, or just even the driving one for journals to not public comments or replication studies,it would certainly be an indictment.

        But I am skeptical as to whether we can rule out that other, more palatable incentives also exist. And then there also seems to me to be a logic problem. If a journal were to get their hands on a paper that thoroughly debunks a widely accepted prior study, that 2nd paper would presumably get a lot of citations. Of course, with comments, it might be a somewhat different scenario.

        • Hi Sameera –

          > What palatable incentives do you mean?

          I don’t have any in particular in mind.

          But I think that training from personal incredulity to ascribe motivations to others is a problematic approach (and borders on a fallacy).

          My assumption is that sometimes people who are in the sciences have a limited motivation to advance science relative to their interest in things like profit or personal gain in some other sense – but that also, sometimes, we can ascribe motivations in error, because of limitations in our own cognitive empathy.

          Cognitive empathy, also known as empathic accuracy, involves “having more complete and accurate knowledge about the contents of another person’s mind, including how the person feels,”

          I think that in general, online discourse would be improved by more circumspection with respect to assigning motivations.

          So, while I certainly get the logic that if the original paper is worth publishing then it’s worthwhile publishing analyses that demonstrates flaws – I’m also skeptical that there aren’t any explanations – at least in the mind of the editors – as to why there’s a policy against publishing comments that isn’t inconsistent with the goal of advancing the science. So isn’t that a worthwhile question to interrogate?

        • I should add, that part of my motivation here is a desire to pushback against what I think is a more general piling on against academic publishing.

          I think that there are certainly a host of sub-optimal aspects related to academic publishing, but IMO, there’s also a lot of politically or ideologically expedient exploitation of those problems, and much of it is closely tied to assigning motivations to editors or scientists via a fallacious pathway of arguing from incredulity. I think that’s a problem of equal or perhaps greater magnitude than the problems that in truth, certainly DO degrade the value of academic publishing.

          So I don’t want to be a tone troll, but again, I do think it’s important to be circumspect when making assumptions about, or even working hard to understand, the motivations of actors in academic publishing and whether the root causes are (the very real) perverse incentives rather than more benign or even unrecognized positive motivations.

        • I’ll respond here because of the embedding. I can’t say that the two comments changed my mind, although as I said, I don’t really know of a solution either. I can see the current situation is the result of trying to make the grant and job process more open. Under an old-boy system, there may not be much of a rush to publish, as the word of a senior (white male) colleague was sufficient. If you make it one of merit, it very quickly becomes one of quantity rather than quality because as the number of job applications, grant proposals, journal submissions increases, one has less and less time to qualitatively evaluate the merits and details of each and every one of them.

          I do think there’s a schizophrenic towards peer-reviewed papers within academica, however (and the process of peer-review more generally). The better journalists now say issue disclaimers with research they discuss that *isn’t* peer-reviewed. I don’t ever recall reading an account in a newspaper that says something along the lines that “just because this research has been peer-reviewed does mean that there are significant errors in it that undermine its conclusions.” I think we academics fudge the issue of what peer-review means to non-academics (and, to be honest, probably to ourselves at times), because it quickly goes from skepticism to cyncism to expert knowledge, and populists are quite willing to weaponize any skeptism towards research that goes against their ideology (e.g., Michel Gove’s statement that “people in this country have had enough with experts”). Still, given that much of published research is publically funded, there is a question about whether we should be much more up front with the general public (and to ourselves and to our students) than I think we are.

        • Sorry, I keep forgetting the central point for which I *do* want to criticize the editor. The comment concerns whether the model used by the researchers of the original articicle is inappropriate for the complex study design of the PISA dataset. Two reviewers agree, and one mentions that an earlier paper published by one of the co-authors in the same journal has already been critiqued on that issue (a critique I’m assuming the authors and reviewers of the new article either are unaware of or ignore). I think both the author of the comment and the reviewers say that economists commonly misuse PISA data because they don’t know how to analyse it properly. The editor’s final comment is that a paper discussing the problems that arise when economists misuse PISA data is better suited for a field journal in education. Probably a new editor just wants a legacy issue off their plate, but the comment isn’t exactly reassuring about the self-correcting nature of academic research.

          (I guess this goes back to the issue that research is becoming complex and interdisciplinary enough that we often have to assume there aren’t major errors on the research we cite, because no one person can chase the details of it all. Unfortunately, peer reviewers aren’t in any better of a position, which is why I think post-publication comments attached to the papers is necessary so people don’t build their own research on sand)

          Anyways thanks for the discussion. If you feel like responding please do, but I think I’ve said everything I have to say on the subject.

        • I’ll respond here rather than wait. Unless editors actually state their reasons, we can’t interrogate that question without ascribing motivations. I don’t think identifying how incentive structures are counter-productive to the advancement of science is a limitation in cultural empathy (I would actually argue it’s the opposite: I realize the bind everyone is in, given that these are the rules we have to play under). Even when specified people are named, I don’t think the criticism is really about pointing flaws of particular people: they’re just illustrations of a dynamic: the situation is common enough that it’s often arbitrary who gets chosen for the illustration. So it’s nothing really personal, even though it’s hard for people to not feel that it is.

          I guess editors who don’t publish comments or replications could think to themselves that the function of *their* journal is to advance ideas that could lead to further research, hence they are only interested in publishing articles that produce novel and surprising results (and generally ignore those that produce null results) and the critiques and replications can be left to others (they might say “more specialized journals”, but I suspect they also think those journals are less prestigious than their own). But it shouldn’t take much reflection to realize that if the most prestigious journals are basically more interested in ideas that can generate more research irrespective of the accuracy or generalizability of the results, the result probably isn’t an advancement of science as much as it is to produce more and more flawed studies.

          I honestly don’t know how we walk back from this, barring some catastrophe that forces us to begin anew from first principles (and which might eventually us back to precisely where we are now). II’m not a historian, but I wonder how much the current state resembles a period of decadence (again, without any moral judgement intended)

        • Joe –

          Thanks for the response. I’d like to give it some thought. So as not to dominate the “recent comments, I’ll respond later.

        • Joe –

          I don’t really have much to offer that Dale and gec haven’t said below better than I might have, still…

          > I don’t think identifying how incentive structures are counter-productive to the advancement of science is a limitation in cultural empathy (I would actually argue it’s the opposite: I realize the bind everyone is in, given that these are the rules we have to play under).

          I wasn’t speaking of cultural empathy, but cognitive empathy – which I suppose aren’t quite the same. Still, I don’t think that identifying incentive structures is in any way a limitation in either realm. My point is that assuming, or making conclusions in those regards reflects limitations. I think that the Internet as a domain for the exchange of views is largely a vast pool of antipathy based in a lack of cognitive (and I suppose cultural) empathy. I think it’s a problem. That doesn’t mean that identifying potential perverse incentives is wrong or even necessarily counterproductive. Not at all.

          > Even when specified people are named, I don’t think the criticism is really about pointing flaws of particular people: they’re just illustrations of a dynamic: the situation is common enough that it’s often arbitrary who gets chosen for the illustration. So it’s nothing really personal, even though it’s hard for people to not feel that it is.

          Theoretically, that might be true. But in reality, IMO, the way it plays out is that it is usually perceived to be personal in nature – by those to whom the perverse incentive is assigned as explanatory as well as by many who look at the assertion as a signal to pile on. I see this all over the Internet. Even when accompanies by a “I’m just sayin.” The fact that an identification might be accompanied by a plausible deniability (“I didn’t actually say individual A was only motivated by profit to look the other way at the deaths of millions, I’m just pointing out it certainly looks like that was the case.”)

          > I guess editors who don’t publish comments or replications could think to themselves that the function of *their* journal is to advance ideas that could lead to further research, hence they are only interested in publishing articles that produce novel and surprising results (and generally ignore those that produce null results) and the critiques and replications can be left to others (they might say “more specialized journals”, but I suspect they also think those journals are less prestigious than their own). But it shouldn’t take much reflection to realize that if the most prestigious journals are basically more interested in ideas that can generate more research irrespective of the accuracy or generalizability of the results, the result probably isn’t an advancement of science as much as it is to produce more and more flawed studies.

          I’m not unsympathetic to that argument – but I’m wondering if Dale and gec’s comments below change your view at all in that regard.

          > I honestly don’t know how we walk back from this, barring some catastrophe that forces us to begin anew from first principles (and which might eventually us back to precisely where we are now). II’m not a historian, but I wonder how much the current state resembles a period of decadence (again, without any moral judgement intended)

          IMO, despite the ubiquity of the framing of all of this as a “crisis,” I have yet to see a quantified analysis of all of this. Sure, we’re more aware, perhaps, of the flaws and the impact of perverse incentives, but is the ratio of bad information we’re getting relative to the good information really increased? More bad information may more be a reflection of more information rather than an indication of a shift in the ratio. In which case, maybe the increase in good information is greater than the increase in bad information? I dunno. And I’ve never seen anyone address that question.

          People often think things were better back in the day, and we’re going to hell in a handbasket because “kids today,” and my impression is that people often think that in error.

        • Well, I can’t find the original article, so I am just relying on memory, but this actual was a journal editor’s stated reason, and it’s one that sorta makes sense to me and seems consistent with other issues (for example, not publishing negative effects and an emphasis on novelty. Ultimately, it’s an empirical question about whether journals that publish replications have much lower impact factors than those that don’t, or whether original studies get cited less after a failed replication (I remember reading an article purporting to show that it had no effect: I wouldn’t surprised if it hadn’t because people probably don’t check whether all the articles they cite in a paper have or have not been replicated: it’s a shame we don’t have notekeeping software with fields for DOIs that would automatically notify us when another work refers to it in more than trivial ways).

          I think your logical problem would only occur to a few number of studies: I suspect most research doesn’t fall into them (and I also suspect that there wouldn’t be as much resistance to publishing replications of them.

          What other more palpable incentives do you think there are, or citation metrics (e.g., impact factors) isn’t a major one?

        • Joe –

          > What other more palpable incentives do you think there are, or citation metrics (e.g., impact factors) isn’t a major one?

          See my comment above to Sameera.

          > …and it’s one that sorta makes sense to me and seems consistent with other issues (for example, not publishing negative effects and an emphasis on novelty.

          Re not publishing negative effects: One day I had a convo with an international graduate student who seemed very tired. I asked him about that and he explained that on top of all his other responsibilities he was spending hours and hours in lab conducting experiments and he felt a lot of pressure about those experiments. I asked him why, and he said that none of the ongoing experiments he was conducting confirmed the results that his supervising professor was hoping to confirm. He felt that he was letting his supervising professor down because he wasn’t generating positive results. I was struck that at no time did it occur to him that there was a greater scientific value in that he was at some level disconfirming the expected results.

          Now I think that kind of approach is in some ways more common with international students, but still, I think it points to a more pervasive and easily understandable pressure that students and researchers feel, that causes them to have a positivity bias. To some extent, I’m sure that’s driven by the perverse incentives in the academic publishing industry and the sub-optimal, profit-driven perverse incentives in academia more generally. But I also think there’s a “natural” psychological bias hidden in there somewhere, and a kind of “cultural” bias as well that in some ways might actually play out as a positive force when seen from a 10,000 foot level look at the advancement of science.

          I think that the same could be related to an emphasis on novelty. In some ways, “naturally,” more novel results are more compelling.

          I don’t say any of that to suggest that perverse incentives don’t exist, or that positive incentives can’t have unintended negative consequences. I just think that checking assumptions about all of that is an important step to improving the scientific enterprise.

          > it’s a shame we don’t have notekeeping software with fields for DOIs that would automatically notify us when another work refers to it in more than trivial ways).

          That’s a great idea.

          > I think your logical problem would only occur to a few number of studies.

          Maybe – but I would imagine that, also, could be subject to an empirical study. IOW, to what degree do studies that disprove or point out flaws in widely accepted findings get cited? I would imagine there’s a lot of variability depending on which papers we’re talking about, which field we’re talking about. Interestingly, my sense is that a lot of the material I’ve seen about the “replication crisis” is, in itself, flawed. So then there are the studies that push back on the studies that push back.

    • EJ probably doesn’t accept comments because most comments are corollaries, rather than a plain demonstration of central flaws in the original paper, as this comment apparently was. The editor should have recognized the distinction and said, “Let’s call this a short article rather than a comment and then we’ll publish.”

      • The comments I’ve seen published in, say, the American Economic Review, all go to the heart of the article commented on. We don’t know, of course, how many comments they reject as not important enough. But if the Editor thought Theorem 6 was important enough to keep in a paper he publishes, he should think that a disproof of Theorem 6 is important enough to justify publishing a correction. He’s already established what’s important enough to take up space in his journal.

    • “But I would imagine there might also be reasons, consistent with a goal of advancing science, to have a policy of not publishing comments.”

      If the goal is to advance science, then correcting mistakes is one of the best ways to do that, no? One does not advance science by only publishing new stuff. Errors in that new stuff must be addressed as well.

      • JFA –

        > If the goal is to advance science, then correcting mistakes is one of the best ways to do that, no?

        No doubt. Just sayin’ that there could be thinking running in the other direction that’s also consistent with advancing science. If we miss non-perverse incentives for what takes place, then there’s a potential opportunity cost in focusing only on the perverse incentives. But maybe I’m searching for angels on the head of a pin here.

        • “there could be thinking running in the other direction that’s also consistent with advancing science”

          I’m open to suggestions for why editors wouldn’t publish corrections/critiques of articles they also published, but if the goal is advancement of knowledge, what are those reasons? I can’t come up with any off the top of my head, but I really haven’t thought much about it either.

  4. I like the parallel role of “stars” in this story:
    “gold stars [given out] to academics who write the sorts of big-impact papers they want to publish”, and
    “I think we have to strip DL of the stars in their tables” in reviewer comments on Troccoli’s manuscript (per the twitter thread). Maybe journals could use a gold font colour for the asterisks in tables of significance tests to indicate the big-impact results.

    Also two tweeted replies summarize a recurring theme of this blog. One person responded,
    “Would reaching out to the authors of the original study be an option? Usually colleagues are responsive, & grateful, when notified of errors in a collegial manner”,
    Immediately followed by a second person who replied,
    “They are grateful then politely do absolutely nothing for the public record.”

  5. Perhaps because I studied history before I turned to environmental science, I like to read the early papers on topics I’m investigating. Back in the day when papers were read at meetings before they were published, some editors routinely published papers together with comments by two or three “discussants,” plus a response from the author. Some of these discussions were interesting reading. With electronic publishing, you could do something like this by having reviewers write versions of their reviews that would get published on-line together with the papers. If the reviewers were willing to put their names on the reviews, they could then get some credit.

    • I’m not sure I’d be willing to (for free) put my name on a document where I critique content on which I’m not expert enough to publish myself, that is too short to contain all my caveats and hedges, that I have to turn around in very short order, and is published in the same medium as rigorous research and commentary despite receiving no oversight or editing itself. The opportunity to get credit is not nearly so motivating as the risk of looking like a fool. Don’t get me wrong: I’m fine looking like a fool on blogs or sites that do post-publication review, which is essentially just a conversation and everybody knows it. But a review document a) determines whether something gets published, and b) cannot be amended ad infinitum.

      I think a better solution might be for journals to turn into wikis, with article reviews crowdsourced. Review would consist of an ongoing, documented conversation during a period when the author could continue to edit the article, with the ability to hide old edits (but not old reviews) or to withdraw it entirely. After a consensus of some kind is reached, the paper is “published” and no more edits can be made to the article or reviews directly; but the reviews can be continued in a new document, ad infinitum, and serious flaws can be flagged and the paper moved to a “retracted” section. The original author could make corrections in a new article, which supersedes the original in searches but is still linked to it.

      In other words, an article would carry its own history. But this would require several changes to our culture, even greater than moving to open access. Publishing would become less of a single event, and less a product of the original authors, which would then mean rebalancing the nature of prestige in a field where prestige is our primary compensation.

  6. This post and the comments would make an excellent text for an advanced high school class on the scientific method. The ideal and the real world are so, so far apart.

  7. This situation makes me think that different people have different ideas about what “the literature” is supposed to be, and maybe these ideas have shifted over time.

    The current structure for journal publication seems to be related to how results were disseminated in the kinds of scientific societies that emerged in the late 19th century. Among those societies, an article in a journal had basically the same status as a talk at a meeting—it was intended to expose some result to the scientific public and engage with some ongoing debate within that society. In that sense, journals were more like blogs and individual articles weren’t expected to be anything more than reflective of the current state of knowledge (coupled with the reasoned opinions of an individual).

    But this runs into conflict with a different perspective on “the literature”, which is that each published article should in itself be at least reliable and preferably “true”. In other words, the idea seems to be that anything published should be able to be safely cited for its central claims long into the future, regardless of what we learn in that time. Although this belief seems silly when stated “out loud”—after all, the whole point of doing science is that we *know* that our current state of understanding is at best incomplete—it seems widely held. This is why it is viewed as important to retract terrible articles and also why people feel comfortable continuing to cite those terrible articles as long as they are not retracted (the vaccines/autism case is a good example here).

    In summary, the two perspectives are: 1) “the literature” represents ongoing debates in which each article represents a particular viewpoint; 2) “the literature” represents our current best state of knowledge in which each article makes or supports claims that are generally accepted.

    But where these two perspectives really collide is with the journal editors. On the one hand, they want to publish articles that “start a conversation”, and in that sense they are in keeping with the tradition in which journals were originally developed. But on the other hand, they want to publish articles that are “true” and admitting fault or even debate would undermine that goal. The result is a situation as described here: articles “start conversations” that are never allowed to continue (at least, not in the same venue).

    It seems to me that the easiest resolution would be to “demote” the status of published articles; after all, no matter how good a conversation they start, they are all guaranteed to be wrong in the end. But then how does someone outside the field know what to do if they can’t just pick any old article and cite it for support? I suppose they might actually have to ask an expert in that field…

    • ‘different people have different ideas about what “the literature” is supposed to be’

      This is true in some measure but I’m not sure that “supposed to be” is really the right framing. Different people and groups utilize the literature for different purposes and in doing so claim (implicitly or explicitly) that the literature is intended to serve or is robust for their use case.

      There are people who use the literature carelessly, as though every published claim was, by the fact that it’s published, indisputably true with no further consideration; and others that use it cautiously, treating each new contribution as tentative and expecting new claims to be exhaustively tested under different conditions and with different methods; and still others that operate between the two extremes. Some people operate on both extremes, using the extreme that suits their particular purpose at any given moment or for any given issue.

      Beyond that, however, to serve the purpose of the advancement of science overall, it is sensible that in some measure published papers should expected to be free of gross and simple errors: spreadsheet and code screwups; obvious data misrepresentations (like Andrew’s favorite Walker-sleep-bar-chart); blatant graphical errors; use of discredited methods etc, and that if such clear and blatant errors are found, a paper should simply be retracted.

      Part of the problem at the moment is that it’s easier than ever, and there are more people qualified than ever, to review the details of others’ work. That has revealed a disturbing degree of sloppiness and carelessness. If we can get that corrected, then we might not need to worry so much about how people use the literature.

    • Your two views of journals are:
      1. Start a conversation.
      2. Only publish dependable true results.

      These are very different in what gets published, but identical in their response to corrections: they’d both publish them eagerly.The correction continues the conversation, and it makes the articles dependable. Indeed, in view (2) corrections are *more* important, since truth is more important and standards for acceptance of articles is higher.

      The problem is a third view of the purpose of journals:
      3. To certify how intelligent and creative the authors are, so as to help departments decide who to tenure.

      For purpose (3), you don’t want to publish true but “easy” results, and it doesn’t matter if there are fatal flaws, so long as the article accurately signalled IQ.

      • Eric:

        I don’t think most journals are about “signaling IQ.” Rather, I’d say that they think they’re in the business of publishing true and important claims and they have too much of a faith in the peer-review process.

        Basically, the journal will take seriously just about any comment, no matter how ridiculous, that comes in the pre-publication peer review process. But after the article has been published, it pretty much doesn’t matter what is found, the journal doesn’t care. Because the article made it through the peer-review process!

        There are some exceptions, of course: for example, after 12 years, Lancet retracted that notorious vaccine-denial paper. After about a zillion years, a few journals retracted a few papers by that pizzagate guy. Etc. But it takes a huge amount for that to happen.

        What I’m saying is, I think that, deep down, these journal editors believe that just about every paper ever published by their journal is fundamentally true, and that any flaws found by post-publication reviewers are minor technicalities, not deserving of publication. In their view, post-publication reviewers are parasites, second-stringers, Stasi, terrorists, etc. And, in the very rare instances when editors admit there’s been a serious error, then, hey, that’s cool, it’s the self-correcting nature of science!

        The awkward thing is that “journal editors” aren’t some special species. They’re people like us! I have friends who have been journal editors. Sure, there are some cases of journal editing being about logrolling and soft corruption, but my impression is that most academic journal editors are doing it for free, trying their best. They just get sucked into that defend-the-system role. Put that “journal editor” suit on them, and they start acting like “journal editors.” They want to publish exciting new work, they don’t want to spend their time correcting old work. Also, publishing new work is easy. Correcting old work is hard, because authors of erroneous articles will sometimes be difficult.

        • I think that all journal editors do in their conscious minds think that their purpose is to publish new and important results. That includes, though, publishing articles that debunk important old theories, whether from their own journal or others. So it isn’t just that they don’t want to correct the past. They do see that correcting error is important.

          The source of our dissatisfaction is in their unconscious minds. As you say, they think that a PhD student who did a replication and found a coding error is just a parasite, somebody who got lucky and did literally 5% of the work of the average article author (tho that is still a lot of hours!). That person does not deserve a “Journal of Great Thoughts” line on their vitae, one that university bean-counting deans will count as just as good as the original article for showing their talent. This boils down to “This article does not certify brilliance and hard work in the way most articles in this journal do, whether it’s just as important or not. The author didn’t put in his time; he doesn’t deserve it.”

          I think editors need *conscious* effort to resist that *unconscious* thought. So I hope they read this blog article and thread!

          ps– I know we economics PhDs are like this. The field I know next best is Math, and I think it may be similar there, since that field puts even greater emphasis on IQ as being a big part of scholarly quality (while recognizing that grit and hard work is another big part). I don’t know about statistics.

  8. This story contains one of the more interesting variations on the ‘corrections to the big journals go into the small journals’ theme. It’s very obvious from the correspondence and process that the editor handling the manuscript was prepared to publish the comment. Otherwise, why solicit the reviewers and wait for their reports. So an intervention happened somewhere along the line. My guess is the editorial board pushed back when the ‘comment’ paper was put in the queue.

    What is unclear to me is whether (a) it really is true that the Economic Journal editors are toeing the line regarding a no comment policy, perhaps in fear that they will be inundated with similar such ‘correction papers’, or (b) that the authors of the original paper are being specifically protected.

  9. No need to impugn the motivations of the editors (though there might be reasons to do so) – I think publishing comments/corrections is a threat to the academic structure and publishing’s place in it. A century ago, academics were few enough, and empirical work was difficult enough, that many practices could work well for bringing new ideas forth as well as questioning those ideas. Today the world is quite different. Data is so readily available, and methods have become so sophisticated, that a general practice of publishing comments could result in a journal becoming like a blog! Perish the thought! In other words, there would be so many comments that the editorial and referee functions would be undermined. Perhaps they should be, but that is a different discussion. It is hard for me to see how a modern journal can survive the onslaught of comments/replication attempts/corrections without a major change in their processes.

    I would note that journals such as the NEJM do publish comments regularly, along with the authors’ responses. I believe this is in recognition of the importance of the research and the need to make sure that the “truth” is being advanced. While that practice is admirable, it is severely limited by two factors: first, the comments must be submitted within a very short time (something like 2 weeks, I believe) after initial publication, and second, the raw data is rarely available for much meaningful comment. Rather than speculating on the editorial motivations, I think this is straightforward evidence of the difficulties (yes, a slippery slope) associated with opening this can of worms. If comments are readily accepted, then editors would need to decide which ones to include – and they would probably need referees to assist with this. It’s hard enough to get good referees, just imagine needing them to screen all the comments. The role of journals could grind to a halt, and along with it, much of the academic structure (promotions, tenure, grants, etc.).

    Perhaps it is time for new models to emerge. Certainly there are many such attempts – a number of which I have learned about through this blog. However, the progress seems slow and haphazard to me. There are few authoritative bodies able to “control” what journals evolve into – and the whole idea of “control” is anathema to the academic culture. It will take many more stories such as the one in the above post before we see sustained progress, I fear.

  10. The reason for journals not publishing such comments is very simple, it negatively impacts the IF. The reason is it increases the denominator while it is unlikely to be cited so adds nothing to the numerator.

  11. I wrote an article in the Times Higher Education on my experience of two similar cases in economic journals. Journals must stop blocking critical comments
    Allowing faulty papers to go unchallenged damages integrity and threatens dangerous real-world consequences, says Peter Bowbrick
    May 27, 2021
    Critical comments are of fundamental importance to academic publishing. They do more than peer reviewers do to keep academics from knowingly submitting bad or faked research. And they can be the first step in shifting scientific paradigms.
    When I started writing critical comments in 1973, there was a presumption that editors would publish them if possible. They would check them, but people who write comments face a reply from the author, so they check their own work more carefully than any referee or editor ever would. Besides, errors are usually obvious once they have been pointed out.
    Comments continue to be read and cited for years. In the week of writing, for instance, more than 200 researchers and students from 12 countries have read 10 of my comments and refutations; seven of these were written more than 20 years ago.
    But 15 years ago, the climate in publishing began to change. Some American journals decided that critical comments showed up the fallibility of their vetting process, which could affect the commercial value of their brands. Instead of accepting that all refereeing is fallible, they started running comments past two referees, as well as the editor. Inevitably one of the three would object to something trivial, and this could then be used as an excuse to reject the comment.
    A few years ago, I submitted a comment to a journal showing that a paper it had published broke all the rules for surveys. However, one referee disagreed with my statement that it was improper to use members of one organisation to interview members of another when comparing the two organisations (which is just one of the rules). The comment was suppressed, so I put it up on ResearchGate and Academia.edu instead. Now lecturers ask their students to read it before conducting a survey.
    More recently, a UK journal developed a cast iron defence against criticisms. When I objected that one of its papers was littered with errors of commission and omission, my comment was considered by the entire editorial board – of 49 people, according to the journal’s website. This level of pre-publication scrutiny is unprecedented.
    The 49 – including the editor – found no fault with my economics or facts, but would not even consider publishing my comment because, I was told, the editorial board “expects all comments and rejoinders to be worded in a collegial and constructive manner, tone and style. We feel that yours does fall rather below our usual conventions in this regard.” I did not make any ad hominem attacks or snide remarks – unlike the author of the paper I was criticising – so I asked for clarification. But all I received was a reiteration that I must rewrite my comment “in the sort of tone and style that is normal for contributions to this journal”.
    No rational writer would comply with such a hostile demand. I am asked to rewrite my comment to meet secret criteria about toning down criticisms. And if by some miracle I meet these criteria without the criticisms vanishing, the editorial board will send my comment to two referees. Given that the journal’s referees accepted the appalling paper in the first place, how much faith can I have in them?
    Even if they pass my comment, the editor may still reject it. Research shows that a significant proportion of editors in the sciences overrule the referees or rewrite their comments. And if the author makes a reply, however wrong, meaningless, or ad hominem, I have no confidence that I will be allowed to write a rejoinder.
    I surmise that the journal’s objection is actually to my conclusions, rather than the way I present them; this is not the first time I have seen the words “collegial” and “constructive” used as a demand for omerta.
    The journal says that “the priority of the Editorial Board is the journal”. Clearly, we have totally different moral stances. The paper I object to is on the economics of famine. Bad economics kills people – and I am squeamish about killing people. I do not see any moral justification for the continued existence of a journal that refuses to correct falsehoods in any field, but particularly this one. Less seriously, the journal is arguably committing an offence under the UK’s Consumer Protection from Unfair Trading Regulations 2008 by continuing to sell a paper it knows to be full of falsehoods.
    It is still possible to submit refutations of entire research programmes to a range of journals. In practice, though, such refutations develop out of a few critical comments on specific papers – and these can only be submitted to the journal that published the paper in the first place.
    This means that to avoid the risk of wasting their time, comment writers these days would be wise to seek assurances from the journal about publication before they even begin. The implications for academic integrity – not to mention people’s willingness to attempt to set the record straight in the first place – are obvious.

    I have published a lot of refutations and comments over the years, but it is getting harder.

    I would be grateful to receive evidence from others, particularly about economics and the social sciences.

  12. I wonder if there is a “coordination problem” – i.e., no individual journal wants to adopt a policy of admitting to past mistakes unless all peer journals adopt the same policy. If there were a “regulator” that enforces a common policy of acknowledging and correcting error, might we overcome this problem?

    Perhaps the various professional bodies/associations & leading universities could take the lead by at least publishing a set of journals that have committed to a common policy of “publication integrity”, and discourage academics from publishing in journals that don’t commit to this policy or don’t submit voluntarily to enforcement for breach of the policy.

    Just a suggesiton!

Leave a Reply

Your email address will not be published. Required fields are marked *