Skip to content
 

IEEE’s Refusal to Issue Corrections

This is Jessica. The following was written by a colleague Steve Haroz on his attempt to make corrections to a paper he wrote published by IEEE (which, according to Wikipedia, publishes “over 30% of the world’s literature in the electrical and electronics engineering and computer science fields.”)

One of the basic Mertonion norms of science is that it is self-correcting. And one of the basic norms of being an adult is acknowledging when you make a mistake. As an author, I would like to abide by those norms. Sadly, IEEE conference proceedings do not abide by the standards of science… or of being an adult.

Two years ago Robert Kosara and I published a position paper titled, “Skipping the Replication Crisis in Visualization: Threats to Study Validity and How to Address Them”, in the proceedings of “Evaluation and Beyond – Methodological Approaches for Visualization”, which goes by “BELIV”. It describes a collection of problems with studies, how they may arise, and measures to mitigate them. It broke down threats to validity from data collection, analysis mistakes, poorly formed research questions, and a lack of replication publication opportunities. There was another validity threat that we clearly missed… a publisher that doesn’t make corrections.

Requesting to fix a mistake

A few months after the paper was published, a colleague, Pierre Dragicevic, noticed a couple problems. We immediately corrected and annotated them on the OSF postprint, added an acknowledgment to Pierre, and then sent an email to the paper chairs summarizing the issues and asking for a correction to be issued.

Dear organizers of Evaluation and Beyond – Methodological Approaches for Visualization (BELIV),

This past year, we published a paper titled “Skipping the Replication Crisis in Visualization: Threats to Study Validity and How to Address Them”. Since then, we have been made aware of two mistakes in paper:

  1. The implications of a false positive rate

In section 3.1, we wrote:

…a 5% false positive rate means that one out of every 20 studies in visualization (potentially several each year!) reports on an effect that does not exist.

But a more accurate statement would be:

…a 5% false positive rate means that one out of every 20 non-existent effects studied in visualization (potentially several each year!) is incorrectly reported as being a likely effect.

  1. The magnitude of p-values

In section 3.2, we wrote:

…p-values between 0.1 and 0.5 are actually much less likely than ones below 0.1 when the effect is in fact present…

But the intended statement was:

…p-values between 0.01 and 0.05 are actually much less likely than ones below 0.01 when the effect is in fact present…

As the main topic of the paper is the validity of research publications, we feel that it is important to correct these mistakes, even if seemingly minor. We have uploaded a new version to OSF with interactive comments highlighting the original errors (https://osf.io/f8qey/). We would also like to update the IEEE DL with the version attached. Please let us know how we can help accomplish that.

Thank you,

Steve Haroz and Robert Kosara

Summary of what we wanted to fix

  1. We should have noted that the false positive rate applies to non-existent effects. (A sloppy intro-to-stats level mistake.)
  2.  We put some decimals in the wrong place. (It probably happened when hurriedly moving from a Google doc to latex right before the deadline.)

We knew better than this, but we made a couple mistakes. They’re minor mistakes that don’t impact conclusions, but mistakes nonetheless. Especially in a paper that is about the validity of scientific publications, we should correct them. And for a scientific publication, the process for making corrections should be in place.

Redirected to IEEE

The paper chairs acknowledged receiving the email but took some time to get back to us. Besides arriving during everyone’s summer vacation, there was apparently no precedence for requesting a corrigendum (corrections for mistakes made by the authors) at this publication venue, so they needed a couple months to figure out how to go about it. Here was what IEEE eventually told them:

Generally updates to the final PDF files are not allowed once they are posted in Xplore. However, the author may be able to add an addendum to address the issue. They should contact ieee-mce@ieee.org to make the request. 

So we contacted that email address and after a month and a half got the following reply:

We have received your request to correct an error in your work published in the IEEE Xplore digital library. IEEE does not allow for corrections within the full-text publication document (e.g., PDF) within IEEE Xplore, and the IEEE Xplore metadata must match the PDF exactly.  Unfortunately, we are unable to change the information on your paper at this time.  We do apologize for any inconveniences this may cause.

This response is absurd. For any publisher of scientific research, there is always some mechanism for corrigenda. But IEEE has a policy against it.

Trying a different approach

I emailed IEEE again asking how this complies with the IEEE code of ethics:

I am surprised by this response, as it does not appear consistent with the IEEE code of ethics (https://www.ieee.org/about/corporate/governance/p7-8.html), which states that IEEE members agree:

“7 … to acknowledge and correct errors…”

I would appreciate advice on how we can comply with an ethical code that requires correcting errors when IEEE does not allow for it. 

And one of the BELIV organizers, to their credit, backed us up by replying as well:

As the organizer of the scientific event for which the error is meant to be reported, […] I am concerned about the IEEE support response that there are NO mechanisms in place to correct errors in published articles. I have put the IEEE ethics board in the cc to this response and hope for an answer on how to acknowledge and correct errors as an author of an IEEE published paper.

The IEEE ethics board was CCed, but we never heard from them. However, we did hear from someone involved in “Board Governance & Intellectual Property Operations”:

IEEE conference papers are published as received. The papers are submitted by the conference organizers after the event has been held, and are not edited by IEEE. Each author assumes complete responsibility for the accuracy of the paper at the time of publication. Each conference is considered a stand-alone publication and thus there is no mechanism for publishing corrections (e.g., in a later issue of a journal). The conference proceedings serves as a ‘snapshot’ of what was distributed at the conference at the time of presentation and must remain as is. IEEE will make metadata corrections (misspelled author name, affiliation, etc) in our database, but per IEEE Publications policy, we do not edit a published PDF unless the PDF is unreadable. 

That said, any conference author who identifies an error in their work is free to build upon and correct a previously published work by submitting to a subsequent conference or journal. We apologize for any inconvenience this may cause.

The problem with IEEE’s suggestion

Rather than follow the norm of scientific publishing and even its own ethics policies, IEEE suggests that we submit an updated version of the paper to another conference or journal. This approach is unworkable for multiple reasons:

1) It doesn’t solve the problem that the incorrect statements are available and citable.

Keeping the paper available potentially spreads misinformation. In our paper, these issues are minor and can be checked via other sources. But what if they substantially impacted the conclusions? This year, IEEE published a number of papers about COVID-19 and pandemics. Are they saying that one of these papers should not be corrected even if the authors and paper chairs acknowledge they include a mistake? 

2) A new version would be rejected for being too similar to the old version.

According to IEEE’s policies, if you update a paper and submit a new version, it must include “substantial additional technical material with respect to the … articles of which they represent an evolution” (see IEEE PSPB 8.1.7 F(2)). Informally, this policy is often described as meaning that papers need 30% new content to be publishable. But some authors have added entire additional experiments to their papers and gotten negative reviews about the lack of major improvements over previous publications. In other words, minor updates would get rejected. And I don’t see any need to artificially inflate the paper with 30% more content just for the heck of it.

It could even be rejected for self-plagiarism unless we specifically cite the original paper somehow. What a great way to bump up your h-index! “And in conclusion, as we already said in last year’s paper…”

3) An obnoxious amount of work for everyone involved.

The new version would need to be handled by a paper chair (conference) or editor (journal), assigned to a program committee member (conference) or action editor (journal), have reviewers recruited, be reviewed, have a meta-review compiled, and be discussed by the paper chairs or editors. What a blatant disregard for other people’s time.

The sledgehammer option

I keep cringing every time I get a Google Scholar alert for the paper. That’s not a good place to be. I looked into options for retracting it, but IEEE doesn’t seem very interested in retracting papers that make demonstrably incorrect statements or that incorrectly convey the authors’ intent:

Under an extraordinary situation, it may be desirable to remove access to the content in IEEE Xplore for a specific article, standard, or press book. Removal of access shall only be considered in rare instances, and examples include, but are not limited to, a fraudulent article, a duplicate copy of the same article, a draft version conference article, a direct threat of legal action, and an article published without copyright transfers. Requests for removal may be submitted to the Director, IEEE Publications. Such requests shall identify the publication and provide a detailed justification for removing access.  -IEEE PSPB 8.1.11-A

So attempting to retract is unlikely to succeed. Also, there’s no guarantee that we would not get accused of self-plagiarism if we retracted it and then submitted the updated version. And really, it’d be such a stupid way to fix a minor problem. I don’t have a better word to describe this situation. Just stupid.

Next steps

  1. Robert and I ask any authors who would cite our paper to cite the updated OSF version. Please do not cite the IEEE version. You can find multiple reference formats on the bottom right of the OSF page.
  2. This policy degrades the trustworthiness and citability of papers in IEEE conference proceedings. And any authors who have published with IEEE would be understandably disturbed by IEEE denigrating the reliability of their work. What if a paper contained substantial errors? And what if it misinformed and endangered the public? It is difficult to see these proceedings as any more trustworthy than a preprint. At least preprints have a chance of authors updating them. So use caution when reading or citing IEEE conference proceedings, as the authors may be aware of errors but unable to correct them.
  3. IEEE needs to make up its mind. It could decide to label conference proceedings as in-progress work and allow them to be republished elsewhere. However, if updated versions of conference papers cannot be resubmitted due to lack of novelty or “self-plagiarism”, IEEE needs to treat these conference papers the way that scientific journals treat their articles. In other words, if IEEE is to be a credible publisher of scientific content, it needs to abide by the basic Mertonian norm of enabling correction and the basic adult norm of acknowledging and correcting mistakes.

24 Comments

  1. AV says:

    Does any journal allow corrections or retractions?

    I have a smallish battle against the argument:”we cannot change print editions retrospectively, so no change or retraction is possible”…

  2. rm bloom says:

    Errata *always* used to be published.
    Is that no longer an option?

  3. Hi Steve and Jessica,

    I’ve been curious about IEEE this month. I’ve been exploring the RetractionWatch database for a project, and IEEE has 7,499 retractions on record, most of which are conference papers. Retracted IEEE conference papers make up 33% of all retractions and 36% of all retractions tagged as “psychology.” Of these, about 6,500 are tagged as “Limited or No Information.”

    You may be interested in this RetractionWatch story: https://retractionwatch.com/2015/06/25/one-publisher-appears-to-have-retracted-thousands-of-meeting-abstracts-yes-thousands/

    Given the volume of retractions IEEE has, at times, processed, and given the generic retraction notices pasted on those retractions, I wonder if IEEE simply doesn’t have the administrative capacity to process corrections? I don’t know much about IEEE, but I am very curious about those thousands of retractions.

    • Steve Haroz says:

      Hi Joseph,

      I’ve also noticed the quantity of retraction in the RetractionWatch database, and it’s very odd. While many are for fraud, IEEE seems unique in in terms of the quantity and ambiguity of retractions.

      With that said, if IEEE is going to be a credible science publisher, and if they want their publications to be taken seriously, they need to abide by the same standards as everyone else. From Sage to Wiley to Elsevier to APA, everyone else can handle the administrative burden.

      Should we be citing literature from a publisher with this policy? I’m not sure.

    • Weird about the number of retractions. Maybe engineers are sloppier than we thought!

      More seriously, what surprised me in Steve’s account is that it’s not even possible to publish an addendum to an IEEE published paper. I can maybe understand that its a big undertaking if they agree to let authors update the PDFs post publication, but it would not be a huge engineering feat to make an optional section of the article template in IEEE Xplore called Corrections. At the very least, if they want to avoid any oversight they could do what PLOS does and allow for reader comments, so authors themselves can add a note about errors they find later.

      • rm bloom says:

        Many of the IEEE conference proceedings, they often include stuff of marginal quality.
        The IEEE Transactions on the other hand are generally quite good however.

        • Steve Haroz says:

          If IEEE allowed authors of conference papers to submit them to journals with minor updates, I’d be OK with that distinction of conferences for low quality or in-progress work and transactions (journals) for high quality work. But if we take their current policy into account, we should view conference proceedings as an untrustworthy sandpit where potentially good work gets stuck.

          • Bob76 says:

            The IEEE does allow conference papers to be submitted to journals with minor updates. I think it is common for the conference version to be an abbreviated version of the paper.

            Here’s a statement of their policy from https://www.comsoc.org/publications/journals/ieee-tcom/information-authors/conference-vs-journal-papers

            Overall rationale for this policy: Authors should be encouraged to present their work to the community at IEEE conferences, which inevitably involves publishing a conference version. A policy that prevented conference papers from being later published in a journal version would have the opposite effect and discourage researchers from presenting their best work at conferences. Similarly, it would damage the quality of IEEE journals if innovative ideas are barred from publication in them, simply because they have been presented in a conference format.

            • Steve Haroz says:

              Thanks for the reference. But I’m not sure the minor updates we described are compatible with what’s described. That site says “‘substantial’ differences”. And the IEEE operations manual that I quoted in the post uses the phrase “substantial additional technical material” (IEEE PSPB 8.1.7 F(2)).

              Any change that is small enough to be fixable in a corrigendum is likely too small to be considered “substantial”. Any issue bigger than that would warrant a retraction or statement of concern.

              I would be curious if any editor of an IEEE journal editor would say that if you have a conference paper missing a 5-word qualifying phrase, you can submit the update to the journal.

              • Bob76 says:

                I’m pretty sure that I’ve seen many IEEE Transactions articles that were negligibly different from the conference version.

                That said, it would be ridiculous if the IEEE did not have a way to link some kind of author’s note to the information that is served up by a search of the IEEE Digital Library. It turns out they do—at least for journal articles.

                I just did a search for the word erratum in the IEEE Digital Library (restricted to publications in 2021). I got 5 hits. The first one I clicked on had been viewed about 290 times. That erratum had a link back to the original paper. The paper had a prominent note stating “An Erratum is Available.” The note was a clickable link to the erratum. See https://ieeexplore.ieee.org/document/9079584

                So, IEEE can do errata for journals.
                Bob76

  4. Rahul says:

    Isn’t the reference to the Mertonian norms a bit ironic? I mean most of them we seem to violate everyday in the modern academic enterprise.

    • Ben says:

      Yeah I just looked them up and sort of thought the same thing. The critique of IEEE seems fine without leaning on this.

      Also this one (taking from Wikipedia) just seems wrong:

      > organized skepticism: scientific claims should be exposed to critical scrutiny before being accepted: both in methodology and institutional codes of conduct.

      It’s a zombie on here, but thinking of science as accept/reject is bad.

  5. > We should have noted that the false positive rate applies to non-existent effects. (A sloppy intro-to-stats level mistake.)
    It certainly is an error that we all have or are at risk of making, but reinforces the false view that the p-value is a posterior probability or the chance the result was just due to chance.

    So it is important to correct.

    And it’s great to see authors trying to correct the sort error given the more usual response https://statmodeling.stat.columbia.edu/2019/12/18/attempts-at-providing-helpful-explanations-of-statistics-must-avoid-instilling-misleading-or-harmful-notions-statistical-significance-just-tells-us-whether-or-not-something-definitely-does-or-defin/
    (with ps,s by Andrew)

    • Thanks for sharing the link to your post Keith, I missed it the first time but the comments are pretty interesting!

    • Steve Haroz says:

      Great post Keith! This point is particularly apt:

      “Unfortunately those are the ones folks are most likely to take away as they are much easier to make sense of and seemingly more profitable for what they want to do.”

      People probably don’t randomly select from all explanations out there. People may be more likely to latch on to the simpler explanations, and that often means that the oversimplifications (like in our original article) are especially prone to misinforming people.

  6. John says:

    IEEE, science needs to be maintained the most correct possible. IEEE, your arguments are ridiculous.

    Authors, I think in this case it’s necessary to be more drastic, I suggest that the publication be excluded from IEEE database.

  7. Michael Nelson says:

    As a practical matter, if someone is searching a general database like Google Scholar, both papers should come up. You could re-title the post-print to begin: “CORRECTED: …” with a note right at the top with the corrections. (Don’t say “CORRIGENDUM” as people like me would think it was a foreign translation!)

  8. Dean Eckles says:

    Having some similar problems getting IEEE to deal with plagiarism in one of their journals…

Leave a Reply