Authors retract the Nature Communications paper on female mentors

The paper “The association between early career informal mentorship in academic collaborations and junior author performance” that I (Jessica) previously blogged about has been retracted from Nature Communications. 

Here’s the authors’ statement:

The Authors are retracting this Article in response to criticisms about the assumptions underpinning the Article in terms of the identification of mentorship relationships and the measure of mentorship quality, challenging the interpretation of the conclusions. These criticisms were raised by readers and confirmed by three experts post-publication as part of a journal-led investigation.

In this Article, we analysed publication records to identify pairs of junior and senior researchers working in the same discipline, at the same institution, who are co-authors on papers with no more than 20 authors. We use co-authorship, as defined above, as a proxy of mentorship by senior researchers, with the support of a survey that was targeted at a random sample of a recent cohort of researchers. We measure the quality of mentorship using the number of citations and the connectedness of the senior investigators.

The three independent experts commented on the validity of the approaches and the soundness of the interpretation in the Article. They supported previous criticisms in relation to the use of co-authorship as a measure of mentorship. Thus, any conclusions that might be drawn on the basis of co-authorship may not be automatically extended to mentorship. The experts also noted that the operationalisation of mentorship quality was not validated in the paper.

Although we believe that all the key findings of the paper with regards to co-authorship between junior and senior researchers are still valid, given the issues identified by reviewers about the validation of key measures, we have concluded that the most appropriate course of action is to retract the Article.

We are an interdisciplinary team of scientists with an unwavering commitment to gender equity, and a dedication to scientific integrity. Our work was designed to understand factors that influence the scientific impact of those who advance in research careers. We feel deep regret that the publication of our research has both caused pain on an individual level and triggered such a profound response among many in the scientific community. Many women have personally been extremely influential in our own careers, and we express our steadfast solidarity with and support of the countless women who have been a driving force in scientific advancement. We hope the academic debate continues on how to achieve true equity in science–a debate that thrives on robust and vivid scientific exchange.

All Authors agree with the retraction.

 

When I read the original study, I balked at how most of the implications suggested by the authors assumed a causal link between female mentors and poor career outcomes, at how the authors failed to mention the existing evidence that female authors are associated with lower citations, and at the label “mentorship” being applied so loosely to coauthorship. 

If I’d reviewed the paper, I would have wanted, at the very least, for the authors to soften the claims about mentorship to better acknowledge the limitations of the measurements. But, given that it was published, as I said in comments on the previous post, I didn’t think the paper needed to be retracted, since reading it, it wasn’t hard to tell how they were operationalizing “mentoring” and even the egregious interpretations about women being worse mentors at the end were hedged enough to be identifiable as speculation. I tend to think retraction should be reserved for cases where the discerning reader cannot tell they are being duped, because of fraud, plagiarism, or errors in analyses that they couldn’t be expected to find. 

I guess I’m comfortable with the idea of retraction having high precision but low recall. Otherwise there would be much more to retract than would seem possible. This would probably also open the door for all sorts of heuristic decisions based on political views, etc. in the less clear cut cases, which, at the extreme, could end up disincentivizing work on any controversial topics. I’m not sure how much retraction adds over issuing a correction attached to the original article in many cases, but if we’re going to do it, it makes sense to me to try to keep it relatively rare and our definition as simple as we can. 

Perhaps what makes me most uncomfortable with retractions for reasons like bad speculation is that retracting some, but not all, science with misleading labels or conclusions seems to play into a myth that the scientific record is a reflection of truth and we have to keep it pure. In cases like this one where a paper makes some seemingly bogus claims but the analysis itself seems to hold, if that paper made it through peer review I’d rather trust the reader figure out which claims or conclusions are exaggerated or not rather than trying to protect them from bad speculation. Realistically, we should expect to see sloppy unsubstantiated claims in much of the science we encounter. Corrections should happen a lot more than they do, and we should not treat them like a huge earthshaking deal when authors, or journals, make them. 

So initially, I was glad to see that the official retraction notice is from the authors themselves, not the journal. I assume that they will still publish their analysis else without those problematic claims. If they chose to hold themselves to a higher standard, good for them. 

But here’s where the impetus behind this retraction gets slightly confusing. While the retraction notice on the website is written by the authors as though they made the decision, it seems Nature Communications did appoint a three person committee to do an additional review after posting a notice on the paper saying they were examining it more closely in November. They describe this process here:

We followed our established editorial processes, which involved recruiting three additional independent experts to evaluate the validity of the approaches and the soundness of the interpretation. They supported previous criticisms and identified further shortcomings in relation to the use of co-authorship as a measure of informal mentorship. They also noted that the operationalisation of mentorship quality, based on the number of citations and network centrality of mentors, was not validated.

According to these criticisms, any conclusions that might be drawn on biases in citations in the context of co-authorship cannot be extended to informal mentorship. As such, the paper’s conclusions in their current form do not stand, and the authors have retracted the paper.

During the investigation, we also received further communications from readers highlighting issues with the paper and are grateful to all the researchers who have contacted us and who have invested their time in reviewing the work.

Simply being uncomfortable with the conclusions of a published paper, would and should not lead to retraction on this basis alone. If the research question is important, and the conclusions sound and valid, however controversial, there can be merit in sharing them with the research community so that a debate can ensue and a range of possible solutions be proposed. In this case, the conclusions turned out not to be supported, and we apologise to the research community for any unintended harm derived from the publication of this paper.

So maybe Nature Communications would have retracted the paper even if the authors didn’t. Or maybe they brought to the authors’ attention that they were considering retracting, and the authors then felt pressed to. I’m not sure.  

At any rate, the backlash against the Nature Communications paper made clear to me how ambiguous language can be in cases like this, and also how different people’s views can be about the responsibility that readers should have. I found the liberal use of the terms mentor and mentorship quality throughout the paper very annoying but not fatal, since the details of what was measured were there for the reader to see, and I read enough social science to be accustomed to authors at times adopting shorthand for what they measure, for better or worse. But if you read many of the sentences in the paper without the mental substitution, they are pretty problematic statements, so there is some space for judgment. Perhaps because of this ambiguity in language, the idea of protecting the reader from misleading claims by retracting everything that someone later points out makes misleading claims seems fraught with challenges to me. 

I’ve also seen some people suggesting it’s unfair for papers like the Nature Comm one to be subject to so much public criticism based on their topics striking a political chord. It seems naive to expect that papers that make strong claims counter to things that many people firmly believe wouldn’t get some extra scrutiny. People simply cared about the topic. I don’t think the paper warranted retraction in this case over something like a statement warning that the measures of mentorship quality weren’t validated. But suggesting that it shouldn’t have gotten so much scrutiny in the first place contributes to a belief that the published record should be venerated. I can sympathize with authors who feel singled out based in part on the topic of the work; I had one of my first published papers ever critiqued by a well-known practitioner in my field. It had won an award, and it made a controversial argument (that sometimes making graphs harder to read is better because it stimulates active cognitive processing on the part of the viewer). Getting critiqued for a paper that I suspected was singled out partly based on the topic wasn’t fun at all. As Andrew recently blogged about, there often seem to be worries in such cases about how public criticism will affect the academic career, and so if the process feels somewhat random it can be unsettling. But there were some valid points made in the critique, as there are in most critiques, and so I learned something, and I assume others did too. The idea that it should have been withheld out of respect or fairness didn’t make sense to me then or now. We shouldn’t be trying to reserve public scrutiny for only the most horrible papers.   

15 thoughts on “Authors retract the Nature Communications paper on female mentors

  1. “Although we believe that all the key findings of the paper with regards to co-authorship between junior and senior researchers are still valid…” Huh???

    • Graham:

      Wow—I hadn’t caught this! This is kind of like Brian Wansink, who stands by his findings even though (a) he can’t provide his data, and (b) there is no possible data that would be consistent with his published numbers; or Donald Trump, who continues to insist he won the election even though he provides no evidence and he lost something like 800 post-election lawsuits. Or lots of examples discussed here.

  2. The misuse of the term “mentorship” was so pervasive that it undermined the entire conclusions of the paper. I understand about how their transparency gives them some leeway regarding the use of the term, but this is too much. Having to mentally substitute so much is such a cognitive burden that even dedicated readers are likely to slip up. There is no easy answer and lots of shades of gray regarding how much leeway to give when using a term in a nonstandard way, but I found the paper to be clearly misleading and not an edge case in this manner.

    • I can respect that. It is a pretty extreme example of careless conclusions (and maybe not just careless but intentionally written the way it was). I guess my pessimistic attitude ultimately makes me more concerned about how we can ever achieve a really fair and effective retraction process than about the potential damage the misleading conclusions of a study like this could cause. But like I said above, I think there’s room for judgment in cases like this. What I wrote is just how I see it at the moment.

  3. I don’t buy the idea that there is a causal link between female mentors and poor career outcomes. Even if there was one many other factors play much more of a role than the gender of your mentor. Also I don’t think simply counting the number of citations is a good way of judging success in academia or the success of the mentorship. Even if you aren’t getting that may citations you could still be doing good work, influencing how other people think even if they don’t cite your work and just because you get a lot of citations doesn’t mean the work you did is good just look at the Social Priming literature for an example. Seems weird to me to evaluate the success of the mentorship only on the number of citations they have after the mentorship.

  4. Nice Post! I wonder what would have happened, if the main finding of the paper was “female mentorship has a strong positive effect on academic career”. TedTalk? TeenVogue? (at least a mention in The Guardian i presume). So are we going to scrutinize some papers more than others, namely, if they make politically inconvenient claims for left of center sensibilities? We will probably do just that, since the proportion of right of the center social scientists is relatively low and not known to be “activists”. And this is what I don’t like about this story: it seems a lot to me like this was a “political activist driven retraction”. My question is whether this type of process will end up inserting a worrying new form of publication bias, or whether this bias has always existed anyway?

    • I would assume it already exists in the review process in cases where reviewers bring their values and experiencs to bear to decide how contentious something is. I just don’t think it’s worth trying to also set up a retraction system that we think will distinguish in some reasonably fair way when conclusions are overblown or misleading enough to retract even when the analysis itself seems valid. Beyond being hard to implement its the kind of assumption behind the idea that most publications present perfectly valid conclusions, we just have to weed out the occassional bad apple.

    • This is a good point.

      I would like to begin by making it clear that I generally favour policies that could be described as feminist, anti-racist, and progressive – I’m not a conservative in disguise. I don’t believe in conspiracy by SJW’s to take over the academia or anything like that. I see the – general – progression towards inclusion positive.

      I think many people here are familiar with the concept of “White Fragility”. There’s a book about it, there’s a subreddit (r/fragilewhiteredditor) and at least in the more feminist circles I’ve seen it used as an argument against – what I would perceive as – sincere criticisms or questions. Often this concept is taken as being “scientifically proven”. If people question the existence of “White Fragility” they are just slyly given the link to the paper – maybe they are told that “it’s been published in a scientific journal, so it’s science”.

      So it is clear that this concept is influential, it is used as an argumentative tactic and – at least some portion of the public – sees it as a scientific fact.

      Now, if people already aren’t familiar with it, you go and read DiAngelo’s original article…

      https://libjournal.uncg.edu/ijcp/article/viewFile/249/116

      …can you be honest about its scientific merits? Do you WANT it to be true? How well do you think the central concept has been defined? How much the author presents evidence for the existence of it?

      I don’t think the paper discussed in this blog post will EVER reach the level of influence that the aforementioned DiAngelo’s paper has. One could maybe argue that misogyny is more prevalent so it is more important to tackle that. Maybe so! I am not against the retraction of the paper mentioned in OP, quite the contrary – it did seem silly and the conclusions far-fetched.

      Ah, I don’t know.

      • I share your politics and have not a shadow of a doubt that if the paper had said women were better academic mentors than men, it would be untouchable and would pass into our lay liberal discourse (NY Times, NPR) as proven fact.

  5. Social science questions are, almost invariably, political questions. Or rather, the reason non-scientists fund social science is to answer inherently political questions about society.

  6. “with an unwavering commitment to gender equity”

    “You have to know the game you’re playing,” says Dan Inosanto. “Don’t box with a boxer,” he advises. Science, politics, and religion (and personal preferences) are different games. Oil and water, my friend.

  7. Why are their unwavering commitments and other political preferences of relevance to the research question? Why not their strong preference for a mother’s right to kill her unborn babies or vice versa? Surely important issues to some people, but why is it urgent to state it in a research report? First Amendment right perhaps?? Indeed, however if we actually read the first amendment it does not require everyone to inform everyone else about everything that they [the right holders] think, feel, or prefer. Well, as they say, “有りの儘々に.”

Leave a Reply to Bob the Demiurge Cancel reply

Your email address will not be published. Required fields are marked *