Physicist Michael Weissman wrote an article, “Methods of Causal Inference in Physics Education Research,” which begins:
Educational policies require sound estimates of their potential effects. Methods for making such estimates, i.e. finding causal estimands, have made great progress in the last few decades. Three recent papers in Physical Review Physics Education Research present explicit methods of causal inference but unfortunately the methods have major errors. Identifying causal mediation, choosing variables to control for, and imputation of missing data are among the topics for which incorrect methods were given. . . .
He sent it to the journal where the flawed papers had appeared, but the journal didn’t want to publish his article. Here was their reasoning, in a letter sent to Weissman by the journal editor:
Re: YK10068
Methods of causal inference in physics education research
by M. B. WeissmanDear Dr. Weissman,
Thank you for submitting your article for consideration in Physical Review Physics Education Research (PRPER).
I have shared the article with the PRPER Editorial Board and regret to inform you that we do not think the article is a good fit for the journal. We have already published one of your articles that is critical of methods used in some specific papers in PRPER. While critique is certainly an important part of improving scientific fields, we do not want to set a precedent or encourage submission of articles that only critique the methods used by others. We would prefer to publish critical articles that offer concrete solutions or alternatives using the same or similar data sets. We also think that it is important for methodology-related papers to discuss the role of quantitative and qualitative data working together to make strong knowledge claims for the field.
We appreciate the effort that you have taken to dig into some articles published in the journal and offer important methodological critiques. While we are not able to publish this manuscript, I think that the authors of these papers would find your critiques valuable as they will likely use similar methods in the future. Would you allow us to send your manuscript draft to these authors?
Yours sincerely,
Weissman’s reply:
You’ve published a series of papers whose methods are utterly false, and which tell other to use those same false methods. To say that your board finds it tiresome to publish corrections means that you’ve abandoned the core principles of science, which require trying to get things more or less right. That requires correcting major mistakes. PRL, PRB, etc. all have mechanisms for doing that. They are still part of science. . . .
I think perhaps it’s appropriate to add some specific responses.
“We would prefer to publish critical articles that offer concrete solutions or alternatives using the same or similar data sets.”
One of the papers I critiqued was a pure methods paper. Therefore the critique necessarily was confined to methods. One was primarily a results paper. I did exactly as you propose, showing what the correct quantitative conclusion would be from the data set in the paper, to the extent that was possible given the incomplete presentation. I also gave a brief primer on how to fix one method (imputation) that was done incorrectly. One was mainly a results paper but with an explicit methods instruction section on mediation. In addition to explaining how to fix the mediation methods, I gave numerical results for a more reasonable mediation picture, and discussed the qualitative implications for policy.
In other words, to the extent that data sets were presented I “offer concrete solutions or
alternatives using the same or similar data sets.”I’m not sure what was meant by “quantitative and qualitative data working together” since the papers were presented as almost entirely quantitative. There are some subjective priors involved in drawing causal diagrams, and I believe that I was more explicit and more reasonable about that than the original papers.
I think that this reply is about right, and I’ve seen this sort of thing before, that a journal will publish an article with clear flaws and then refuse to publish a letter or article pointing out the problems. In this case, the journal editor is very polite and does not deny the substance of the criticism; they just don’t want to publish it. I’ve seen this too. I think one problem is that editors are habituated to thinking of journal publication as this valuable coin that they have minted. They don’t want to just give out journal publications any more than most teachers want to just give out A’s without the student putting in sufficient work. I think that’s a key hangup right there: a misguided sense of equity.
I say that this attitude is misguided because I think the most important stakeholder in the academic publication process is not the editor, not the author, but the reader. And the interests of readers is not advanced when they are served up papers with fatal errors.
As we like to say around here, everybody makes mistakes. It’s too bad the journal editors and reviewers didn’t catch the problem with those articles during the submission process, but mistakes are inevitable. The issue is what you do once the mistakes are pointed out to you.
To return to the question of audience, consider the last line in the editor’s response: “I think that the authors of these papers would find your critiques valuable as they will likely use similar methods in the future. Would you allow us to send your manuscript draft to these authors?” Again, this is a very polite, civilized thing to suggest, but, again, the main point of this correction is not to help out the authors; it’s to help the readers, or, more generally, to help the field of physics education.
This point should be obvious but lots of people seem to miss it. When a paper is published, it is public. If PRPER is willing to share some false claims with its thousands of readers, then damn straight it should be willing to share the correction with those readers too.
That said, journals can’t publish everything, so it’s not like I’m saying they have some sort of absolute duty to publish this new article. My issue is that they seem to be following a lapse in logic. Their implicit reasoning appears to go something like this:
1. Claim A is important, so we should publish a paper that provides evidence in favor of claim A.
2. The negation of claim A is not so important, so we should not publish a paper that discounts the purported evidence in favor of claim A.
OK, that’s what I’m saying is their implicit reasoning. Their explicit reasoning is clear enough; it’s procedural: They’ll publish original research but not criticism of original research. I share with Weissman the opinion that this reasoning is mistaken. Indeed, the very fact that the journal published those earlier, erroneous papers makes it clear that there is value in publishing the criticism.
Unfortunately, there aren’t a lot of journals that will publish criticisms. Two that I can think of are Sociological Science and Econ Journal Watch, but neither sociology nor economics is that close to physics education. I guess the natural place to publish would be a different physics education journal . . . but the whole thing is so exhausting.
Weissman summarizes the episode as, “PhysRev Phys Ed Res. says they DGAF if their papers are true.” That seems going a bit too far. I’m sure they do give a fig if their papers are true. They just don’t want to spend their time handling corrections. Think of a basketball player who loves to score but hates to go back on D. Journal editing is a volunteer position; you do it because you want to shepherd great new work into the field and because you want to develop brilliant new careers. Correcting mistakes in already-published papers is like cleaning up the garbage: it’s not what you signed up for.
Consider a statement like, “We would prefer to publish critical articles that offer concrete solutions or alternatives using the same or similar data sets.” It sounds kinda good, but it inappropriately devalues negative contributions. Consider an engineering analogy. Researcher A publishes a design of a bridge. Researcher B explains how this bridge would collapse if it were built. Is Researcher B offering a “concrete solution” or even an “alternative”? No! He’s just trying to deter someone from building a bridge that collapses. But I think that’s valuable too!
One more thing
As noted above, I have a lot of sympathy for volunteer journal editors doing their best. The “set a precedent” thing, though, that ticks me off. What exactly are they worried about? That people will find other old articles from their journal that are flawed? It would be good for readers to know about this, no? It’s all pencils, never erasers for this crew.
Awkward- I agree with everything Andrew says, and have no clever quibbles.
The journal editor actually was very constructive in guiding the revisions of my first critique paper. There are some vibes of internal conflicts in the field, which I also picked up in interacting with Science Advances on another critique.
At some point it might be interesting for someone to study the sort of closed-circle community of funding agencies, journals, writers, etc. in which systematically goofy work can thrive, not just the occasional messed-up paper. One issue in this PER field, and probably others, is that people who are still active are intimidated into silence. This is not a mere personal surmise but something I’ve heard first hand from multiple researchers. It’s good to be a retired dilettante.
Andrew, can’t you start your own journal? You, Sander Greenland, Keith O’Rourke, and Daniel Lakeland, et al can be editors
Sameera:
I’m on record as saying that journals should be replaced by recommender systems. People can publish their papers on Arxiv or wherever, and the role of the journal would be to recommend good papers to readers. A journal of recommendations could also include some negative recommendations.
You could think of this blog as a sort of journal.
+1 to these ideas.
Andrew,
Thanks for your response. Yeah, after I posted my comment, I realized that your blog is a type of journal.
Twitter, as a social media platform, has become a more complicating information environment. I am not sure just how the quality of conversations is panning out. I try to give some visibility to critically thinking experts by retweeting their tweets.
No doubt, there are some new entrants to social media that are hosting & moderating interviews with different experts that are available on YouTube. Some have 250,000 to 1 million views within a short time. I learn a good deal from Youtube them too.
I would like to see more cogent discussions about COVID in a blog format.
“We would prefer to publish critical articles that offer concrete solutions or alternatives using the same or similar data sets.”
Isn’t this just a contrived way of saying: “we support critisism as long as the newly proposed methods lead to the same conclusions.”
I don’t think it is: Using the same or similar data sets one might reach very different conclusions. However, it seems that what they are saying is that they don’t want to publish papers that show that based on some already published data set, no conclusion is possible. I assume that’s why they added the “similar data sets”.
The 3 papers in my first critique generally reached conclusions where none were possible. Ironically, it was this MS that described cases where conclusions were possible and opposite to those that had been claimed.
Now that’s just depressing.
This is sad to read, but I can’t say I’m surprised, not only because this problem of dismissing criticism is rampant throughout science, but because physics education research is particularly bad. I say this as a physics professor and one who has worked a lot on pedagogy / education, and as a regular attendee of our weekly science teaching journal club! There’s some great stuff in physics education (e.g. better ways to teach intro physics to life scientists from people at U. Maryland and others; I think this is a relevant link https://www.compadre.org/nexusph/). There’s a lot that’s awful, though, and it often seems “the worst of all worlds” — neither physics nor good social science nor good cognitive science. It’s also a quite insular field, but very vocal, and multiple times I’ve experienced its resistance to criticism, even the most gentle.
At first glance that Maryland material looks great. There’s a lot of room for improving physics teaching for bio types. From the last few years helping docs with stats, I think there’s even more room there. That there’s so much good work that needs doing makes the effort wasted on pointless or counterproductive subfields particularly painful.
Raghu, Michael:
Compared to statistics teaching, physics teaching has a couple of built-in advantages:
1. The first-semester curriculum is pretty much settled, so it’s possible to focus on teaching without getting tangled in disagreements and confusion about fundamentals.
2. Physics class is more of an option than a requirement, and I expect you have less of a problem with people being forced to take the class while being overwhelmed by the math.
I’ve taken the ideas of peer instruction and just-in-time teaching from the physics education literature and applied them to statistics. It’s more of a challenge because statistics is a less codified field and there’s less of an expectation that students in an intro physics class will be able and willing to solve difficult problems.
But, yeah, my focus when reading that literature is on the ideas, not on empirical claims.
I guess that my cue to link to my wife’s intro stats class. (UIUC Stat 200)
http://courses.atlas.illinois.edu/fall2020/STAT200/index.html
All the lectures are available via the links in the calendar. I hope that she’ll find a way to make an open-access link to the online homework too, since nobody really learns just by listening.
A less mathematical (Stat 100) version is just starting to go up, links not yet available.
Stat 100 is good stuff although basically frequentist. It’s gone over well with tens of thousands of students, mostly not at all mathematical, including lots of premeds.
It’s descended from Freedman, but with some more vivid examples, and with some of the neurotic kinks smoothed out.
The syllabus looks very much like the syllabus of my course at Duke in 2008 which used the Freedman book. Now, I was banned from showing R code in class and only allowed to show simulation results. I also warned them not to think the course prepared them to do analysis for any real study addressing a serious question.
Trust your wife is replacing the box model with simulation. I have material that uses digits of Pi to make the generation of uniform psuedo-random variables less mysterious and then simple rejection sampling to get any distribution one wants. Most seem to get that fairly immediately – but as with any learning practice is needed to experience it.
I think the syllabus is misguided but no one really has come up with a convincing replacement.
These were good to see “Often we really care about whether X causes Y, not just whether X predicts Y.” and “usually all [aways?!] approximations to the real world”
Feel free to email me if you wish to discuss further.
Hi Andrew,
About (2): “Physics class is more of an option than a requirement” — sort of. A lot of intro physics is taken by pre-meds, who (in general) don’t want to be there.
About “my focus when reading that literature is on the ideas, not on empirical claims” — Yes! That’s also the approach taken by our journal club. The ideas, filtered through our own experiences, make for good discussion.
Well, how many negative relative to positive submissions do the journal publish? I will tho suggest that in my observation of negative critiques, not all that many offer very good alternatives either. It is easier to critique than it is to mount convincing alternatives.
Here I think Ian Harris, author of Surgery, The Ultimate Placebo excels. Dean Ornish too.
I like Michael’s suggestion in his comment. Michael McGee in his book Code Blue identifies the public and private medical partnerships/initiatives that historically have played a role in shaping the US health care system. A really wonderful writer to boot.
This reminds me: We got reviews back for a recently submitted paper in which we develop a new method. We commented that discrepancies in the literature may be because some methods may not be as great as one might think. Reviewer 1: “…these [other papers] are presumably peer-reviewed articles and would not be published if methodology is flawed.” I was stunned.
Oy!
At least 99% of papers published in the last 20 years use flawed methodology, probably more like 99.9%. I once visited every poster at a conference and kept track. It was something like 10/20k papers did not rely on NHST to draw a conclusion about their research hypothesis.
The same reviewer probably would have blocked those 10 from ever being published until they added in some statistically significant deviations from a strawman model.
But, to be fair, even ignoring the NHST issue I’ve never seen a paper without flaws. People overlook flaws when they like the result and focus on the same flaws when they don’t like it. Compare the response to “real world” vitamin D and smoking covid studies vs vaccines.
What conditions must be met for a post to get labeled “Zombies”? Apologies if this is a dumb question!
Hogg:
I guess you can click on the Zombies label, read through those old posts, and draw your own conclusions. I typically use the category for two sorts of posts. First, posts that are actually about zombies (this may have been the first). Secondly, and more commonly, posts on some topic that refuses to die, no matter how often it seems to have been killed off already.
Wholesale agree with Andrew and physicist Michael.
I’m also a physicist who’s been told by the editor in chief of one of the big astrophysics journals that they won’t publish papers that are only discussing and correcting methodological errors in already published papers in the same journal. (Publish elsewhere perhaps?) And then watched as paper after paper got accepted making the same errors.
And I also worry about the quality standards in many PER papers.
Michael –
I’m curious if there would be a distinction in their reluctance towards publishing articles that are critiques of other articles, as opposed to letters that are critiques?
Aren’t they suggesting some kind of threshold, or limited proportion of articles that are critiques of other articles? They’ve already published one of your critiques so it’s not some kind of blanket refusal to publishing critiques. It seems many of the comments so far suggest they’re voicing a blanket refusal.
While I do think there’s a basic logical problem here as Andrew writes, I also think there’s a meaningful distinction between a blanket refusal and what they wrote in the letter.
I first submitted a comment on one article. That was desk rejected but one board member suggested a review of several articles making similar mistakes. So no, unlike other branches of Phys. Rev., this one isn’t into Comments. That suggestion turned into my first article, for which the editor gave good guidance on revisions. But he warned that some board members did not want to see it published.
Meanwhile, 3 more papers came out that were if anything worse than the ones in the first batch. Now they are saying that methods papers that give flat-out wrong methods will stand without correction. That doesn’t necessarily mean that no critical paper will ever be published again, but I think it does show a depraved indifference to truth.
Obviously there are some differences of opinion within the field, and some would like to see a more scientific attitude. They are not winning.
Michael:
This fits the common pattern in which publishing criticisms takes a huge amount of effort. Journal editors, when not flat-out hostile, often seem genuinely puzzled by why people would put in so much work just to correct the literature. It’s just so different from the usual pattern of publishing papers in order to build a career. And then the Javert paradox rears its ugly head.
Your Javert Paradox seems to be an expression of the same idea as Brandolini’s Law of Bullshit. L. Sprague de Camp said something along the lines of “people throw their dollars at swindlers but won’t pay a penny to the people saving them from swindlers.”
Sean:
We’ve discussed that Brandolini thing here too. The Javert paradox is different in that it addresses not just the persistence of B.S. and the difficulty of countering it with rational argument but also a Catch-22 in the way in which people question those who put in the work to try to provide counterarguments to B.S.
“I think one problem is that editors are habituated to thinking of journal publication as this valuable coin that they have minted. ”
Yeah, I wonder about that. My sense is that it’s more self-serving: If a journal is publishes critiques on a regular basis, that’s a deterrent to researchers to publish their work in that journal, and it’s an invitation for other would-be critiquers to start tearing apart the papers. Who knows what kind of floodgates that might open – especially in the social sciences where erroneous work seems to be rampant.
Noname:
This relates to an issue that Weissman alludes to, but which I didn’t discuss in my post, which is that researchers working with in a field can have an personal or intellectual or ideological reason for supporting a particular position. So it’s not just institutional loyalty but also a specific desire to protect a particular claim that’s been made. We’ve seen this in some fields where leaders will attack critics and otherwise try to derail scientific criticism.
This reminds me of ten years ago how difficult Russ Lyons found it was to publish a critique of the widely-touted theory of social contagion published by Christakis and Fowler. He ended up publishing in a statistics policy journal.
If anyone reading the comments doesn’t know the history, it’s here: https://rdlyons.pages.iu.edu/#CF and this blog covered it as well (https://statmodeling.stat.columbia.edu/2011/06/10/controversy_ove_2/)
Quote from press release: “Both of the leading, prestigious journals that published research by Christakis and Fowler — the New England Journal of Medicine and BMJ (formerly British Medical Journal) — rejected Lyons’ critique, the first declining to give a reason and the second saying the work would be better placed in a specialist journal. Rejections then came from three other leading journals on the grounds that they had not published the original research. A statistics review journal rejected Lyons’ paper on the basis that the original research of Christakis and Fowler was itself not sufficiently important.”
John:
This reminds me of a paradox that arose several years ago when a leading psychology journal published an article claiming evidence in favor of ESP. After that well-publicized article, there were many failed replication attempts. It was really hard for anyone to get these published, because . . . everyone knows there’s no such thing as ESP, so there’s no novelty in publishing such a negative finding.
The desire to publish novel and important results leads to a bias in favor of bad work.
I am an editor of a fairly new journal that specifically encourages these types of critiques. After handling some of them, I am less surprised that editors reject these types of submissions, because they end up being way more work than a typical paper. The authors of the critiqued piece are far more motivated than the typical reviewer/observer to reach out to editors and explain why the critique shouldn’t be published, why various statements are misleading and must be changed, and so on. So that takes a lot of my time.
Also, critiques often make statements that, while technically correct, are likely to lead readers to make unwarranted inferences about the intent and personal character of the original authors. In one recent case, the authors were motivating the choice to critique a paper by pointing to questionable research practices and data fabrication in other papers in other fields. The critique didn’t provide any evidence of such behavior in this paper, but simply motivating the analysis that way seemed like an accusation to the original authors, and I can see why!)
Let me emphasize that I don’t think editors are right to reject critiques for these reasons. Just that I understand why they might to avoid the extra work involved
Couldn’t you lesson some of the work by not allowing the original authors the opportunity to explain why the critique shouldn’t be published? Why are the original authors given that opportunity in the first place? Is that you think people would be reluctant to publish there if they knew their work could be critiqued without their ability to influence the decision to publish said critique?
Physical Review (Letters, B,…) allows responses from the authors, then lets the commenter rethink and maybe revise. It sounds cumbersome but it works. I’ve commented 5 times, 4 published. The other should have been but no system is perfect. But we all share a basic framework, which makes it easier. Social science, to paraphrase Porfirio DIaz, is so far from physical law, so close to the passions.
https://libraries.mit.edu/scholarly/publishing/elsevier-fact-sheet/#:~:text=RELX%2C%20the%20parent%20company%20of,US%20%243.6%20billion%20in%202019.
Journals may simply do what is good for business.
I’ve been struggling with this a bit lately. I’ve started thinking of my JMP in terms of The Prestige (it’s a methodology paper). The Pledge: Here’s RDD. The Turn: It’s much more sensitive to outliers and specification choices than you think. I have both of those, but haven’t submitted it yet, because I need The Prestige: here’s how to fix it. And I doubt I can do that by my dissertation deadlines.