Michael Weissman pointed me to a frustrating exchange he had with the editor of, Physical Review Physics Education Research. Weissman submitted an article criticizing an article that the journal had published, and the editor refused to publish his article. That’s fine—it’s the journal’s decision to decide what to publish!—but I agree with Weissman that some of the reasons they gave for not publishing were bad reasons, for example, “in your abstract, you describe the methods used by the researchers as ‘incorrect’ which seems inaccurate, SEM or imputation are not ‘incorrect’ but can be applied, each time they are applied, it involves choices (which are often imperfect). But making these choices explicit, consistent, and coherent in the application of the methods is important and valuable. However, it is not charitable to characterize the work as incorrect. Challenges are important, but PER has been and continues to be a place where people tend to see the positive in others.”
I would not have the patience to go even 5 minutes into these models with the coefficients and arrows, as I think they’re close to hopeless even in the best of settings and beyond hopeless for observational data, nor do I want to think too hard about terms such as “two-way correlation,” a phrase which I hope never to see again!
I agree with Weissman on these points:
1. It is good for journals to publish critiques, and I don’t think that critiques should be held to higher standards than the publications they are critiquing.
2. I think that journals are too focused on “novel contributions” and not enough on learning from mistakes.
3. Being charitable toward others is fine, all else equal, but not so fine if this is used as a reason for researchers, or an entire field, to avoid confronting the mistakes they have made or the mistakes they have endorsed. Here’s something I wrote in praise of negativity.
4. Often these disputes are presented as if the most important parties are authors of the original paper, the journal editor, and the author of the letter or correction note. But that’s too narrow a perspective. The most important parties are not involved in the discussion at all: these are the readers of the articles—those who will takes its claims and apply them to policy or to further researchers—and all the future students who may be affected by these policies. Often it seems that the goal is to minimize any negative career impact on the authors of the original paper and to minimize any inconvenience to the journal editors. I think that’s the wrong utility function, and to ignore the future impacts of uncorrected mistakes is implicitly an insult to the entire field. If the journal editors think the work they publish has value—not just in providing chits that help scholars get promotions and publicity, but in the world outside the authors of these articles—then correcting errors and learning from mistakes should be a central part of their mission.
I hope Weissman’s efforts in this area have some effect in the physics education community.
As a statistics educator, I’ve been very impressed by the innovation shown by physics educators (for example, the ideas of peer instruction and just-in-time teaching, which I use in my classes), so I hope they can do better in this dimension of evaluating evidence of effectiveness.
Context please. Was the quoted reason the complete reason for them declining to publish the critique? That one statement – objecting to the word “incorrect” I agree is a poor reason. I also think it was the wrong word to use – but it should have been easy for the editor to ask the word to be changed to “inappropriate” or some less objectionable term than “incorrect.” But I am assuming there were other reasons for them to decline to publish the critique. Absent that context, it seems like you are using this to make a point about journal policies towards post-publication review – and I agree with your point. But I can’t tell if that is really the relevant point regarding this particular incident.
Thanks! I think your point 4 is the crux of the matter.
@Dale Lehman- I will scrounge through old emails and try to find the entire exchange. There were no actual technical objections to my points, which were then published (with somewhat harsher language) after peer review: https://econjwatch.org/articles/invalid-methods-and-false-answers-physics-education-research-and-the-use-of-gres?ref=articles. The authors I criticized have a standing invitation from that journal to reply in print, which they have declined.
I later had a long zoom conversation with one of the authors who agreed with the technical points and with whom I have since corresponded on further work. The other two groups are another story. I’ll pst some more one new work later.
@Dale Lehmann:
Here is the final desk rejection letter in its entirety.
Re: YK10068
Methods of causal inference in physics education research
by M. B. Weissman
Dear Dr. Weissman,
Thank you for submitting your article for consideration in Physical
Review Physics Education Research (PRPER).
I have shared the article with the PRPER Editorial Board and regret to
inform you that we do not think the article is a good fit for the
journal. We have already published one of your articles that is
critical of methods used in some specific papers in PRPER. While
critique is certainly an important part of improving scientific
fields, we do not want to set a precedent or encourage submission of
articles that only critique the methods used by others. We would
prefer to publish critical articles that offer concrete solutions or
alternatives using the same or similar data sets. We also think that
it is important for methodology-related papers to discuss the role of
quantitative and qualitative data working together to make strong
knowledge claims for the field.
We appreciate the effort that you have taken to dig into some articles
published in the journal and offer important methodological critiques.
While we are not able to publish this manuscript, I think that the
authors of these papers would find your critiques valuable as they
will likely use similar methods in the future. Would you allow us to
send your manuscript draft to these authors?
Yours sincerely,
Charles Henderson, Ph.D. (he/him/his)
Here are my somewhat intemperate responses (pasted together) in their entirety:
Dear Prof. Henderson-
You’ve published a series of papers whose methods are utterly false, and which tell other to use those same false methods. To say that your board finds it tiresome to publish corrections means that you’ve abandoned the core principles of science, which require trying to get things more or less right. That requires correcting major mistakes. PRL, PRB, etc. all have mechanisms for doing that. They are still part of science.
[2cd email]
I think perhaps it’s appropriate to add some specific responses.
” We would
prefer to publish critical articles that offer concrete solutions or
alternatives using the same or similar data sets. ”
One of the papers I critiqued was a pure methods paper. Therefore the critique necessarily was confined to methods.
One was primarily a results paper. I did exactly as you propose, showing what the correct quantitative conclusion would be from using the data set in the paper, to the extent that was possible given the incomplete presentation. I also gave a brief primer on how to fix one method (imputation) that was done incorrectly.
One was mainly a results paper but with an explicit methods instruction section on mediation. In addition to explaining how to fix the mediation methods, I gave numerical results for a more reasonable mediation picture, and discussed the qualitative implications for policy.
In other words, to the extent that data sets were presented I “offer concrete solutions or
alternatives using the same or similar data sets. ”
I’m not sure what was meant by “quantitative and qualitative data working together ” since the papers were presented as almost entirely quantitative. There are some subjective priors involved in drawing causal diagrams, and I believe that I was more explicit and more reasonable about that than the original papers.
Sincerely,
Michael B. Weissman
Wow: “we do not want to set a precedent or encourage submission of articles that only critique the methods used by others. We would prefer to publish critical articles that offer concrete solutions or alternatives using the same or similar data sets.”
Setting the precedent of publishing wrong articles, that’s ok, but publishing critiques . . . no, that would go too far!
I had virtually the same response from The Astrophysical Journal – arguably one if not the premier journal for astrophysic. Happy to publish poor analysis, unwilling to publish a short critique.
Thanks for the additional information. Their attitude is indeed unfortunate and, frankly, embarrassing. However, I would refrain from labeling someone’s methods “utterly false” and I am no paragon of subtlety. Few things in life are “utterly false” or perhaps they all are. Either way, that language isn’t likely to be received well. But from their rejection, I suspect that more forgiving language would not have changed the response. They apparently want to set the precedent of publishing flawed work and not correcting it. Much like politics.
Michael Weissman: You wrote “@Dale Lehmann” but his last name has but one “n.” It is a common enough name and the extra “n” does make it more exotic.
Tell it to Murray!
One would expect someone called Weissman(n) to be more attentive to this issue than the average man (pun intended).
Thanks for this detailed context. I had the same reaction as Dale (what was the full story here?). Having seen the rejection letter, I feel the editor should have made a revise and resubmit decision, asking for more emphasis on the solutions, not just the criticism. Did I understand correctly that the data are not available? That is outrageous and unacceptable if true.
Just a heads up. PRPER recently published a paper with $3M NSF funding and with sponsorship and coauthors from the American Physical Society, the American Association of Physics Teachers, the Fannie and John Hertz Foundation, and the Gordon and Betty Moore Foundation.
https://journals.aps.org/prper/abstract/10.1103/PhysRevPhysEducRes.19.010126
It forms the basis for a major effort to add some lessons to high school physics classes. I’m writing a critique with help from statisticians to try to avoid making errors myself. Meanwhile, looking for errors in its methods might be a fun exercise for perverse readers with time to spare.
This seems a bit over the top.
I noticed that they created a “Statistical Modeling Review Committee” https://journals.aps.org/prper/edannounce/10.1103/PhysRevPhysEducRes.18.020001
which seems interesting — maybe a response to you?
I
Is the rejected article publicly available on arxiv or somewhere else?
A slightly revised (using harsher language) version was published elsewhere after peer review:
https://econjwatch.org/articles/invalid-methods-and-false-answers-physics-education-research-and-the-use-of-gres?ref=articles
Andrew posted about the exact same paper two years ago but didn’t link it here for some reason:
https://statmodeling.stat.columbia.edu/2021/10/30/while-critique-is-certainly-an-important-part-of-improving-scientific-fields-we-do-not-want-to-set-a-precedent-or-encourage-submission-of-articles-that-only-critique-the-methods-used-by-others/
Weissman’s article is linked in that post and is here:
https://arxiv.org/abs/2110.04266
The entire email exchange is also in that post.
Most of the articles I’ve seen from PRPER are terrible-to be fair, this may be a biased sample. However, I remember Weissman saying something like “PRPER has a policy of specifically resisting the kind of error correction that is used in other Physical Review journals,” so I am not optimistic that the sample is in fact biased. Here’s an example of a comment that was rejected because “it was from a scientific perspective, but the original article was not”: https://arxiv.org/abs/2303.09434
Anon:
A few months ago I was cleaning out my inbox. As part of that, I wrote a bunch of blog posts, some of which are appearing now. I guess what happened is that I’d kept the email from Weissman in my inbox even though I’d already blogged it, and then I blogged it again without remembering.
Basically a lot of disciplinary SoTL is not that great on a technical level. People in the fields are often self taught when it comes to research on humans and organizations and the kinds of statistics they learn for their research are not really what is needed for SotL or observational research generally.
Michael’s efforts are great, and hopefully they will have an effect.
Physics education research (and STEM education research in general) is a strange field. I attend a weekly science teaching journal club which is fascinating. Sometimes, there are excellent papers. More frequently, there are terrible papers, but often with some good ideas or seeds for discussion.
In addition to being plagued by poor statistics and ideological biases, a large underlying problem is that many of the practitioners are doing social science without having the skills of social scientists. The study designs, analysis approaches, awareness of past work, etc., that one would expect of good cognitive science or psychology research — which is what much of this is trying to be — are largely absent. Perhaps it’s another example of physics arrogance.
Two things authors don’t like to do are admit errors and share data. Both take time away from what they really want to do, which is churn out as many articles as possible.
Journals should publish more negative results and critiques. There are a number of reasons for the current biases in the literature. I’ve had mixed results with doing critiques. I published a short note on airfoil parameterization that took down a whole series of papers advocating an inferior method. The editor in fact seemed quite interested in expediting the publication. Very few researchers want to get involved in this kind of publication because they say “people will figure it out soon enough.” I doubt that is really true.
I’ve had more trouble publishing on uncertainty and bias in CFD even with a massive amount of actual data and code results documenting the issues. My statistician collaborators are no longer active in research, so it became impractical for them to do the revisions requested. Maybe this summer I’ll try to get this published again. If anyone here wants to help that would be most welcome. Basically the paper consists of a statistical analysis of tens of thousands of CFD code runs for a standard test case.
In CFD, most people have written codes or are funded to build or apply these codes. They have a vested interest in making the codes look good so the funding will continue. As Andrew points out, there is also a huge bias in favor or “novel” results or results that result in corporate media attention. This is a way to shape the public’s and lawmaker’s view of science or a particular field positively. Cerainly in CFD, the vast majority of the money comes from the government and its agencies such as NASA.