In a recent discussion thread, I mentioned how I’m feeling charitable toward David Brooks, Michael Barone, and various others whose work I’ve criticized over the years, because their responses have been so civilized and moderate.
Consider the following range of responses to an outsider pointing out an error in your published work:
1. Look into the issue and, if you find there really was an error, fix it publicly and thank the person who told you about it.
2. Look into the issue and, if you find there really was an error, quietly fix it without acknowledging you’ve ever made a mistake.
3. Look into the issue and, if you find there really was an error, don’t ever acknowledge or fix it, but be careful to avoid this error in your future work.
4. Avoid looking into the question, ignore the possible error, act as if it had never happened, and keep making the same mistake over and over.
5. If forced to acknowledge the potential error, actively minimize its importance, perhaps throwing in an “everybody does it” defense.
6. Attempt to patch the error by misrepresenting what you’ve written, introducing additional errors in an attempt to protect your original claim.
7. Attack the messenger: attempt to smear the people who pointed out the error in your work, lie about them, and enlist your friends in the attack.
We could probably add a few more rungs to the latter, but the basic idea is that response 1 is optimal, responses 2 and 3 are unfortunate but understandable, response 4 represents at the very least a lost opportunity for improvement, and responses 5, 6, and 7 increasingly pollute the public discourse.
David Brooks is a pretty solid 4 on that scale, which isn’t great but in retrospect is like a breath of fresh air, given the 6’s and 7’s we’ve been encountering lately.
Most of the responses I’ve seen, in academic research and also the news media, have been 1’s. Or, at worst, 2’s and 3’s. From that perspective, Brooks’s stubbornness (his 4 on the above scale) has been frustrating. But it can, and has, been much worse. So I appreciate that, however Brooks handles criticism of his own writing, he does not go on the attack. Similarly, I was annoyed when Gregg Easterbrook did response 2, but, in retrospect, that 2 doesn’t seem so bad at all.
As I said, I put the above into a comment thread, but I thought it’s something we might want to refer to more generally, so it’s convenient to give it its own post.
This might be a nice place to park some thoughts from Peirce on this topic.
“The interaction [criticism and response] with fellow inquirers is crucial for filtering out the various idiosyncrasies that individual inquirers bring to the table—it allows us, as he [Peirce] puts it, “to grind off the arbitrary and the individualistic character of thought”… Peirce defines the private self not in terms of anything exquisite or divine, but in terms of error and ignorance. What makes our private selves unique is that we differ from others in that we are wrong about different things and that we are ignorant about different things. Hence, for Peirce, scientific inquiry—which seeks to alleviate error and ignorance—is in essence a process of self-effacement.”
Peirce: A Guide for the Perplexed. Cornelis de Waal
h are decidedly anti-Cartesian. Peirce defines
the private self not in terms of anything exquisite or divine, but in terms of error and ignorance. What makes our private selves unique is that
we differ from others in that we are wrong about different things and that we are ignorant about different things. Hence, for Peirce, scientific
inquiry—which seeks to alleviate error and ignorance—is in essence a process of self-effacement. I Hence, for Peirce, scientific
inquiry—which seeks to alleviate error and ignorance—is in essence a process of self-effacement. I
Opps – sorry for what was below the preview and left in.
I once found an error in a book by mathematician Irving Kaplansky (the guy who wrote the pi song). I wrote him a note pointing it out, and got back what must have been by return mail (this was before email) a note that started, “Oh, is my face red!”
I would guess that this was his accustomed reply when someone pointed out a blooper he’d made. So maybe it’s a good idea for people to think in advance about what to say when someone points out an error in their work.
What a great list. I plan to reflect on them. IF ANY of YOU see me committing any of them PLEASSSE tell me so I can rationalize it. LOL. Just kidding. Seriously I’m not an academic celebrity. So I’ll stick with effacement.
Regards
Good examples of response #1 by Zeke Hausfather here: https://twitter.com/hausfath/status/1087390124025905156 and by Ralph Keeling here: https://www.realclimate.org/index.php/archives/2018/11/resplandy-et-al-correction-and-response/
I hope that the more people see scientists like Keeling and Hausfather modeling this kind of response, the more the rest of us will feel inclined to respond similarly if someone points out errors in our own work.
+1
A possible example of “4. Avoid looking into the question, ignore the possible error, act as if it had never happened, and keep making the same mistake over and over.” can perhaps be found in somewhat more general criticism such as the decades old criticism regarding NHST or low statistical power that seems to have not resulted in any or much change.
Some more individual ans specific cases of at least the ignoring part can perhaps be found in the cases of “Priming intelligent behavior: An elusive phenomenon” published in PlosONE where a comment can be found that indicates that the original author might have not responded (appropriately) to questions or criticism regarding their original findings.
A further example of this ignoring behavior might also be found in the case of the retracted Protzko et al. (2023) paper which can be read in a blog post by an author that was critical of the paper and process if I understand and describe things correctly, and notes that: “So far, there has been no engagement with any of the evidence and only continued statements that conflict with it.”
https://joebakcoleman.com/blog/2024/nosek/
A possible example of “5. If forced to acknowledge the potential error, actively minimize its importance, perhaps throwing in an “everybody does it” defense.” might be found in the Levelt et al. (2012) investigative report concerning the fraud case of Stapel. On page 48 of that report the following can be read:
“Another clear sign is that when interviewed, several co-authors who did perform the analyses themselves, and were not all from Stapel’s ‘school’, defended the serious and less serious violations of proper scientific method with the words: that is what I have learned in practice; everyone in my research environment does the same, and so does everyone we talk to at international conferences.”
A further example of the minimizing of the error can perhaps also be found in the case of the retracted Protzko et al. (2023) paper:
https://statmodeling.stat.columbia.edu/2024/09/26/whats-the-story-behind-that-paper-by-the-center-for-open-science-team-that-just-got-retracted/
A possible example of “6. Attempt to patch the error by misrepresenting what you’ve written, introducing additional errors in an attempt to protect your original claim.” can perhaps be found in a replication study of some so-called priming findings. The title of the replcication paper is “Priming intelligent behavior: An elusive phenomenon” and is published in PloSONE where there are also comment-options. You can read some comments from some people involved there which may provide some examples of this #6 response option:
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0056515
A possible example of “7. Attack the messenger: attempt to smear the people who pointed out the error in your work, lie about them, and enlist your friends in the attack.” might be partly exemplified by the Fiske response to general criticism which has been mentioned on this blog as well:
https://statmodeling.stat.columbia.edu/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/
0. Look into the issue and, if you find there really was an error, fix it publicly and thank the person who told you about it, and be careful to avoid this error in your future work.
Quote from above: “0. Look into the issue and, if you find there really was an error, fix it publicly and thank the person who told you about it, and be careful to avoid this error in your future work.”
Additional note I: This should be the base level for a scientist according to my reasoning and view.
Additional note II: From a certain perspective the “base level” comment might be seen as amusing or fitting in light of “the ladder” of responses 1 to 7 depicted in the blog post and the possible addition of this 0.
Anon:
What you’re calling step 0 is the same as what I’m calling step 1, except you’re also adding, “be careful to avoid this error in your future work,” which is fine; I just thought that was implicit in my step 1. Once you fix the error publicly, of course you’d want to avoid it in the future!
Yes, I copied your 1. text and added the part about avoiding because I noticed that general topic at points 3 and 4 (“, but be careful to avoid this error in your future work.” & “, and keep making the same mistake over and over.”).
I thought there was a more strict adherence to certain aspects of the reponses in that I thought response options 1-4 consisted of certain factors that differed more systematically (if that’s the appropriate word):
-“avoid looking in to” VS. “look in to”
-“ignore the possible error” VS.”if you find there really was an error”
-“don’t acknowlegde” VS. “acknowledge”
-“don’t fix” VS. “fix” (“fix quietly” VS. “fix publicly”)
-“make same mistakes” VS. “be careful to avoid this error”
I still wonder about the usefulness of making the ladder response-options in line with all those factors in a way that all options are accounted for but that’s a very minor point I guess. I reason that would also depend a lot on the plausibility of the assumption that it is fair to assume people are indeed careful to avoid the error when they acknowledged and fixed the error. I am not sure that is always the case, which likely contributed to me thinking about that and adding it to option 1 after reading something about that in options 3 and 4. That and possibly reading too fast and not thinking about it some more.
I appreciate your reply!
Quote from the blog post: “As I said, I put the above into a comment thread, but I thought it’s something we might want to refer to more generally, so it’s convenient to give it its own post.”
After some searching about error-learning just now, and in light of the quote above, I wondered whether the response-options ladder might be a great basis, or starting point, for a paper about scientific criticism.
I don’t know if such papers already exist, but in light of the quote above such a paper might be something to more easily and perhaps better refer to when useful and needed. Should Mr. Gelman, or someone else, need any more paper-topic ideas, that’s something to consider perhaps.
I assume several, recent, cases might be found that might fit with the individual response ladder-options for some (perhaps even already familiar) examples. And after a brief search about error-learning just now, I can see several papers about that general topic which might be appropriate and useful for such a paper. And, you could possibly combine that all with some more general views on how science could or should work, and the role of mistakes and criticism in that process.