A coauthor writes:
I really like the paper [we are writing] as it is. My only criticism of it perhaps would be that we present this great new method and discuss all of its merits, but we do not really discuss when it fails / what its downsides are. Are there any cases where the traditional analyses or some other analysis are more appropriate? Should we say that in the main body of the paper?
Good point! I’m gonna add a section to the paper called Failure Modes or something like that, to explore where our method makes things worse.
And, no, this is not the same as the traditional “Threats to Validity” section. The Threats to Validity section, like the Robustness Checks, are typically a joke in that the purpose is usually not to explore potential problems but rather to rule out potential objections.
Now, don’t get me wrong, I love our recommended method and so does my coauthor. It’s actually hard for us to think of examples where our approach would be worse than what people were diong before. But I’ll think about it. I agree we should write something about failure modes.
I love my collaborators. They’re just great.
Hopefully Mayo will do that with her new SEV method.
Applause!
However, in a properly functioning scientific universe, a discussion of the benefits *and* flaws of a new method would be obligatory, because other scientists would immediately dive in to explore and expose the method’s limitations and embarrass (not intentionally or with malice, but jst in the course of business) the authors who failed to do so themselvs.
Isn’t it a standard recommendation from all paper writing guides to include drawbacks and limitations discussion in conclusion? At least this is something I usually request from authors as a reviewer.
I recall that at some point, in some fields, a Ph.D. dissertation was required to include a section on “limitations and delimitations”.
That – “explore where our method makes things worse” – is likely what I was not clear enough about here “I think the statistical discipline needs to take more responsibility for the habits of inference they instill in others” https://statmodeling.stat.columbia.edu/2017/11/01/missed-fixed-effects-plural/
I can see the distinction between talking about limitations–like assumptions that must be met–and talking about modes of failure–like the results of a stress test. “This car was designed to get you from one place to another when the two places are connected by standard roads” vs “this car will overheat if you drive it too fast, even on a road, so use a sports car for that.”
Chances are I will never fund a journal, but I have thought for a long time that if I do it, I will make a condition for acceptance that at least one “situation in which one may think it reasonable to apply the new method but in which the new method fails” is outlined.
Nice posting!
I meant “found a journal” of course!