Deborah Mayo writes:
How should journal editors react to heated disagreements about statistical significance tests in applied fields, such as conservation science, where statistical inferences often are the basis for controversial policy decisions? They should avoid taking sides. They should also avoid obeisance to calls for author guidelines to reflect a particular statistical philosophy or standpoint. The question is how to prevent the misuse of statistical methods without selectively favoring one side.
This is from an article called, “The statistics wars and intellectual conflicts of interest.” The concept of an intellectual conflict of interest is interesting, and it’s all over statistics and its applications; I wouldn’t know where to start, and there’s definitely no place to stop once you get started on it.
Mayo got several people to comment on this article, and she put it all on her blog, for example here. She suggests we discuss it here, as she (accurately, I think) suspects that our readership would have a much different take on these issues.
The particular discussion I linked to is by John Park, who warns of “poisoned priors” in medical research. My response to this is that all parts of an analysis, including data model, prior distributions, and estimates or assumptions of costs and benefits, should be explicitly justified. Conflict of interest is a real problem no matter what, and I don’t think the solution is to use a statistical approach that throws away data. To put it another way: As Park notes, the tough problems come when data are equivocal and the correct medical decision is not clear. In that case, much will come down to assessed costs and benefits. I think it’s best to minimize conflict of interest through openness and feedback mechanisms (for example, predictive markets, which are kind of a crude idea here but at least provide a demonstration in principle that it’s possible to disincentivize statistical cheating). I mean, sure, if your data are clean enough and your variability is low enough that you can get away with simple classical approach, then go for it—why not?—but we’re talking here about the tougher calls.
I won’t go through the discussions on Mayo’s blog one by one, but, yeah, I have something to disagree with about each of them!
A lot of the discussion is about p-values, so I’ll remind everyone that I think the problems with p-values are really problems with null hypothesis significance testing and naive confirmationism. I discuss this in my article, The problems with p-values are not just with p-values, and my post, Confirmationist and falsificationist paradigms of science. The trouble is that, in practice, null hypothesis significance testing and naive confirmationism are often what p-values are used for!
There’s also a separate question about whether p-values should be “banned” or whatever. I don’t think any statistical method should be banned. I say this partly because I used to work at a statistics department where they pretty much tried to ban my methods! So I have strong feelings on that one. The flip side of not banning methods is that I should feel no obligation to believe various Freakonomics, Ted-talk crap about beauty and sex ratio or the critical positivity ratio or the latest brilliant nudge study, just cos it happens to be attached to “p less than 0.05.” Nor should anyone feel obliged to believe some foolish analysis just because it has the word “Bayes” written on it. Or anything else.
Anyway, feel free to follow the above links and draw your own conclusions.