More on that horrible statistical significance grid

Regarding this horrible Table 4:

Eric Loken writes:

The clear point or your post was that p-values (and even worse the significance versus non-significance) are a poor summary of data.

The thought I’ve had lately, working with various groups of really smart and thoughtful researchers, is that Table 4 is also a model of their mental space as they think about their research and as they do their initial data analyses. It’s getting much easier to make the case that Table 4 is not acceptable to publish. But I think it’s also true that Table 4 is actually the internal working model for a lot of otherwise smart scientists and researchers. That’s harder to fix!

Good point. As John Carlin and I wrote, we think the solution is not to reform p-values or to replace them with some other statistical summary or threshold, but rather to move toward a greater acceptance of uncertainty and embracing of variation.

3 thoughts on “More on that horrible statistical significance grid

  1. Re: A common conceptual error is that researchers take the rejection of a straw-man null as evidence in favor of their preferred alternative.

    Hehehe, ‘strawman’ is the EXACT characterization came to my mind several years ago. I don’t think one needs any statistical training to discern it. It surfaces when statisticians informally talk about NHST and p-values. So again it’s in the process of conceiving hypotheses that are lapses of analytic power.

  2. “… the solution is not to reform p-values or to replace them with some other statistical summary or threshold, but rather to move toward a greater acceptance of uncertainty and embracing of variation.”

    Yes, yes, yes.

    However, I suspect that this solution (acceptance of uncertainty and variation) needs to start much earlier than when people typically first learn statistics — perhaps in secondary school, but probably earlier.

  3. More evidence in favor of Loken’s idea: I used to have a collaborator whom I’d tease for fetishizing statistical significance. One day we were meeting to look at some new analysis, and in addition to making some regression tables, as a joke I made a regression table where I removed all the coefficients and standard errors and replace them with stars (or no stars). My collaborator loved it and wanted me to do that for all future analysis!

    It wasn’t his complete mental model–he still wanted to see the full results, but those were secondary. His primary filter was to look at the stars. Like Loken, I don’t think this is an unusual case; I think it’s the norm in many fields.

Leave a Reply to John Cancel reply

Your email address will not be published. Required fields are marked *