“The harm done by tests of significance” (article from 1994 in the journal, “Accident Analysis and Prevention”)

Ezra Hauer writes:

In your January 2013 Commentary (Epidemiology) you say that “…misunderstanding persists even in high-stakes settings.” Attached is an older paper illustrating some such.

“It is like trying to sink a battleship by firing lead shot at it for a long time”—well put!

5 thoughts on ““The harm done by tests of significance” (article from 1994 in the journal, “Accident Analysis and Prevention”)

  1. Ultimately, the ammunition Bayesians throw at NHST’ing comes from their foundational understanding of statistics. If their ammunition is like using lead shot against a battleship then what does that tell you about Bayesian’s understanding of their foundations?

    Similarly, the teaching of statistics depends almost entirely on statistician’s understanding of the foundations (and not as some think on classroom gimmicks or data analysis of teaching methods). If that teaching is as screwed up as everyone laments than what does that tell you about statistician’s understanding of their own subject?

    One thing all statisticians implicitly agree on is that all the big ideas in statistics have been discovered. There may be some epsilon developments here and there, and there may be argument over which big ideas are true, but most statisticians think there are no major additions, surprises, or surprising new big ideas.

    This is false.

    Statistics is having all these troubles because there are big ideas which aren’t being included in the foundations. Statistics today is like Ptolemy trying to predict the motion of the planets without Newton’s Laws of gravity. The errors Ptolemy’s predictions kept producing weren’t minor irritations to be fixed with better teaching or more epicycles, they were symptoms that Ptolemy was missing some big important ideas.

    Instead of fiddle farting around with whatever lame blabitty blah statisticians like to waste time on, they should get busy finding those missing big ideas. The place to start by the way is Jaynes’s papers (not his book).

    • It is worth looking at the article. That comment is preceded by “As to the habit of subjecting the data from each study to the NHST separately, as if no previous
      knowledge existed …”. I think it is a very fair of typical practice and one that most frequentists would agree with (else why would frequentist meta-analysis be so common).

  2. Hi Andrew,

    This is a great paper that highlights the minefield of NHST – how NHST smuggles in a theory of decision making that may not be appropriate.

    It sounds like the traffic engineers looked at the number of accidents at a particular site. For the right-turn-on-red case, they probably could have greatly reduced the variance of their sample data by considering all the cars that drove through the same site that did not have accidents, perhaps modeling the auto accidents as a Poisson process. (I recognize that data might not be available.) In other words the true sample size is probably very large and therefore the accident rates per car-use are probably more precise and stable than they appeared in the original analysis.

  3. For those who don’t care to read the paper: the cases described therein are all instances of what Mayo calls the fallacy of acceptance — that is, if the null is not rejected, then it must be accepted. A little multilevel modelling would have avoided this pitfall altogether…

Leave a Reply to annlia Cancel reply

Your email address will not be published. Required fields are marked *