Ryan Ickert writes:
I was wondering if you’d seen this post, by a particle physicist with some degree of influence. Dr. Dorigo works at CERN and Fermilab.
The penultimate paragraph is:
From the above expression, the Frequentist researcher concludes that the tracker is indeed biased, and rejects the null hypothesis H0, since there is a less-than-2% probability (P’<α) that a result as the one observed could arise by chance! A Frequentist thus draws, strongly, the opposite conclusion than a Bayesian from the same set of data. How to solve the riddle?
He goes on to not solve the riddle. Perhaps you can?
Surely with the large sample size they have (n=10^6), the precision on the frequentist p-value is pretty good, is it not?
The first comment on the site (by Anonymous [who, just to be clear, is not me; I have no idea who wrote that comment], 22 Feb 2012, 21:27pm) pretty much nails it: In setting up the Bayesian model, Dorigo assumed a silly distribution on the underlying parameter. All sorts of silly models can work in some settings, but when a model gives nonsensical results—in this case, stating with near-certainty that a parameter equals zero, when the data clearly reject that hypothesis—then, it’s time to go back and figure out what in the model went wrong.
It’s called posterior predictive checking and we discuss it in chapter 6 of Bayesian Data Analysis. Our models are approximations that work reasonably well in some settings but not in others.
P.S. Dorigo also writes:
A Bayesian researcher will need a prior probability density function (PDF) to make a statistical inference: a function describing the pre-experiment degree of belief on the value of R. From a scientific standpoint, adding such a “subjective” input is questionable, and indeed the thread of arguments is endless; what can be agreed upon is that in science a prior PDF which contains as little information as possible is mostly agreed to be the lesser evil, if one is doing things in a Bayesian way.
No. First, in general there is nothing more subjective about a prior distribution than about a data model: both are based on assumptions. Second, if you have information, then it’s not “the lesser evil” to include it. It’s not evil at all! See, for example, the example in Section 2.8 of Bayesian Data Analysis.
P.P.S. I ran the above by a couple of physicist types. They said it was ok, but one wrote:
There is a lot of poop being thrown these days between bayesians and frequentists. I am not sure why; that fight seems so 2003 to me. But anyway, I guess it is worth responding.
I agree and have no desire to throw any poop. It can be useful to highlight the differences between different approaches, and I think it can also be useful to clear up misconceptions, such as the idea that Bayesian inference is particularly “subjective” (or, to put it another way, the idea that non-Bayesian inference is particularly “objective”). One thing I like about Bayesian methods is how they force you to put your assumptions out there, potentially to be criticized. But I understand that others prefer a mode of inference that makes minimal assumptions.