Arnaud Trolle (no relation) writes:
I have a question about the interpretation of (non-)overlapping of 95% credibility intervals. In a Bayesian ANOVA (a within-subjects one), I computed 95% credibility intervals about the main effects of a factor. I’d like to compare two by two the main effects across the different conditions of the factor. Can I directly interpret the (non-)overlapping of these credibility intervals and make the following statements: “As the 95% credibility intervals do not overlap, both conditions have significantly different main effects” or conversely “As the 95% credibility intervals overlap, the main effects of both conditions are not significantly different, i.e. equivalent”?
I heard that, in the case of classical confidence intervals, the second statement is false, but what happens when working within a Bayesian framework?
My reply:
I think it makes more sense to directly look at inference for the difference. Also, your statements about equivalence are all in the context of whatever prior distribution you are using. If you use a very weak prior distribution, you will be more likely to make strong probability statements, for example with a posterior distribution implying 90% certainty, say, that theta_1 > theta_2. Such a comparison is not statistically significant at the conventional 95% level but is a strong statement as a guide to action. However, such probabilities can’t be taken too seriously as they often are the result of implausible priors.
If you’ve done a bayesian analysis most likely you have a bunch of samples of the coefficients, so it would be relatively easy to look at the distribution of the samples of the difference in coefficients and give a 95% credible interval on that difference (which is a simple version of what I think Andrew means when he says to “directly look at inference for the difference”).
Can you clarify this part please;
If you use a very weak prior distribution, you will be more likely to make strong probability statements
Surely if you use a strong prior that supports all the differences being large, the posterior is more likely to make strong probability statements of the sort you describe?
Fred:
By a strong probability statement, I’m talking about a claim that an effect is positive (or negative) with high probability, a statement such as Pr(theta>0) = 0.98. See my 2000 paper with Tuerlinckx for more on this.