Alicia Arneson writes:
I am a PhD student at Virginia Tech studying quantitative ecology. This semester, I am taking Deborah Mayo’s Philosophy of Statistics course, so I can’t help but to think more critically about statistical methods in some of the papers I read. To admit my current statistical bias – I do work in a lab that is primarily Bayesian (though this is my first year so I am still somewhat new to it), but Dr. Mayo does have me questioning some aspects of Bayesian practice. One of those questions is the topic of this letter!
Recently, I read a paper that aimed to determine the effect of increased foraging costs on passerine immune function. The experiment seemed really well designed, but I was somewhat frustrated when I got to the statistical analysis section. The authors used Bayesian univariate response models that fit each immune outcome to upwards of 26 parameters that included up to four-way interactions. My initial feeling was that there is no good way to (a) interpret these or (b) to feel at all confident about the results.
In investigating those thoughts, I came across your blog post entitled “You need 16 times the sample size to estimate an interaction than to estimate a main effect.” I thought this was a very interesting read and, while it applies more to frequentist frameworks, I noticed in the comments that you suggested not that we shouldn’t try to estimate interactions, but rather that it would be better to estimate them using a Bayesian approach. I can somewhat understand this suggestion given the examples you used to demonstrate how standard errors can change so much, but what is less clear to me is how Bayes provides a better (or at least more clear) approach when estimating interaction effects.
Therein lies my questions. If you have some time, I am curious to know what you think about:
(a) how a Bayesian approach for estimating interactions is better than doing so under a frequentist methodology, and
(b) can researchers use Bayesian methods to “go too far,” so to speak, when trying to estimate interaction effects that their design would not have captured well (thinking along the lines of classical experimental design and higher order effects being masked when sample sizes are too small), i.e. should a relatively small experiment ever attempt to quantify complex interactions (like a 4-way interaction), regardless of the framework?
Lots to chew on! Here are my responses:
1. As discussed, estimates of interactions tend to be noisy. But interactions are important! Setting them to zero is not always a good solution. The Bayesian approach with zero-centered priors partially pools the interactions toward zero, which can make more sense.
2. We need to be more willing to live with uncertainty. Partial pooling toward zero reduces the rate of “statistical significance”—estimates that are more than two posterior standard deviations from zero—as Francis Tuerlinckx and I discussed in our article from 2000 on Type M and Type S errors. The point is, if you do a Bayesian (or non-Bayesian) estimate, we don’t recommend acting as if non-statistically-significant parameters are zero.
3. I think the Bayesian method will “go too far,” in the sense of apparently finding big things that aren’t really there, if it uses weak priors. With strong priors, everything gets pulled toward zero, and the only things that remain far from zero are those where there is strong evidence.
4. Bayesian or otherwise, design matters! If you’re interested in certain interactions, design your study accordingly, with careful measurement and within-person (or, in your case, within-animal) measurements; see discussion here. There are problems with design and data collection that analysis can’t rescue.
5. To look at it another way, here’s an article from 2000 where we used frequentist analysis of a Bayesian procedure to recommend a less ambitious design, on the grounds that inferences from the more ambitious design would be too noisy to be useful.
I’m guessing it is this 2022 paper:
https://pubmed.ncbi.nlm.nih.gov/36268692/
This is the model used to “predict” (post-dict) plasma nitric oxide shown in table S1:
Then it looks like the model coefficients are shown in table S3. Great, so now we can test the predictive skill of this model on new data.
First repeat the experiment as exactly as possible, plug in the data, and see if the model predictions are consistent with the new nitric oxide observations.
Second, see if the model can extrapolate outside the ranges used to fit the coefficients. Eg, “Age at start of treatment” can be varied to be much younger or older than done for the current model.
If it still seems to work, great. We have a model with predictive skill. This can provide usable information. But there can still be other models with essentially the same skill, or even better.
Eg, what about including the weight of the bird in the model? Or the amount of sleep they get? There are many possibilities. Adding a new variable is also going to change all the other coefficients. So which set of coefficients, if any, is correct?
Once you realize the values of these coefficients are arbitrary, like weights in a neural network, you’ll understand why your initial intuition is correct:
There simply is no good way to interpret arbitrary numbers. And checking if the p-values (or bayes factor, or whatever) derived from these arbitrary numbers are higher/lower than another arbitrary number (eg, 0.05) is even sillier.
If you want meaningful coefficients/parameters, you need to derive the model from a set of plausible assumptions. Then the coefficients have meaning to anyone willing to accept your assumptions at least approximate reality.
If you do a Bayesian analysis, then it will tell you what your data tells you, given your model (where “model” includes your prior). Adding a prior to a bad model won’t make it a good model.
> Adding a prior to a bad model won’t make it a good model.
This is way too broad a statement to be true. It definitely depends on why the model was bad in the first place.
As just one of many possible counter examples, a model may be bad because it is nonidentifiable. Adding a proper prior can make it identifiable.
I’m delighted to come across a letter from a student of mine, Alicia, on this post! (Since I’ve already given her the highest grade in our class, I can’t raise it.)
One thing: Andrew writes: “strong priors, everything gets pulled toward zero, and the only things that remain far from zero are those where there is strong evidence”. But this is not so if strong priors are placed on non-zero effects. It’s also not clear if Andrew intends these “strong priors” are to be subjective, conventional, empirical or something else.
Deborah:
My statement, “strong priors, everything gets pulled toward zero, and the only things that remain far from zero are those where there is strong evidence,” is true if those strong priors are placed on non-zero effects.
At least, I think that statement of mine is correct. Do you have a counterexample?
Also, the statement is just math, and the math does not depend on the source of the prior information. For that matter, the math does not depend on if you intend the data model to be “subjective, conventional, empirical or something else.” (Remember the camel.)
I think she meant, eg, your prior on the coefficient for “age at start foraging treatment” need not be centered at zero.
You can have a strong prior that it is between 0.5-1, or whatever.
That seems rather obvious though and not what AG meant, as he stated he was talking about “zero-centered priors”.
I don’t understand Deborah Mayo’s statement, as this is the purpose of priors like regularized horseshoe in p>>>n problems.
Honestly, I think I understand her position very well.
She just needs to try applying her ideas to a real world problem without committing any of the fallacies she notes. I think it is impossible.
Either you figure it out and stop using this NHST logic (that ignores normalizing to all the plausible explanations along with costs/risks/benefits), or you commit a fallacy.