I’ve written this explanation on the board often enough that I thought I’d put it in a blog post.
Bayes factors
Bayes factors compare the data density (sometimes called the “evidence”) of one model against another. Suppose we have two Bayesian models for data 
, one model 
with parameters 
and a second model 
with parameters 
The Bayes factor is defined to be the ratio of the marginal probability density of the data in the two models,
where we have
![p_1(y) = \mathbb{E}[p_1(y \mid \Theta_1)] \ = \ \int p_1(y \mid \theta_1) \cdot p_1(\theta_1) \, \textrm{d}\theta_1](https://statmodeling.stat.columbia.edu/wp-content/ql-cache/quicklatex.com-e5f7f69b0078945f7ced307ebb9b9b18_l3.png "Rendered by QuickLaTeX.com")
and
![p_2(y) = \mathbb{E}[p_2(y \mid \Theta_2)] \ = \ \int p_2(y \mid \theta_2) \cdot p_2(\theta_2) \, \textrm{d}\theta_2.](https://statmodeling.stat.columbia.edu/wp-content/ql-cache/quicklatex.com-945783ca00f23e2e7013ef1941dcbfc4_l3.png "Rendered by QuickLaTeX.com")
The distributions 
and 
are known as prior predictive distributions because they integrate the likelihood over the prior.
There are ad-hoc guidelines from Harold Jeffreys of “uninformative” prior fame, classifying Bayes factor values as “decisive,” “very strong,” “strong,” “substantial,” “barely worth mentioning,” or “negative”; see the Wikipedia on Bayes factors. These seem about as useful as a 5% threshold on p-values before declaring significance.
Held-out validation
Held-out validation tries to evaluate prediction after model estimation (aka training). It works by dividing the data into two pieces, 
and then training on 
and testing on 
. The held out validation values are
![p_1(y'' \mid y') = \mathbb{E}[p_1(y'' \mid \Theta_1) \mid y'] = \int p_1(y'' \mid \theta_1) \cdot p_1(\theta_1 \mid y') \, \textrm{d}\theta_1](https://statmodeling.stat.columbia.edu/wp-content/ql-cache/quicklatex.com-9af4570cd2fcd0548410687634122c68_l3.png "Rendered by QuickLaTeX.com")
and
![p_2(y'' \mid y') = \mathbb{E}[p_2(y'' \mid \Theta_2) \mid y'] = \int p_2(y'' \mid \theta_2) \cdot p_2(\theta_2 \mid y') \, \textrm{d}\theta_2.](https://statmodeling.stat.columbia.edu/wp-content/ql-cache/quicklatex.com-805c47c594e6d0f9be1e7b9f2a9d1cd2_l3.png "Rendered by QuickLaTeX.com")
The distributions 
and 
are known as posterior predictive distributions because they integrate the likelihood over the posterior from earlier training data.
This can all be done on the log scale to compute either the log expected probability or the expected log probability (which are different because logarithms are not linear). We will use expected log probability in the next section.
(Leave one out) cross validation
Suppose our data is 
. Leave-one-out cross validation works by successively taking 
and 
and then averaging on the log scale.
and
Leave-one-out cross validation is interpretable as the expected log posterior density (ELPD) for a new data item. Estimating ELPD is (part of) the motivation for various information criteria such as AIC, DIC, and WAIC.
Conclusion and a question
The main distinction between Bayes factors and cross validation is that the former uses prior predictive distributions whereas the latter uses posterior predictive distributions. This makes Bayes factors very sensitive to features of the prior that have almost no effect on the posterior. With hundreds of data points, the difference between a normal(0, 1) and normal(0, 100) prior is negligible if the true value is in the range (-3, 3), but it can have a huge effect on Bayes factors.
This matters because pragmatic Bayesians like Andrew Gelman tend to use weakly informative priors that determine the rough magnitude, but not the value of parameters. You can’t get good Bayes factors this way. The best way to get a good Bayes factor is to push the prior toward the posterior, which you get for free with cross validation.
My question is whether the users of Bayes factors really believe so strongly in their priors. I’ve been told that’s true of the hardcore “subjective” Bayesians, who aim for strong priors, and also the hardcore “objective” Bayesians, who try to use “uninformative” priors, but I don’t think I’ve ever met anyone who claimed to follow either approach. It’s definitely not the perspective we’ve been pushing in our “pragmatic” Bayesian approach, for instance as described in the Bayesian workflow paper. We flat out encourage people to start with weakly informative priors and then add more information if the priors turn out to be too weak for either inference or computation.
Further reading
For more detail on these methods and further examples, see Gelman et al.’s Bayesian Data Analysis, 3rd Edition, which is available free online through the link, particularly Section 7.2 (“Information criteria and cross-validation,” p. 175) and section 7.4 (“Model comparison using Bayes factors,” page 183). I’d also recommend Vehtari, Gelman, and Gabry’s paper, Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC.
Bob:
That’s an interesting framing. Also relevant is my article with Rubin, Avoiding model selection in Bayesian social research, from Sociological Methodology 1995.
“How can BIC conclude that θ=0 is the better model? The answer lies in the implicit improper prior distribution on A that is assumed by BIC – a mixture of a point mass at θ=0 and a uniform density on the real line. We use the term “proper prior distribution” in its technical sense to mean a distribution for the parameter that integrates to 1 and does not depend on the data.”
The answer cannot really lie in the improperness of the prior: the situation would be quite the same if the prior had been a mixture of a point mass at θ=0 and a (proper) uniform density on an interval from minus one gazillion plus one gazillion.
Good point. They should have written “implicit prior distribution” to better bring across the point that this prior doesn’t make any sense.
Their focus is not the improperness of the prior, but the same aspect that you mentioned, the mass it places implausibly far from zero.
If they wanted to put the focus on the improperness of the prior why discuss that extensively?
Why devote a full sentence to focus on the technical definition of proper prior distribution – which includes distributions with most of their mass implausibly far from zero – when they could use those 29 words to talk about the problem being caused by distributions with most of their mass implausibly far from zero?
Why suggest in the conclusions that a possible solution to the problems posed by multiple models is to calculate Bayes factors using proper distributions?
Jackson, Carlos:
We wrote that article in 1995, in response to a different article in that journal recommending the use of the method called BIC to select among models. That article included two applied examples. The main points of our article were: (1) model selection of the sort demonstrated in the paper we were discussing does not generally make sense in social science and in particular did not make sense in those two applied examples, and (2) if you want to do Bayesian model averaging or model selection using the marginal likelihood, BIC does not make sense. This nonsensicality is particularly clear with improper priors, which is what were used in the paper being discussed, so that was our focus. Similar problems arise with priors that are weak but not improper. We didn’t discuss that issue in our paper. There’s only so much that you can put into one paper, also back in 1995 we hadn’t completely thought things through to that extent. At that time, we were already a pretty extreme position to not recommend integration over the prior, and since then our thinking has become more clear.
A recent paper has shown that the prior predictive density has a deeper connection to cross-validation, one not described in the above post.
https://academic.oup.com/biomet/article/107/2/489/5715611?login=false&s=09
I’ve seen this argument before, with Bayes factor fans claiming that “if you support cross-validation, you need to support Bayes factors, becasue they are cross-validation”. I think the result is neat, but the cross-validation scheme corresponding to Bayes factors (the sum of leave one out + leave two out + … + leave all data out) is a pretty weird one. Yes, if your model and prior is already so well specified, that you can use it to predict future observations without needing any extra data, then you maybe are somewhat interested in predicting datasets of varying sizes after observing no or few additional observations, But that is AFAIK a rare situation outside physics.
Contrast that with the general recommendations for CV that the structure of the folds (leave one out; leave one group out; leave future out; …) should match the prediction task you are interested in. Using Bayes Factors thus binds you to a specific (and IMHO weird) fold structure instead of letting you use an appropriate one…
It is also unclear how the result translates to situations where the data have more structure and cannot be just treated as a vector.
The beautifully true part of Bayes rule is the denominator. The equation gives a way to compare one explanation to all the others.
If by denominator, you mean the integral of p(theta, y) = p(y | theta) * p(theta) w.r.t. theta, to get p(y), then yes, that’s the cool part that’s lets you fully invert—the rest is just the chain rule.
The problem with using p(y) to compare models is that it’s very prior sensitive.
> There are ad-hoc guidelines from Harold Jeffreys of “uninformative” prior fame, classifying Bayes factor values as “decisive,” “very strong,” “strong,” “substantial,” “barely worth mentioning,” or “negative”; see the Wikipedia on Bayes factors. These seem about as useful as a 5% threshold on p-values before declaring significance.
There are also ad-hoc guidelines for comparing ELPD values stating that differences smaller than 4 are small and then the models have very similar predictive performance: https://avehtari.github.io/modelselection/CV-FAQ.html#se_diff
I’d lump those in the same category. It all depends on the application.
> The best way to get a good Bayes factor is to push the prior toward the posterior, which you get for free with cross validation.
If “the Bayes factor is defined to be the ratio of the marginal probability density of the data in the two models” what does it mean to “get a good Bayes factor”? Whatever you think you get for free with cross validation wouldn’t you get it for both models?
I think there’s a typo in the equation starting with
p_2(y” | y’) = …
It should be \theta_2 (instead of \theta_1).
Fixed. Thanks.
I’d rather say that Bayes factors evaluate predictions; cross validation evaluates postdictions.
I think most people instinctively know which one should be valued. Would anyone be impressed if I built a model that successfully postdicted yesterday’s football results?
I’d be very impressed if you could build a model that predicts the results of football games based solely on a prior and not based on the previous games in the season. When the predictions are sequential, like in football games, you need to use leave-future-out cross-validation. See, for example, Approximate leave-future-out cross-validation for Bayesian time series models> by Paul Bürkner, Jonah Gabry, and Aki Vehtari.
Exactly – successful prediction is truly challenging and should be rewarded in any sensible evaluation of a model.
You complain that the Bayes factor is sensitive to priors, but this is a reflection of the fact that it’s sensitive to what models predict. As any sensible evaluation must be sensitive to what a model predicts, it must be sensitive to the prior.
Suppose I built a model that predicted with certainty that Arsenal would be unbeaten in 38 games. And indeed we observed over the following year that they were unbeaten in 38 games.
Now suppose I made a second model that made a vague prediction: Arsenal win or draw with an unknown probability p. Applying LOO cross validation, based on any 37 games (they were unbeaten in them all) we estimate p ~= 1 and successfully postdict that Arsenal were unbeaten in game 38 with probability p ~= 1. This leads to a score of log p ~= 0.
Shouldn’t the observed 38 unbeaten games, which matched exactly the prediction of the first model, not significantly favour the first model by any sensible measure?
What you’re saying is that if 37/38 games have already happened, and you’re building a model to predict the 38th, you should pick a model based on which one best predicted the 37 games without any data, and completely ignore how the model learns from games that have already happened.
The amazing (amusing) thing is that crossvalidation is referred to without considering the data generation process. We strongly suggest this aspect is addressed. It is essential for generalization of findings.
We call this befitting cross validation (BCV), see https://xwdeng80.github.io/BCV2022.pdf
Cross-validation is defined w.r.t. the full joint model p(y, theta). So when I write p(y” | y’), that’s implicitly marginalizating out the theta as shown in the integral.
> The best way to get a good Bayes factor is to push the prior toward the posterior
Hey Bob, I guess some extensions of the usual Bayes factors could be useful to this aim. As you claim, BFs measure prior predictive performance, and the role of the prior could be too dominant there. However, many scholars, such as O’Hagan in this old paper:
https://rss.onlinelibrary.wiley.com/doi/abs/10.1111/j.2517-6161.1995.tb02017.x
show how properly informing the prior distribution through a posterior distribution derived from some training data, and they compare then two models in terms of some held-out data. Moreover, I like the idea of using the posterior predictive distribution of the BFs rather than the prior-predictive distribution when calibrating the BFs, as suggested here:
https://xianblog.wordpress.com/2014/10/08/posterior-predictive-distributions-of-bayes-factors/
Then, what about using BFs with a more ‘predictive’ flavour?
My feeling is that both tools, LOO cross-validation and BFs, could be useful and able to interact one each other: for sure, in order to wisely use BFs, one should move more toward the (posterior) predictive framework.
Nowadays BFs still represent a model comparison milestone in many appllied fields, such as computational biology and genomics: for such reason I would not completely discard them and I support a decisive renovation of their practical use.
From your description, it sounds like O’Hagan is suggesting cross-validation! If you take some data, train on it, then use the posterior p(theta | y) as a prior for new data y’, then the prior predictive inference is p(y’ | y). So it just reduces to cross-validation.
Now I’m all for informing priors with old data.
Leo says, “BFs still represent a model comparison milestone in many appllied fields, such as computational biology and genomics: for such reason I would not completely discard them…” I don’t like these “everyone else is doing it” arguments. If we bought that line of reasoning, we’d all be doing hypothesis testing with multiple comparison adjustments!
Bayes factors are just a logical consequence of Bayes rule. You can’t really argue against it, without arguing against Bayesian inference in general.
Moreover, the Bayes factor is meaningless without a prior distribution on the models.
Finally, there is nothing special about Bayes factors per se. Using weak or non-informative priors is inconsequential. The fact that people think it matters is just an artifact of incoherent model selection practices that have nothing to do with Bayes factors.
Jon:
Bayesian inference, like all mathematical models, makes more sense in some settings than in others. There is a large class of problems for which Bayesian inference conditional on the model makes sense, but Bayesian inference integrating over the models (Bayes factors) does not make sense. This is the class of problems with strong data and noninformative or weakly informative priors. As is well understood, the marginal likelihood and Bayes factor can depend very strongly on aspects of a noninformative or weakly informative prior that have essentially no effect on the posterior distribution conditional on the model. You can see this, for example, by considering a parameter on the unit scale and changing its prior from normal(0, 10) to normal(0,10000), a change which for any reasonably-informative data will have essentially no effect on the posterior but which will change the marginal likelihood under the model by approximately a factor of 1000.
Bayesian inference isn’t a mathematical model; it’s a rule of inductive inference. We can’t pick and choose when it applies based on whether the results make us feel uncomfortable anymore than we can pick and choose when the rules of addition apply based on whether we like the results of particular sums.
Besides, the example you give isn’t surprising or counterintuitive at all. Of course, a model that makes a sharp, correct prediction should be favoured relative to one that makes a vague, diffuse prediction. Sensitivity to a prior is nothing but sensitivity to what a model predicts. You should be more concerned by methods aren’t sensitive to the prior as they aren’t sensitive to what a model predicts.
Other:
Thanks for commenting. There’s a lot of confusion over this general topic so it’s good to bring these discussions out in the open.
I think you’re replying to this comment of mine above.
To briefly reply to your comment:
1. That’s right, Bayesian inference is a mathematical rule, not a model. So let me rephrase my statement, “Bayesian inference, like all mathematical models, makes more sense in some settings than in others,” to “Bayesian inference, like all mathematical rules, makes more sense in some settings than in others.” We can definitely pick and choose when it applies! When Bayesian inference doesn’t apply, it doesn’t mean that Bayesian inference is wrong, it just implies that the model it is attached to has problems. To put it another way: we never use “Bayesian inference” on its own; what we use is “Bayesian inference plus a model.”
2. As you said, consider addition. 1 + 1 = 2, that’s always true. But there are lots of examples of nonadditivity in social science. For example, suppose that a refrigerator or a freezer would be equally valuable to you. You could say that the utility of the fridge and the utility of the freezer are each equal to 1. But U(fridge + freezer) > U(2 fridges) or U(2 freezers). To say this doesn’t mean that math is wrong; it just means that a simple additive utility model, of the sort that is often used by default, doesn’t make sense in this situation. But in other settings an additive utility model can be just fine. I would not pick choose when to use the rules of addition—but I might well decline to use the rules of addition if they are tied to a model that does not make sense in context. To put it another way, I would use the failure of the rules of addition to investigate what went wrong with the assumed model.
3. I agree that the example I gave in that comment is not surprising or counterintuitive. It’s simple math, no surprise at all and it completely fits the intuitions of Bayesian inference.
4. I agree with your final point about sensitivity to prediction. This is a theme that comes up a lot in applied statistics. No model is perfect, and a model can give good predictions in some settings and not others. A familiar example is that interpolative predictions tend to be more stable than extrapolative predictions. In the example I gave in my above comment of a parameter theta that is well estimated with a flat or weak prior, Bayesian inference under such a model can give good predictions for future data from the same experiment (that is, the value of theta that generated the raw data) but terrible predictions for future experiments (that is, new values of theta drawn from the prior). Often a model is set up just to draw inference from the local theta, in which case it can be just fine to use a noninformative or weakly informative prior—but it would be a mistake to use that model to predict new thetas, or to use it to compute a marginal likelihood or Bayes factor.
There is no conflict between Bayes factors and posteriors if you define your priors appropriately.
The fact that a BF can change by 1000 if you spread the mass of the prior on regions not supported by the data is what should be expected , it baffles me that people find this surprising or counter intuitive.
Jon:
Don’t worry, nobody in this discussion seems find it surprising or counterintuitive that a Bayes factor or marginal likelihood can change by 1000 if you spread the mass of the prior on regions not supported by the data.
> My question is whether the users of Bayes factors really believe so strongly in their priors. I’ve been told that’s true of the hardcore “subjective” Bayesians, who aim for strong priors, and also the hardcore “objective” Bayesians, who try to use “uninformative” priors, but I don’t think I’ve ever met anyone who claimed to follow either approach.
Bob, you’ve met me, and I (meaning my co-authors and I) use informative priors in Bayes factors calculations.
See, e.g.,
1. Kate Stone, Bruno Nicenboim, Shravan Vasishth, and Frank Roesler. Understanding the effects of constraint and predictability in ERP. Neurobiology of Language, 2023.
https://direct.mit.edu/nol/article/4/2/221/114205/Understanding-the-Effects-of-Constraint-and
2. Daniela Mertzen, Dario Paape, Brian W. Dillon, Ralf Engbert, and Shravan Vasishth. Syntactic and semantic interference in sentence comprehension: Support from English and German eye-tracking data. Glossa Psycholinguistics, 2, 2023.
https://escholarship.org/uc/item/0p94125d
3. Shravan Vasishth, Himanshu Yadav, Daniel Schad, and Bruno Nicenboim. Sample size determination for Bayesian hierarchical models commonly used in psycholinguistics. Computational Brain and Behavior, 2022
https://link.springer.com/article/10.1007/s42113-021-00125-y
4. Daniel J. Schad, Bruno Nicenboim, Paul-Christian Bürkner, Michael Betancourt, and Shravan Vasishth. Workflow Techniques for the Robust Use of Bayes Factors. Psychological Methods, 2022.
https://psycnet.apa.org/doiLanding?doi=10.1037%2Fmet0000472
5. Bruno Nicenboim, Shravan Vasishth, and Frank Rösler. Are words pre-activated probabilistically during sentence comprehension? Evidence from new data and a Bayesian random-effects meta-analysis using publicly available data. Neuropsychologia, 142, 2020.
https://www.sciencedirect.com/science/article/pii/S0028393220300981?via%3Dihub
I think it is basically pointless trying to do a Bayes factors analysis with uninformative or default or weakly informative priors alone on the target parameter. It’s OK if it’s part of a sensitivity analysis.
I don’t understand why people publish BF-based papers using default priors alone.