David Zimmerman writes:
I am a grad student in biophysics and basically a novice to Bayesian methods. I was wondering if you might be able to clarify something that is written in section 7.2 of Bayesian Data Analysis. After introducing the log pointwise predictive density as a scoring rule for probabilistic prediction, you say:
The advantage of using a pointwise measure, rather than working with the joint posterior predictive distribution … is in the connection of the pointwise calculation to cross-validation, which allows some fairly general approaches to approximation of out-of-sample fit using available data.
But would it not be possible to do k-fold cross validation, say, with a loss function based on the joint predictive distribution over each full validation set? Can you explain why (or under what circumstances) it is preferable to use a pointwise measure rather than something based on the joint predictive?
My reply: Yes, you can do k-fold cross validation. One reason we do leave-one-out (LOO) is for computational efficiency: you don’t need to re-fit the model each time, instead you can repurpose a single set of draws from the full posterior distribution using Pareto smoothed importance sampling. Ultimately, though, yes, it can make sense to look at predictive accuracy for aggregate quantities as well as for individual observations, and this comes up with multilevel models. Wei Wang and I discussed some of these challenges in this article from 2015.
You can also take a look at Aki’s faq on cross validation.
Lauren’s and my grad student Alex Cooper wrote about this at length in the context of time-series data (with help from Aki and Catherine Forbes). The short version is that when there is dependence in the data, you get less variability if you use joint densities in your CV https://arxiv.org/abs/2301.08276. It’s a pretty nice paper.
I’ll just comment that this question addresses an important conceptual question in any kind of modeling, namely, what is the “unit of replication”? I think this is also what the work Dan points to above is getting at.
For example, imagine a classroom full of students who take several quizzes over the course of a semester. We can build a model to predict the probability correct for each question answered by each student on each quiz. But then, what is the “unit of replication”? If it is the individual response by each student on each quiz, then LOOCV makes sense, where what is “left out” is a response on an individual question by an individual student on an individual quiz. But the unit of replication could instead be considered to be each student’s performance on a whole quiz, in which can you would “leave out” all the questions from an individual student on an individual quiz. The unit could also be the trajectory of performance across the semester for a student, in which case you would “leave out” all the responses from an individual student, etc.
The “unit of replication” should, in my view, ultimately reflect the theory behind the model—is it a model of how students respond to single questions on quizzes? Of how responses are produced across an entire quiz? Of how student performance evolves over time?
Crossvalidation is more suttle than just accounting for dependence in the data. It needs to account for the data generation process. We address this in a paper on befitting cross validation (BCV)
Kenett, R. S., Gotwalt, C., Freeman, L., & Deng, X. (2022). Self‐supervised cross validation using data generation structure. Applied Stochastic Models in Business and Industry, 38(5), 750-765
BCV applies three principles:
BCV Principle 1: The formation of training and hold-out datasets should reflect the goal of the study. What is the goal of the study?
BCV Principle 2: The training dataset and the hold-out dataset should have the same data generation structure as the whole dataset. What is the data structure?
BCV Principle 3: The construction of the hold-out dataset should reflect the data generation structure needed for the predictive model. Is the hold out set with a matching structure?