Paul Bürkner, Jonah Gabry, and Aki Vehtari write:

One of the common goals of time series analysis is to use the observed series to inform predictions for future observations. In the absence of any actual new data to predict, cross-validation can be used to estimate a model’s future predictive accuracy, for instance, for the purpose of model comparison or selection. Exact cross-validation for Bayesian models is often computationally expensive, but approximate cross-validation methods have been developed, most notably methods for leave-one-out cross-validation (LOO-CV). If the actual prediction task is to predict the future given the past, LOO-CV provides an overly optimistic estimate because the information from future observations is available to influence predictions of the past. To properly account for the time series structure, we can use leave-future-out cross-validation (LFO-CV). Like exact LOO-CV, exact LFO-CV requires refitting the model many times to different subsets of the data. Using Pareto smoothed importance sampling, we propose a method for approximating exact LFO-CV that drastically reduces the computational costs while also providing informative diagnostics about the quality of the approximation.

A useful generalization, I’d say!

More should be possible, although I’m not quite sure how. One concern with leave-future-out is that you’re evaluating model fit for the early points in the series in a different way than how you’re evaluating later points in the series. Which makes me think that this sort of marginal evaluation could be done in other settings as well. Maybe one way to understand what’s possible would be to apply this mixed marginal/conditional predictive idea (something sliding between prior and posterior predictive checking) for the simpler problem of regression with independent data points. In some sense, leave-one-out or other forms of cross validation are already intermediate steps along this continuum, so maybe this is a good opportunity to explore these ideas further. From the other direction, we can consider problems such as spatial and network models in which there is no ordering, and cross-validation is related to the pseudo-likelihood idea of Besag (1974).