I happened to come across this post from 2004 on cross validation for Bayesian multilevel modeling. In it, I list some problems that, in the past 17 years, Aki and others have solved! It’s good to know that we make progress.
Here’s how that earlier post concludes:
Cross-validation is an important technique that should be standard, but there is no standard way of applying it in a Bayesian context. . . . I don’t really know what’s the best next step toward routinizing Bayesian cross-validation.
And now we have a method: Pareto-smoothed importance sampling. Aki assures me that we’ll be solving more problems about temporal, spatial and hierarchical models.
I agree that CV is important in a lot of areas, but I’m not convinced that it should be “standard”. To be clear, I think CV is a valuable tool to keep in the shed, just not something that is relevant or meaningful in all situations.
CV is meant to estimate how closely a model will come to predicting future data from the same data generating processes that produced the data you already have. So it is clearly useful when that’s how the model is going to be used. It is also useful as a model comparison tool when the models under consideration are extremely flexible relative to the constraints provided by the data at hand. Many ML models fall under this heading because, like deep networks, they are overparameterized relative to the data. So CV is necessary to know whether good predictive performance is due to fitting consistent features of the data or just due to over-fitting noise.
But CV is not so useful when future data aren’t guaranteed to look like past data. This can happen when the data generating process is not stationary. For example, it is not clear to me whether CV would be able to say anything about whether a model could predict future election outcomes more than a few years out.
CV, at least as typically practiced, is also not useful when your main interest is in generalizing to situations where you know the data generating process will differ systematically but you cannot (yet) say how. This is the case for most scientific models. Although a good model in science should be able to predict data of the type we have already seen (and this could be assessed with CV), it should *also* be able to predict data from entirely different experiments using different conditions/manipulations that haven’t been studied yet. This is a different kind of predictive ability than is assessed by CV.
Finally, CV is not particularly relevant in situations where the model is identifiable with existing data and the goal is to estimate parameters of the model. Then, you should just do Bayesian estimation. The joint posterior distribution over parameters will, as a side effect, contain the information you need to detect whether there may be over-fitting, like multi-modality or strongly correlated estimates suggesting trade-offs. Okay, that’s true “in principle”; in practice, it is still hard to get a good picture of the joint posterior distribution for complex models. My point is only that CV doesn’t have any particular role to play in that situation.
Thanks for the comment. It seems we agree on most things except how often valuable tools should be taken out of the shed :)
CV can weighted to assume alternate data generating processes. Of course that means we need to make some assumption about the future data generating process, but hold for any approach trying to predict future, and the future can’t be too much different that weighting would work, and for further extrapolation more modelling is needed as in any extrapolation.
It seems predicting election outcomes in many countries a few years out is very difficult and not related to CV. Otherwise for time series check also leave-future-out cross-validation.
First, CV is still useful as if the performance assuming stationarity is not sufficient then we can’t expect the performance be sufficient in case of nonstationairy. Second, no approach works well if you cannot say anything about the future (aka no free lunch theorem)
We also advocate looking at the posterior instead of using CV when applicable (see e.g. Bayesian workflow paper and talks and case studies at https://avehtari.github.io/modelselection/). But… First, identifiability doesn’t guarantee that model is well specified and looking at the identified posterior of mis-specified model can lead to wrong conclusions, and thus CV is useful for model checking before looking at the posterior. Second, evne if the model is well specified, the posterior may have such dependencies between parameters that inferring what is going on from the posterior can be very difficult, and for example, marginal posteriors are often misinterpreted (see. many examples at https://avehtari.github.io/modelselection/)
To conclude, you list many challenges for CV, but they are not unique for CV, and we agree that CV is not the only valuable tool (and bring out the valuable tools out of the shed and use them!)
I generally agree, and thanks for checking in! I agree that many of the issues I talked about are not just problems for CV. I wanted to point them out, however, because there is a tendency to treat CV as a panacea. In pointing out these limitations, I agree that they are, to a large degree, shared by all quantitative model evaluation methods.
I think a lot of misapplications of CV arise from people either forgetting about or failing to confront the fact that they really want to generalize to non-stationary or unknown data generating processing. Essentially, they know that CV is a measure of “out-of-sample” predictive accuracy, and they assume too broad a definition of “out-of”. This is why I focused on those issues especially.
> First, identifiability doesn’t guarantee that model is well specified and looking at the identified posterior of mis-specified model can lead to wrong conclusions, and thus CV is useful for model checking before looking at the posterior.
I don’t see why this is a CV problem, per se, rather than one to be addressed by posterior predictive checks. I guess I view such predictive checks in general as a “standard” part of modeling, whether Bayesian or not. Definitely a tool that should see a lot of use!
> the posterior may have such dependencies between parameters that inferring what is going on from the posterior can be very difficult, and for example, marginal posteriors are often misinterpreted
Strong agreement on this point! I was careful to specify that the *joint* posterior distribution conveys information about dependencies, but I agree that often people just look at marginals which can be quite misleading.
On another positive note (related to our discussion of Stan the other day), getting good samples from the joint posterior is another area in which efforts from you, Andrew, Bob, etc. have made real progress. Though, to be fair, I have room in my heart—and my shed—for JAGS too!
It was my impression that cross validation has a slight bias that decreases with the number of folds due to the reduction in sample size and a very large sampling variance due to dependence between folds, and that the variance is high enough that it’s not generally not actually a good estimator of out-of-sample performance except in very-tall-data situations. However, the cross validated estimator is monotonic with the expected loss for a particular population and fitting algorithm/model family, so it’s still useful for comparison between multiple models and can still detect things going drastically wrong in cases like MLE linear regression where n_coefs > len(data). Though trying many hyperparameters and models and selecting the lowest cross validated loss still constitutes a fitting-algorithm in a meta-sense, and the cross validation estimators are not monotonic with expected loss across these meta-fitting algorithms, and will always reward trying more things.
I think this is tautologically in the m-closed case where your model family contains the “true” model, but if you have multiple possible models you’d still want to cross validate. However, I think it’s true that fully bayesian estimation is inherently more robust to traditional overfitting because they ideally explore all of the likelihood function, so I think posterior predictive intervals are probably fine if you’re only trying a couple of different things.
> I think this is tautologically in the m-closed case where your model family contains the “true” model, but if you have multiple possible models you’d still want to cross validate
I wasn’t talking about model comparison there, just about estimating the parameters of some chosen model (though arguably this could be considered model comparison in a continuous sense).
Whether CV is useful in any particular model comparison depends on what you value about the models being compared. If you want to prioritize predictive performance on the same type of data to which the models are being fit, then CV makes sense. But there can be other criteria (interpretability, prior predictive flexibility, generalizability to novel settings, alternative measures of complexity, practicality, etc.) that CV won’t capture. As Aki points out, all quantitative model comparison methods have shortcomings, not just CV! My point is only that one should try to pick the right tool for the right job, and sometimes CV ain’t the right tool.
The biggest problem with CV is that people:
1) make a model
2) do CV and check the performance
3) play with the model
4) go to #2
Then you end up overfitting anyway. You still need to use a real holdout dataset collected more recently than the training data. For publication purposes the final CV performance is pretty worthless since you have to assume this type of “p-hacking” went on.
Indeed! And there’s a relevant xkcd that gets at the essence of this issue:
https://xkcd.com/221