10 thoughts on “Cross-validation FAQ”

1. 7 When is cross-validation valid?

   Unless you only run it once (so don’t use the output to do model selection, tune hyperparameters or adjust the data prep) cv skill pretty much always exaggerates the actual skill.

   Whether that makes it invalid depends on the use-case. If your final skill is assessed on new data generated after the training data, it could be fine as a preliminary step.

   Reply ↓

2. I must admit that many of the answers I wrote there were not for questions but rebuttals for false statements.

   > “7 When is cross-validation valid?”
   >
   > Unless you only run it once (so don’t use the output to do model selection, tune hyperparameters or adjust the data prep) cv skill pretty much always exaggerates the actual skill.

   This is a myth and false. There are plenty of papers illustrating the model selection bias, but it’s not happening “pretty much always”. When the best model is better with probability approaching 1, then the expected model selection induced bias approaches 0. Based on various CV-based model comparisons I’ve seen in the wild, there are plenty of cases where there is no “exaggeration”. I think I should clarify when the model selection induced bias is negligible in “5 How to use cross-validation for model selection?”. The section “7 When is cross-validation valid?” used the term valid as I often saw comments claiming CV to be invalid for something, but instead of that dichotomy, it would be better to focus on continuous aspects such as bias and variance. I think I should divide Section 7 to first discuss when it will provide almost unbiased estimates for single models (no selection), second to remind about the possibly non-negligible selection induced bias, and then discuss that we shouldn’t focus just on bias but also care about the variance especially in the selection as reducing variance with slight increase in bias (especially if the bias for two models mostly cancel out) can improve selection.

   Birthday case study is an example where most of the model comparisons are safe and the probability of selection induced bias is practically 0. The last two models have less difference in performance, and there is possibility of selection induced bias.

   Our preprint illustrates how using CV-approach that may have higher bias for a single model but has lower variance can be better in model selection.

   We are working on more papers on illustrating the cases when CV is safe to use for model selection and how we can diagnose when it’s not safe (and e.g. model averaging or better priors are needed). I’ll post here when we get them arxived.

   > Whether that makes it invalid depends on the use-case. If your final skill is assessed on new data generated after the training data, it could be fine as a preliminary step.

   Collecting more data is always useful, but not always possible. It is common to divide the existing data so that some part is used only in the end, but this is less efficient in the cases where use of CV is safe.

   Reply ↓

   • I looked at (but didn’t run) the birthday case study. It trains on data up to 1988. We want to know what happens when you use your final model to predict 1989-2023 and compare to the actual data for those years.

     If I’m reading it right, you chose to summarize the model skill using loo-cv R^2 = 0.94. But what is it for the 1989-2023 holdout?

     Actually, after looking around I’m not finding a continuous daily births dataset. I’m fairly sure I used such data about 5 years ago, but anyway there is 1994-2003 and 2000-2014 data here:

     https://github.com/fivethirtyeight/data/tree/master/births

     It is possible your simpler models (eg, model 1) will not be overfit. However, as soon as the model is flexible enough, choosing models/hyperparameters based on CV performance becomes guaranteed to overfit. It is no myth.

     Reply ↓

     • Anoneuoid, can you link a paper supporting this theoretical result? I’ll look at the preprint Avi linked but would also be interested on something arguing to the contrary.

       Reply ↓

       • It isn’t a theoretical result. It is an empirical result that anyone who does a few analyses using a true holdout will soon discover. When I first learned ML, it was by doing kaggle competitions (where there is a leaderboard using one holdout, then a final score using another holdout). This issue quickly became obvious to me and is considered common knowledge in the ML community.

       • Eg:

         Recall that validation is meant to be a measure of how the model does on data that it hasn’t considered before. You can corrupt this process in subtle ways if the validation data affects the preprocessing behavior. This is sometimes called train-test contamination.

         For example, imagine you run preprocessing (like fitting an imputer for missing values) before calling train_test_split(). The end result? Your model may get good validation scores, giving you great confidence in it, but perform poorly when you deploy it to make decisions.

         After all, you incorporated data from the validation or test data into how you make predictions, so the may do well on that particular data even if it can’t generalize to new data. This problem becomes even more subtle (and more dangerous) when you do more complex feature engineering.

         If your validation is based on a simple train-test split, exclude the validation data from any type of fitting, including the fitting of preprocessing steps. This is easier if you use scikit-learn pipelines. When using cross-validation, it’s even more critical that you do your preprocessing inside the pipeline!

         https://www.kaggle.com/code/alexisbcook/data-leakage

     • > We want to know what happens when you use your final model to predict 1989-2023 and compare to the actual data for those years.

       Why do you think so? What is the application you have in mind? The original question was whether Halloween and Valentine’s Day are special in the given historical time range, and thus there doesn’t seem to be benefit in predicting the future, especially as we assume the behavior is changing in time.

       > But what is it for the 1989-2023 holdout?

       If we were interested in predicting the future, say the next month to plan the needed hospital resources, we would like to use also the most recent data available. Leaving that much out seems to me as waste of data. Of course quite many time series predictions are now very difficult as the years 2020-2022 were quite special.

       > However, as soon as the model is flexible enough, choosing models/hyperparameters based on CV performance becomes guaranteed to overfit. It is no myth.

       It seems the myth is strong. It’s not just about the flexibility, but what is the difference and uncertainty in the comparison. When the difference is big enough and the uncertainty is negligible, there is no overfit, and it’s very easy to repeat such simulations. It is also possible to diagnose when that overfit might happen. I’ll post a link to a manuscript discussing this here soon. Meanwhile, a related classic paper demonstrating the flexibility part is Occam’s razor paper, and there are scattered examples elsewhere, but we’re writing a paper that focuses specifically on CV and overfit as this comes up often.

       EDIT: The above depends naturally on the definition of flexibility, but I think we have reasonable examples where flexibility is increasing and examples where the overfit is mainly for other reasons than CV. And the above depends also on the definition of overfit and whether negligible overfit is stated as no overfit.

       Reply ↓

       • If we iteratively search for whatever hyperparameters (eg, stepsize for MCMC) to max out CV performance, then we are feeding information about the holdouts back into the model. This will overfit.

         The only time it won’t is because it can’t when the model is limited to very few parameters relative to the data available.

         It is not a myth. If there is some mathematical proof it is a myth, then the proof relies on at least one assumption that doesn’t apply to how people use these algorithms in real life (eg, they only run it once).

       • The original question was whether Halloween and Valentine’s Day are special in the given historical time range, and thus there doesn’t seem to be benefit in predicting the future, especially as we assume the behavior is changing in time.

         There is no point to a model that can only post-dict the past, because there are infinitely many of those. Fitting past data is a first step, but then the model needs to be tested on pre-dicting the future, which narrows down the useful candidates considerably.

3. Yes, this CV FAQ has been super helpful to me over the last couple of years! Also, thanks to Aki for taking the time to answer so many Stan forum questions on these (and other) topics.

   Reply ↓