Cross-validation is a technique for estimating how well a predictive model will perform on data it has never seen before. The core idea is simple: instead of training a model on all your data and then hoping it works in the real world, you repeatedly hold out a portion of your data, train on the rest, and test on the held-out portion. The averaged performance across those rounds gives you a more honest picture of how the model generalizes. Despite being one of the most widely used tools in machine learning and statistics, its behavior is surprisingly complex, and common mistakes in applying it can produce results that look great on paper but fall apart in practice.
Why You Cannot Just Test on Your Training Data
Every predictive model faces the same fundamental tension: it needs to learn patterns from data, but some of those “patterns” are actually noise specific to the dataset at hand. A model that memorizes noise will score perfectly on its training data and then fail on new observations. This is overfitting, and it is the central problem cross-validation was designed to detect.
The oldest and most intuitive fix is to split your data into a training set and a test set, say 80/20. You train on the 80% and evaluate on the 20%. The trouble is that a single split is noisy. Depending on which observations land in which group, your estimate of performance can swing substantially. You also lose 20% of your data for training, which hurts when data is scarce. Cross-validation addresses both problems by rotating which portion gets held out, so every observation eventually serves as both training data and test data.
How K-Fold Cross-Validation Works
In k-fold cross-validation, you divide your dataset into k equally sized groups called folds. The model trains on k − 1 folds and is tested on the remaining fold. This process repeats k times, each time holding out a different fold. The performance scores from all k rounds are averaged to give a single estimate. Resampling methods like k-fold cross-validation and bootstrapping are generally recommended for arriving at realistic estimates of how a model will perform on unseen data during the training phase itself.
1PubMed. Foundations of Machine Learning-Based Clinical Prediction Modeling: Part II-Generalization and OverfittingFive-fold and ten-fold cross-validation are the most common choices. With five folds, each round trains on 80% of the data and tests on 20%. With ten folds, each round trains on 90% and tests on 10%. The choice matters more than many practitioners realize, because it directly controls a tradeoff between two sources of error in your performance estimate.
Picking the Right Number of Folds
When you use fewer folds (say, 2 or 3), each training set is substantially smaller than the full dataset. The model sees less data, so it tends to underperform compared to a model trained on everything. Your error estimate ends up pessimistically biased, overstating how bad the model is. On the other hand, when you use many folds (say, 20 or more), each training set is nearly the size of the full dataset, so bias shrinks. But now the training sets across folds overlap heavily, and the individual fold estimates become correlated with each other, which inflates variance.
Research examining this tradeoff across multiple algorithms and datasets confirms the pattern: variance increased as k grew, meaning larger k values produced less stable fold-to-fold estimates. The bias trends, however, depended on the specific algorithm and dataset. Some algorithms showed increasing bias with larger k while others remained comparatively balanced.
2PubMed Central. The impact of K selection in K‑fold cross-validation on bias and variance in supervised learning modelsFive or ten folds tend to land in a reasonable middle ground for most problems. Ten-fold is the more popular default in practice, but five-fold is often preferred when computation is expensive or datasets are small enough that losing even 10% to a test fold creates instability. There is no universally correct k; the best choice depends on how much data you have, how computationally expensive the model is to train, and how sensitive the algorithm is to training set size.
Repeating Cross-Validation for Stability
One way to reduce the noise in a k-fold estimate without pushing k to extreme values is to repeat the entire procedure multiple times, each time shuffling the data into a new random partition. Repeated k-fold cross-validation, commonly 10 folds repeated 10 times, produces 100 individual performance scores. Research on this approach has found that repeated k-fold cross-validation results in lower variance than the non-repeated version, and that this variance reduction becomes more pronounced with smaller sample sizes.
3Elsevier / Pattern Recognition. Error estimation based on variance analysis of k-fold cross-validationThe cost is computational: if training a single model already takes minutes or hours, running 100 rounds may not be practical. For lightweight algorithms, though, repeated cross-validation is a straightforward way to get a more reliable performance estimate without the downsides of using an extremely high k.
The Problem With Leave-One-Out
Leave-one-out cross-validation (LOOCV) is the extreme case where k equals the number of observations. Each round holds out exactly one data point, trains on all the rest, and predicts that single point. It has the appeal of using almost all your data for training in every round. But it comes with serious drawbacks beyond just computational expense.
A 2025 study in Science Advances identified a structural problem called distributional bias. Every time you remove a single observation from the dataset, you slightly shift the average of the remaining training labels. Across all the folds, this creates a perfect negative correlation between the training set’s label distribution and the held-out test label. That correlation can leak information about the test label into the training process, compromising the independence that cross-validation relies on.
4Science Advances. Distributional bias compromises leave-one-out cross-validationIn practical terms, this means LOOCV can give misleadingly optimistic or pessimistic results depending on the structure of your data and model. For most applications, standard k-fold cross-validation is a safer default.
Stratified Cross-Validation for Imbalanced Data
Imagine you are building a fraud detection model where only 2% of transactions are fraudulent. If you randomly split data into folds, some folds might end up with no fraudulent cases at all, making it impossible to meaningfully evaluate the model on those folds. Stratified cross-validation solves this by ensuring each fold preserves roughly the same class proportions as the original dataset.
5PubMed Central. A Comparative Study of the Use of Stratified Cross-Validation and Distribution-Balanced Stratified Cross-Validation in Imbalanced LearningStratification is so widely recognized as a good practice for classification tasks that many machine learning libraries now use it by default. For regression problems, stratification can also be applied by binning the continuous target variable and ensuring each fold gets a representative spread of values, though this is less standardized.
Group-Level Splitting
One of the subtler mistakes in cross-validation happens when your data contains multiple observations from the same source, such as multiple medical images from the same patient, multiple readings from the same sensor, or multiple transactions from the same user. If observations from one patient appear in both the training and test folds, the model can effectively “cheat” by recognizing patient-specific patterns rather than learning generalizable features.
Group k-fold cross-validation handles this by ensuring that all observations from a given group (a patient, a sensor, a user) stay together in the same fold. In medical imaging, for instance, researchers have adopted stratified group k-fold schemes that enforce strict patient-level isolation, preventing any data from the same patient from appearing in both training and test sets.
6PubMed Central. Slice-level vs. volumetric radiomics for molecular subtyping of breast cancer: impact of patient-level data partitioning on model performanceA similar issue arises with geospatial data. When you are building a model to predict some environmental variable across a landscape, nearby locations tend to have similar values simply because of spatial autocorrelation. Standard random cross-validation will scatter nearby points across training and test sets, making the model look far more accurate than it would be when predicting at genuinely new locations. Spatial cross-validation methods address this by creating folds that are geographically separated, though even these methods struggle to guarantee that the validation set is truly representative of the locations where the model will eventually be deployed.
7International Journal of Applied Earth Observation and Geoinformation. Spatial+: A new cross-validation method to evaluate geospatial machine learning modelsData Leakage and Feature Selection
The single most common and damaging mistake in cross-validation is data leakage: letting information from the test fold influence the training process. The most frequent way this happens is performing feature selection or data preprocessing on the entire dataset before splitting it into folds. When you select the “best” features using all the data and then cross-validate, the model has already seen statistical patterns from the test data during the feature selection step. The resulting performance estimate is optimistically biased, sometimes dramatically so.
A study in radiomics quantified this effect directly: applying feature selection before cross-validation inflated the AUC score by up to 0.15 compared to correctly performing feature selection within each fold.
8PubMed Central. Measuring the bias of incorrect application of feature selection when using cross-validation in radiomicsResearch on neuropsychiatric biomarkers found the same problem: prediction accuracies were considerably higher when feature selection was done outside of cross-validation compared to when it was correctly done within each fold, and the gap between the two can be interpreted as the amount of overfitting due to selection bias.
9PubMed Central. Inflated prediction accuracy of neuropsychiatric biomarkers caused by data leakage in feature selectionThe fix is straightforward in principle but easy to get wrong in practice: every preprocessing step that uses information from the target variable, including feature selection, normalization based on target statistics, oversampling for class balance, and imputation that uses label information, must happen inside the cross-validation loop. Each fold should treat its training set as if the test set does not exist. Many machine learning pipelines now provide tools (like scikit-learn’s Pipeline class) specifically to prevent this kind of leakage, but it remains a persistent source of inflated results in published research.
Nested Versus Flat Cross-Validation
When you are not just evaluating a single model but also tuning its hyperparameters, there is a question of whether you need a second layer of cross-validation inside the first. The standard advice is to use nested cross-validation: an outer loop estimates how well the overall modeling procedure performs, while an inner loop selects the best hyperparameters for each outer fold. The concern with the simpler “flat” approach, where one cross-validation loop handles both tuning and evaluation, is that the performance estimate might be optimistically biased because the hyperparameters were chosen to do well on the very data used for evaluation.
A large-scale study tested both approaches using 12 algorithms across 115 real-world binary classification datasets. The conclusion was that flat cross-validation generally selected algorithms of similar quality to those chosen by nested cross-validation, provided the algorithms had relatively few hyperparameters to optimize.
10Elsevier. Nested cross-validation when selecting classifiers is overzealous for most practical applicationsThis does not mean nested cross-validation is never warranted. When you are searching over a large hyperparameter space, or when you need a very precise estimate of out-of-sample performance for regulatory or clinical reporting, the extra computational cost of nesting is justified. But for routine model comparison where each candidate has only a handful of tuning knobs, the simpler approach often gives you a practically equivalent answer in a fraction of the time.
Small Samples and the Limits of Cross-Validation
Cross-validation is sometimes treated as a universal fix for limited data: since you reuse all observations for both training and testing, it should give you a reliable estimate even with a small dataset. This faith is misplaced. Simulations have shown that k-fold cross-validation produces strongly biased performance estimates with small sample sizes, and the bias persists even with sample sizes up to 1,000.
11PubMed Central. Machine learning algorithm validation with a limited sample sizeThe reason is that with few observations, each fold’s training set may not capture the true data-generating pattern well. The model trained on each fold is substantially worse than one trained on the full dataset, so cross-validation underestimates the performance of the final model. Conversely, if the model is flexible enough to overfit small training sets, the error estimates across folds can swing wildly. With small data, cross-validation results should be treated as rough guidance rather than precise measurements, and external validation on a completely independent dataset becomes even more important.
What Cross-Validation Actually Estimates
There is a subtle but important question that most practitioners never think about: when you get a cross-validation score, what exactly is it an estimate of? You might assume it estimates the prediction error of your specific trained model, the one fit to your full dataset. But cross-validation actually does something slightly different. Each fold’s model is trained on a subset of the data, so each fold’s error reflects a model trained on less data than your final model. What cross-validation estimates is better described as the expected prediction error of the modeling procedure applied to a dataset of the same size as your training folds, not the error of the particular model you will deploy.
12PubMed Central. Cross-validation: what does it estimate and how well does it do it?In practice, this distinction is often small for large datasets, since training on 90% of the data versus 100% makes little difference. But for small datasets, where each fold’s training set is meaningfully smaller than the full dataset, cross-validation can systematically overestimate the error of the final deployed model. Understanding this gap matters most when you are reporting results in a high-stakes context, or when comparing models where the difference in performance is close to the noise floor of your cross-validation estimate.
Out-of-Bag Error as an Alternative
For ensemble methods like random forests that are built from many bootstrap samples, there is a built-in alternative to cross-validation called out-of-bag (OOB) error estimation. Each individual tree in a random forest is trained on a bootstrap sample, which typically includes about two-thirds of the observations. The remaining one-third, the “out-of-bag” observations, were not used to build that tree and can be used as a natural test set. By aggregating predictions from only the trees that did not see a given observation, you get an error estimate without any explicit cross-validation step.
The appeal is efficiency: you only build one forest instead of k forests. However, OOB error has its own quirks. Because each observation’s OOB prediction comes from only a subset of trees (roughly a third of them), the effective ensemble making each prediction is smaller and less accurate than the full forest. Research has noted this: the complete original sample is used for both building the forest and estimating error, but because only a fraction of trees contribute to each OOB prediction, the resulting error can overestimate what the full forest would produce.
13PLoS One. On the overestimation of random forest’s out-of-bag errorOOB error is a reasonable quick check for random forests and related bagging methods, but it is specific to that family of models and should not replace cross-validation when you need to compare across fundamentally different algorithm types.
Cross-Fitting in Causal Inference
Cross-validation originated in prediction problems, but a closely related technique called cross-fitting has become important in a very different setting: estimating causal effects from observational data. In causal inference, you often need to estimate “nuisance” quantities, like the probability of receiving a treatment given a person’s characteristics, before you can estimate the treatment effect itself. If you use a flexible machine learning model for those nuisance estimates, you risk introducing bias from overfitting.
Cross-fitting addresses this by splitting the data into folds and using one fold’s data to train the nuisance model while estimating causal effects on the other fold, then switching. The technique allows researchers to use flexible machine learning for the messy parts of the estimation while retaining valid statistical inference about the causal parameter of interest.
14PubMed Central. Machine learning for causal inference: on the use of cross-fit estimatorsThis application of the cross-validation idea, using data splitting to prevent one stage of an analysis from contaminating another, is conceptually the same insight that drives all of cross-validation: honest evaluation requires a firewall between what the model learns from and what it is tested on. The specific implementation looks different from k-fold cross-validation, but the underlying logic is identical.
A Brief History
The idea of holding out data to assess prediction quality has been around for a long time, but the formal cross-validation framework most researchers use today traces back to a 1974 paper by M. Stone in the Journal of the Royal Statistical Society. Stone proposed a generalized cross-validation criterion and demonstrated its application across several problem areas including estimation, regression, and analysis of variance.
15Oxford Academic. Cross-Validatory Choice and Assessment of Statistical PredictionsIn the decades since, cross-validation has gone from a specialized statistical tool to arguably the default method for model evaluation across all of applied machine learning. Its popularity exploded alongside the growth of machine learning itself in the 1990s and 2000s, and today it is built into virtually every major machine learning library. The core principle, though, has not changed: if you want to know how your model will do tomorrow, the best proxy is to simulate tomorrow by hiding some of today’s data and seeing what happens.