Cross-Validation
1. Purpose & Motivation
1.1 Why cross-validation exists
- A single train/test split can give misleading performance estimates
- Small datasets make it hard to afford a large held-out test set
- Model selection requires reliable estimates of generalization error
1.2 The core problem it solves
Provide a robust, low-variance estimate of model performance by systematically rotating which portion of the data is used for training and which for evaluation.
2. Definition & Core Idea
2.1 Formal definition
Cross-validation is an evaluation technique that partitions the dataset into k complementary subsets (folds), trains the model on k-1 folds, evaluates on the remaining fold, and repeats this process k times so every data point serves as both training and validation.
2.2 Intuitive mental model
"Instead of one exam with one answer key, take k different exams where each covers different questions, then average the scores for a fairer assessment."
3. k-Fold Cross-Validation
3.1 Algorithm
- Shuffle the dataset and split it into k equal-sized folds
- For each fold i = 1, ..., k:
- Use fold i as the validation set
- Train on the remaining k-1 folds
- Record the validation score
- Report the mean and standard deviation of the k scores
3.2 Choosing k
| k value | Behavior | Common use |
|---|---|---|
| k = 5 | Good balance of bias and variance | Default choice |
| k = 10 | Slightly lower bias, higher computational cost | Standard practice |
| k = n | Leave-One-Out (LOO): minimal bias, maximum variance | Very small datasets |
3.3 Typical choice
- k = 5 or k = 10 is standard for most applications
- Larger k reduces bias but increases computation and variance of the estimate
4. Variants
4.1 Stratified k-Fold
- Ensures each fold has the same class distribution as the full dataset
- Essential for imbalanced classification problems
4.2 Leave-One-Out (LOO)
- k equals the number of samples; each sample is a fold
- Nearly unbiased but computationally expensive and high variance
4.3 Repeated k-Fold
- Run k-fold multiple times with different random splits
- Averages out randomness in the fold assignment
- More stable estimate at the cost of extra computation
4.4 Time Series Split
- Respects temporal ordering: training set always precedes the validation set
- Prevents data leakage from future to past
- Essential for sequential and financial data
5. Cross-Validation for Model Selection
5.1 Hyperparameter tuning
- Evaluate each hyperparameter configuration using cross-validation
- Select the configuration with the best average validation score
- Grid search and random search both rely on CV for fair comparison
5.2 Nested cross-validation
- Outer loop: Estimates generalization performance
- Inner loop: Tunes hyperparameters
- Prevents optimistic bias from using the same data for tuning and evaluation
6. Common Pitfalls
- Data leakage: Preprocessing (scaling, feature selection) must be done inside each fold, not before splitting
- Ignoring grouping: If samples are correlated (e.g., multiple images from the same patient), use Group k-Fold to prevent leakage
- Reporting only mean: Always report standard deviation to convey estimate reliability
7. One-Sentence Summary
Cross-validation rotates training and validation splits across k folds to produce a reliable, low-variance estimate of how a model will perform on unseen data.