Cross-Validation

1. Purpose & Motivation

1.1 Why cross-validation exists

  • A single train/test split can give misleading performance estimates
  • Small datasets make it hard to afford a large held-out test set
  • Model selection requires reliable estimates of generalization error

1.2 The core problem it solves

Provide a robust, low-variance estimate of model performance by systematically rotating which portion of the data is used for training and which for evaluation.

2. Definition & Core Idea

2.1 Formal definition

Cross-validation is an evaluation technique that partitions the dataset into k complementary subsets (folds), trains the model on k-1 folds, evaluates on the remaining fold, and repeats this process k times so every data point serves as both training and validation.

2.2 Intuitive mental model

"Instead of one exam with one answer key, take k different exams where each covers different questions, then average the scores for a fairer assessment."

3. k-Fold Cross-Validation

3.1 Algorithm

  1. Shuffle the dataset and split it into k equal-sized folds
  2. For each fold i = 1, ..., k:
    • Use fold i as the validation set
    • Train on the remaining k-1 folds
    • Record the validation score
  3. Report the mean and standard deviation of the k scores

3.2 Choosing k

k value Behavior Common use
k = 5 Good balance of bias and variance Default choice
k = 10 Slightly lower bias, higher computational cost Standard practice
k = n Leave-One-Out (LOO): minimal bias, maximum variance Very small datasets

3.3 Typical choice

  • k = 5 or k = 10 is standard for most applications
  • Larger k reduces bias but increases computation and variance of the estimate

4. Variants

4.1 Stratified k-Fold

  • Ensures each fold has the same class distribution as the full dataset
  • Essential for imbalanced classification problems

4.2 Leave-One-Out (LOO)

  • k equals the number of samples; each sample is a fold
  • Nearly unbiased but computationally expensive and high variance

4.3 Repeated k-Fold

  • Run k-fold multiple times with different random splits
  • Averages out randomness in the fold assignment
  • More stable estimate at the cost of extra computation

4.4 Time Series Split

  • Respects temporal ordering: training set always precedes the validation set
  • Prevents data leakage from future to past
  • Essential for sequential and financial data

5. Cross-Validation for Model Selection

5.1 Hyperparameter tuning

  • Evaluate each hyperparameter configuration using cross-validation
  • Select the configuration with the best average validation score
  • Grid search and random search both rely on CV for fair comparison

5.2 Nested cross-validation

  • Outer loop: Estimates generalization performance
  • Inner loop: Tunes hyperparameters
  • Prevents optimistic bias from using the same data for tuning and evaluation

6. Common Pitfalls

  • Data leakage: Preprocessing (scaling, feature selection) must be done inside each fold, not before splitting
  • Ignoring grouping: If samples are correlated (e.g., multiple images from the same patient), use Group k-Fold to prevent leakage
  • Reporting only mean: Always report standard deviation to convey estimate reliability

7. One-Sentence Summary

Cross-validation rotates training and validation splits across k folds to produce a reliable, low-variance estimate of how a model will perform on unseen data.

8. Related Knowledge Pages