Bias-Variance Tradeoff

1. Purpose & Motivation

1.1 Why the bias-variance tradeoff matters

  • Every model's prediction error can be decomposed into interpretable components
  • Understanding this decomposition guides model selection and tuning
  • It explains why complex models sometimes perform worse than simple ones

1.2 The core problem it addresses

Prediction error comes from two competing sources -- bias and variance -- and reducing one typically increases the other. Finding the right balance is central to building models that generalize well.

2. Definition & Core Idea

2.1 Formal definition

For a given input x, the expected prediction error of a model can be decomposed as:

Error = Bias^2 + Variance + Irreducible Noise

  • Bias: Error from oversimplifying the true relationship (wrong assumptions)
  • Variance: Error from sensitivity to fluctuations in the training data
  • Irreducible noise: Inherent randomness in the data that no model can eliminate

2.2 Intuitive mental model

"Bias is consistently missing the target in the same direction; variance is scattering shots all over the target. The best model finds the sweet spot between the two."

3. Bias in Detail

3.1 What causes high bias

  • Model is too simple for the underlying data pattern
  • Important features or interactions are missing
  • Strong regularization constrains the model too much

3.2 Symptoms

  • High error on both training and validation sets
  • Model fails to capture known relationships
  • This condition is called underfitting

4. Variance in Detail

4.1 What causes high variance

  • Model is too complex relative to the amount of training data
  • Too many parameters or too little regularization
  • Model memorizes training data noise

4.2 Symptoms

  • Low training error but high validation/test error
  • Predictions change significantly with different training samples
  • This condition is called overfitting

5. The Tradeoff Visualized

Model complexity Bias Variance Total error
Very low High Low High (underfitting)
Optimal Medium Medium Minimum
Very high Low High High (overfitting)

The total error curve is U-shaped: decreasing bias drives it down, then increasing variance drives it back up. The minimum point is the optimal complexity.

6. How to Navigate the Tradeoff

6.1 Reduce bias

  • Use a more expressive model (more parameters, deeper trees)
  • Add relevant features or feature interactions
  • Reduce regularization strength

6.2 Reduce variance

  • Collect more training data
  • Use regularization (L1, L2, dropout)
  • Ensemble methods (bagging, Random Forest)
  • Simplify the model
  • Cross-validation for hyperparameter tuning

7. Connection to Model Selection

  • Simple models (Linear Regression, shallow trees): typically high bias, low variance
  • Complex models (deep networks, unpruned trees): typically low bias, high variance
  • Ensembles (Random Forest, Gradient Boosting): designed to reduce variance or both

8. One-Sentence Summary

The bias-variance tradeoff states that prediction error arises from model simplicity (bias) and data sensitivity (variance), and the best model minimizes their combined effect.

9. Related Knowledge Pages