Bias-Variance Tradeoff
1. Purpose & Motivation
1.1 Why the bias-variance tradeoff matters
- Every model's prediction error can be decomposed into interpretable components
- Understanding this decomposition guides model selection and tuning
- It explains why complex models sometimes perform worse than simple ones
1.2 The core problem it addresses
Prediction error comes from two competing sources -- bias and variance -- and reducing one typically increases the other. Finding the right balance is central to building models that generalize well.
2. Definition & Core Idea
2.1 Formal definition
For a given input x, the expected prediction error of a model can be decomposed as:
Error = Bias^2 + Variance + Irreducible Noise
- Bias: Error from oversimplifying the true relationship (wrong assumptions)
- Variance: Error from sensitivity to fluctuations in the training data
- Irreducible noise: Inherent randomness in the data that no model can eliminate
2.2 Intuitive mental model
"Bias is consistently missing the target in the same direction; variance is scattering shots all over the target. The best model finds the sweet spot between the two."
3. Bias in Detail
3.1 What causes high bias
- Model is too simple for the underlying data pattern
- Important features or interactions are missing
- Strong regularization constrains the model too much
3.2 Symptoms
- High error on both training and validation sets
- Model fails to capture known relationships
- This condition is called underfitting
4. Variance in Detail
4.1 What causes high variance
- Model is too complex relative to the amount of training data
- Too many parameters or too little regularization
- Model memorizes training data noise
4.2 Symptoms
- Low training error but high validation/test error
- Predictions change significantly with different training samples
- This condition is called overfitting
5. The Tradeoff Visualized
| Model complexity | Bias | Variance | Total error |
|---|---|---|---|
| Very low | High | Low | High (underfitting) |
| Optimal | Medium | Medium | Minimum |
| Very high | Low | High | High (overfitting) |
The total error curve is U-shaped: decreasing bias drives it down, then increasing variance drives it back up. The minimum point is the optimal complexity.
6. How to Navigate the Tradeoff
6.1 Reduce bias
- Use a more expressive model (more parameters, deeper trees)
- Add relevant features or feature interactions
- Reduce regularization strength
6.2 Reduce variance
- Collect more training data
- Use regularization (L1, L2, dropout)
- Ensemble methods (bagging, Random Forest)
- Simplify the model
- Cross-validation for hyperparameter tuning
7. Connection to Model Selection
- Simple models (Linear Regression, shallow trees): typically high bias, low variance
- Complex models (deep networks, unpruned trees): typically low bias, high variance
- Ensembles (Random Forest, Gradient Boosting): designed to reduce variance or both
8. One-Sentence Summary
The bias-variance tradeoff states that prediction error arises from model simplicity (bias) and data sensitivity (variance), and the best model minimizes their combined effect.