Overfitting / Underfitting

1. Purpose & Motivation

1.1 Why understanding overfitting and underfitting matters

  • The goal of ML is generalization -- performing well on unseen data
  • Most training failures trace back to one of these two failure modes
  • Diagnosing which problem is occurring determines the correct fix

1.2 The core problem it addresses

A model may fail to learn the underlying pattern (underfitting) or memorize the training data instead of generalizing (overfitting). Recognizing and correcting these issues is essential.

2. Definition & Core Idea

2.1 Formal definitions

  • Overfitting: The model performs well on training data but poorly on new data. It has learned noise and specific training examples rather than the true pattern.
  • Underfitting: The model performs poorly on both training and new data. It is too simple to capture the underlying relationship.

2.2 Intuitive mental model

"Underfitting is studying too little for an exam and failing everything. Overfitting is memorizing the practice exam word-for-word and failing the real exam because the questions are slightly different."

3. How to Diagnose

3.1 Learning curves

Condition Training error Validation error Gap
Underfitting High High Small
Good fit Low Low Small
Overfitting Very low High Large

3.2 Visual inspection

  • Plot training loss and validation loss over epochs
  • Underfitting: Both curves plateau at a high value
  • Overfitting: Training loss keeps decreasing while validation loss increases
  • The point where validation loss starts rising is the optimal stopping point

4. Causes

4.1 Causes of underfitting

  • Model is too simple (too few parameters, shallow architecture)
  • Insufficient or irrelevant features
  • Too much regularization
  • Not enough training iterations

4.2 Causes of overfitting

  • Model is too complex relative to the data
  • Training data is too small or not representative
  • Too little regularization
  • Training for too many epochs
  • Noisy or mislabeled data

5. Remedies

5.1 Fixing underfitting

  • Use a more complex model (deeper network, higher-degree polynomial)
  • Add more relevant features (feature engineering)
  • Reduce regularization
  • Train longer

5.2 Fixing overfitting

  • Regularization: L1, L2, dropout, weight decay
  • More data: Collect more samples or use data augmentation
  • Early stopping: Stop training when validation error starts increasing
  • Simpler model: Fewer parameters, shallower architecture
  • Cross-validation: More robust estimate of generalization performance
  • Ensemble methods: Combine multiple models to reduce variance

6. The Goldilocks Zone

The ideal model sits between underfitting and overfitting:

  • Complex enough to capture the true data pattern
  • Simple enough to ignore noise and generalize to new data
  • Found through systematic experimentation with model complexity, regularization, and validation

7. One-Sentence Summary

Overfitting memorizes training noise while underfitting ignores the signal; the goal is a model complex enough to capture patterns but simple enough to generalize.

8. Related Knowledge Pages