Regularization

1. Purpose & Motivation

1.1 Why regularization matters

  • Deep networks have millions of parameters and can memorize training data
  • Overfitting occurs when the model performs well on training data but poorly on unseen data
  • Regularization techniques constrain the model to learn generalizable patterns

1.2 The bias-variance perspective

  • Too little regularization -- high variance (overfitting)
  • Too much regularization -- high bias (underfitting)
  • The goal is to find the sweet spot where the model captures real patterns without fitting noise

2. Definition & Core Idea

2.1 Formal definition

Regularization is any technique that constrains or penalizes model complexity to improve generalization performance on unseen data.

2.2 Intuitive mental model

"Regularization is like telling a student they cannot just memorize answers -- they must understand the underlying concepts well enough to handle new questions."

3. Explicit Regularization

3.1 L2 regularization (weight decay)

  • Adds a penalty proportional to the squared magnitude of weights: L_total = L_data + lambda * sum(w^2)
  • Pushes weights toward zero but rarely makes them exactly zero
  • Encourages smooth, distributed representations

3.2 L1 regularization

  • Adds a penalty proportional to the absolute value of weights: L_total = L_data + lambda * sum(|w|)
  • Encourages sparsity -- many weights become exactly zero
  • Useful for feature selection

3.3 Comparison

Property L1 L2
Penalty Absolute value Squared value
Sparsity Yes No
Feature selection Natural No
Common use Sparse models General training

4. Dropout

4.1 How it works

  • During training, randomly set a fraction of activations to zero (typically 20--50%)
  • During inference, use all activations but scale them
  • Forces the network to learn redundant representations

4.2 Why it works

  • Prevents co-adaptation of neurons
  • Approximates an ensemble of many sub-networks
  • Acts as a strong regularizer with minimal computational cost

Key insight: Dropout works because it forces each neuron to be useful independently, not just in combination with specific other neurons.

5. Data Augmentation

5.1 Core idea

  • Artificially expand the training set by applying random transformations to inputs
  • The model sees more variation and learns invariances

5.2 Common augmentations for images

  • Random crop, horizontal flip, rotation
  • Color jitter, Gaussian blur
  • Advanced: CutOut, MixUp, CutMix, RandAugment

5.3 Why it is powerful

  • Directly addresses the root cause of overfitting -- insufficient data diversity
  • Often the single most effective regularization strategy

6. Early Stopping

6.1 How it works

  • Monitor validation loss during training
  • Stop training when validation loss stops improving (or starts increasing)
  • Restore the best-performing checkpoint

6.2 Practical considerations

  • Use a patience parameter (e.g., stop after 10 epochs without improvement)
  • Simple and effective, but requires a held-out validation set

7. Batch Normalization as Regularizer

7.1 Primary purpose

  • Normalizes activations within each mini-batch to stabilize training

7.2 Regularization side effect

  • The noise from mini-batch statistics acts as a mild regularizer
  • Often allows reducing or removing dropout when batch norm is present
  • This effect diminishes with larger batch sizes

8. Practical Guidelines

8.1 When to apply what

Technique When to use
L2 / weight decay Almost always (default)
Dropout FC layers; less common in conv layers
Data augmentation Always for vision tasks
Early stopping Always as a safety net
Batch normalization Standard in most architectures

8.2 Combining techniques

  • Start with data augmentation + weight decay
  • Add dropout if still overfitting
  • Use early stopping as a final safeguard
  • Monitor the gap between training and validation loss

9. One-Sentence Summary

Regularization prevents overfitting by constraining model complexity through weight penalties, dropout, data augmentation, and early stopping.

10. Related Knowledge Pages