Regularization
1. Purpose & Motivation
1.1 Why regularization matters
- Deep networks have millions of parameters and can memorize training data
- Overfitting occurs when the model performs well on training data but poorly on unseen data
- Regularization techniques constrain the model to learn generalizable patterns
1.2 The bias-variance perspective
- Too little regularization -- high variance (overfitting)
- Too much regularization -- high bias (underfitting)
- The goal is to find the sweet spot where the model captures real patterns without fitting noise
2. Definition & Core Idea
2.1 Formal definition
Regularization is any technique that constrains or penalizes model complexity to improve generalization performance on unseen data.
2.2 Intuitive mental model
"Regularization is like telling a student they cannot just memorize answers -- they must understand the underlying concepts well enough to handle new questions."
3. Explicit Regularization
3.1 L2 regularization (weight decay)
- Adds a penalty proportional to the squared magnitude of weights: L_total = L_data + lambda * sum(w^2)
- Pushes weights toward zero but rarely makes them exactly zero
- Encourages smooth, distributed representations
3.2 L1 regularization
- Adds a penalty proportional to the absolute value of weights: L_total = L_data + lambda * sum(|w|)
- Encourages sparsity -- many weights become exactly zero
- Useful for feature selection
3.3 Comparison
| Property | L1 | L2 |
|---|---|---|
| Penalty | Absolute value | Squared value |
| Sparsity | Yes | No |
| Feature selection | Natural | No |
| Common use | Sparse models | General training |
4. Dropout
4.1 How it works
- During training, randomly set a fraction of activations to zero (typically 20--50%)
- During inference, use all activations but scale them
- Forces the network to learn redundant representations
4.2 Why it works
- Prevents co-adaptation of neurons
- Approximates an ensemble of many sub-networks
- Acts as a strong regularizer with minimal computational cost
Key insight: Dropout works because it forces each neuron to be useful independently, not just in combination with specific other neurons.
5. Data Augmentation
5.1 Core idea
- Artificially expand the training set by applying random transformations to inputs
- The model sees more variation and learns invariances
5.2 Common augmentations for images
- Random crop, horizontal flip, rotation
- Color jitter, Gaussian blur
- Advanced: CutOut, MixUp, CutMix, RandAugment
5.3 Why it is powerful
- Directly addresses the root cause of overfitting -- insufficient data diversity
- Often the single most effective regularization strategy
6. Early Stopping
6.1 How it works
- Monitor validation loss during training
- Stop training when validation loss stops improving (or starts increasing)
- Restore the best-performing checkpoint
6.2 Practical considerations
- Use a patience parameter (e.g., stop after 10 epochs without improvement)
- Simple and effective, but requires a held-out validation set
7. Batch Normalization as Regularizer
7.1 Primary purpose
- Normalizes activations within each mini-batch to stabilize training
7.2 Regularization side effect
- The noise from mini-batch statistics acts as a mild regularizer
- Often allows reducing or removing dropout when batch norm is present
- This effect diminishes with larger batch sizes
8. Practical Guidelines
8.1 When to apply what
| Technique | When to use |
|---|---|
| L2 / weight decay | Almost always (default) |
| Dropout | FC layers; less common in conv layers |
| Data augmentation | Always for vision tasks |
| Early stopping | Always as a safety net |
| Batch normalization | Standard in most architectures |
8.2 Combining techniques
- Start with data augmentation + weight decay
- Add dropout if still overfitting
- Use early stopping as a final safeguard
- Monitor the gap between training and validation loss
9. One-Sentence Summary
Regularization prevents overfitting by constraining model complexity through weight penalties, dropout, data augmentation, and early stopping.