Optimizers
1. Purpose & Motivation
1.1 Why optimizers matter
- Training a neural network means minimizing a loss function over millions of parameters
- Vanilla gradient descent is slow and fragile
- Modern optimizers adapt step sizes and directions to make training faster and more robust
1.2 The core challenge
Loss landscapes are high-dimensional, non-convex, and full of saddle points, plateaus, and sharp valleys. A good optimizer navigates these efficiently.
2. Definition & Core Idea
2.1 Formal definition
An optimizer is an algorithm that updates model parameters by using gradient information (and often historical gradient statistics) to minimize the loss function.
2.2 Intuitive mental model
"An optimizer is a smart hiker who remembers past terrain to decide not just which direction to step, but how far and how fast."
3. Key Optimizers
3.1 SGD (Stochastic Gradient Descent)
- The simplest optimizer:
w = w - lr * grad - Noisy updates from mini-batches
- Requires careful learning rate tuning
- Can generalize well but converges slowly
3.2 SGD with Momentum
- Accumulates a velocity from past gradients:
v = beta * v + grad; w = w - lr * v - Smooths out oscillations and accelerates along consistent gradient directions
- Typical momentum beta = 0.9
- Still the preferred optimizer for many vision tasks (with good scheduling)
3.3 Adam (Adaptive Moment Estimation)
- Maintains both first moment (mean) and second moment (variance) of gradients
- Adapts the learning rate per parameter
- Fast convergence with minimal tuning
- Default: lr = 1e-3, beta1 = 0.9, beta2 = 0.999
3.4 AdamW (Adam with Decoupled Weight Decay)
- Fixes a subtle issue: in Adam, L2 regularization interacts poorly with adaptive learning rates
- Decouples weight decay from the gradient-based update
- The recommended variant for most modern training recipes
3.5 Other notable optimizers
- RMSProp -- precursor to Adam; adapts per-parameter lr using gradient variance
- Adagrad -- adapts lr based on cumulative gradients; good for sparse data
- LARS / LAMB -- designed for very large batch training
- Lion -- memory-efficient optimizer using only sign of momentum
4. Comparison Table
| Optimizer | Adaptive lr | Memory overhead | Best for |
|---|---|---|---|
| SGD | No | None | Vision, when well-tuned |
| SGD + Momentum | No | 1x params | Vision with scheduling |
| Adam | Yes | 2x params | Fast prototyping, NLP |
| AdamW | Yes | 2x params | Modern default |
| RMSProp | Yes | 1x params | RNNs (legacy) |
| LARS / LAMB | Yes | 2x params | Large-batch training |
5. When to Use Which
5.1 General guidelines
- Starting a new project: AdamW with cosine schedule -- fast convergence, minimal tuning
- Maximizing vision accuracy: SGD + momentum with step/cosine decay -- often best final accuracy
- NLP / Transformers: AdamW is the standard
- Very large batch: LARS (vision) or LAMB (NLP)
5.2 Common pitfalls
- Using Adam with L2 regularization instead of AdamW with weight decay
- Setting Adam's epsilon too small (can cause instability; default 1e-8 is usually fine)
- Not adjusting the learning rate when switching optimizers
Key insight: AdamW has become the de facto default because it combines fast adaptive convergence with proper weight decay, but SGD + momentum can still win on final accuracy for vision tasks when carefully tuned.
6. Practical Tips
- Always pair the optimizer choice with an appropriate learning rate schedule
- Monitor gradient norms -- sudden spikes signal instability
- For fine-tuning, reduce lr by 10--100x regardless of optimizer
- AdamW's memory cost (2x parameter count) matters for very large models
7. One-Sentence Summary
Optimizers like SGD, Adam, and AdamW update network weights using gradient information and adaptive strategies to efficiently minimize the training loss.