Learning Rate
1. Purpose & Motivation
1.1 Why the learning rate matters
- The learning rate controls how much weights change in response to each gradient
- Too large and training diverges; too small and training stalls
- Often cited as the single most important hyperparameter in deep learning
1.2 The optimization landscape
Gradient descent navigates a high-dimensional loss surface. The learning rate determines the step size -- take steps that are too big and you overshoot minima; too small and you waste compute or get stuck in poor local optima.
2. Definition & Core Idea
2.1 Formal definition
The learning rate (often denoted alpha or lr) is a scalar multiplier applied to the gradient during parameter updates:
w_new = w_old - lr * gradient(w_old)
2.2 Intuitive mental model
"The learning rate is the size of each step you take while walking downhill in fog -- too large and you stumble past the valley, too small and you never reach it."
3. Effects of Learning Rate Choice
3.1 Too high
- Loss oscillates or diverges (NaN)
- Weights overshoot good solutions
- Training is unstable
3.2 Too low
- Training converges extremely slowly
- May get stuck in suboptimal local minima or saddle points
- Wastes computational resources
3.3 Just right
- Loss decreases steadily
- Reaches a good solution in reasonable time
- Some noise is acceptable -- it can help escape sharp minima
| Learning rate | Behavior |
|---|---|
| ~0.1--1.0 | Often too high for Adam |
| ~1e-3 | Common starting point for Adam |
| ~0.01--0.1 | Common starting point for SGD |
| ~1e-5 | Typical for fine-tuning pretrained models |
4. Learning Rate Schedules
4.1 Why schedules help
A single fixed learning rate is rarely optimal throughout training. Schedules adapt the learning rate over time to balance fast initial progress with fine-grained convergence later.
4.2 Common schedules
4.2.1 Step decay
- Multiply lr by a factor (e.g., 0.1) at predefined epochs
- Simple and effective; widely used in vision (e.g., drop at epoch 30, 60, 90)
4.2.2 Cosine annealing
- Smoothly decreases lr following a cosine curve from initial value to near zero
- Avoids sudden drops; often used with warm restarts
4.2.3 Warmup
- Start with a very small lr and linearly increase to the target over the first few epochs
- Stabilizes early training, especially with large batch sizes or Adam-type optimizers
- Often combined with cosine decay (warmup + cosine)
4.2.4 One-cycle policy
- Ramp up lr to a peak, then decay; simultaneously adjust momentum inversely
- Can achieve faster convergence with good final performance
Key insight: Cosine annealing with warmup has become the de facto default schedule for modern training recipes.
5. Learning Rate Finder
5.1 How it works
- Start with a very small lr (e.g., 1e-7)
- Gradually increase lr over one epoch
- Plot loss vs. learning rate
- Choose the lr where loss decreases most steeply (before it diverges)
5.2 Practical use
- Popularized by Leslie Smith and the fastai library
- Gives a good initial estimate; fine-tune from there
- Run before each new experiment or architecture change
6. Relationship to Batch Size
6.1 Linear scaling rule
- When batch size increases by factor k, multiply lr by k
- Justification: larger batches reduce gradient noise, allowing larger steps
6.2 When it breaks down
- Very large batch sizes (thousands+) require warmup and specialized optimizers (LARS, LAMB)
- The linear rule is an approximation, not a guarantee
7. Practical Guidelines
- Start with lr = 1e-3 for Adam, lr = 0.01--0.1 for SGD with momentum
- Use warmup + cosine annealing as a strong default schedule
- Run a learning rate finder when in doubt
- Scale lr with batch size using the linear scaling rule
- For fine-tuning pretrained models, use 10--100x smaller lr than training from scratch
8. One-Sentence Summary
The learning rate sets the step size for weight updates and is best managed through schedules like cosine annealing with warmup for stable, efficient training.