Learning Rate

1. Purpose & Motivation

1.1 Why the learning rate matters

  • The learning rate controls how much weights change in response to each gradient
  • Too large and training diverges; too small and training stalls
  • Often cited as the single most important hyperparameter in deep learning

1.2 The optimization landscape

Gradient descent navigates a high-dimensional loss surface. The learning rate determines the step size -- take steps that are too big and you overshoot minima; too small and you waste compute or get stuck in poor local optima.

2. Definition & Core Idea

2.1 Formal definition

The learning rate (often denoted alpha or lr) is a scalar multiplier applied to the gradient during parameter updates:

w_new = w_old - lr * gradient(w_old)

2.2 Intuitive mental model

"The learning rate is the size of each step you take while walking downhill in fog -- too large and you stumble past the valley, too small and you never reach it."

3. Effects of Learning Rate Choice

3.1 Too high

  • Loss oscillates or diverges (NaN)
  • Weights overshoot good solutions
  • Training is unstable

3.2 Too low

  • Training converges extremely slowly
  • May get stuck in suboptimal local minima or saddle points
  • Wastes computational resources

3.3 Just right

  • Loss decreases steadily
  • Reaches a good solution in reasonable time
  • Some noise is acceptable -- it can help escape sharp minima
Learning rate Behavior
~0.1--1.0 Often too high for Adam
~1e-3 Common starting point for Adam
~0.01--0.1 Common starting point for SGD
~1e-5 Typical for fine-tuning pretrained models

4. Learning Rate Schedules

4.1 Why schedules help

A single fixed learning rate is rarely optimal throughout training. Schedules adapt the learning rate over time to balance fast initial progress with fine-grained convergence later.

4.2 Common schedules

4.2.1 Step decay

  • Multiply lr by a factor (e.g., 0.1) at predefined epochs
  • Simple and effective; widely used in vision (e.g., drop at epoch 30, 60, 90)

4.2.2 Cosine annealing

  • Smoothly decreases lr following a cosine curve from initial value to near zero
  • Avoids sudden drops; often used with warm restarts

4.2.3 Warmup

  • Start with a very small lr and linearly increase to the target over the first few epochs
  • Stabilizes early training, especially with large batch sizes or Adam-type optimizers
  • Often combined with cosine decay (warmup + cosine)

4.2.4 One-cycle policy

  • Ramp up lr to a peak, then decay; simultaneously adjust momentum inversely
  • Can achieve faster convergence with good final performance

Key insight: Cosine annealing with warmup has become the de facto default schedule for modern training recipes.

5. Learning Rate Finder

5.1 How it works

  1. Start with a very small lr (e.g., 1e-7)
  2. Gradually increase lr over one epoch
  3. Plot loss vs. learning rate
  4. Choose the lr where loss decreases most steeply (before it diverges)

5.2 Practical use

  • Popularized by Leslie Smith and the fastai library
  • Gives a good initial estimate; fine-tune from there
  • Run before each new experiment or architecture change

6. Relationship to Batch Size

6.1 Linear scaling rule

  • When batch size increases by factor k, multiply lr by k
  • Justification: larger batches reduce gradient noise, allowing larger steps

6.2 When it breaks down

  • Very large batch sizes (thousands+) require warmup and specialized optimizers (LARS, LAMB)
  • The linear rule is an approximation, not a guarantee

7. Practical Guidelines

  • Start with lr = 1e-3 for Adam, lr = 0.01--0.1 for SGD with momentum
  • Use warmup + cosine annealing as a strong default schedule
  • Run a learning rate finder when in doubt
  • Scale lr with batch size using the linear scaling rule
  • For fine-tuning pretrained models, use 10--100x smaller lr than training from scratch

8. One-Sentence Summary

The learning rate sets the step size for weight updates and is best managed through schedules like cosine annealing with warmup for stable, efficient training.

9. Related Knowledge Pages