Backpropagation
1. Purpose & Motivation
1.1 Why backpropagation exists
- Neural networks have millions of parameters that need to be updated
- We need an efficient way to compute how each parameter affects the loss
- Backpropagation computes all gradients in a single backward pass
Backpropagation is the algorithm that makes training deep networks computationally feasible.
1.2 The credit assignment problem
- When a network makes an error, which weights are responsible?
- Backpropagation answers this by tracing the error backward through the network
- Each weight receives a gradient proportional to its contribution to the error
2. Definition & Core Idea
2.1 Formal definition
Backpropagation is an algorithm that computes the gradient of the loss function with respect to every parameter in the network by repeated application of the chain rule from calculus.
2.2 Intuitive mental model
"Backpropagation pushes blame for prediction errors backward through the network, so every weight knows how to adjust itself."
3. The Chain Rule
3.1 Mathematical foundation
For composed functions L(f(g(x))), the chain rule gives:
dL/dx = dL/df * df/dg * dg/dx
3.2 Application to networks
Each layer is a function composition. The chain rule lets us decompose the gradient of the loss into a product of local gradients at each layer.
| Layer | Local gradient | Accumulated gradient |
|---|---|---|
| Output | dL/dz_output | dL/dz_output |
| Hidden 2 | dz_output/dz_hidden2 | Product of above |
| Hidden 1 | dz_hidden2/dz_hidden1 | Product of all above |
4. Forward and Backward Pass
4.1 Forward pass
- Input data enters the network
- Each layer computes:
z = Wx + b, thena = f(z) - Intermediate values are cached for later use
- The final output is compared to the target via the loss function
4.2 Backward pass
- Compute the gradient of the loss with respect to the output
- Propagate gradients backward through each layer
- At each layer, compute gradients for weights, biases, and the input
- Cache-stored activations are used to compute weight gradients
Forward: Input → Layer 1 → Layer 2 → ... → Loss
Backward: Loss → Layer N → Layer N-1 → ... → Layer 1
Key insight: The forward pass computes predictions; the backward pass computes how to improve them.
5. Computational Graph Perspective
5.1 What is a computational graph
- Every operation in the network is a node in a directed graph
- Edges represent data flow
- Backpropagation is a reverse traversal of this graph
5.2 Why it matters
- Modern frameworks (PyTorch, TensorFlow) build computational graphs automatically
- Automatic differentiation handles the chain rule for arbitrary architectures
- This enables rapid experimentation with novel network designs
6. Gradient Flow Problems
6.1 Vanishing gradients
- Gradients shrink exponentially as they pass through many layers
- Caused by: sigmoid/tanh activations, poor initialization
- Effect: early layers stop learning
- Solutions: ReLU, residual connections, batch normalization
6.2 Exploding gradients
- Gradients grow exponentially, causing unstable updates
- Caused by: large weight values, deep unrolled RNNs
- Effect: NaN losses, divergent training
- Solutions: gradient clipping, careful initialization, LSTM/GRU
| Problem | Direction | Cause | Key solution |
|---|---|---|---|
| Vanishing | Too small | Saturating activations | ReLU, skip connections |
| Exploding | Too large | Multiplicative growth | Gradient clipping |
7. Backpropagation in Practice
7.1 Mini-batch training
- Gradients are averaged over a batch of examples
- Reduces variance while remaining computationally efficient
- Batch size affects gradient noise and training dynamics
7.2 What happens after backprop
- Computed gradients are passed to an optimizer (SGD, Adam, etc.)
- The optimizer applies the update rule to adjust weights
- Backpropagation computes the direction; the optimizer decides the step size
8. One-Sentence Summary
Backpropagation efficiently computes gradients for all network parameters by applying the chain rule backward through the computational graph, enabling gradient-based learning.