Initialization
1. Purpose & Motivation
1.1 Why initialization matters
- Neural networks learn by iteratively updating weights via gradient descent
- The starting values of weights determine the initial loss landscape the optimizer navigates
- Poor initialization can make training impossibly slow or completely unstable
1.2 The core problem
If all weights start at the same value, every neuron computes the same gradient and learns the same features -- this is called the symmetry problem. Initialization must break symmetry while keeping signal magnitudes stable across layers.
2. Definition & Core Idea
2.1 Formal definition
Weight initialization is the strategy for setting the initial values of learnable parameters before training begins, typically by sampling from a carefully chosen probability distribution.
2.2 Intuitive mental model
"Give every neuron a unique, reasonably-sized starting guess so that gradients flow smoothly through the entire network from the very first step."
3. What Goes Wrong with Bad Initialization
3.1 Zero initialization
- All neurons compute identical outputs
- Gradients are identical -- no symmetry breaking
- The network effectively has one neuron per layer
3.2 Too-large values
- Activations explode exponentially with depth
- Gradients overflow, causing NaN losses
- Known as the exploding gradient problem
3.3 Too-small values
- Activations shrink toward zero with depth
- Gradients vanish, and early layers stop learning
- Known as the vanishing gradient problem
4. Standard Initialization Methods
4.1 Xavier / Glorot initialization
Designed for sigmoid and tanh activations.
| Variant | Distribution | Variance |
|---|---|---|
| Uniform | U(-a, a) | a = sqrt(6 / (fan_in + fan_out)) |
| Normal | N(0, sigma^2) | sigma^2 = 2 / (fan_in + fan_out) |
- Balances variance of activations and gradients across layers
- Assumes linear activation regime
4.2 He / Kaiming initialization
Designed for ReLU and its variants.
| Variant | Distribution | Variance |
|---|---|---|
| Uniform | U(-a, a) | a = sqrt(6 / fan_in) |
| Normal | N(0, sigma^2) | sigma^2 = 2 / fan_in |
- Accounts for the fact that ReLU zeros out roughly half of activations
- The default choice for most modern CNN and MLP architectures
4.3 Other methods
- Orthogonal initialization -- preserves gradient norms exactly; useful for RNNs
- LSUV (Layer-Sequential Unit Variance) -- data-driven normalization of each layer's output variance
5. Practical Guidelines
5.1 Quick decision table
| Activation | Recommended init |
|---|---|
| ReLU / Leaky ReLU | He (Kaiming) |
| Sigmoid / Tanh | Xavier (Glorot) |
| GELU / SiLU | He (Kaiming) |
| Linear output | Xavier (Glorot) |
5.2 Framework defaults
- PyTorch uses Kaiming uniform by default for
nn.Linearandnn.Conv2d - In practice, the defaults work well for most architectures
- Override only when using unusual activations or seeing training instability
5.3 Biases
- Biases are typically initialized to zero
- Exception: set the bias of the final layer to the log-prior of class frequencies for imbalanced classification
Key insight: Good initialization does not find a good solution -- it ensures the optimizer can find one by keeping gradients healthy.
6. One-Sentence Summary
Initialization sets weight starting values using principled distributions (Xavier for sigmoid/tanh, He for ReLU) to ensure stable gradient flow from the first training step.