Initialization

1. Purpose & Motivation

1.1 Why initialization matters

  • Neural networks learn by iteratively updating weights via gradient descent
  • The starting values of weights determine the initial loss landscape the optimizer navigates
  • Poor initialization can make training impossibly slow or completely unstable

1.2 The core problem

If all weights start at the same value, every neuron computes the same gradient and learns the same features -- this is called the symmetry problem. Initialization must break symmetry while keeping signal magnitudes stable across layers.

2. Definition & Core Idea

2.1 Formal definition

Weight initialization is the strategy for setting the initial values of learnable parameters before training begins, typically by sampling from a carefully chosen probability distribution.

2.2 Intuitive mental model

"Give every neuron a unique, reasonably-sized starting guess so that gradients flow smoothly through the entire network from the very first step."

3. What Goes Wrong with Bad Initialization

3.1 Zero initialization

  • All neurons compute identical outputs
  • Gradients are identical -- no symmetry breaking
  • The network effectively has one neuron per layer

3.2 Too-large values

  • Activations explode exponentially with depth
  • Gradients overflow, causing NaN losses
  • Known as the exploding gradient problem

3.3 Too-small values

  • Activations shrink toward zero with depth
  • Gradients vanish, and early layers stop learning
  • Known as the vanishing gradient problem

4. Standard Initialization Methods

4.1 Xavier / Glorot initialization

Designed for sigmoid and tanh activations.

Variant Distribution Variance
Uniform U(-a, a) a = sqrt(6 / (fan_in + fan_out))
Normal N(0, sigma^2) sigma^2 = 2 / (fan_in + fan_out)
  • Balances variance of activations and gradients across layers
  • Assumes linear activation regime

4.2 He / Kaiming initialization

Designed for ReLU and its variants.

Variant Distribution Variance
Uniform U(-a, a) a = sqrt(6 / fan_in)
Normal N(0, sigma^2) sigma^2 = 2 / fan_in
  • Accounts for the fact that ReLU zeros out roughly half of activations
  • The default choice for most modern CNN and MLP architectures

4.3 Other methods

  • Orthogonal initialization -- preserves gradient norms exactly; useful for RNNs
  • LSUV (Layer-Sequential Unit Variance) -- data-driven normalization of each layer's output variance

5. Practical Guidelines

5.1 Quick decision table

Activation Recommended init
ReLU / Leaky ReLU He (Kaiming)
Sigmoid / Tanh Xavier (Glorot)
GELU / SiLU He (Kaiming)
Linear output Xavier (Glorot)

5.2 Framework defaults

  • PyTorch uses Kaiming uniform by default for nn.Linear and nn.Conv2d
  • In practice, the defaults work well for most architectures
  • Override only when using unusual activations or seeing training instability

5.3 Biases

  • Biases are typically initialized to zero
  • Exception: set the bias of the final layer to the log-prior of class frequencies for imbalanced classification

Key insight: Good initialization does not find a good solution -- it ensures the optimizer can find one by keeping gradients healthy.

6. One-Sentence Summary

Initialization sets weight starting values using principled distributions (Xavier for sigmoid/tanh, He for ReLU) to ensure stable gradient flow from the first training step.

7. Related Knowledge Pages