Batch Normalization
1. Purpose & Motivation
1.1 Historical context
- Introduced in 2015 in Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift by Sergey Ioffe and Christian Szegedy at Google
- Addressed the internal covariate shift problem: as network weights update, the distribution of each layer's inputs changes, forcing subsequent layers to continuously adapt
- Enabled training of much deeper networks at higher learning rates
- Became a standard component in virtually every modern deep learning architecture (ResNet, MobileNet, EfficientNet, DenseNet)
1.2 Why it matters
- Dramatically accelerates training — allows 14× fewer training steps to achieve the same accuracy
- Acts as a regularizer — reduces the need for Dropout in many architectures
- Enables higher learning rates without divergence
- Makes networks less sensitive to weight initialization
- Now a universal building block: used in ResNet, MobileNet, EfficientNet, DenseNet, and most modern CNNs
2. Definition & Core Idea
2.1 Formal definition
Batch Normalization normalizes the inputs to a layer by computing the mean and variance over the current mini-batch, then applying a learned affine transformation (scale γ and shift β) to restore representational capacity.
For a mini-batch B = {x₁, ..., xₘ}:
μ_B = (1/m) Σ xᵢ (batch mean)
σ²_B = (1/m) Σ (xᵢ - μ_B)² (batch variance)
x̂ᵢ = (xᵢ - μ_B) / √(σ²_B + ε) (normalize)
yᵢ = γ x̂ᵢ + β (scale and shift)
- ε: Small constant (e.g., 1 × 10⁻⁵) for numerical stability
- γ, β: Learnable parameters — allow the network to undo normalization if needed
2.2 Intuitive mental model
"Before each layer processes its inputs, standardize them to have zero mean and unit variance — like normalizing test scores to a standard scale. Then let the network learn whether to rescale them back."
3. How Batch Normalization Works
3.1 Training vs. inference
| Phase | Mean and variance source | Behavior |
|---|---|---|
| Training | Computed from current mini-batch | Stochastic — varies per batch |
| Inference | Running statistics (exponential moving average) | Deterministic — fixed |
During training, BN maintains running estimates of mean and variance:
running_mean = momentum × running_mean + (1 - momentum) × μ_B
running_var = momentum × running_var + (1 - momentum) × σ²_B
At inference, these running statistics replace the batch statistics.
3.2 Placement in the network
Two conventions exist:
Original (post-activation): Conv → BN → Activation
- Used in original BN paper and early ResNets
Pre-activation (modern): BN → Activation → Conv
- Used in ResNet v2, DenseNet
- Better gradient flow — BN is applied before the non-linearity
- Generally preferred in modern architectures
3.3 BN in convolutional layers
For a convolutional layer with output shape (N, C, H, W):
- Normalize over the N × H × W dimensions per channel C
- Each channel has its own γ and β parameters
- Total learnable parameters: 2 × C (one γ and one β per channel)
4. Core Effects
4.1 Reduces internal covariate shift
- Without BN: each layer must adapt to the changing distribution of its inputs as earlier layers update
- With BN: inputs to each layer are always approximately zero-mean, unit-variance
- Layers can focus on learning the transformation, not adapting to shifting input distributions
4.2 Enables higher learning rates
- Without BN: high learning rates cause exploding/vanishing gradients
- With BN: normalization prevents activations from saturating or exploding
- Allows learning rates 10–100× higher than without BN
- Faster convergence: fewer epochs needed to reach the same accuracy
4.3 Acts as regularization
- The stochasticity of batch statistics (different batches → different normalization) adds noise
- This noise acts as a regularizer — similar to Dropout but implicit
- Reduces overfitting, often allowing Dropout to be removed or reduced
- Effect is stronger with smaller batch sizes (more noise per batch)
4.4 Reduces sensitivity to initialization
- Without BN: poor weight initialization can cause vanishing/exploding gradients from the start
- With BN: normalization corrects for poor initialization — training is more robust
- Enables use of simpler initialization schemes
5. Technical Details
5.1 Learnable parameters
| Parameter | Shape | Purpose |
|---|---|---|
| γ (scale) | (C,) | Learned scale — can restore original scale |
| β (shift) | (C,) | Learned shift — can restore original mean |
Initialized: γ = 1, β = 0 (identity transformation at start of training)
5.2 Batch size dependency
- BN statistics are computed over the mini-batch — accuracy depends on batch size
- Large batches (≥32): Stable statistics, BN works well
- Small batches (<8): Noisy statistics, BN can hurt performance
- Batch size = 1: BN is undefined (variance = 0)
For small-batch or single-sample inference scenarios, alternatives are preferred.
5.3 Computational cost
- Forward pass: 2 passes over the batch (mean, then variance)
- Backward pass: Gradients through normalization are well-defined
- Memory: Stores running mean and variance (2 × C floats per BN layer)
- Overhead: ~10–20% additional computation — generally worth the training benefits
6. Batch Normalization Variants
6.1 Normalization comparison
| Method | Normalizes over | Best for |
|---|---|---|
| Batch Norm (BN) | N × H × W per channel | CNNs with large batches |
| Layer Norm (LN) | C × H × W per sample | Transformers, NLP, small batches |
| Instance Norm (IN) | H × W per sample/channel | Style transfer, image generation |
| Group Norm (GN) | Groups of channels per sample | Small batches, detection |
Layer Normalization is the standard for Transformers (BERT, GPT, ViT) because it doesn't depend on batch size.
Group Normalization is preferred for object detection where batch size per GPU is small (1–2 images).
6.2 When to use each
- BN: Default for CNNs with batch size ≥ 16 (ResNet, MobileNet, EfficientNet)
- LN: Default for Transformers and RNNs
- GN: Object detection, video models, medical imaging (small batch sizes)
- IN: Style transfer, image-to-image translation (per-image statistics)
7. Strengths & Limitations
7.1 Strengths
- Dramatically accelerates training
- Enables higher learning rates
- Acts as implicit regularization
- Reduces sensitivity to initialization
- Universal — works across CNNs, RNNs (with care), and some transformer variants
7.2 Limitations
- Batch size dependency: Degrades with small batches (< 8)
- Train/inference discrepancy: Running statistics may not match batch statistics exactly
- Not suitable for online learning: Single-sample inference requires running statistics
- Sequence models: BN is awkward for variable-length sequences (LN preferred)
- Memory: Stores running statistics for each BN layer
8. One-Sentence Summary
Batch Normalization normalizes layer inputs to zero mean and unit variance over each mini-batch, then applies learned scale and shift — dramatically accelerating training, enabling higher learning rates, and acting as implicit regularization in virtually every modern deep learning architecture.
9. Related Knowledge Pages
10. Sources
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift — Original BN paper by Ioffe & Szegedy (ICML 2015)
- Identity Mappings in Deep Residual Networks — Pre-activation BN by He et al. (ECCV 2016)
- Group Normalization — Wu & He (ECCV 2018)
- Batch Normalization — Wikipedia
- PyTorch BatchNorm2d documentation