Batch Normalization

1. Purpose & Motivation

1.1 Historical context

  • Introduced in 2015 in Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift by Sergey Ioffe and Christian Szegedy at Google
  • Addressed the internal covariate shift problem: as network weights update, the distribution of each layer's inputs changes, forcing subsequent layers to continuously adapt
  • Enabled training of much deeper networks at higher learning rates
  • Became a standard component in virtually every modern deep learning architecture (ResNet, MobileNet, EfficientNet, DenseNet)

1.2 Why it matters

  • Dramatically accelerates training — allows 14× fewer training steps to achieve the same accuracy
  • Acts as a regularizer — reduces the need for Dropout in many architectures
  • Enables higher learning rates without divergence
  • Makes networks less sensitive to weight initialization
  • Now a universal building block: used in ResNet, MobileNet, EfficientNet, DenseNet, and most modern CNNs

2. Definition & Core Idea

2.1 Formal definition

Batch Normalization normalizes the inputs to a layer by computing the mean and variance over the current mini-batch, then applying a learned affine transformation (scale γ and shift β) to restore representational capacity.

For a mini-batch B = {x₁, ..., xₘ}:

μ_B = (1/m) Σ xᵢ                    (batch mean)
σ²_B = (1/m) Σ (xᵢ - μ_B)²         (batch variance)
x̂ᵢ = (xᵢ - μ_B) / √(σ²_B + ε)     (normalize)
yᵢ = γ x̂ᵢ + β                       (scale and shift)
  • ε: Small constant (e.g., 1 × 10⁻⁵) for numerical stability
  • γ, β: Learnable parameters — allow the network to undo normalization if needed

2.2 Intuitive mental model

"Before each layer processes its inputs, standardize them to have zero mean and unit variance — like normalizing test scores to a standard scale. Then let the network learn whether to rescale them back."

3. How Batch Normalization Works

3.1 Training vs. inference

Phase Mean and variance source Behavior
Training Computed from current mini-batch Stochastic — varies per batch
Inference Running statistics (exponential moving average) Deterministic — fixed

During training, BN maintains running estimates of mean and variance:

running_mean = momentum × running_mean + (1 - momentum) × μ_B
running_var  = momentum × running_var  + (1 - momentum) × σ²_B

At inference, these running statistics replace the batch statistics.

3.2 Placement in the network

Two conventions exist:

Original (post-activation): Conv → BN → Activation

  • Used in original BN paper and early ResNets

Pre-activation (modern): BN → Activation → Conv

  • Used in ResNet v2, DenseNet
  • Better gradient flow — BN is applied before the non-linearity
  • Generally preferred in modern architectures

3.3 BN in convolutional layers

For a convolutional layer with output shape (N, C, H, W):

  • Normalize over the N × H × W dimensions per channel C
  • Each channel has its own γ and β parameters
  • Total learnable parameters: 2 × C (one γ and one β per channel)

4. Core Effects

4.1 Reduces internal covariate shift

  • Without BN: each layer must adapt to the changing distribution of its inputs as earlier layers update
  • With BN: inputs to each layer are always approximately zero-mean, unit-variance
  • Layers can focus on learning the transformation, not adapting to shifting input distributions

4.2 Enables higher learning rates

  • Without BN: high learning rates cause exploding/vanishing gradients
  • With BN: normalization prevents activations from saturating or exploding
  • Allows learning rates 10–100× higher than without BN
  • Faster convergence: fewer epochs needed to reach the same accuracy

4.3 Acts as regularization

  • The stochasticity of batch statistics (different batches → different normalization) adds noise
  • This noise acts as a regularizer — similar to Dropout but implicit
  • Reduces overfitting, often allowing Dropout to be removed or reduced
  • Effect is stronger with smaller batch sizes (more noise per batch)

4.4 Reduces sensitivity to initialization

  • Without BN: poor weight initialization can cause vanishing/exploding gradients from the start
  • With BN: normalization corrects for poor initialization — training is more robust
  • Enables use of simpler initialization schemes

5. Technical Details

5.1 Learnable parameters

Parameter Shape Purpose
γ (scale) (C,) Learned scale — can restore original scale
β (shift) (C,) Learned shift — can restore original mean

Initialized: γ = 1, β = 0 (identity transformation at start of training)

5.2 Batch size dependency

  • BN statistics are computed over the mini-batch — accuracy depends on batch size
  • Large batches (≥32): Stable statistics, BN works well
  • Small batches (<8): Noisy statistics, BN can hurt performance
  • Batch size = 1: BN is undefined (variance = 0)

For small-batch or single-sample inference scenarios, alternatives are preferred.

5.3 Computational cost

  • Forward pass: 2 passes over the batch (mean, then variance)
  • Backward pass: Gradients through normalization are well-defined
  • Memory: Stores running mean and variance (2 × C floats per BN layer)
  • Overhead: ~10–20% additional computation — generally worth the training benefits

6. Batch Normalization Variants

6.1 Normalization comparison

Method Normalizes over Best for
Batch Norm (BN) N × H × W per channel CNNs with large batches
Layer Norm (LN) C × H × W per sample Transformers, NLP, small batches
Instance Norm (IN) H × W per sample/channel Style transfer, image generation
Group Norm (GN) Groups of channels per sample Small batches, detection

Layer Normalization is the standard for Transformers (BERT, GPT, ViT) because it doesn't depend on batch size.

Group Normalization is preferred for object detection where batch size per GPU is small (1–2 images).

6.2 When to use each

  • BN: Default for CNNs with batch size ≥ 16 (ResNet, MobileNet, EfficientNet)
  • LN: Default for Transformers and RNNs
  • GN: Object detection, video models, medical imaging (small batch sizes)
  • IN: Style transfer, image-to-image translation (per-image statistics)

7. Strengths & Limitations

7.1 Strengths

  • Dramatically accelerates training
  • Enables higher learning rates
  • Acts as implicit regularization
  • Reduces sensitivity to initialization
  • Universal — works across CNNs, RNNs (with care), and some transformer variants

7.2 Limitations

  • Batch size dependency: Degrades with small batches (< 8)
  • Train/inference discrepancy: Running statistics may not match batch statistics exactly
  • Not suitable for online learning: Single-sample inference requires running statistics
  • Sequence models: BN is awkward for variable-length sequences (LN preferred)
  • Memory: Stores running statistics for each BN layer

8. One-Sentence Summary

Batch Normalization normalizes layer inputs to zero mean and unit variance over each mini-batch, then applies learned scale and shift — dramatically accelerating training, enabling higher learning rates, and acting as implicit regularization in virtually every modern deep learning architecture.

9. Related Knowledge Pages

10. Sources