Batch Size
1. Purpose & Motivation
1.1 Why batch size matters
- Training data is too large to compute gradients over the entire dataset at once
- The number of samples processed per weight update affects speed, memory, and generalization
- Batch size is one of the most impactful hyperparameters, yet often under-tuned
1.2 The fundamental tradeoff
Larger batches give more accurate gradient estimates but cost more memory and may generalize worse. Smaller batches are noisy but can act as implicit regularizers.
2. Definition & Core Idea
2.1 Formal definition
Batch size is the number of training samples used to compute a single gradient estimate and parameter update during one optimization step.
2.2 Intuitive mental model
"Batch size controls how many examples the model looks at before adjusting its weights -- like studying one flashcard at a time versus reviewing the whole deck before making notes."
3. Gradient Estimation Strategies
3.1 Full-batch gradient descent
- Uses the entire dataset per update
- Exact gradient, but prohibitively expensive for large datasets
- Converges smoothly but slowly
3.2 Stochastic gradient descent (SGD, batch size = 1)
- Uses a single sample per update
- Very noisy gradient estimates
- High variance but frequent updates
3.3 Mini-batch gradient descent
- Uses a subset (typically 32--512 samples) per update
- Balances noise and computational efficiency
- The standard approach in modern deep learning
| Strategy | Batch size | Gradient noise | GPU utilization |
|---|---|---|---|
| Full batch | N (all data) | None | Often underutilized |
| Mini-batch | 32--512 | Moderate | Good |
| Stochastic | 1 | High | Poor |
4. Effects of Batch Size
4.1 On convergence
- Small batches -- noisier gradients help escape sharp minima; may converge to flatter, more generalizable solutions
- Large batches -- smoother gradients converge faster in wall-clock time but may get trapped in sharp minima
4.2 On generalization
- Several studies show that very large batch training can degrade test accuracy (the "generalization gap")
- Techniques like learning rate warmup and LARS/LAMB optimizers mitigate this
4.3 On memory
- Batch size directly scales GPU memory consumption
- Activations stored for backpropagation grow linearly with batch size
- Gradient accumulation simulates larger batches without extra memory
Key insight: Doubling the batch size roughly halves the number of parameter updates per epoch, so the learning rate typically needs to scale accordingly.
5. Practical Guidelines
5.1 Common choices
| Domain | Typical batch size |
|---|---|
| Image classification | 32--256 |
| Object detection | 2--16 |
| NLP (Transformers) | 8--64 (tokens: thousands) |
| Fine-tuning | 4--32 |
5.2 The linear scaling rule
- When increasing batch size by factor k, multiply learning rate by k
- Use learning rate warmup for the first few epochs to stabilize training
5.3 Gradient accumulation
- Accumulate gradients over multiple forward passes before updating
- Effective batch size = micro-batch size x accumulation steps
- Allows large effective batches on limited hardware
5.4 Decision process
- Start with the largest batch that fits in GPU memory
- If generalization suffers, reduce batch size or add regularization
- Use gradient accumulation if the desired batch size exceeds memory
6. One-Sentence Summary
Batch size controls the number of samples per gradient update, trading off between gradient accuracy, memory cost, training speed, and generalization quality.