Loss Function
1. Purpose & Motivation
1.1 Why loss functions matter
- A loss function quantifies how wrong the model's predictions are
- It provides the signal that drives all learning via gradient descent
- Without a loss function, there is no objective to optimize
Training a model without a loss function is like navigating without a destination.
1.2 Loss as the optimization target
- The model adjusts its weights to minimize the loss
- Lower loss means predictions are closer to ground truth
- The choice of loss function shapes what the model learns to prioritize
2. Definition & Core Idea
2.1 Formal definition
A loss function L(y_hat, y) maps a prediction y_hat and a ground truth label y to a non-negative scalar measuring their disagreement.
2.2 Intuitive mental model
"The loss function is a score card that tells the model how badly it performed on each example."
3. Common Loss Functions
3.1 Mean Squared Error (MSE)
- Formula:
L = (1/n) * sum((y_hat - y)^2) - Use case: regression tasks
- Properties: penalizes large errors heavily (quadratic), smooth gradient
- Sensitivity: outliers have disproportionate influence
3.2 Cross-Entropy Loss
- Formula:
L = -sum(y * log(y_hat)) - Use case: multi-class classification
- Properties: measures divergence between predicted and true distributions
- Paired with softmax output activation
3.3 Binary Cross-Entropy (BCE)
- Formula:
L = -[y * log(y_hat) + (1-y) * log(1-y_hat)] - Use case: binary classification, multi-label classification
- Properties: treats each output independently
- Paired with sigmoid output activation
3.4 Other notable losses
| Loss function | Use case | Key property |
|---|---|---|
| L1 / MAE | Regression | Robust to outliers |
| Huber | Regression | Combines MSE and MAE |
| Focal Loss | Imbalanced classification | Down-weights easy examples |
| Dice Loss | Segmentation | Overlap-based metric |
| Contrastive | Embedding learning | Pulls similar pairs close |
4. Choosing a Loss Function
4.1 Match loss to task type
| Task | Recommended loss | Output activation |
|---|---|---|
| Regression | MSE or Huber | Linear |
| Binary classification | BCE | Sigmoid |
| Multi-class classification | Cross-Entropy | Softmax |
| Multi-label classification | BCE (per label) | Sigmoid |
| Semantic segmentation | Cross-Entropy + Dice | Softmax |
Key insight: The loss function, output activation, and task type must be consistent with each other.
5. Loss and Optimization
5.1 Connection to gradients
- Backpropagation computes
dL/dwfor every weight - The gradient of the loss tells each parameter which direction reduces error
- Smooth, well-behaved loss surfaces lead to more stable training
5.2 Loss landscape
- The loss function defines a high-dimensional surface over parameter space
- Training is the process of navigating this surface toward a minimum
- Local minima, saddle points, and flat regions all affect training dynamics
6. Loss in Practice
6.1 Training vs. validation loss
- Training loss decreases as the model fits the data
- Validation loss reveals generalization ability
- Divergence between the two signals overfitting
6.2 Common pitfalls
| Pitfall | Symptom | Fix |
|---|---|---|
| Wrong loss for task | Poor convergence | Match loss to task type |
| Class imbalance | Model ignores minority class | Weighted loss, focal loss |
| Numerical instability | NaN losses | Log-sum-exp trick, clipping |
7. One-Sentence Summary
A loss function measures the gap between predictions and ground truth, providing the optimization signal that drives all neural network learning.