Loss Function

1. Purpose & Motivation

1.1 Why loss functions matter

  • A loss function quantifies how wrong the model's predictions are
  • It provides the signal that drives all learning via gradient descent
  • Without a loss function, there is no objective to optimize

Training a model without a loss function is like navigating without a destination.

1.2 Loss as the optimization target

  • The model adjusts its weights to minimize the loss
  • Lower loss means predictions are closer to ground truth
  • The choice of loss function shapes what the model learns to prioritize

2. Definition & Core Idea

2.1 Formal definition

A loss function L(y_hat, y) maps a prediction y_hat and a ground truth label y to a non-negative scalar measuring their disagreement.

2.2 Intuitive mental model

"The loss function is a score card that tells the model how badly it performed on each example."

3. Common Loss Functions

3.1 Mean Squared Error (MSE)

  • Formula: L = (1/n) * sum((y_hat - y)^2)
  • Use case: regression tasks
  • Properties: penalizes large errors heavily (quadratic), smooth gradient
  • Sensitivity: outliers have disproportionate influence

3.2 Cross-Entropy Loss

  • Formula: L = -sum(y * log(y_hat))
  • Use case: multi-class classification
  • Properties: measures divergence between predicted and true distributions
  • Paired with softmax output activation

3.3 Binary Cross-Entropy (BCE)

  • Formula: L = -[y * log(y_hat) + (1-y) * log(1-y_hat)]
  • Use case: binary classification, multi-label classification
  • Properties: treats each output independently
  • Paired with sigmoid output activation

3.4 Other notable losses

Loss function Use case Key property
L1 / MAE Regression Robust to outliers
Huber Regression Combines MSE and MAE
Focal Loss Imbalanced classification Down-weights easy examples
Dice Loss Segmentation Overlap-based metric
Contrastive Embedding learning Pulls similar pairs close

4. Choosing a Loss Function

4.1 Match loss to task type

Task Recommended loss Output activation
Regression MSE or Huber Linear
Binary classification BCE Sigmoid
Multi-class classification Cross-Entropy Softmax
Multi-label classification BCE (per label) Sigmoid
Semantic segmentation Cross-Entropy + Dice Softmax

Key insight: The loss function, output activation, and task type must be consistent with each other.

5. Loss and Optimization

5.1 Connection to gradients

  • Backpropagation computes dL/dw for every weight
  • The gradient of the loss tells each parameter which direction reduces error
  • Smooth, well-behaved loss surfaces lead to more stable training

5.2 Loss landscape

  • The loss function defines a high-dimensional surface over parameter space
  • Training is the process of navigating this surface toward a minimum
  • Local minima, saddle points, and flat regions all affect training dynamics

6. Loss in Practice

6.1 Training vs. validation loss

  • Training loss decreases as the model fits the data
  • Validation loss reveals generalization ability
  • Divergence between the two signals overfitting

6.2 Common pitfalls

Pitfall Symptom Fix
Wrong loss for task Poor convergence Match loss to task type
Class imbalance Model ignores minority class Weighted loss, focal loss
Numerical instability NaN losses Log-sum-exp trick, clipping

7. One-Sentence Summary

A loss function measures the gap between predictions and ground truth, providing the optimization signal that drives all neural network learning.

8. Related Knowledge Pages