Supervised Learning

1. Purpose & Motivation

1.1 Why supervised learning exists

  • Most real-world prediction tasks have historical examples with known outcomes
  • Humans naturally learn from corrective feedback; supervised learning formalizes this
  • It provides a direct path from labeled data to actionable predictions

1.2 The core problem it solves

Given a dataset of input-output pairs, learn a mapping function that generalizes to unseen inputs. This is the most widely used and best-understood learning paradigm.

2. Definition & Core Idea

2.1 Formal definition

Supervised learning is a paradigm where a model learns a function f: X -> Y from a training set of labeled examples {(x_i, y_i)}, where x_i is an input and y_i is the corresponding target label or value.

2.2 Intuitive mental model

"A student learns from an answer key: see the question, check the correct answer, adjust understanding, repeat."

3. Two Main Tasks

3.1 Classification

  • Goal: Assign inputs to discrete categories
  • Output: Class label (e.g., spam / not spam)
  • Loss functions: Cross-entropy, hinge loss
  • Examples: Email filtering, medical diagnosis, image recognition

3.2 Regression

  • Goal: Predict a continuous numerical value
  • Output: Real number (e.g., house price)
  • Loss functions: Mean Squared Error (MSE), Mean Absolute Error (MAE)
  • Examples: Stock price forecasting, temperature prediction, age estimation

4. How Training Works

4.1 The learning loop

  1. Forward pass: Model makes a prediction given input
  2. Loss computation: Compare prediction against ground truth label
  3. Parameter update: Adjust model weights to reduce the loss
  4. Repeat over many examples and epochs

4.2 Key ingredients

Ingredient Role
Training data Labeled examples the model learns from
Loss function Quantifies prediction error
Optimizer Updates parameters to minimize loss
Validation data Monitors generalization during training
Test data Final unbiased evaluation of performance

5. Strengths & Limitations

5.1 Strengths

  • Well-defined objective makes optimization straightforward
  • Strong theoretical guarantees (e.g., PAC learning)
  • Extensive tooling and established best practices

5.2 Limitations

  • Requires large amounts of labeled data, which is expensive to obtain
  • Labels can contain noise or human bias
  • Model can memorize labels rather than learn patterns (overfitting)

6. Common Algorithms

  • Linear Regression, Logistic Regression
  • k-NN, SVM, Decision Trees
  • Random Forests, Gradient Boosting
  • Neural Networks (MLPs, CNNs, Transformers)

7. One-Sentence Summary

Supervised learning trains a model on labeled input-output pairs so it can predict correct outputs for new, unseen inputs.

8. Related Knowledge Pages