Supervised Learning
1. Purpose & Motivation
1.1 Why supervised learning exists
- Most real-world prediction tasks have historical examples with known outcomes
- Humans naturally learn from corrective feedback; supervised learning formalizes this
- It provides a direct path from labeled data to actionable predictions
1.2 The core problem it solves
Given a dataset of input-output pairs, learn a mapping function that generalizes to unseen inputs. This is the most widely used and best-understood learning paradigm.
2. Definition & Core Idea
2.1 Formal definition
Supervised learning is a paradigm where a model learns a function f: X -> Y from a training set of labeled examples {(x_i, y_i)}, where x_i is an input and y_i is the corresponding target label or value.
2.2 Intuitive mental model
"A student learns from an answer key: see the question, check the correct answer, adjust understanding, repeat."
3. Two Main Tasks
3.1 Classification
- Goal: Assign inputs to discrete categories
- Output: Class label (e.g., spam / not spam)
- Loss functions: Cross-entropy, hinge loss
- Examples: Email filtering, medical diagnosis, image recognition
3.2 Regression
- Goal: Predict a continuous numerical value
- Output: Real number (e.g., house price)
- Loss functions: Mean Squared Error (MSE), Mean Absolute Error (MAE)
- Examples: Stock price forecasting, temperature prediction, age estimation
4. How Training Works
4.1 The learning loop
- Forward pass: Model makes a prediction given input
- Loss computation: Compare prediction against ground truth label
- Parameter update: Adjust model weights to reduce the loss
- Repeat over many examples and epochs
4.2 Key ingredients
| Ingredient | Role |
|---|---|
| Training data | Labeled examples the model learns from |
| Loss function | Quantifies prediction error |
| Optimizer | Updates parameters to minimize loss |
| Validation data | Monitors generalization during training |
| Test data | Final unbiased evaluation of performance |
5. Strengths & Limitations
5.1 Strengths
- Well-defined objective makes optimization straightforward
- Strong theoretical guarantees (e.g., PAC learning)
- Extensive tooling and established best practices
5.2 Limitations
- Requires large amounts of labeled data, which is expensive to obtain
- Labels can contain noise or human bias
- Model can memorize labels rather than learn patterns (overfitting)
6. Common Algorithms
- Linear Regression, Logistic Regression
- k-NN, SVM, Decision Trees
- Random Forests, Gradient Boosting
- Neural Networks (MLPs, CNNs, Transformers)
7. One-Sentence Summary
Supervised learning trains a model on labeled input-output pairs so it can predict correct outputs for new, unseen inputs.