Semi-Supervised Learning
1. Purpose & Motivation
1.1 Why semi-supervised learning exists
- Labeling data is expensive; collecting unlabeled data is cheap
- In many domains (medical imaging, NLP), only a small fraction of data has labels
- Purely supervised methods underperform when labeled data is scarce
1.2 The core problem it solves
Leverage a small set of labeled examples together with a large pool of unlabeled examples to achieve better generalization than either supervised or unsupervised learning alone.
2. Definition & Core Idea
2.1 Formal definition
Semi-supervised learning uses a training set consisting of a small labeled subset {(x_i, y_i)} and a large unlabeled subset {x_j} to learn a function f: X -> Y that benefits from the structure revealed by the unlabeled data.
2.2 Intuitive mental model
"A teacher labels a few example essays, then lets the student study thousands of unlabeled essays to understand writing patterns before grading new ones."
3. Key Assumptions
3.1 Why unlabeled data helps
Semi-supervised methods rely on structural assumptions about the data:
| Assumption | Meaning |
|---|---|
| Smoothness | Nearby points in input space likely share the same label |
| Cluster | Points in the same cluster likely share the same label |
| Low-density | Decision boundaries should pass through low-density regions |
| Manifold | High-dimensional data lies on a lower-dimensional manifold |
If these assumptions hold, unlabeled data reveals the data geometry and improves decision boundaries.
4. Common Techniques
4.1 Self-training
- Train a model on labeled data
- Use it to predict labels for unlabeled data (pseudo-labels)
- Add high-confidence pseudo-labeled examples to the training set
- Retrain and repeat
4.2 Consistency regularization
- Enforce that the model produces the same output for an input and its augmented versions
- Examples: Temporal Ensembling, Mean Teacher, UDA
4.3 Graph-based methods
- Build a graph where nodes are data points and edges encode similarity
- Propagate labels from labeled to unlabeled nodes along the graph
4.4 Generative approaches
- Model the joint distribution P(X, Y) using both labeled and unlabeled data
- Variational Autoencoders (VAEs) with label information
5. Strengths & Limitations
5.1 Strengths
- Dramatically reduces labeling requirements
- Often matches fully supervised performance with 5-10% of the labels
- Works across domains (vision, NLP, speech)
5.2 Limitations
- Performance depends on assumption validity
- Pseudo-labels can propagate errors (confirmation bias)
- More complex training pipelines than pure supervised learning
6. One-Sentence Summary
Semi-supervised learning combines a small amount of labeled data with a large amount of unlabeled data, using data structure to improve predictions beyond what labels alone provide.