Semi-Supervised Learning

1. Purpose & Motivation

1.1 Why semi-supervised learning exists

  • Labeling data is expensive; collecting unlabeled data is cheap
  • In many domains (medical imaging, NLP), only a small fraction of data has labels
  • Purely supervised methods underperform when labeled data is scarce

1.2 The core problem it solves

Leverage a small set of labeled examples together with a large pool of unlabeled examples to achieve better generalization than either supervised or unsupervised learning alone.

2. Definition & Core Idea

2.1 Formal definition

Semi-supervised learning uses a training set consisting of a small labeled subset {(x_i, y_i)} and a large unlabeled subset {x_j} to learn a function f: X -> Y that benefits from the structure revealed by the unlabeled data.

2.2 Intuitive mental model

"A teacher labels a few example essays, then lets the student study thousands of unlabeled essays to understand writing patterns before grading new ones."

3. Key Assumptions

3.1 Why unlabeled data helps

Semi-supervised methods rely on structural assumptions about the data:

Assumption Meaning
Smoothness Nearby points in input space likely share the same label
Cluster Points in the same cluster likely share the same label
Low-density Decision boundaries should pass through low-density regions
Manifold High-dimensional data lies on a lower-dimensional manifold

If these assumptions hold, unlabeled data reveals the data geometry and improves decision boundaries.

4. Common Techniques

4.1 Self-training

  1. Train a model on labeled data
  2. Use it to predict labels for unlabeled data (pseudo-labels)
  3. Add high-confidence pseudo-labeled examples to the training set
  4. Retrain and repeat

4.2 Consistency regularization

  • Enforce that the model produces the same output for an input and its augmented versions
  • Examples: Temporal Ensembling, Mean Teacher, UDA

4.3 Graph-based methods

  • Build a graph where nodes are data points and edges encode similarity
  • Propagate labels from labeled to unlabeled nodes along the graph

4.4 Generative approaches

  • Model the joint distribution P(X, Y) using both labeled and unlabeled data
  • Variational Autoencoders (VAEs) with label information

5. Strengths & Limitations

5.1 Strengths

  • Dramatically reduces labeling requirements
  • Often matches fully supervised performance with 5-10% of the labels
  • Works across domains (vision, NLP, speech)

5.2 Limitations

  • Performance depends on assumption validity
  • Pseudo-labels can propagate errors (confirmation bias)
  • More complex training pipelines than pure supervised learning

6. One-Sentence Summary

Semi-supervised learning combines a small amount of labeled data with a large amount of unlabeled data, using data structure to improve predictions beyond what labels alone provide.

7. Related Knowledge Pages