Dimensionality Reduction
1. Purpose & Motivation
1.1 Why dimensionality reduction exists
- High-dimensional data is difficult to visualize, store, and process
- Many features are redundant or noisy, adding no predictive value
- The curse of dimensionality causes models to degrade as feature count grows
1.2 The core problem it solves
Project high-dimensional data into a lower-dimensional space while preserving as much meaningful structure as possible.
2. Definition & Core Idea
2.1 Formal definition
Dimensionality reduction transforms data from a high-dimensional space R^d into a lower-dimensional space R^k (where k << d), either by selecting a subset of features or by constructing new composite features that capture the most variance or structure.
2.2 Intuitive mental model
"Flatten a 3D globe into a 2D map: you lose some information, but the important relationships between locations are preserved."
3. Two Families of Methods
3.1 Feature extraction (creates new features)
These methods construct new lower-dimensional features from the original ones.
| Method | Approach | Preserves |
|---|---|---|
| PCA | Linear projection onto max-variance axes | Global variance |
| t-SNE | Non-linear embedding for visualization | Local neighborhood |
| UMAP | Non-linear embedding, faster than t-SNE | Local + global structure |
| Autoencoders | Neural network learns compressed encoding | Non-linear patterns |
3.2 Feature selection (keeps original features)
These methods choose the most informative original features and discard the rest. (See the dedicated Feature Selection page for details.)
4. Principal Component Analysis (PCA)
4.1 How PCA works
- Center the data (subtract the mean)
- Compute the covariance matrix
- Find its eigenvectors (principal components) and eigenvalues
- Project data onto the top-k eigenvectors (directions of maximum variance)
4.2 Choosing k
- Plot explained variance ratio vs. number of components
- Select k where cumulative explained variance reaches a threshold (e.g., 95%)
- Scree plot: look for the "elbow" where gains diminish
5. Non-Linear Methods
5.1 t-SNE
- Optimizes pairwise similarity preservation in 2D/3D
- Excellent for visualization of clusters
- Not suitable for downstream modeling (non-parametric, stochastic)
5.2 UMAP
- Similar goal to t-SNE but faster and better at preserving global structure
- Can be used for both visualization and as preprocessing for models
5.3 Autoencoders
- Neural network with a bottleneck layer that forces compressed representation
- Encoder maps to low dimensions; decoder reconstructs the original
- Can capture complex non-linear manifolds
6. Strengths & Limitations
6.1 Strengths
- Reduces computation and storage costs
- Mitigates the curse of dimensionality
- Improves visualization of high-dimensional data
- Can remove noise encoded in minor dimensions
6.2 Limitations
- Information loss is inevitable
- Linear methods (PCA) miss non-linear relationships
- Transformed features may lose interpretability
- Choosing the right number of dimensions requires judgment
7. One-Sentence Summary
Dimensionality reduction projects high-dimensional data into fewer dimensions, preserving essential structure while reducing noise, computation, and the curse of dimensionality.