SVM
1. Purpose & Motivation
1.1 Why SVMs exist
- Many classification problems require finding the best boundary between classes
- Maximizing the margin between classes improves generalization
- Non-linear problems need a principled way to operate in higher-dimensional spaces
1.2 The core problem it solves
Find the hyperplane that maximizes the margin between two classes, optionally mapping data into higher dimensions to handle non-linear boundaries.
2. Definition & Core Idea
2.1 Formal definition
A Support Vector Machine finds the hyperplane w^T x + b = 0 that maximizes the margin (distance between the hyperplane and the nearest data points from each class). The nearest points are called support vectors.
2.2 Intuitive mental model
"Place a wide road between two groups of houses; the widest road that separates them is the best boundary, and the houses closest to the road define its edges."
3. How It Works
3.1 Linear SVM
- Finds the maximum-margin hyperplane in the original feature space
- Only the support vectors (points closest to the boundary) matter
- Optimization problem: minimize ||w||^2 subject to correct classification constraints
3.2 Soft margin (C parameter)
| C value | Behavior |
|---|---|
| Large C | Narrow margin, fewer misclassifications allowed |
| Small C | Wide margin, more misclassifications tolerated |
The parameter C controls the tradeoff between margin width and training error.
3.3 The kernel trick
- Maps data into a higher-dimensional space where a linear separator exists
- Key insight: The mapping is done implicitly via a kernel function, avoiding the cost of computing in high dimensions
3.4 Common kernels
| Kernel | Use case |
|---|---|
| Linear | Linearly separable data |
| RBF (Gaussian) | General-purpose non-linear |
| Polynomial | Interaction features up to degree d |
| Sigmoid | Neural network-like behavior |
4. SVM for Regression (SVR)
- Fits a tube of width epsilon around the data
- Points inside the tube incur no loss
- Points outside contribute linearly to the loss
5. Strengths & Limitations
5.1 Strengths
- Effective in high-dimensional spaces
- Memory-efficient (only stores support vectors)
- Versatile through different kernels
- Strong theoretical foundation (margin maximization)
5.2 Limitations
- Slow training on large datasets: O(n^2) to O(n^3)
- Does not naturally output probabilities (requires calibration)
- Kernel and hyperparameter selection can be tricky
6. One-Sentence Summary
SVMs classify data by finding the maximum-margin hyperplane between classes, using the kernel trick to handle non-linear boundaries in high-dimensional space.