SVM

1. Purpose & Motivation

1.1 Why SVMs exist

  • Many classification problems require finding the best boundary between classes
  • Maximizing the margin between classes improves generalization
  • Non-linear problems need a principled way to operate in higher-dimensional spaces

1.2 The core problem it solves

Find the hyperplane that maximizes the margin between two classes, optionally mapping data into higher dimensions to handle non-linear boundaries.

2. Definition & Core Idea

2.1 Formal definition

A Support Vector Machine finds the hyperplane w^T x + b = 0 that maximizes the margin (distance between the hyperplane and the nearest data points from each class). The nearest points are called support vectors.

2.2 Intuitive mental model

"Place a wide road between two groups of houses; the widest road that separates them is the best boundary, and the houses closest to the road define its edges."

3. How It Works

3.1 Linear SVM

  • Finds the maximum-margin hyperplane in the original feature space
  • Only the support vectors (points closest to the boundary) matter
  • Optimization problem: minimize ||w||^2 subject to correct classification constraints

3.2 Soft margin (C parameter)

C value Behavior
Large C Narrow margin, fewer misclassifications allowed
Small C Wide margin, more misclassifications tolerated

The parameter C controls the tradeoff between margin width and training error.

3.3 The kernel trick

  • Maps data into a higher-dimensional space where a linear separator exists
  • Key insight: The mapping is done implicitly via a kernel function, avoiding the cost of computing in high dimensions

3.4 Common kernels

Kernel Use case
Linear Linearly separable data
RBF (Gaussian) General-purpose non-linear
Polynomial Interaction features up to degree d
Sigmoid Neural network-like behavior

4. SVM for Regression (SVR)

  • Fits a tube of width epsilon around the data
  • Points inside the tube incur no loss
  • Points outside contribute linearly to the loss

5. Strengths & Limitations

5.1 Strengths

  • Effective in high-dimensional spaces
  • Memory-efficient (only stores support vectors)
  • Versatile through different kernels
  • Strong theoretical foundation (margin maximization)

5.2 Limitations

  • Slow training on large datasets: O(n^2) to O(n^3)
  • Does not naturally output probabilities (requires calibration)
  • Kernel and hyperparameter selection can be tricky

6. One-Sentence Summary

SVMs classify data by finding the maximum-margin hyperplane between classes, using the kernel trick to handle non-linear boundaries in high-dimensional space.

7. Related Knowledge Pages