Logistic Regression
1. Purpose & Motivation
1.1 Why logistic regression exists
- Classification (assigning categories) is one of the most common ML tasks
- Linear regression outputs unbounded values, making it unsuitable for probabilities
- A simple, probabilistic classifier is needed as a reliable baseline
1.2 The core problem it solves
Predict the probability that an input belongs to a particular class, producing outputs bounded between 0 and 1.
2. Definition & Core Idea
2.1 Formal definition
Logistic regression applies a sigmoid function to a linear combination of inputs: P(y=1|x) = sigma(w^T x + b), where sigma(z) = 1 / (1 + e^(-z)). The sigmoid squashes the linear output into the range (0, 1), interpreted as a probability.
2.2 Intuitive mental model
"Draw a line that separates two classes, then use distance from the line to express confidence in the classification."
3. How It Works
3.1 The sigmoid function
- Maps any real number to (0, 1)
- Output > 0.5 is typically classified as positive; below 0.5 as negative
- The steepness at the boundary reflects classification confidence
3.2 Training with cross-entropy loss
- Binary cross-entropy: -[y * log(p) + (1-y) * log(1-p)]
- No closed-form solution; optimized via gradient descent
- Convex loss function guarantees a global minimum
3.3 Decision boundary
- The boundary is a linear hyperplane in feature space
- Points on one side are classified as positive, the other as negative
- The model is inherently a linear classifier
4. Multiclass Extension
| Variant | Approach |
|---|---|
| One-vs-Rest (OvR) | Train one binary classifier per class |
| One-vs-One (OvO) | Train one classifier per class pair |
| Softmax Regression | Generalize sigmoid to multiple classes directly |
Softmax regression (multinomial logistic regression) outputs a probability distribution over all classes.
5. Regularization
- L2 (Ridge): Prevents large weights, improves generalization
- L1 (Lasso): Encourages sparsity, acts as feature selection
- Regularization strength is controlled by hyperparameter C (inverse of lambda)
6. Strengths & Limitations
6.1 Strengths
- Outputs calibrated probabilities, not just class labels
- Fast training and prediction
- Highly interpretable (weight per feature indicates direction and strength)
6.2 Limitations
- Limited to linear decision boundaries
- Struggles with complex, non-linear patterns
- Assumes features are roughly independent (multicollinearity hurts)
7. One-Sentence Summary
Logistic regression applies a sigmoid to a linear model to produce class probabilities, serving as the foundational algorithm for binary classification.