Logistic Regression

1. Purpose & Motivation

1.1 Why logistic regression exists

  • Classification (assigning categories) is one of the most common ML tasks
  • Linear regression outputs unbounded values, making it unsuitable for probabilities
  • A simple, probabilistic classifier is needed as a reliable baseline

1.2 The core problem it solves

Predict the probability that an input belongs to a particular class, producing outputs bounded between 0 and 1.

2. Definition & Core Idea

2.1 Formal definition

Logistic regression applies a sigmoid function to a linear combination of inputs: P(y=1|x) = sigma(w^T x + b), where sigma(z) = 1 / (1 + e^(-z)). The sigmoid squashes the linear output into the range (0, 1), interpreted as a probability.

2.2 Intuitive mental model

"Draw a line that separates two classes, then use distance from the line to express confidence in the classification."

3. How It Works

3.1 The sigmoid function

  • Maps any real number to (0, 1)
  • Output > 0.5 is typically classified as positive; below 0.5 as negative
  • The steepness at the boundary reflects classification confidence

3.2 Training with cross-entropy loss

  • Binary cross-entropy: -[y * log(p) + (1-y) * log(1-p)]
  • No closed-form solution; optimized via gradient descent
  • Convex loss function guarantees a global minimum

3.3 Decision boundary

  • The boundary is a linear hyperplane in feature space
  • Points on one side are classified as positive, the other as negative
  • The model is inherently a linear classifier

4. Multiclass Extension

Variant Approach
One-vs-Rest (OvR) Train one binary classifier per class
One-vs-One (OvO) Train one classifier per class pair
Softmax Regression Generalize sigmoid to multiple classes directly

Softmax regression (multinomial logistic regression) outputs a probability distribution over all classes.

5. Regularization

  • L2 (Ridge): Prevents large weights, improves generalization
  • L1 (Lasso): Encourages sparsity, acts as feature selection
  • Regularization strength is controlled by hyperparameter C (inverse of lambda)

6. Strengths & Limitations

6.1 Strengths

  • Outputs calibrated probabilities, not just class labels
  • Fast training and prediction
  • Highly interpretable (weight per feature indicates direction and strength)

6.2 Limitations

  • Limited to linear decision boundaries
  • Struggles with complex, non-linear patterns
  • Assumes features are roughly independent (multicollinearity hurts)

7. One-Sentence Summary

Logistic regression applies a sigmoid to a linear model to produce class probabilities, serving as the foundational algorithm for binary classification.

8. Related Knowledge Pages