Linear Regression

1. Purpose & Motivation

1.1 Why linear regression exists

  • Many real-world relationships are approximately linear
  • Predicting continuous outcomes is a fundamental task (prices, temperatures, scores)
  • A simple, interpretable baseline is essential before trying complex models

1.2 The core problem it solves

Find the best-fitting straight line (or hyperplane) through data points to predict a continuous target variable from one or more input features.

2. Definition & Core Idea

2.1 Formal definition

Linear regression models the relationship between input features x and a continuous target y as a linear function: y = w^T x + b, where w are learned weights and b is a bias term.

2.2 Intuitive mental model

"Draw the line through a scatterplot that minimizes the total distance of all points to the line."

3. How It Works

3.1 Ordinary Least Squares (OLS)

The most common approach minimizes the Mean Squared Error (MSE):

  • MSE = (1/n) * sum of (y_i - y_hat_i)^2
  • Has a closed-form solution: w = (X^T X)^(-1) X^T y
  • Fast to compute for moderate-sized datasets

3.2 Gradient descent alternative

  • For large datasets, iteratively update weights in the direction that reduces MSE
  • Scales better than the closed-form solution
  • Foundation for training more complex models

4. Key Assumptions

Assumption Meaning
Linearity Relationship between features and target is linear
Independence Observations are independent of each other
Homoscedasticity Constant variance of residuals
Normality Residuals are approximately normally distributed

Violations of these assumptions can degrade predictions and invalidate confidence intervals.

5. Regularized Variants

5.1 Ridge Regression (L2)

  • Adds lambda * ||w||^2 penalty to the loss
  • Shrinks weights toward zero, reducing overfitting
  • Keeps all features but dampens their influence

5.2 Lasso Regression (L1)

  • Adds lambda * ||w||_1 penalty to the loss
  • Can shrink some weights exactly to zero (feature selection)
  • Produces sparse models

5.3 Elastic Net

  • Combines L1 and L2 penalties
  • Balances sparsity with stability

6. Strengths & Limitations

6.1 Strengths

  • Highly interpretable (each weight quantifies feature importance)
  • Fast to train, even on large datasets
  • Strong baseline for many regression problems

6.2 Limitations

  • Cannot capture non-linear relationships without feature engineering
  • Sensitive to outliers
  • Assumes linear separability of the feature-target relationship

7. One-Sentence Summary

Linear regression fits a straight line to data by minimizing squared errors, providing a fast, interpretable model for predicting continuous values.

8. Related Knowledge Pages