Gradient Boosting

1. Purpose & Motivation

1.1 Why gradient boosting exists

  • Bagging (Random Forest) reduces variance but does not directly address bias
  • A method that sequentially corrects errors can reduce both bias and variance
  • Many competition-winning models and production systems rely on boosted trees

1.2 The core problem it solves

Build a strong predictive model by iteratively adding weak learners (typically shallow trees), where each new learner focuses on correcting the residual errors of the ensemble so far.

2. Definition & Core Idea

2.1 Formal definition

Gradient Boosting constructs an additive model F(x) = sum of f_t(x) by sequentially fitting each new weak learner f_t to the negative gradient of the loss function (the residuals), performing gradient descent in function space.

2.2 Intuitive mental model

"Each new team member focuses exclusively on fixing the mistakes the current team is making, gradually eliminating all errors."

3. How It Works

3.1 Training algorithm

  1. Initialize model with a constant prediction (e.g., mean of targets)
  2. For each round t = 1, ..., T:
    • Compute the residuals (negative gradient of the loss)
    • Fit a shallow decision tree to the residuals
    • Add the tree to the ensemble, scaled by a learning rate
  3. Final prediction is the sum of all trees

3.2 Key hyperparameters

Parameter Effect Typical range
Number of trees (T) More rounds = lower bias, risk of overfitting 100-10000
Learning rate Shrinks each tree's contribution 0.01-0.3
Max depth Controls individual tree complexity 3-8
Subsample ratio Fraction of data used per round (stochastic GB) 0.5-1.0
Min child weight Regularization on leaf node size 1-10

3.3 Learning rate and number of trees

  • Smaller learning rate requires more trees but often yields better performance
  • The two parameters trade off: halving the learning rate roughly doubles the needed trees

4. Popular Implementations

Library Key innovation
XGBoost Regularized objective, parallel tree building
LightGBM Leaf-wise growth, histogram-based splits
CatBoost Native categorical feature handling

These implementations dominate tabular data tasks and ML competitions.

5. Gradient Boosting vs Random Forest

Aspect Random Forest Gradient Boosting
Strategy Parallel (bagging) Sequential (boosting)
Bias/Variance Reduces variance Reduces both bias and variance
Tuning Relatively easy More hyperparameters
Overfitting Resistant Can overfit without regularization
Speed Easily parallelized Inherently sequential

6. Strengths & Limitations

6.1 Strengths

  • State-of-the-art on tabular data
  • Handles mixed feature types and missing values
  • Built-in feature importance
  • Flexible loss functions (regression, classification, ranking)

6.2 Limitations

  • Sequential training is slower than parallel methods
  • Requires careful hyperparameter tuning
  • Less interpretable than a single decision tree
  • Sensitive to noisy labels

7. One-Sentence Summary

Gradient boosting sequentially trains shallow decision trees on residual errors, combining them into a powerful ensemble that excels on structured tabular data.

8. Related Knowledge Pages