Gradient Boosting
1. Purpose & Motivation
1.1 Why gradient boosting exists
- Bagging (Random Forest) reduces variance but does not directly address bias
- A method that sequentially corrects errors can reduce both bias and variance
- Many competition-winning models and production systems rely on boosted trees
1.2 The core problem it solves
Build a strong predictive model by iteratively adding weak learners (typically shallow trees), where each new learner focuses on correcting the residual errors of the ensemble so far.
2. Definition & Core Idea
2.1 Formal definition
Gradient Boosting constructs an additive model F(x) = sum of f_t(x) by sequentially fitting each new weak learner f_t to the negative gradient of the loss function (the residuals), performing gradient descent in function space.
2.2 Intuitive mental model
"Each new team member focuses exclusively on fixing the mistakes the current team is making, gradually eliminating all errors."
3. How It Works
3.1 Training algorithm
- Initialize model with a constant prediction (e.g., mean of targets)
- For each round t = 1, ..., T:
- Compute the residuals (negative gradient of the loss)
- Fit a shallow decision tree to the residuals
- Add the tree to the ensemble, scaled by a learning rate
- Final prediction is the sum of all trees
3.2 Key hyperparameters
| Parameter | Effect | Typical range |
|---|---|---|
| Number of trees (T) | More rounds = lower bias, risk of overfitting | 100-10000 |
| Learning rate | Shrinks each tree's contribution | 0.01-0.3 |
| Max depth | Controls individual tree complexity | 3-8 |
| Subsample ratio | Fraction of data used per round (stochastic GB) | 0.5-1.0 |
| Min child weight | Regularization on leaf node size | 1-10 |
3.3 Learning rate and number of trees
- Smaller learning rate requires more trees but often yields better performance
- The two parameters trade off: halving the learning rate roughly doubles the needed trees
4. Popular Implementations
| Library | Key innovation |
|---|---|
| XGBoost | Regularized objective, parallel tree building |
| LightGBM | Leaf-wise growth, histogram-based splits |
| CatBoost | Native categorical feature handling |
These implementations dominate tabular data tasks and ML competitions.
5. Gradient Boosting vs Random Forest
| Aspect | Random Forest | Gradient Boosting |
|---|---|---|
| Strategy | Parallel (bagging) | Sequential (boosting) |
| Bias/Variance | Reduces variance | Reduces both bias and variance |
| Tuning | Relatively easy | More hyperparameters |
| Overfitting | Resistant | Can overfit without regularization |
| Speed | Easily parallelized | Inherently sequential |
6. Strengths & Limitations
6.1 Strengths
- State-of-the-art on tabular data
- Handles mixed feature types and missing values
- Built-in feature importance
- Flexible loss functions (regression, classification, ranking)
6.2 Limitations
- Sequential training is slower than parallel methods
- Requires careful hyperparameter tuning
- Less interpretable than a single decision tree
- Sensitive to noisy labels
7. One-Sentence Summary
Gradient boosting sequentially trains shallow decision trees on residual errors, combining them into a powerful ensemble that excels on structured tabular data.