Random Forest
1. Purpose & Motivation
1.1 Why random forests exist
- Single decision trees are unstable and prone to overfitting
- Combining multiple models (ensembling) reduces variance and improves robustness
- Practitioners need a strong out-of-the-box model with minimal tuning
1.2 The core problem it solves
Reduce the high variance of individual decision trees by training many trees on different random subsets of data and features, then aggregating their predictions.
2. Definition & Core Idea
2.1 Formal definition
A Random Forest is an ensemble of decision trees, each trained on a bootstrap sample of the data with a random subset of features considered at each split. Predictions are made by majority vote (classification) or averaging (regression) across all trees.
2.2 Intuitive mental model
"Ask a crowd of diverse decision-makers (each seeing different evidence) and trust the majority opinion over any single individual."
3. How It Works
3.1 Training algorithm (Bagging + feature randomness)
- For each of B trees:
- Draw a bootstrap sample (random sample with replacement) from the training data
- Grow a decision tree, but at each split consider only a random subset of m features
- Grow the tree fully (no pruning)
- Store all B trees
3.2 Prediction
- Classification: Each tree votes; the class with the most votes wins
- Regression: Average the predictions of all trees
3.3 Key hyperparameters
| Parameter | Effect | Typical default |
|---|---|---|
| Number of trees (B) | More trees = lower variance, diminishing returns | 100-500 |
| Max features (m) | Controls tree diversity | sqrt(p) for classification, p/3 for regression |
| Max depth | Limits individual tree complexity | None (fully grown) |
| Min samples per leaf | Regularization via minimum leaf size | 1-5 |
4. Why It Works
4.1 Variance reduction through averaging
- Individual trees overfit, but their errors are decorrelated due to randomness
- Averaging decorrelated predictions reduces variance without increasing bias
4.2 Feature importance
- Permutation importance: Measure accuracy drop when a feature's values are shuffled
- Impurity-based importance: Sum of impurity decreases across all splits using that feature
- Built-in feature ranking is a major practical benefit
5. Out-of-Bag (OOB) Evaluation
- Each bootstrap sample leaves out ~37% of the data
- These "out-of-bag" samples serve as a free validation set
- OOB error approximates cross-validation error without extra computation
6. Strengths & Limitations
6.1 Strengths
- Excellent performance with minimal hyperparameter tuning
- Resistant to overfitting (more trees never hurts)
- Handles missing values and mixed feature types
- Provides built-in feature importance
6.2 Limitations
- Less interpretable than a single decision tree
- Large memory and inference cost for many trees
- Struggles with very high-dimensional sparse data
- Cannot extrapolate beyond the training data range (regression)
7. One-Sentence Summary
Random forests combine many decision trees trained on random data and feature subsets, using aggregation to achieve robust, high-accuracy predictions.