Feature Selection
1. Purpose & Motivation
1.1 Why feature selection exists
- Not all features are equally useful; some are irrelevant or redundant
- Fewer features mean faster training, simpler models, and better interpretability
- Removing noisy features can improve generalization, not just reduce cost
1.2 The core problem it solves
Identify and retain the most relevant subset of original features while discarding those that contribute little or degrade model performance.
2. Definition & Core Idea
2.1 Formal definition
Feature selection is the process of choosing a subset of d' features from the original d features (d' < d) that maximizes predictive performance, using relevance criteria, model-based importance, or search strategies.
2.2 Intuitive mental model
"Packing for a trip: keep only the items you will actually use, leave behind what adds weight without value."
3. Three Families of Methods
3.1 Filter methods
- Evaluate features independently of the model
- Fast and scalable; used as a preprocessing step
| Technique | What it measures |
|---|---|
| Correlation coefficient | Linear relationship with target |
| Mutual information | Any statistical dependency with target |
| Chi-squared test | Association between categorical variables |
| Variance threshold | Removes near-constant features |
3.2 Wrapper methods
- Use a model to evaluate subsets of features
- More accurate but computationally expensive
| Technique | Strategy |
|---|---|
| Forward selection | Start empty, add one feature at a time |
| Backward elimination | Start with all, remove one at a time |
| Recursive Feature Elimination (RFE) | Iteratively remove least important |
3.3 Embedded methods
- Feature selection happens during model training
- Balance between filter speed and wrapper accuracy
| Technique | How it selects |
|---|---|
| Lasso (L1) | Shrinks irrelevant feature weights to zero |
| Tree-based importance | Features ranked by split contribution |
| Elastic Net | Combines L1 sparsity with L2 stability |
4. Feature Selection vs Dimensionality Reduction
| Aspect | Feature Selection | Dimensionality Reduction |
|---|---|---|
| Output | Subset of original features | New constructed features |
| Interpretability | Preserved (original features) | Often lost (transformed space) |
| Information loss | Discards entire features | Compresses all features |
| Methods | Filter, Wrapper, Embedded | PCA, t-SNE, Autoencoders |
5. Best Practices
- Start with filter methods to remove obviously useless features
- Use embedded or wrapper methods for fine-grained selection
- Always validate selection via cross-validation on held-out data
- Be aware of multicollinearity: correlated features can mask each other's importance
- Re-evaluate selected features when data distribution changes
6. Strengths & Limitations
6.1 Strengths
- Improves model interpretability by keeping original features
- Reduces training time and memory
- Can improve generalization by removing noise
6.2 Limitations
- Wrapper methods are computationally expensive for many features
- Filter methods ignore feature interactions
- Optimal subset selection is NP-hard; heuristics are used in practice
7. One-Sentence Summary
Feature selection identifies and retains the most informative original features, improving model performance, interpretability, and efficiency.