Feature Selection

1. Purpose & Motivation

1.1 Why feature selection exists

  • Not all features are equally useful; some are irrelevant or redundant
  • Fewer features mean faster training, simpler models, and better interpretability
  • Removing noisy features can improve generalization, not just reduce cost

1.2 The core problem it solves

Identify and retain the most relevant subset of original features while discarding those that contribute little or degrade model performance.

2. Definition & Core Idea

2.1 Formal definition

Feature selection is the process of choosing a subset of d' features from the original d features (d' < d) that maximizes predictive performance, using relevance criteria, model-based importance, or search strategies.

2.2 Intuitive mental model

"Packing for a trip: keep only the items you will actually use, leave behind what adds weight without value."

3. Three Families of Methods

3.1 Filter methods

  • Evaluate features independently of the model
  • Fast and scalable; used as a preprocessing step
Technique What it measures
Correlation coefficient Linear relationship with target
Mutual information Any statistical dependency with target
Chi-squared test Association between categorical variables
Variance threshold Removes near-constant features

3.2 Wrapper methods

  • Use a model to evaluate subsets of features
  • More accurate but computationally expensive
Technique Strategy
Forward selection Start empty, add one feature at a time
Backward elimination Start with all, remove one at a time
Recursive Feature Elimination (RFE) Iteratively remove least important

3.3 Embedded methods

  • Feature selection happens during model training
  • Balance between filter speed and wrapper accuracy
Technique How it selects
Lasso (L1) Shrinks irrelevant feature weights to zero
Tree-based importance Features ranked by split contribution
Elastic Net Combines L1 sparsity with L2 stability

4. Feature Selection vs Dimensionality Reduction

Aspect Feature Selection Dimensionality Reduction
Output Subset of original features New constructed features
Interpretability Preserved (original features) Often lost (transformed space)
Information loss Discards entire features Compresses all features
Methods Filter, Wrapper, Embedded PCA, t-SNE, Autoencoders

5. Best Practices

  • Start with filter methods to remove obviously useless features
  • Use embedded or wrapper methods for fine-grained selection
  • Always validate selection via cross-validation on held-out data
  • Be aware of multicollinearity: correlated features can mask each other's importance
  • Re-evaluate selected features when data distribution changes

6. Strengths & Limitations

6.1 Strengths

  • Improves model interpretability by keeping original features
  • Reduces training time and memory
  • Can improve generalization by removing noise

6.2 Limitations

  • Wrapper methods are computationally expensive for many features
  • Filter methods ignore feature interactions
  • Optimal subset selection is NP-hard; heuristics are used in practice

7. One-Sentence Summary

Feature selection identifies and retains the most informative original features, improving model performance, interpretability, and efficiency.

8. Related Knowledge Pages