Feature Engineering
1. Purpose & Motivation
1.1 Why feature engineering exists
- Raw data is rarely in the best format for a model to learn from
- The right features can make a simple model outperform a complex one on raw data
- Domain knowledge encoded as features bridges the gap between data and predictions
1.2 The core problem it solves
Transform raw data into informative, discriminative, and well-structured input features that maximize a model's predictive performance.
2. Definition & Core Idea
2.1 Formal definition
Feature engineering is the process of using domain knowledge and data analysis to create, transform, and select input variables (features) that make machine learning algorithms more effective.
2.2 Intuitive mental model
"A chef does not serve raw ingredients; feature engineering is the preparation that turns raw data into a dish a model can digest."
3. Common Techniques
3.1 Numerical transformations
| Technique | Purpose | Example |
|---|---|---|
| Normalization | Scale features to [0, 1] | Min-max scaling |
| Standardization | Zero mean, unit variance | Z-score scaling |
| Log transform | Reduce skewness | log(income) |
| Binning | Convert continuous to categorical | Age ranges |
| Polynomial features | Capture non-linear relationships | x^2, x1 * x2 |
3.2 Categorical encoding
- One-hot encoding: Creates binary columns per category
- Label encoding: Assigns integer to each category (ordinal data)
- Target encoding: Replaces category with mean of target variable
- Frequency encoding: Replaces category with its occurrence count
3.3 Temporal features
- Extract day, month, hour, day-of-week from timestamps
- Compute time since an event (recency)
- Rolling statistics (moving averages, rolling counts)
3.4 Text features
- Bag-of-words, TF-IDF
- Word count, character count, sentiment scores
- Embeddings from pre-trained language models
4. Feature Interaction
4.1 Creating interaction features
- Multiply or combine features to capture joint effects
- Example: price_per_sqft = price / square_footage
- Domain knowledge guides which interactions are meaningful
4.2 Automated feature generation
- Tools like Featuretools automate interaction and aggregation feature creation
- Deep learning can learn features automatically, reducing manual engineering
5. Best Practices
- Start with exploratory data analysis (EDA) to understand distributions
- Engineer features iteratively: create, evaluate, refine
- Validate that new features improve model performance via cross-validation
- Document feature definitions for reproducibility
- Watch for data leakage (features that encode target information)
6. Strengths & Limitations
6.1 Strengths
- Often the highest-leverage activity in an ML project
- Enables simple models to compete with complex ones
- Encodes domain expertise directly into the model
6.2 Limitations
- Time-consuming and requires domain knowledge
- Risk of overfitting to training data patterns
- Deep learning partially automates this for unstructured data
7. One-Sentence Summary
Feature engineering transforms raw data into informative model inputs, and is often the single most impactful step in building an effective machine learning system.