Activation Function
1. Purpose & Motivation
1.1 Why activation functions exist
- Without activation functions, stacking layers produces only a linear transformation
- Non-linearity enables networks to learn complex, curved decision boundaries
- They are what make deep learning deep in a meaningful sense
A neural network without activation functions is just a single linear transformation, no matter how many layers it has.
1.2 The linearity problem
- Linear functions composed together remain linear:
f(g(x)) = Axfor some matrix A - Real-world data has non-linear relationships
- Activation functions break linearity at each layer, enabling hierarchical feature learning
2. Definition & Core Idea
2.1 Formal definition
An activation function f is a (typically non-linear) function applied element-wise to the pre-activation values of a layer:
a = f(z) = f(Wx + b)
2.2 Intuitive mental model
"An activation function decides how much of the computed signal a neuron should pass forward."
3. Common Activation Functions
3.1 Sigmoid
- Formula:
f(z) = 1 / (1 + e^(-z)) - Output range: (0, 1)
- Use case: output layer for binary classification
- Problem: vanishing gradients for large |z|
3.2 Tanh
- Formula:
f(z) = (e^z - e^(-z)) / (e^z + e^(-z)) - Output range: (-1, 1)
- Advantage: zero-centered, stronger gradients than sigmoid
- Problem: still suffers from vanishing gradients
3.3 ReLU (Rectified Linear Unit)
- Formula:
f(z) = max(0, z) - Output range: [0, infinity)
- Advantage: fast computation, reduces vanishing gradient
- Problem: dead neurons (zero gradient for negative inputs)
3.4 Leaky ReLU
- Formula:
f(z) = max(alpha * z, z)where alpha is small (e.g., 0.01) - Fixes the dead neuron problem by allowing small negative gradients
3.5 GELU (Gaussian Error Linear Unit)
- Formula:
f(z) = z * Phi(z)where Phi is the Gaussian CDF - Smooth approximation of ReLU
- Default in Transformers (BERT, GPT)
4. Comparison Table
| Function | Range | Zero-centered | Vanishing gradient | Computation |
|---|---|---|---|---|
| Sigmoid | (0, 1) | No | Yes | Moderate |
| Tanh | (-1, 1) | Yes | Yes | Moderate |
| ReLU | [0, inf) | No | Partial | Fast |
| Leaky ReLU | (-inf, inf) | No | No | Fast |
| GELU | (-inf, inf) | No | No | Moderate |
5. Choosing an Activation Function
5.1 Hidden layers
- Default choice: ReLU or its variants (Leaky ReLU, GELU)
- Transformer models: GELU is standard
- Older architectures: tanh was common in RNNs
5.2 Output layer
| Task | Activation | Reason |
|---|---|---|
| Binary classification | Sigmoid | Output in (0, 1) |
| Multi-class classification | Softmax | Outputs sum to 1 |
| Regression | None (linear) | Unrestricted output range |
| Multi-label classification | Sigmoid | Independent probabilities |
Key insight: The hidden layer activation controls learning dynamics; the output layer activation must match the task and loss function.
6. Activation Functions and Gradient Flow
6.1 Vanishing gradients
- Sigmoid and tanh saturate for large inputs, producing near-zero gradients
- Gradients shrink exponentially through layers, preventing early layers from learning
6.2 Why ReLU helps
- Gradient is exactly 1 for positive inputs -- no shrinkage
- Enables training of much deeper networks
- Trade-off: dead neurons when inputs are consistently negative
7. One-Sentence Summary
Activation functions introduce the non-linearity that allows neural networks to learn complex patterns, with ReLU and its variants being the modern default for hidden layers.