Activation Function

1. Purpose & Motivation

1.1 Why activation functions exist

  • Without activation functions, stacking layers produces only a linear transformation
  • Non-linearity enables networks to learn complex, curved decision boundaries
  • They are what make deep learning deep in a meaningful sense

A neural network without activation functions is just a single linear transformation, no matter how many layers it has.

1.2 The linearity problem

  • Linear functions composed together remain linear: f(g(x)) = Ax for some matrix A
  • Real-world data has non-linear relationships
  • Activation functions break linearity at each layer, enabling hierarchical feature learning

2. Definition & Core Idea

2.1 Formal definition

An activation function f is a (typically non-linear) function applied element-wise to the pre-activation values of a layer:

a = f(z) = f(Wx + b)

2.2 Intuitive mental model

"An activation function decides how much of the computed signal a neuron should pass forward."

3. Common Activation Functions

3.1 Sigmoid

  • Formula: f(z) = 1 / (1 + e^(-z))
  • Output range: (0, 1)
  • Use case: output layer for binary classification
  • Problem: vanishing gradients for large |z|

3.2 Tanh

  • Formula: f(z) = (e^z - e^(-z)) / (e^z + e^(-z))
  • Output range: (-1, 1)
  • Advantage: zero-centered, stronger gradients than sigmoid
  • Problem: still suffers from vanishing gradients

3.3 ReLU (Rectified Linear Unit)

  • Formula: f(z) = max(0, z)
  • Output range: [0, infinity)
  • Advantage: fast computation, reduces vanishing gradient
  • Problem: dead neurons (zero gradient for negative inputs)

3.4 Leaky ReLU

  • Formula: f(z) = max(alpha * z, z) where alpha is small (e.g., 0.01)
  • Fixes the dead neuron problem by allowing small negative gradients

3.5 GELU (Gaussian Error Linear Unit)

  • Formula: f(z) = z * Phi(z) where Phi is the Gaussian CDF
  • Smooth approximation of ReLU
  • Default in Transformers (BERT, GPT)

4. Comparison Table

Function Range Zero-centered Vanishing gradient Computation
Sigmoid (0, 1) No Yes Moderate
Tanh (-1, 1) Yes Yes Moderate
ReLU [0, inf) No Partial Fast
Leaky ReLU (-inf, inf) No No Fast
GELU (-inf, inf) No No Moderate

5. Choosing an Activation Function

5.1 Hidden layers

  • Default choice: ReLU or its variants (Leaky ReLU, GELU)
  • Transformer models: GELU is standard
  • Older architectures: tanh was common in RNNs

5.2 Output layer

Task Activation Reason
Binary classification Sigmoid Output in (0, 1)
Multi-class classification Softmax Outputs sum to 1
Regression None (linear) Unrestricted output range
Multi-label classification Sigmoid Independent probabilities

Key insight: The hidden layer activation controls learning dynamics; the output layer activation must match the task and loss function.

6. Activation Functions and Gradient Flow

6.1 Vanishing gradients

  • Sigmoid and tanh saturate for large inputs, producing near-zero gradients
  • Gradients shrink exponentially through layers, preventing early layers from learning

6.2 Why ReLU helps

  • Gradient is exactly 1 for positive inputs -- no shrinkage
  • Enables training of much deeper networks
  • Trade-off: dead neurons when inputs are consistently negative

7. One-Sentence Summary

Activation functions introduce the non-linearity that allows neural networks to learn complex patterns, with ReLU and its variants being the modern default for hidden layers.

8. Related Knowledge Pages