RNN
What an RNN Does
A Recurrent Neural Network (RNN) processes an ordered sequence while carrying information from one step to the next. This makes its current prediction depend on both the current input and a summary of earlier inputs.
RNNs can accept variable-length text, speech, sensor readings, or other time series. They remain useful when data arrives as a stream and predictions must be updated incrementally.
Core Idea
The same recurrent cell is reused at every time step. Its hidden state acts as working memory: the cell reads , combines it with , and produces a new state .
A vanilla RNN commonly uses
The matrices are shared across all positions, so the model applies the same learned update whether it is processing the first step or the thousandth.
How It Works
1. Start with an Initial State
The model begins with , often a vector of zeros or a learned value. For each input, it updates that state before moving to the next position.
2. Unroll the Recurrent Computation
“Unrolling” draws one copy of the shared cell for each time step. The copies are not separate models: they use the same parameters, but each receives a different input and the preceding state.
An RNN may return an output at every step for sequence labeling, only the final output for sequence classification, or feed its states into another sequence model.
3. Train Through Time
Backpropagation Through Time (BPTT) treats the unrolled computation as a deep network. Gradients flow backward from later outputs through each hidden-state transition. Memory and compute therefore grow roughly linearly with sequence length; truncated BPTT limits training to shorter windows when necessary.
4. Adapt the Direction and Depth
A bidirectional RNN combines forward and backward passes when the full sequence is available. Stacked RNNs pass one recurrent layer's sequence of states to another, increasing representational capacity.
Why It Works
- Persistent context: The hidden state gives later predictions access to earlier information.
- Order sensitivity: Changing the order of inputs changes the sequence of state updates.
- Parameter sharing: One cell handles every position, allowing variable sequence lengths without adding parameters.
- Streaming operation: Inference needs only the current input and previous state rather than the entire history.
Limitations
- Recurrent steps are sequential, which limits parallelism during training and can make long sequences slow.
- Repeated Jacobian multiplication can make gradients vanish or explode. Vanishing gradients prevent early information from receiving a useful learning signal; exploding gradients destabilize optimization.
- A single fixed-size hidden state can become an information bottleneck for long or complex sequences.
- Vanilla RNNs usually learn short-range dependencies more reliably than long-range ones. LSTM and GRU cells add gates to improve memory, while Transformers create shorter paths between distant positions.
Gradient clipping can control exploding gradients, but it does not solve long-term credit assignment caused by vanishing gradients.