LSTM / GRU
What LSTM and GRU Do
Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) networks are recurrent cells designed to retain useful information over more time steps than a vanilla RNN. Learned gates control how much old state survives and how much new evidence enters.
Both process sequences incrementally, making them practical for time series, audio, and streaming systems. An LSTM offers separate cell and hidden states; a GRU combines them into a simpler update.
Core Idea
A gate is a sigmoid output between 0 and 1 that acts as a soft switch. Multiplying state values by a gate can preserve them, suppress them, or blend them with a candidate update.
The important change from a vanilla RNN is an additive state path. Instead of replacing all memory through one nonlinear transformation, gated cells can carry selected values forward with small changes.
How It Works
LSTM: Separate Memory and Output
An LSTM maintains a cell state for longer-term storage and a hidden state exposed to the rest of the network. Three gates regulate them:
The forget gate scales old memory, the input gate controls the candidate update, and the output gate decides which memory is exposed:
GRU: One State and Two Gates
A GRU merges cell and hidden state into . Its reset gate controls how much previous state contributes to a candidate; its update gate blends that candidate with retained state:
Gate naming and the update equation's convention can vary by source or implementation, but the role is the same: interpolate between retained memory and new content.
Why It Works
- Additive memory updates: The state can travel through many steps without being completely rewritten.
- Learned retention: Gates adapt memory duration to the data rather than imposing a fixed window.
- Selective exposure: The model can store information without exposing all of it at every output.
- Shared recurrent structure: Like an RNN, the same cell works across variable-length sequences and supports online inference.
When an LSTM forget gate remains near 1 and its input gate near 0, the cell state changes little. This creates a more direct path for information and gradients than repeated vanilla-RNN transformations, although it does not guarantee unlimited memory.
LSTM or GRU?
| Aspect | LSTM | GRU |
|---|---|---|
| Gates | Forget, input, output | Reset, update |
| State | Cell state and hidden state | Hidden state only |
| Parameter count | Higher | Lower |
| Per-step computation | More | Less |
| Practical choice | More explicit memory control | Simpler baseline and faster iteration |
Neither cell is universally better. The choice is empirical: compare validation quality, latency, and memory use for the actual sequence length and dataset.
Limitations
- Recurrence still prevents full parallelization across time steps.
- Gates reduce vanishing-gradient problems but do not remove difficulty on extremely long dependencies.
- Training and inference costs still grow with sequence length.
- More gates add parameters and computation compared with a vanilla RNN, especially for LSTMs.
- Transformers often perform better when global context, large datasets, and parallel training matter more than streaming efficiency.