Transformer
What a Transformer Does
A Transformer builds context-aware representations by letting each sequence position gather information from other positions. It can encode an input, generate an output one token at a time, or combine an encoder and decoder for sequence-to-sequence tasks.
Unlike an RNN, a Transformer has no recurrent hidden state that must pass through every intermediate step. During training, this allows many positions to be processed in parallel and gives distant positions a direct interaction path.
Core Idea
Self-attention computes a weighted mixture of token information for every position. The weights are content-dependent: each token asks a query, compares it with other tokens' keys, and collects their values.
For query, key, and value matrices , , and , scaled dot-product attention is
The optional mask can hide padding or prevent a decoder from attending to future tokens. Dividing by keeps large dot products from pushing softmax into saturated regions.
How It Works
1. Represent Tokens and Positions
Inputs become vectors through an embedding or projection. Because self-attention alone does not encode order, the model adds or applies positional information such as learned embeddings, sinusoidal encodings, or rotary position embeddings (RoPE).
2. Create Queries, Keys, and Values
Learned linear projections produce , , and . Query-key dot products measure compatibility between positions. Softmax normalizes those scores, and the weighted values form a context vector for each query.
3. Attend in Multiple Subspaces
Multi-head attention runs several learned projections in parallel:
Different heads can represent different interaction patterns, although a head does not necessarily correspond to one clean linguistic or visual concept.
4. Transform Each Position
After attention mixes information across positions, a feed-forward network applies the same MLP independently to each position. Residual connections preserve earlier representations and improve gradient flow; layer normalization stabilizes optimization. Repeating this block builds progressively richer representations.
5. Choose an Encoder, Decoder, or Both
| Structure | Attention pattern | Typical role |
|---|---|---|
| Encoder-only | Bidirectional self-attention | Representation and understanding |
| Decoder-only | Causal self-attention | Autoregressive generation |
| Encoder-decoder | Encoder self-attention plus decoder cross-attention | Input-to-output transformation |
BERT and ViT are encoder-oriented examples, GPT is decoder-only, and the original Transformer and T5 use both components.
Why It Works
- Direct context access: Any unmasked position can aggregate information from any other in one attention layer.
- Content-dependent routing: Attention weights change with the input instead of applying one fixed local kernel.
- Training parallelism: All known token positions can compute a layer together, unlike recurrent state updates.
- Scalable blocks: Repeating attention and MLP layers works across language, vision, audio, and other tokenized data.
- Specialized masking: The same mechanism supports bidirectional encoding, causal generation, and cross-attention.
Parallel training does not mean generation is fully parallel: an autoregressive decoder still emits tokens sequentially because each new token depends on previous outputs.
Limitations
- Full self-attention forms an score matrix, giving quadratic time and memory growth with sequence length.
- The architecture does not inherently know order, locality, or hierarchy; these biases must come from position mechanisms, data, or architectural modifications.
- Large Transformers often require substantial training data, compute, and memory.
- Attention weights show routing strength but are not automatically faithful explanations of a model's decision.
- Context length remains finite, and fitting information into the window does not guarantee that the model will use it reliably.