Attention
What It Does
Attention lets a model gather information according to the content of the current input. Instead of using one fixed aggregation pattern, each position can assign different weights to other positions or features.
In vision, this can connect distant image patches, emphasize useful channels, or combine image features with another representation such as text.
Core Idea
Each item produces a query, key, and value. A query scores the available keys, and the normalized scores determine how strongly their values contribute to the output.
Scaled dot-product attention is:
The factor keeps large dot products from pushing softmax into saturated regions as key dimension grows.
How It Works
1. Project Inputs into Queries, Keys, and Values
Learned linear projections create three views of each representation.
| Component | Question it answers |
|---|---|
| Query () | What information does this position need? |
| Key () | What kind of information does this position offer? |
| Value () | What content should be passed onward? |
2. Score Relationships
Each query takes a dot product with every key in its attention scope. A larger score means the pair is more compatible under the learned projections.
3. Normalize and Aggregate
Softmax turns a query's scores into weights that sum to one. The output is the weighted sum of the corresponding values, so its content changes with the input.
4. Use Multiple Heads
Multi-head attention repeats this process with different projections. Heads can learn different relationships, then their outputs are concatenated and projected back into the model dimension.
Attention Patterns in Vision
Self-Attention
Queries, keys, and values come from the same token set. Full self-attention lets every image patch interact directly with every other patch, providing global context in one layer.
Cross-Attention
Queries come from one representation while keys and values come from another. Decoders use it to read encoder features; text-to-image systems use it to condition image features on text.
Spatial and Channel Attention
Spatial attention produces weights over positions. Channel attention, as in Squeeze-and-Excitation networks, weights feature channels. CBAM applies channel and spatial attention in sequence.
| Pattern | Scope | Typical role |
|---|---|---|
| Channel attention | Feature channels | Reweight CNN feature types |
| Spatial attention | Image positions | Emphasize relevant regions |
| Full self-attention | All token pairs | Global interaction in Vision Transformers |
| Windowed attention | Tokens within windows | Limit cost in models such as Swin Transformer |
| Cross-attention | Two representations | Fuse modalities or encoder-decoder features |
Why It Works
- Content-dependent routing: The model selects information based on the current representation.
- Short paths between distant regions: Full attention connects any two patches without many local layers between them.
- Flexible relation modeling: Different heads can represent distinct spatial or semantic relationships.
- Composable fusion: Cross-attention gives unlike token sets a shared interaction mechanism.
Limitations and Design Trade-offs
Full attention over tokens forms an score matrix. Its quadratic time and memory growth becomes expensive for high-resolution images, where token count rises with image area.
Windowed, sparse, or linearized variants reduce cost but also restrict available interactions or approximate the original operation. Convolution may offer a stronger locality bias and greater data efficiency in some settings.
Attention weights are not guaranteed explanations. They reveal one internal weighting step, but residual paths, value projections, and later layers also affect the prediction.