Vision Transformer (ViT)
What ViT Does
The Vision Transformer applies a standard transformer encoder to images. It converts an image into a sequence of patch tokens, models relationships among them, and produces features for classification or downstream vision tasks.
ViT showed that attention could match or exceed convolutional networks when trained with enough data and compute. It helped establish transformers as general-purpose backbones for vision and multimodal systems.
Core Idea
Cut an image into a regular grid of patches and treat each patch like a token in a sentence. Embed those tokens, add their positions, and let self-attention learn which regions should exchange information.
Unlike a convolution, global self-attention can directly connect distant patches in the first encoder layer. ViT learns spatial patterns from data rather than hard-coding locality and translation equivariance.
How It Works
1. Turn the Image into Tokens
For an image of height H, width W, and patch size P, ViT creates N = HW/P² non-overlapping patches. Each flattened patch is projected into a D-dimensional embedding.
Image (224×224) → 196 patches (16×16) → 196 embedding vectors
The linear patch projection is equivalent to a convolution whose kernel and stride both equal the patch size. Larger patches shorten the sequence but discard finer spatial detail.
2. Add Identity and Position
A learnable [CLS] token is prepended for classification. Learnable positional embeddings are added because attention alone does not know where patches occurred in the original grid.
When fine-tuning at a new resolution, the patch grid changes. Positional embeddings are therefore interpolated to the new grid before training continues.
3. Process Tokens with an Encoder
The sequence passes through repeated pre-normalized transformer blocks:
z′_l = MSA(LN(z_{l-1})) + z_{l-1}
z_l = MLP(LN(z′_l)) + z′_l
Multi-head self-attention lets different heads model different relationships. The feed-forward network transforms each token independently, and residual connections preserve information across depth.
For N patch tokens, global attention builds an N × N interaction matrix. This offers direct global context but makes attention cost grow quadratically with image resolution.
4. Read Out a Prediction
For classification, the final [CLS] representation passes through an MLP head to produce class logits. Other tasks can reshape patch tokens into a spatial feature map for a detector or segmentation decoder.
5. Pre-train and Transfer
ViT has weaker built-in visual priors than a CNN, so training only on a modest dataset such as ImageNet can underperform a comparable convolutional model.
Pre-training on larger datasets such as JFT-300M or LAION lets ViT learn locality, shape, and texture from examples. The resulting encoder transfers effectively to classification, detection, segmentation, and multimodal tasks.
Why It Works
- Global context: Any patch can interact directly with any other patch in one attention layer.
- Content-dependent routing: Attention weights adapt to the current image rather than using a fixed kernel.
- Scalable architecture: Model width, depth, token count, data, and compute can grow within a uniform design.
- Transferable features: Large-scale pre-training produces representations that adapt across many tasks.
- Simple modality bridge: Representing images as tokens makes it easier to combine visual and language transformers.
CNNs still provide useful locality and translation priors for limited data. ViT’s weaker priors become an advantage mainly when scale supplies enough evidence to learn those patterns.
ViT Compared with CNNs
| Aspect | CNN | ViT |
|---|---|---|
| Inductive bias | Strong locality and translation bias | Weak; learned from data |
| Global context | Emerges through deep stacking | Available from the first layer |
| Data efficiency | Usually better on small datasets | Often needs large pre-training data |
| Scaling | Strong but architecture-specific | Predictable gains with data and compute |
| Spatial hierarchy | Natural multi-scale feature maps | Base ViT is usually single-scale |
| Computation | Local and regular | Global attention is quadratic in patch count |
Extensions and Uses
| Model | Approach |
|---|---|
| DeiT | Data-efficient training and distillation from a CNN teacher |
| Swin | Hierarchical transformer with shifted local windows |
| CoAtNet | Depthwise convolution combined with attention |
| EfficientFormer | Transformer-style vision model optimized for mobile deployment |
ViT variants support image classification, DETR and ViTDet-style object detection, SegFormer-style semantic segmentation, medical imaging, and multimodal systems. CLIP commonly uses ViT as its image encoder.
Hybrid models use early CNN layers for local feature extraction, then pass feature tokens into a transformer. This combines convolutional priors with long-range, content-dependent attention.
Limitations
- Data hungry: From-scratch training can be inefficient without large datasets or strong augmentation.
- Quadratic attention: Increasing image resolution sharply increases token interactions and memory use.
- Coarse patches: Large patches can hide small objects and precise boundaries.
- Weak native hierarchy: Base ViT does not naturally produce the multi-scale pyramid expected by many dense prediction systems.
- Pre-training cost: Strong transfer performance may depend on expensive data and compute.
- Interpretability limits: Attention maps show interactions, but they are not complete explanations of a prediction.
Windowed attention, token pruning, hierarchical stages, distillation, masked-image pre-training, and convolutional stems address different parts of these limitations.