Vision Transformer (ViT)

What ViT Does

The Vision Transformer applies a standard transformer encoder to images. It converts an image into a sequence of patch tokens, models relationships among them, and produces features for classification or downstream vision tasks.

ViT showed that attention could match or exceed convolutional networks when trained with enough data and compute. It helped establish transformers as general-purpose backbones for vision and multimodal systems.

Core Idea

Cut an image into a regular grid of patches and treat each patch like a token in a sentence. Embed those tokens, add their positions, and let self-attention learn which regions should exchange information.

Unlike a convolution, global self-attention can directly connect distant patches in the first encoder layer. ViT learns spatial patterns from data rather than hard-coding locality and translation equivariance.

How It Works

1. Turn the Image into Tokens

For an image of height H, width W, and patch size P, ViT creates N = HW/P² non-overlapping patches. Each flattened patch is projected into a D-dimensional embedding.

Image (224×224) → 196 patches (16×16) → 196 embedding vectors

The linear patch projection is equivalent to a convolution whose kernel and stride both equal the patch size. Larger patches shorten the sequence but discard finer spatial detail.

2. Add Identity and Position

A learnable [CLS] token is prepended for classification. Learnable positional embeddings are added because attention alone does not know where patches occurred in the original grid.

When fine-tuning at a new resolution, the patch grid changes. Positional embeddings are therefore interpolated to the new grid before training continues.

3. Process Tokens with an Encoder

The sequence passes through repeated pre-normalized transformer blocks:

z′_l = MSA(LN(z_{l-1})) + z_{l-1}
z_l  = MLP(LN(z′_l)) + z′_l

Multi-head self-attention lets different heads model different relationships. The feed-forward network transforms each token independently, and residual connections preserve information across depth.

For N patch tokens, global attention builds an N × N interaction matrix. This offers direct global context but makes attention cost grow quadratically with image resolution.

4. Read Out a Prediction

For classification, the final [CLS] representation passes through an MLP head to produce class logits. Other tasks can reshape patch tokens into a spatial feature map for a detector or segmentation decoder.

5. Pre-train and Transfer

ViT has weaker built-in visual priors than a CNN, so training only on a modest dataset such as ImageNet can underperform a comparable convolutional model.

Pre-training on larger datasets such as JFT-300M or LAION lets ViT learn locality, shape, and texture from examples. The resulting encoder transfers effectively to classification, detection, segmentation, and multimodal tasks.

Why It Works

  • Global context: Any patch can interact directly with any other patch in one attention layer.
  • Content-dependent routing: Attention weights adapt to the current image rather than using a fixed kernel.
  • Scalable architecture: Model width, depth, token count, data, and compute can grow within a uniform design.
  • Transferable features: Large-scale pre-training produces representations that adapt across many tasks.
  • Simple modality bridge: Representing images as tokens makes it easier to combine visual and language transformers.

CNNs still provide useful locality and translation priors for limited data. ViT’s weaker priors become an advantage mainly when scale supplies enough evidence to learn those patterns.

ViT Compared with CNNs

Aspect CNN ViT
Inductive bias Strong locality and translation bias Weak; learned from data
Global context Emerges through deep stacking Available from the first layer
Data efficiency Usually better on small datasets Often needs large pre-training data
Scaling Strong but architecture-specific Predictable gains with data and compute
Spatial hierarchy Natural multi-scale feature maps Base ViT is usually single-scale
Computation Local and regular Global attention is quadratic in patch count

Extensions and Uses

Model Approach
DeiT Data-efficient training and distillation from a CNN teacher
Swin Hierarchical transformer with shifted local windows
CoAtNet Depthwise convolution combined with attention
EfficientFormer Transformer-style vision model optimized for mobile deployment

ViT variants support image classification, DETR and ViTDet-style object detection, SegFormer-style semantic segmentation, medical imaging, and multimodal systems. CLIP commonly uses ViT as its image encoder.

Hybrid models use early CNN layers for local feature extraction, then pass feature tokens into a transformer. This combines convolutional priors with long-range, content-dependent attention.

Limitations

  • Data hungry: From-scratch training can be inefficient without large datasets or strong augmentation.
  • Quadratic attention: Increasing image resolution sharply increases token interactions and memory use.
  • Coarse patches: Large patches can hide small objects and precise boundaries.
  • Weak native hierarchy: Base ViT does not naturally produce the multi-scale pyramid expected by many dense prediction systems.
  • Pre-training cost: Strong transfer performance may depend on expensive data and compute.
  • Interpretability limits: Attention maps show interactions, but they are not complete explanations of a prediction.

Windowed attention, token pruning, hierarchical stages, distillation, masked-image pre-training, and convolutional stems address different parts of these limitations.

Related