Swin Transformer

What Swin Transformer Does

Swin is a general-purpose vision backbone for classification, object detection, and segmentation. It keeps attention affordable at high resolution and produces the multi-scale feature maps expected by dense prediction systems.

Introduced in 2021 in Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, it won the Best Paper Award at ICCV 2021.

Swin-T reached 81.3% ImageNet top-1 accuracy with 28M parameters. Swin backbones were widely paired with DINO, Mask R-CNN, and UPerNet for detection and segmentation from 2021 to 2023.

Core Idea

ViT compares every patch with every other patch, so attention grows quadratically with token count. Swin instead computes self-attention inside small, fixed windows, making cost linear in image size for a fixed window size.

Local windows alone would isolate neighboring groups. Swin shifts the window grid in alternating blocks so patches separated by one partition can share a window in the next.

Swin Transformer Architecture

Swin Transformer Architecture — hierarchical stages with patch merging

Source: Liu et al., 2021 — Swin Transformer (ICCV)

Shifted Window Attention

Regular window (W-MSA) vs. Shifted window (SW-MSA) attention

Source: Liu et al., 2021 — Swin Transformer (ICCV)

How It Works

1. Partition the Image into Fine Patches

Swin begins with 4 × 4 pixel patches, finer than the 16 × 16 patches used by the original ViT. A linear layer projects each patch into the model’s channel dimension.

For a 224 × 224 × 3 image, Swin-T starts with a 56 × 56 × 96 embedded feature map.

2. Attend Within Regular Windows

Window multi-head self-attention, or W-MSA, divides the feature map into non-overlapping M × M windows. With M = 7, each attention operation covers 49 patches.

Because the window size stays fixed, total attention work grows as O(N) with the number of image patches, rather than the O(N²) cost of global attention.

3. Shift the Windows

The next block uses SW-MSA, shifting the partition by roughly (M/2, M/2). Its windows cross the previous boundaries, allowing formerly separated patches to exchange information.

A cyclic shift and attention mask implement the operation efficiently without irregular padding. Alternating W-MSA and SW-MSA gradually propagates information across the full image.

4. Merge Patches into a Hierarchy

Between stages, patch merging concatenates each 2 × 2 neighborhood and applies a linear projection. Spatial resolution halves while channel count doubles.

Swin therefore produces feature maps at 1/4, 1/8, 1/16, and 1/32 of input resolution. This hierarchy connects directly to Feature Pyramid Networks used in detection and segmentation.

Stage Operation Resolution Channels
Patch partition Split into 4×4 patches 56 × 56 48
Linear embed Project patch vectors 56 × 56 96
Stage 1 2 Swin blocks 56 × 56 96
Stage 2 Merge + 2 Swin blocks 28 × 28 192
Stage 3 Merge + 6 Swin blocks 14 × 14 384
Stage 4 Merge + 2 Swin blocks 7 × 7 768
Head Global average pool + linear 1,000 classes —

5. Encode Relative Position

Swin adds a learned relative-position bias B to attention scores:

Attention(Q, K, V) = softmax(QKᵀ / √d + B)V

The bias depends on the offset between query and key patches inside a window. It expresses spatial relationships more directly than a single absolute embedding and can be adapted to new window sizes.

6. Train the Backbone

The original Swin-T recipe used AdamW, weight decay 0.05, an initial learning rate of 1 × 10⁻³ with cosine decay, batch size 1,024, and 300 epochs.

RandAugment, Mixup, CutMix, random erasing, and stochastic depth regularized training. Pre-normalization places layer normalization before attention and feed-forward operations.

Why It Works

  • Bounded attention cost: Fixed local windows keep high-resolution processing practical.
  • Cross-window communication: Shifted partitions remove the hard boundaries created by ordinary window attention.
  • Feature hierarchy: Patch merging gives downstream heads both fine detail and coarse semantic context.
  • Spatial bias: Relative positions encode geometry within each window.
  • Scalable family: Stage depth and channel width create Tiny, Small, Base, Large, and larger variants.
Model Blocks per stage Base channels Parameters Top-1 accuracy
Swin-T 2–2–6–2 96 28M 81.3%
Swin-S 2–2–18–2 96 50M 83.0%
Swin-B 2–2–18–2 128 88M 83.5%
Swin-L 2–2–18–2 192 197M 86.3%
Swin-V2-G 2–2–42–4 512 3B 90.2%

Swin Transformer V2

Swin V2 uses scaled cosine attention for stability, log-spaced continuous position bias for transfer across resolutions, and residual-post-normalization to support very large models.

Progressive training begins with smaller images and windows, then increases them. These changes enabled scaling to 3B parameters while transferring across image and window sizes.

Limitations

  • Restricted first-layer context: Objects larger than a window require several blocks before distant regions interact.
  • Fixed window hyperparameter: A window size that suits one scale may be inefficient at another.
  • Implementation complexity: Cyclic shifts, masks, relative biases, and staged merging are more involved than plain ViT blocks.
  • Hardware efficiency: Window operations may be less optimized than convolutions on edge and mobile accelerators.
  • Training cost: Strong results still depend on substantial data, augmentation, and long schedules.
  • Competing designs: ConvNeXt showed that a modernized CNN can match much of Swin’s accuracy with a simpler convolutional structure.

Related

Sources