Swin Transformer
What Swin Transformer Does
Swin is a general-purpose vision backbone for classification, object detection, and segmentation. It keeps attention affordable at high resolution and produces the multi-scale feature maps expected by dense prediction systems.
Introduced in 2021 in Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, it won the Best Paper Award at ICCV 2021.
Swin-T reached 81.3% ImageNet top-1 accuracy with 28M parameters. Swin backbones were widely paired with DINO, Mask R-CNN, and UPerNet for detection and segmentation from 2021 to 2023.
Core Idea
ViT compares every patch with every other patch, so attention grows quadratically with token count. Swin instead computes self-attention inside small, fixed windows, making cost linear in image size for a fixed window size.
Local windows alone would isolate neighboring groups. Swin shifts the window grid in alternating blocks so patches separated by one partition can share a window in the next.
Swin Transformer Architecture — hierarchical stages with patch merging
Regular window (W-MSA) vs. Shifted window (SW-MSA) attention
How It Works
1. Partition the Image into Fine Patches
Swin begins with 4 × 4 pixel patches, finer than the 16 × 16 patches used by the original ViT. A linear layer projects each patch into the model’s channel dimension.
For a 224 × 224 × 3 image, Swin-T starts with a 56 × 56 × 96 embedded feature map.
2. Attend Within Regular Windows
Window multi-head self-attention, or W-MSA, divides the feature map into non-overlapping M × M windows. With M = 7, each attention operation covers 49 patches.
Because the window size stays fixed, total attention work grows as O(N) with the number of image patches, rather than the O(N²) cost of global attention.
3. Shift the Windows
The next block uses SW-MSA, shifting the partition by roughly (M/2, M/2). Its windows cross the previous boundaries, allowing formerly separated patches to exchange information.
A cyclic shift and attention mask implement the operation efficiently without irregular padding. Alternating W-MSA and SW-MSA gradually propagates information across the full image.
4. Merge Patches into a Hierarchy
Between stages, patch merging concatenates each 2 × 2 neighborhood and applies a linear projection. Spatial resolution halves while channel count doubles.
Swin therefore produces feature maps at 1/4, 1/8, 1/16, and 1/32 of input resolution. This hierarchy connects directly to Feature Pyramid Networks used in detection and segmentation.
| Stage | Operation | Resolution | Channels |
|---|---|---|---|
| Patch partition | Split into 4×4 patches |
56 × 56 |
48 |
| Linear embed | Project patch vectors | 56 × 56 |
96 |
| Stage 1 | 2 Swin blocks | 56 × 56 |
96 |
| Stage 2 | Merge + 2 Swin blocks | 28 × 28 |
192 |
| Stage 3 | Merge + 6 Swin blocks | 14 × 14 |
384 |
| Stage 4 | Merge + 2 Swin blocks | 7 × 7 |
768 |
| Head | Global average pool + linear | 1,000 classes | — |
5. Encode Relative Position
Swin adds a learned relative-position bias B to attention scores:
Attention(Q, K, V) = softmax(QKᵀ / √d + B)V
The bias depends on the offset between query and key patches inside a window. It expresses spatial relationships more directly than a single absolute embedding and can be adapted to new window sizes.
6. Train the Backbone
The original Swin-T recipe used AdamW, weight decay 0.05, an initial learning rate of 1 × 10⁻³ with cosine decay, batch size 1,024, and 300 epochs.
RandAugment, Mixup, CutMix, random erasing, and stochastic depth regularized training. Pre-normalization places layer normalization before attention and feed-forward operations.
Why It Works
- Bounded attention cost: Fixed local windows keep high-resolution processing practical.
- Cross-window communication: Shifted partitions remove the hard boundaries created by ordinary window attention.
- Feature hierarchy: Patch merging gives downstream heads both fine detail and coarse semantic context.
- Spatial bias: Relative positions encode geometry within each window.
- Scalable family: Stage depth and channel width create Tiny, Small, Base, Large, and larger variants.
| Model | Blocks per stage | Base channels | Parameters | Top-1 accuracy |
|---|---|---|---|---|
| Swin-T | 2–2–6–2 | 96 | 28M | 81.3% |
| Swin-S | 2–2–18–2 | 96 | 50M | 83.0% |
| Swin-B | 2–2–18–2 | 128 | 88M | 83.5% |
| Swin-L | 2–2–18–2 | 192 | 197M | 86.3% |
| Swin-V2-G | 2–2–42–4 | 512 | 3B | 90.2% |
Swin Transformer V2
Swin V2 uses scaled cosine attention for stability, log-spaced continuous position bias for transfer across resolutions, and residual-post-normalization to support very large models.
Progressive training begins with smaller images and windows, then increases them. These changes enabled scaling to 3B parameters while transferring across image and window sizes.
Limitations
- Restricted first-layer context: Objects larger than a window require several blocks before distant regions interact.
- Fixed window hyperparameter: A window size that suits one scale may be inefficient at another.
- Implementation complexity: Cyclic shifts, masks, relative biases, and staged merging are more involved than plain ViT blocks.
- Hardware efficiency: Window operations may be less optimized than convolutions on edge and mobile accelerators.
- Training cost: Strong results still depend on substantial data, augmentation, and long schedules.
- Competing designs: ConvNeXt showed that a modernized CNN can match much of Swin’s accuracy with a simpler convolutional structure.
Related
- Vision Transformer
- Transformer
- ResNet
- EfficientNet
- Attention
- Detection
- Segmentation
- Swin Transformer V2 (2022) — scales to 3B parameters
- Swin-UNETR (2022) — Swin Transformer for 3D medical segmentation
- ConvNeXt (2022) — CNN adopting transformer design choices
- BEiT (2021) — BERT-style pre-training for vision transformers
Sources
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows — original paper by Liu et al. (ICCV 2021, Best Paper)
- Swin Transformer V2: Scaling Up Capacity and Resolution — Liu et al. (CVPR 2022)
- Swin Transformer — Wikipedia
- PyTorch Swin Transformer pretrained models
- Microsoft Swin Transformer GitHub