Vision Transformers: ViT and Swin

Attention from Scratch built the Transformer for sentences. This post applies it to images. The idea sounds almost too simple: cut the image into patches and treat each patch as a word. It works, but with two catches: Transformers need much more training data than CNNs, and their cost explodes with image resolution. The Swin Transformer addresses the second catch by bringing back some of the structure of a CNN.

Along the way, a real Vision Transformer runs in your browser, so you can look at what it attends to.

1. An Image Is Worth 16 × 16 Words

The Vision Transformer (ViT) (Dosovitskiy et al., 2021) changes as little as possible about the Transformer encoder:

Patches. A 224 × 224 image is cut into 14 × 14 = 196 patches of 16 × 16 pixels.
Patch embedding. Each patch is flattened (16 · 16 · 3 = 768 numbers) and multiplied by a learned matrix $E$ . This is exactly a convolution with a 16 × 16 kernel and stride 16, and that is how it is implemented.
[CLS] token and positions. A learned classification token is prepended, and a learned position embedding is added to every token, because attention by itself ignores order (see the previous post).
Transformer encoder. $L$ identical layers of multi-head self-attention and an MLP with GELU, both pre-norm with residual connections.
Head. The final [CLS] output goes to a classifier.

z_{0} = [x_{class}; x_{p}^{1} E; \dots; x_{p}^{N} E] + E_{pos}, z_{ℓ}^{'} = MSA (LN (z_{ℓ - 1})) + z_{ℓ - 1}, z_{ℓ} = MLP (LN (z_{ℓ}^{'})) + z_{ℓ}^{'}

The paper defines three sizes. The name ViT-L/16 means Large, with 16 × 16 patches. Smaller patches give more tokens and a finer grid, at a higher cost:

Model	Layers	Width $D$	MLP size	Heads	Parameters
ViT-Base	12	768	3072	12	86 M
ViT-Large	24	1024	4096	16	307 M
ViT-Huge	32	1280	5120	16	632 M

How much image structure is left?

Very little, by design. A CNN builds locality (a 3 × 3 kernel sees neighbours), the 2D neighbourhood and translation equivariance into every layer. In ViT, as the authors put it, only the MLP layers are local and translation-equivariant. The 2D structure is used only when cutting the patches, and when the position embeddings are resized for a different image resolution. The position embeddings start out random; that patch 15 sits directly below patch 1 has to be learned. And it is learned:

2. A Vision Transformer in Your Browser

Below, DeiT-Tiny runs in plain JavaScript: the same architecture as ViT-Base with 192 instead of 768 channels and 3 heads (5.7 M parameters, 72.2% ImageNet top-1). The weights are the original ones, downloaded from Hugging Face. I checked that the code reproduces PyTorch’s outputs: the logits match to four decimals on the same input. Pick an image, press the button, then explore:

Things to try:

Layer 1 vs. layer 12. Click a patch and step through the heads of layer 1. Some heads attend only to the patch itself and its neighbours, much like a small convolution; others spread over the whole image in the very first layer. A CNN cannot do that.
The distance chart. It reproduces an analysis from the ViT paper: the average distance (in pixels) over which each head gathers information. In low layers the heads range from local to global; in deep layers they are all global, close to the dashed line of a head that attends uniformly to every patch.
[CLS] in the last layer often looks at background. On the cat, the strongest patches are a few spots of sofa, not the cat. Attention shows where one layer reads from; by layer 12 the object information has already been mixed into many tokens. Large ViTs even recycle low-information background patches as scratch space for global information (Darcet et al., 2024).
Rollout. Attention rollout (Abnar & Zuidema, 2020) multiplies the attention maps of all layers (adding the residual connection) to estimate how much each patch flows into [CLS] in total. On the street photo it highlights the dog, the predicted class.
The mistakes. The black cat comes out as schipperke, a small black dog breed. DeiT-Tiny is the smallest model of its family and was trained on ImageNet-1k only. (The probabilities differ by a few points from Python because the browser resizes the image slightly differently.)

Attention maps are not explanations. They are easy to show and often look plausible, but a high attention weight does not prove that a patch caused the decision. The cleanest object maps come from self-supervised ViTs such as DINO (Caron et al., 2021), not from supervised classifiers like this one.

3. The Catch: Data

ViT’s first results were disappointing. Trained on ImageNet-1k (1.3 M images), ViT-Base was a few points worse than a ResNet of similar size, and the larger ViT-Large was worse than ViT-Base. Only with more pre-training data did the ranking flip:

With JFT-300M (303 M images, Google-internal), ViT-H/14 reached 88.55% on ImageNet, better than the best CNNs of the time (BiT-L, a ResNet-152×4, 87.54%), with about 4× less pre-training compute (2.5 k vs 9.9 k TPUv3-core-days). The paper’s summary: large-scale training trumps inductive bias.

The explanation is the inductive bias from the start of this post. A CNN assumes locality and translation equivariance, which is a good assumption for images, so it needs fewer examples. A ViT must learn these regularities from data. With little data, the assumptions win; with a lot of data, learning them (and anything else useful) wins. In a controlled experiment on random subsets of JFT (few-shot evaluation), ResNets were better at 9 M images and ViTs better from 90 M on.

DeiT: the same ViT, trained on ImageNet only

Is it really the data, or the training? DeiT (Touvron et al., 2021) trained the unchanged ViT-B architecture on ImageNet-1k only, on one machine with 8 GPUs, in 53 hours, and reached 81.8%, compared with 77.9% in the ViT paper. The difference is the training recipe: AdamW, 300 epochs, and heavy augmentation and regularisation. Removing any single ingredient shows how much each one carries:

Augmentations like Mixup and CutMix (which blend or paste images and their labels) effectively create new training examples; random erasing blanks out a random rectangle; stochastic depth randomly skips residual blocks during training. Without either of the last two, the model did not train at all with the default hyper-parameters.

DeiT added one more idea: distillation through attention. A second special token learns to reproduce the predictions of a CNN teacher:

With distillation, DeiT-B reached 83.4%, and 85.2% with longer training and 384-pixel images, all from ImageNet-1k. So “ViTs need 300 M images” is too strong: they need more data or stronger augmentation and regularisation than CNNs, and a CNN teacher can pass on some of the missing inductive bias.

Model (ImageNet-1k only)	Parameters	Top-1
ViT-B/16 (ViT paper, 384 px)	86 M	77.9%
DeiT-Ti (in the demo above)	5 M	72.2%
DeiT-S	22 M	79.8%
DeiT-B	86 M	81.8%
DeiT-B, distilled	87 M	83.4%
DeiT-B, distilled, 1000 epochs, 384 px	87 M	85.2%

4. The Second Catch: Resolution

Global attention compares every token with every other one. For an image of side $d$ and patch size $P$ , there are $n = (d / P)^{2}$ tokens and $n^{2} = (d / P)^{4}$ pairs: double the image side and the attention cost grows 16×. For classification at 224 × 224 with 16 × 16 patches this is fine (196 tokens). But detection and segmentation want high resolution and small patches:

At 1024 × 1024 with 4-pixel tokens (the first stage of a Swin model), one layer of global attention would need about 830 billion multiply-adds, and a single attention map would take 17 GB of memory. Restricting attention to local windows makes the cost linear in the number of pixels, but then distant windows never talk to each other. The Swin Transformer is about solving that.

5. Swin Transformer: Shifted Windows

The Swin Transformer (Liu et al., 2021) makes two changes to ViT, both familiar from CNNs.

1. A hierarchy. Instead of one 14 × 14 grid, Swin starts with small 4 × 4-pixel tokens (56 × 56 of them) and merges neighbouring tokens between stages, like the stride-2 layers of a CNN:

2. Attention inside windows, shifted every other layer. Self-attention is computed only inside non-overlapping windows of $M \times M = 7 \times 7$ tokens. In the next layer, the window grid is shifted by $⌊ M /2 ⌋ = 3$ tokens, so each new window straddles the borders of the old ones. Click through the four views:

The cost of one layer, for an $h \times w$ map of $C$ channels:

Ω (MSA) = 4 h w C^{2} + 2 (h w)^{2} C, Ω (W-MSA) = 4 h w C^{2} + 2 M^{2} h wC

The first term (the projections) is the same; the second becomes linear in $h w$ because each token attends to only $M^{2} = 49$ others. The cyclic shift (third view) is the implementation trick that keeps the shifted layer as fast as the regular one: instead of 9 windows of uneven size, the map is rolled so there are again 4 full windows, and an attention mask stops tokens from attending to neighbours that only appear adjacent because of the roll.

One more detail: Swin replaces ViT’s absolute position embeddings with a relative position bias $B$ added inside every attention layer, $softmax (Q K^{⊤} / d + B) V$ , where $B$ depends only on the offset between two tokens in the window (a small table of $(2 M - 1)^{2} = 169$ learned values per head).

Does each idea matter?

The Swin paper ablated both on Swin-T:

Variant of Swin-T	ImageNet top-1	COCO box AP	ADE20K mIoU
windows, no shifting	80.2	47.7	43.3
shifted windows (default)	81.3	50.5	46.1
no position information	80.1	49.2	43.8
absolute position embedding (as ViT)	80.5	49.0	43.2
relative position bias (default)	81.3	50.5	46.1

Shifting matters most for the dense tasks (+2.8 box AP, +2.8 mIoU), which depend on connections across window borders.

Results

Model	Parameters	FLOPs	ImageNet-1k top-1
DeiT-S	22 M	4.6 G	79.8%
Swin-T ( $C$ = 96, layers 2-2-6-2)	29 M	4.5 G	81.3%
DeiT-B	86 M	17.5 G	81.8%
Swin-B	88 M	15.4 G	83.5%
EfficientNet-B7 (600 px)	66 M	37.0 G	84.3%
Swin-B (384 px)	88 M	47.0 G	84.5%

With ImageNet-22k pre-training, Swin-L reaches 87.3%. More important than classification: Swin is a drop-in backbone for detection and segmentation. In the same Cascade Mask R-CNN detector, replacing a ResNet-50 with Swin-T raises COCO box AP from 46.3 to 50.5 at similar FLOPs. The largest system set records at the time: 58.7 box AP / 51.1 mask AP on COCO and 53.5 mIoU on ADE20K.

6. CNNs Strike Back: ConvNeXt

If Swin borrowed CNN ideas, how much of its advantage comes from attention itself? ConvNeXt (Liu et al., 2022) took a ResNet-50 and changed it, one step at a time, toward Swin’s design choices, while keeping only convolutions:

The largest single gain (+2.7 points) is the training recipe, with no change to the network. Then come macro choices copied from Swin: a 4 × 4 “patchify” stem, Swin’s stage ratios and widths, 7 × 7 depthwise convolutions (matching Swin’s 7 × 7 windows), inverted bottlenecks (like the Transformer MLP), GELU, fewer normalisations, LayerNorm. The result, ConvNeXt-T, reaches 82.1% at 4.5 GFLOPs, above Swin-T’s 81.3% at the same cost, and runs slightly faster.

The lesson is not “CNNs beat Transformers” or the reverse. Many of the gains credited to Vision Transformers came from training recipes and design details that transfer to CNNs. What attention offers on top is flexibility: it scales well with data, handles any number of tokens, and mixes easily with text tokens, which is why most large vision models today are Transformers.

7. Where Vision Transformers Went Next

Self-supervised pre-training removes the need for huge labelled datasets. DINO (Caron et al., 2021) trains a ViT to match its own outputs on different crops; its [CLS] attention maps segment objects without ever seeing a mask. MAE (He et al., 2022) hides 75% of the patches and trains the ViT to reconstruct them, which only works because a Transformer can simply drop the hidden tokens.
Images and text together. CLIP (Radford et al., 2021) trains a ViT image encoder against a text encoder on 400 M image–caption pairs.
Dense prediction. DETR (see Object Detection) uses a Transformer decoder with cross-attention from object queries to image features; the Segment Anything model (Kirillov et al., 2023) uses an MAE-pre-trained ViT-H as its image encoder.

8. Common Claims, Checked

Claim	Verdict
”The Vision Transformer works entirely without convolutions.”	Almost. The patch embedding (16 × 16 kernel, stride 16, 768 outputs) is a convolution and is implemented as one. After that, there are none.
”When trained only on the 14 M images of ImageNet-21k, ViT falls behind the ResNet.”	The gap is at ImageNet-1k (1.3 M images). On ImageNet-21k the ViTs are in the same range as the BiT ResNets, and ViT-L/16 reaches 85.3%. ViT pulls ahead with JFT-300M.
”Vision Transformers need hundreds of millions of images.”	Only with a plain training recipe. DeiT reached 81.8% (83.4% distilled) with ImageNet-1k alone, using strong augmentation, regularisation and 300 epochs.
”ViT’s training time is surprisingly short.”	Short relative to CNNs of the same accuracy: ViT-L/16 on JFT took 0.68 k TPUv3-core-days vs 9.9 k for BiT-L. In absolute terms ViT-H still took 2.5 k core-days, far beyond a typical budget.
”Global attention on a $d \times d$ image costs $O (d^{4})$ ; windows avoid the exponential growth.”	The $O (d^{4})$ is right ( $n = d^{2} / P^{2}$ tokens, $n^{2}$ pairs). But the growth is polynomial (quadratic in the number of tokens), not exponential. Windows make it linear in the number of pixels.
”Swin performs similar to EfficientNet on ImageNet-1k and vastly outperforms ViT.”	Similar to EfficientNet: yes (Swin-B 84.5% at 47 GFLOPs vs EfficientNet-B7 84.3% at 37 GFLOPs). “Vastly outperforms ViT” compares with the 77.9% ViT trained with the old recipe; against DeiT, trained on the same data the same way, the margin is 1.5–1.7 points.
”On COCO, Swin outperforms YOLOv4.”	The 58.7 AP system is a Swin-L (284 M parameters) inside an HTC++ cascade detector with multi-scale testing; Scaled-YOLOv4-P7 reported 55.8 AP, but YOLO is built for real-time speed. The fair evidence is the backbone swap: same detector, ResNet-50 → Swin-T, 46.3 → 50.5 AP.
”Attention maps show what the network looks at (emergent localisation).”	The clean object maps of the DINO paper come from self-supervised training. In supervised ViTs like the demo above, [CLS] attention is often noisy and lands on background patches, and attention weights alone do not prove which patches caused a decision.

Summary

Idea	Why
Patches as tokens + learned position embeddings	Reuse the Transformer unchanged; the 2D structure is learned, not assumed
Large pre-training data, or strong augmentation (DeiT)	Replace the inductive bias a CNN gets for free
Distillation token	Let a CNN teacher pass on its bias through its predictions
Window attention	Cost linear, not quadratic, in the number of pixels
Shifted windows	Let information cross window borders
Hierarchy + patch merging	Multi-scale features for detection and segmentation, like a CNN
ConvNeXt	Much of the gain was the recipe; a modernised CNN matches Swin

Next: Learning Without Labels: Self-Supervised Vision: how SimCLR, MoCo, BYOL, DINO and MAE learn image features without a single label, and why DINO’s attention finds objects on its own.

References

Vision Transformers

Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. (2021). An image is worth 16×16 words: Transformers for image recognition at scale. ICLR.
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jégou, H. (2021). Training data-efficient image transformers & distillation through attention. ICML.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021). Swin Transformer: Hierarchical vision Transformer using shifted windows. ICCV.
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., & Xie, S. (2022). A ConvNet for the 2020s. CVPR.

Baselines and training

Kolesnikov, A., Beyer, L., Zhai, X., et al. (2020). Big Transfer (BiT): General visual representation learning. ECCV.
Tan, M., & Le, Q. V. (2019). EfficientNet: Rethinking model scaling for convolutional neural networks. ICML.
Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2018). mixup: Beyond empirical risk minimization. ICLR.
Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., & Yoo, Y. (2019). CutMix: Regularization strategy to train strong classifiers with localizable features. ICCV.
Huang, G., Sun, Y., Liu, Z., Sedra, D., & Weinberger, K. Q. (2016). Deep networks with stochastic depth. ECCV.
Wang, C.-Y., Bochkovskiy, A., & Liao, H.-Y. M. (2021). Scaled-YOLOv4: Scaling cross stage partial network. CVPR.

Looking inside, and what came next

Abnar, S., & Zuidema, W. (2020). Quantifying attention flow in Transformers. ACL.
Darcet, T., Oquab, M., Mairal, J., & Bojanowski, P. (2024). Vision Transformers need registers. ICLR.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021). Emerging properties in self-supervised vision Transformers. ICCV.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., & Girshick, R. (2022). Masked autoencoders are scalable vision learners. CVPR.
Radford, A., Kim, J. W., Hallacy, C., et al. (2021). Learning transferable visual models from natural language supervision. ICML.
Kirillov, A., Mintun, E., Ravi, N., et al. (2023). Segment anything. ICCV.

Notes

Explorer