What the position embeddings learned

Click (or drag over) a patch position.

Loading the 197 position embeddings of DeiT-Tiny (151 kB, read straight out of the model file)…

ImageNet accuracy vs. size of the pre-training data
ViT-B/16 (86 M)ViT-L/16 (307 M)ViT-H/14 (632 M)DeiT-B (86 M), ImageNet-1k only

ViT numbers: Dosovitskiy et al. (2021), Table 5, fine-tuned on ImageNet at 384 px. DeiT: Touvron et al. (2021), 224 px; the ringed point is DeiT-B with distillation from a CNN teacher.

DeiT-B on ImageNet-1k: remove one ingredient of the training recipe

Touvron et al. (2021), Table 8; accuracy after training at 224 × 224. Two variants did not train at all with the default hyper-parameters (marked ✕).

The cost of global attention vs. window attention
global MSAwindow MSA, M = 7

Shifted windows, token by token

8 × 8 tokens, windows of M × M = 4 × 4, shift of ⌊M/2⌋ = 2 (Swin uses 7 × 7 windows and a shift of 3). Click a token.

Modernising a ResNet-50, one step at a time (ImageNet-1k top-1)

Liu et al. (2022), Table 10 (mean of 3 runs). Each row adds one change to the row above; all models ≈ 4–5 GFLOPs. Grey bars: steps that were kept even though they did not help on their own (they made the next step possible).