Learning Without Labels: Self-Supervised Vision

Vision Transformers ended with a problem: a ViT needs far more training data than a CNN, and more data usually means more labels. Labelling is the expensive part. Photos are cheap, but someone has to say what each one shows, box every object, or outline every pixel. Self-supervised learning removes that step: the training signal comes from the images themselves. The network learns general-purpose features from unlabelled photos, and labels are needed only at the end, in small numbers, for the task you actually care about.

This post follows the idea from its first attempts (puzzles the network has to solve) through contrastive learning (SimCLR, MoCo), methods without negative examples (BYOL, SimSiam, Barlow Twins), to the two approaches behind most of today’s large vision encoders: DINO and masked autoencoders (MAE). Along the way, I train some of these methods myself on CIFAR-10, show what happens when the trick that prevents collapse is removed, and compare frozen features of supervised and self-supervised Vision Transformers.

1. What We Want: Features That Transfer

A self-supervised method is judged by what it leaves behind: an encoder, a network that turns an image into a feature vector. The training task itself is thrown away. To measure how good the features are, the encoder is trained without labels and then evaluated on a labelled dataset in one of three ways:

Protocol	What is trained on the labels	What it measures
kNN	nothing: classify each test image by the labels of its nearest training images in feature space	whether similar images already sit close together
Linear probe	one linear layer on top of the frozen encoder	whether the classes are linearly separable in feature space
Fine-tuning	the whole network, starting from the pre-trained weights	how good a starting point the weights are

A fourth measure is the most practical one: how many labels are needed. If the features are good, a linear classifier trained on 1% of the labels gets close to a network trained on all of them.

The reference point is a supervised network. A ResNet-50 trained on the 1.28 M labelled images of ImageNet reaches about 76% top-1 accuracy. For years, the question for self-supervised learning was how close a linear probe on its frozen features could get to that number without using a single label during pre-training.

2. First Attempts: Pretext Tasks

The first idea was to invent a task whose answer is known for free, the pretext task, and hope that solving it requires understanding the image:

Context prediction (Doersch et al., 2015): given two patches of an image, predict where the second one sits relative to the first (one of 8 positions).
Jigsaw puzzles (Noroozi & Favaro, 2016): cut the image into 3 × 3 tiles, shuffle them with one of 1,000 fixed permutations, and predict which one.
Colourisation (Zhang et al., 2016): predict the colours from the grey image.
Inpainting (Pathak et al., 2016): fill in a missing region (the “context encoder”).
Rotation (Gidaris et al., 2018): rotate by 0°, 90°, 180° or 270°, predict which. To tell which way is up, the network has to know what a cat or a tree looks like.

These tasks have two problems. The first is shortcuts. Doersch et al. found that their network located patches partly by chromatic aberration, the slight colour fringing that camera lenses produce towards the image border. It is a perfect cue for “where in the image is this patch”, and it has nothing to do with objects. Jigsaw networks likewise exploit edges that continue across tile borders, which is why the tiles are separated by random gaps.

The second problem is that the features become specialised for the pretext task. The last layers learn rotation angles or permutations, not objects; the most useful features are usually found in the middle of the network. In a large comparison, Kolesnikov et al. (2019) found that the best of these methods (rotation prediction with a 16× wider RevNet-50) reached 55.4% ImageNet linear-probe accuracy, far below the 76% of supervised training. Solving the pretext task well was not enough.

3. Contrastive Learning: Same Image or Not?

The idea that closed the gap is simpler: instead of a hand-made puzzle, ask the network to recognise the same image under different disguises. Take an image, make two random views of it (crops, colour changes, blur), and train the network so that the two views get similar embeddings while views of other images get dissimilar ones. Every image is its own class (“instance discrimination”, Dosovitskiy et al., 2014; Wu et al., 2018).

The views define what the network learns to ignore. If two crops of the same photo must match, the representation has to be stable under position, scale and colour changes, and the only way to match a crop of a cat’s ear with a crop of its body is to know that both belong to a cat.

The InfoNCE loss

The embeddings $z$ are normalised to unit length, so the similarity of two embeddings is their cosine, $sim (z_{i}, z_{j}) = z_{i}^{⊤} z_{j}$ . For an anchor view $z_{i}$ with its positive $z_{j}$ (the other view of the same image) and a set of negatives $z_{k}$ (views of other images), the loss is

L_{i} = - lo g \frac{exp ( sim ( z _{i} , z _{j} ) / τ )}{\sum _{k \neq = i} exp ( sim ( z _{i} , z _{k} ) / τ )}

This is softmax cross-entropy, exactly the loss of a classifier, where the “classes” are the other images in the batch and the correct class is the positive. Van den Oord et al. (2018) named it InfoNCE and showed that $lo g N - L$ is a lower bound on the mutual information between the two views ( $N$ = number of candidates), one reason why more negatives help.

The temperature $τ$ is not a detail. The gradient pushes each negative away in proportion to its softmax probability, so a small $τ$ concentrates the push on the few hard negatives that already look most like the anchor, and a large $τ$ spreads it evenly:

The calculator above uses the same formula as PyTorch’s F.cross_entropy on the scaled cosines; for the default layout the two agree to 7 decimal places for τ = 0.05, 0.1, 0.5 and 1.

4. SimCLR: A Simple Recipe

SimCLR (Chen et al., 2020) put these pieces together in the simplest possible way:

What “pull together, push apart” does to the embeddings is easiest to see on a small batch. Below, 4 images give 8 views, embedded on a circle (2-d instead of 128-d). Press train and the NT-Xent loss moves them: each view is pulled towards its partner and pushed away from the other six, most strongly from the nearest ones. Wang and Isola (2020) showed that this is exactly what the loss optimises in the limit of many negatives: alignment (matching views close together) and uniformity (all images spread evenly over the sphere, so no information is thrown away).

For a batch of $N$ images, both views of every image go through the same encoder $f$ and a small projection head $g$ . Each of the $2 N$ views has one positive and $2 N - 2$ negatives: the views of all other images in the batch. The loss is InfoNCE, averaged over all $2 N$ anchors (the paper calls it NT-Xent). Three findings made it work:

The augmentations matter most. Random cropping alone is not enough; the network can then match two crops by their colour histogram, which is the same for both. Adding strong colour distortion forces it to use shape and texture. Crop plus colour jitter was by far the most effective pair.
The projection head. The loss is computed on $z = g (h)$ , but the representation that is kept is $h$ , the layer before the head. A linear probe on $h$ was more than 10 points better than one on $z$ : the head throws away information (like colour) that the loss tells it to ignore but that downstream tasks may need.
Large batches and long training. More negatives help. SimCLR used batches of 4,096 images (8,190 negatives per anchor) on TPUs, and up to 1,000 epochs.

The temperature matters too: in SimCLR’s ablation, τ = 0.1 gave the best linear-probe accuracy, and τ = 1 was more than 6 points worse.

With a ResNet-50, SimCLR reached 69.3% linear-probe accuracy on ImageNet, and a 4× wider ResNet-50 reached 76.5%, matching a supervised ResNet-50. Fine-tuned with only 1% of the labels (about 13 images per class), the standard ResNet-50 reached 48.3% top-1, against 25.4% for the same network trained from scratch on those labels.

5. MoCo: Many Negatives, Small Batches

A batch of 4,096 images is out of reach for most people. MoCo (Momentum Contrast; He et al., 2020) decouples the number of negatives from the batch size. The negatives come from a queue that stores the embeddings (“keys”) of the last 65,536 images; each batch adds its keys and the oldest are dropped.

The catch: keys from earlier batches were computed by an earlier version of the encoder, and if the encoder changes quickly, old and new keys are not comparable. MoCo therefore computes keys with a separate momentum encoder, a slowly moving average of the query encoder that is not trained by backpropagation:

θ_{k} \leftarrow m θ_{k} + (1 - m) θ_{q}, m = 0.999

The ablation in the paper shows how important the slowness is: with $m = 0.9$ accuracy dropped by about 4 points, and with $m = 0$ (keys from a copy of the current encoder) training failed. MoCo v1 reached 60.6% with a ResNet-50 and a batch of 256. MoCo v2 (Chen et al., 2020) added SimCLR’s projection head and stronger augmentation and reached 71.1%, still with batches of 256. On object detection and segmentation (PASCAL VOC, COCO), MoCo’s features already transferred better than supervised ImageNet pre-training in most of the tasks tested.

6. Without Negatives: Why Doesn’t It Collapse?

Negatives are there for a reason. If the loss only asked the two views to agree, there is a trivial solution: map every image to the same vector. The two views then always agree perfectly, the loss is at its minimum, and the features are useless. This is called collapse, and every method needs something that rules it out:

BYOL (Grill et al., 2020) uses no negatives at all. An “online” network predicts, through an extra small MLP called the predictor, the output of a “target” network for the other view; the target is a momentum average of the online network, as in MoCo, and receives no gradient. It reached 74.3% with a ResNet-50, better than SimCLR, and was much less sensitive to batch size.

SimSiam (Chen & He, 2021) showed that even the momentum average is not needed. Both branches share the same weights; one branch has a predictor $h$ , and the other branch is treated as a constant through a stop-gradient:

L = - \frac{1}{2} cos (h (z_{1}), sg (z_{2})) - \frac{1}{2} cos (h (z_{2}), sg (z_{1}))

Without the stop-gradient, SimSiam collapses: the paper reports a linear-probe accuracy of 0.1% (chance level on ImageNet). Removing the predictor collapses it too. Why the stop-gradient works is still debated; Chen and He interpret SimSiam as an alternating optimisation (similar to k-means), where the stop-gradient keeps one set of variables fixed while the other is updated.

I trained SimSiam on CIFAR-10 with and without the stop-gradient (ResNet-18, details in §7). The “spread” below is the standard deviation of the normalised embeddings across the batch, scaled so that 1 means spread over the whole sphere and 0 means all images map to the same point:

Without the stop-gradient, the loss drops to −0.98 in the second epoch and ends at −0.99: the two branches agree almost perfectly. But the spread of the embeddings falls from 0.39 to 0.11 at the same time, and the kNN accuracy of the backbone stays at 31%, the level of an untrained network. The network has found the trivial solution. With the stop-gradient, the same network spreads its outputs over the whole sphere (spread 1.00 at the end) and its features reach 70% kNN accuracy. The only difference between the two runs is one .detach().

Barlow Twins (Zbontar et al., 2021) takes a third route, borrowed from neuroscience (Horace Barlow’s redundancy-reduction principle). It computes the cross-correlation matrix $C$ between the outputs of the two views over the batch, dimension by dimension, and pushes it towards the identity:

L = i \sum (1 - C_{ii})^{2} + λ i \sum j \neq = i \sum C_{ij}^{2}

The diagonal term makes each output dimension agree between views (invariance); the off-diagonal term makes different dimensions carry different information, which rules out collapse to a single direction. It reached 73.2% and, unusually, improved with very high-dimensional outputs (8,192). VICReg (Bardes et al., 2022) splits the same idea into explicit variance, invariance and covariance terms.

7. Trying It: SimCLR and SimSiam on CIFAR-10

To see these methods in action, I pre-trained a ResNet-18 on the 50,000 CIFAR-10 training images without their labels, using SimCLR (τ = 0.5) and SimSiam, for 100 epochs each, and then evaluated the frozen encoder. For reference, the same network was trained with all labels (same augmentations, 100 epochs). To fit a laptop GPU (Apple M1 Pro), the first convolution has stride 2, which makes training 3× faster at some cost in accuracy; one seed per method.

Encoder	kNN · linear probe (all labels)	Linear probe with 500 · 5,000 labels
SimCLR, 100 epochs, no labels	76.1% · 79.4%	71.2% · 75.7%
SimSiam, 100 epochs, no labels	69.8% · 74.7%	66.3% · 72.5%
SimSiam without stop-gradient (collapsed)	30.9% · 35.0%	24.1% · 30.4%
Trained from scratch on the labelled images only		47.2% · 74.4%
Supervised with all 50,000 labels (reference)	91.2% · 91.0%

(Few-label columns: mean of 3 random draws of labelled images; the draws differed by at most 2.1 points.)

The headline is the few-label column. With only 500 labelled images, a linear classifier on SimCLR features reaches 71.2%, while the same ResNet-18 trained from scratch on those 500 images reaches 47.2%: 24 points gained from 50,000 images nobody labelled. With 5,000 labels the advantage has shrunk to about a point, and with all labels, end-to-end supervised training is clearly better (91% vs 79%). That is the practical case for self-supervised learning: many images, few labels.

The absolute numbers are well below published CIFAR-10 results (around 90% linear accuracy for SimCLR with a ResNet-18), for clear reasons: 100 epochs instead of 800–1,000, the 3× cheaper stride-2 stem, and batches of 512. SimSiam is behind SimCLR at 100 epochs; it is known to need longer training (its paper reports CIFAR-10 results after 800 epochs).

The temperature matters. A shorter sweep (20 epochs each) over SimCLR’s temperature gave:

τ	kNN accuracy	Linear probe
0.05	56.9%	63.0%
0.1	62.2%	66.5%
0.2	60.0%	65.3%
0.5	54.1%	62.0%
1.0	48.1%	56.7%

As in SimCLR’s own ablation, τ = 0.1 was best, and τ = 1 was 14 points worse on kNN. The 100-epoch run above used τ = 0.5, chosen before the sweep; with τ = 0.1 it would likely have done better.

8. DINO: Self-Distillation with Vision Transformers

The methods so far were designed for ResNets. DINO (Caron et al., 2021), “self-distillation with no labels”, was designed for Vision Transformers:

A teacher and a student network have the same architecture. Both output a probability distribution over $K = 65, 536$ dimensions (softmax), and the student is trained with cross-entropy to match the teacher on a different view of the same image. As in BYOL, the teacher is a momentum average of the student and receives no gradient. Two details prevent collapse, and they work against each other:

Centring: a running mean $c$ of the teacher’s outputs is subtracted before the softmax, so no single dimension can dominate for every image.
Sharpening: the teacher’s softmax uses a low temperature (0.04, against 0.1 for the student), so its output is peaked rather than uniform.

Centring alone would make the output uniform, sharpening alone would make one dimension win; together they stay in between. DINO also uses multi-crop (from SwAV, Caron et al., 2020): the teacher sees only two large “global” crops, while the student also sees several small “local” crops, so it has to infer the whole image from a part of it.

With a ViT-S/16, DINO reached 77.0% linear-probe and 74.5% kNN accuracy on ImageNet; a ViT-B/8 reached 80.1% linear. The kNN number is the remarkable one: without any training on labels, nearest neighbours in DINO’s feature space are almost as good as a linear classifier.

The most striking result is visual. The last-layer attention of DINO’s [CLS] token outlines objects, although the network never saw a label or a mask. Compare it with a supervised ViT of the same size:

The supervised DeiT-S scatters its attention: to name one of 1,000 classes, a few discriminative spots are enough. DINO’s heads concentrate on the cat’s head and on the three dogs, and different heads pick different objects or parts. DINOv2 (§10) is sharper still, but some of its heads put a large share of their attention on a handful of isolated background patches. Darcet et al. (2024) showed that large ViTs recycle such low-information patches as internal scratch space, and fixed it by adding a few extra “register” tokens.

9. MAE: Hide Most of the Image, Then Reconstruct It

In language, the dominant self-supervised method is masked language modelling (BERT): hide 15% of the words and predict them. The same idea for images, masked image modelling, had been tried before but only took off with Vision Transformers, because a ViT can simply drop the hidden patches. The masked autoencoder (MAE; He et al., 2022) does exactly that:

Two choices make it work:

A very high masking ratio, 75%. Words carry a lot of information each; neighbouring image patches are highly redundant, so hiding 15% of an image can be undone by interpolation. Hiding 75% forces the network to reason about what the missing parts are. In MAE’s ablation, 75% was the best ratio for both fine-tuning and linear probing.
An asymmetric design. The large encoder processes only the visible 25% of the patches, without mask tokens; a small, light decoder fills in the rest. This makes pre-training about 3× faster, which is what allowed huge ViTs to be trained this way.

The target is simply the pixels of the hidden patches (normalised per patch), and the loss is the mean squared error on the hidden patches only. Fine-tuned on ImageNet-1k, with no extra data, MAE reached 83.6% with ViT-B, 85.9% with ViT-L and 87.8% with ViT-H at 448 px, the best results at the time using only ImageNet-1k.

MAE has a different character from the contrastive methods, though. Its features are not organised so that a linear layer can read off the class: the MAE paper itself reports that linear probing lags behind contrastive methods, and that fine-tuning even only the last one or two blocks closes much of the gap. MAE learns how the parts of an image fit together, which is an excellent starting point for fine-tuning; contrastive and self-distillation methods learn which images are alike, which is what a frozen kNN or linear probe needs. The next section shows this difference directly.

10. Frozen Features Compared

How do these encoders behave on a dataset they never saw? I took seven publicly available ImageNet-pretrained Vision Transformers (from timm), kept them frozen, and evaluated their [CLS] features on CIFAR-10 (images resized to 224 × 224): kNN with no training at all, a linear probe on 2,000 labelled images per class, and a linear probe with only 5 labels per class (mean of 3 random draws).

Frozen encoder (pre-training)	kNN · linear probe	5 labels per class
ViT-S/16, random weights	30.4% · 40.1%	19.2%
DeiT-S/16 (ImageNet-1k labels)	91.6% · 92.2%	83.1%
DINO ViT-S/16 (no labels)	91.3% · 94.3%	77.6%
DeiT-B/16 (ImageNet-1k labels)	92.7% · 94.3%	84.1%
DINO ViT-B/16 (no labels)	94.3% · 95.9%	83.4%
MAE ViT-B/16 (no labels)	40.5% · 85.2%	43.7%
DINOv2 ViT-S/14 (142 M curated images, no labels)	96.3% · 96.6%	86.8%

Three observations. First, DINO matches supervised pre-training of the same size on a new dataset without having seen a single label: slightly behind on kNN for ViT-S, ahead for ViT-B, and ahead on every linear probe. With only 5 labels per class the supervised models are a little better, plausibly because ImageNet’s labelled classes include close relatives of CIFAR-10’s (cars, dogs, birds, ships).

Second, MAE’s frozen features are almost useless for kNN (40.5%, close to random weights) but reach 85.2% with a linear probe. Its features contain the information, but not organised so that similar images are near each other. This is exactly the difference between reconstruction and contrastive objectives described in §9, and why MAE is meant to be fine-tuned.

Third, DINOv2, the smallest model in the table, is the best on every measure: 96.3% kNN with no training at all. The larger and more carefully curated pre-training data shows.

DINOv2 (Oquab et al., 2024) is where this line of work led. It combines DINO’s self-distillation with an iBOT-style masked-token loss, and was trained on LVD-142M, 142 million images selected automatically from a larger web crawl to resemble curated datasets. Its largest model (ViT-g/14) reached 86.5% ImageNet linear-probe accuracy with frozen features, and the same frozen features work for depth estimation and segmentation with only a light head. This is why DINOv2 is a common choice today as a general-purpose image encoder, the role that ImageNet-supervised ResNets played a decade ago.

A different source of free supervision is text: CLIP (Radford et al., 2021) learns from 400 million image–caption pairs with the same InfoNCE loss as above, contrasting matching against non-matching captions. That is “weakly supervised” rather than self-supervised, since the captions are written by people, but it is the same contrastive machinery.

11. Common Claims, Checked

Claim	Verdict
”Self-supervised learning is a kind of unsupervised learning.”	In the sense of “no human labels”, yes. But it is trained exactly like supervised learning, with a loss and targets; the targets are just computed from the data. The name was chosen to make that difference clear.
”If the network solves the pretext task well, its features are good.”	No. Networks solve rotation or jigsaw tasks with shortcuts (chromatic aberration, tile edges), and their last layers specialise in the pretext task. The best pretext-task model in a large comparison reached 55.4% on ImageNet linear probing, against 76% supervised.
”Contrastive learning needs very large batches.”	SimCLR did (4,096). MoCo gets 65,536 negatives from a queue with batches of 256, and BYOL, SimSiam and Barlow Twins need no negatives at all.
”Without negative examples, the representation must collapse.”	Not with the right asymmetry. BYOL and SimSiam use a predictor and a stop-gradient; Barlow Twins decorrelates dimensions; DINO centres and sharpens. Remove the stop-gradient from SimSiam and it does collapse (§6).
”The output of the projection head is the learned representation.”	The representation is taken before the head. In SimCLR, a linear probe on it is more than 10 points better than on the head’s output.
”The temperature is a minor hyper-parameter.”	It decides how strongly the hardest negatives are weighted. In SimCLR’s ablation, τ = 1 was more than 6 points worse than τ = 0.1; in my CIFAR-10 sweep, 14 points worse on kNN (§7).
”Self-supervised features are worse than supervised ones.”	It depends on the protocol. On ImageNet linear probing they matched supervised ResNet-50 from 2020 (SimCLR 4×, BYOL). For detection, MoCo transferred better than supervised pre-training. With few labels, they win clearly: 71% vs 47% with 500 CIFAR-10 labels (§7).
”Masking 75% of an image leaves too little to learn from.”	The opposite: images are so redundant that low masking ratios make the task too easy. MAE’s ablation found 75% best.
”MAE features can be used frozen, like DINO features.”	Not well: MAE features need fine-tuning. Frozen on CIFAR-10, MAE ViT-B reached 40.5% kNN accuracy against DINO ViT-B’s 94.3% (§10).
”The attention maps of self-supervised ViTs are object segmentations.”	They often outline the salient objects, but they are unlabelled, not every object gets its own head, and larger models show background artifacts (§8). DINO’s paper measures the overlap with real masks; it is good, not perfect.

Summary

Idea	Why
Pretext tasks (rotation, jigsaw, colour, inpainting)	Labels for free, but shortcuts and task-specific features
Instance discrimination with InfoNCE	Every image is its own class; the views define what to ignore
Strong augmentation, projection head (SimCLR)	Block easy shortcuts; keep the features before the head
Queue + momentum encoder (MoCo)	Many negatives with small batches
Predictor + stop-gradient (BYOL, SimSiam)	No negatives, no collapse
Decorrelating dimensions (Barlow Twins, VICReg)	Rule out collapse by making every output dimension carry information
Self-distillation with centring + sharpening, multi-crop (DINO)	Strong frozen features; attention that finds objects
Masking 75% and reconstructing pixels (MAE)	Cheap pre-training of very large ViTs; best when fine-tuned

Next: From a 3D Point to a Pixel: Image Formation: how a camera turns the 3D world into an image, the geometry that the following posts on feature matching and 3D reconstruction build on.

References

Pretext tasks

Doersch, C., Gupta, A., & Efros, A. A. (2015). Unsupervised visual representation learning by context prediction. ICCV.
Noroozi, M., & Favaro, P. (2016). Unsupervised learning of visual representations by solving jigsaw puzzles. ECCV.
Zhang, R., Isola, P., & Efros, A. A. (2016). Colorful image colorization. ECCV.
Pathak, D., Krähenbühl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016). Context encoders: Feature learning by inpainting. CVPR.
Gidaris, S., Singh, P., & Komodakis, N. (2018). Unsupervised representation learning by predicting image rotations. ICLR.
Kolesnikov, A., Zhai, X., & Beyer, L. (2019). Revisiting self-supervised visual representation learning. CVPR.

Contrastive learning

Dosovitskiy, A., Springenberg, J. T., Riedmiller, M., & Brox, T. (2014). Discriminative unsupervised feature learning with convolutional neural networks. NeurIPS.
Wu, Z., Xiong, Y., Yu, S. X., & Lin, D. (2018). Unsupervised feature learning via non-parametric instance discrimination. CVPR.
van den Oord, A., Li, Y., & Vinyals, O. (2018). Representation learning with contrastive predictive coding. arXiv:1807.03748.
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020). A simple framework for contrastive learning of visual representations. ICML.
Wang, T., & Isola, P. (2020). Understanding contrastive representation learning through alignment and uniformity on the hypersphere. ICML.
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020). Momentum contrast for unsupervised visual representation learning. CVPR.
Chen, X., Fan, H., Girshick, R., & He, K. (2020). Improved baselines with momentum contrastive learning. arXiv:2003.04297.

Without negatives

Grill, J.-B., Strub, F., Altché, F., et al. (2020). Bootstrap your own latent: A new approach to self-supervised learning. NeurIPS.
Chen, X., & He, K. (2021). Exploring simple Siamese representation learning. CVPR.
Zbontar, J., Jing, L., Misra, I., LeCun, Y., & Deny, S. (2021). Barlow Twins: Self-supervised learning via redundancy reduction. ICML.
Bardes, A., Ponce, J., & LeCun, Y. (2022). VICReg: Variance-invariance-covariance regularization for self-supervised learning. ICLR.

Vision Transformers

Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., & Joulin, A. (2020). Unsupervised learning of visual features by contrasting cluster assignments (SwAV). NeurIPS.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021). Emerging properties in self-supervised vision Transformers. ICCV.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., & Girshick, R. (2022). Masked autoencoders are scalable vision learners. CVPR.
Bao, H., Dong, L., Piao, S., & Wei, F. (2022). BEiT: BERT pre-training of image Transformers. ICLR.
Oquab, M., Darcet, T., Moutakanni, T., et al. (2024). DINOv2: Learning robust visual features without supervision. TMLR.
Darcet, T., Oquab, M., Mairal, J., & Bojanowski, P. (2024). Vision Transformers need registers. ICLR.
Radford, A., Kim, J. W., Hallacy, C., et al. (2021). Learning transferable visual models from natural language supervision. ICML.

Datasets and tools

Krizhevsky, A. (2009). Learning multiple layers of features from tiny images. Technical report (CIFAR-10).
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jégou, H. (2021). Training data-efficient image Transformers & distillation through attention. ICML (DeiT, the supervised ViTs above).
timm for the pre-trained weights; the scripts for every experiment are in ssl (zip).

Notes

Explorer