Self-Supervised Lab

Two random views of one image: the only "label" self-supervision gets

original, with the two crops

InfoNCE on a circle: what the temperature does

anchorpositive (other view)negatives (other images)

temperature τ

angle between the two views

One SimCLR batch: 4 images, 8 views, trained live with NT-Xent

temperature τ

Click any point to make it the anchor. Here the embeddings themselves are the trainable parameters (2-d, on the unit circle) instead of the outputs of a network, so you can watch the loss act on them directly. Loss and gradients are the same as PyTorch autograd on F.cross_entropy of the scaled cosine matrix (checked to 1e-12).

Where the [CLS] token looks, last layer

show only the patches holding 60% of the attention

SimSiam with and without the stop-gradient (ResNet-18, CIFAR-10, no labels)

Pre-trained without labels, then evaluated with few labels (CIFAR-10 test accuracy)

Frozen ImageNet features on CIFAR-10: supervised vs. self-supervised Vision Transformers