Self-attention, one query at a time
Query word:

Why divide by √dk?
softmax(q·k)softmax(q·k / √dk)

Sinusoidal positional encoding
dimension 0 (fast) →dimension 63 (slow)

Rows: positions 0–99 (top to bottom). Columns: the 64 dimensions. blue = −1, orange = +1. The marked row is position .

Dot product PE(p) · PE(q) for every position q. It is largest at q = p, lower for distant positions, and depends only on the offset q − p: each sine/cosine pair contributes cos(ω(q − p)). Move p and the curve slides without changing shape.

Three small sentiment models, running in your browser
Colour = how much the attention model's [CLS] token attends to each word, in

The curves show each model's output after every word, i.e. on unfinished sentences like “the course was not”, which never appear in training, so the middle of a curve can jump around.

plain RNNLSTMattention
Words the models know

How much does a word far back affect the output? (at the start of training)
plain RNNLSTMattention

‖∂ output / ∂ word embedding‖, averaged over 200 sentences with 8 filler phrases (42–60 words), freshly initialised models. Log scale: each grid line is a factor of 100.