Attention from Scratch

Every network in this series so far has built in an assumption about which inputs belong together. A convolution assumes nearby pixels matter most: a 3×3 kernel only looks at its neighbours, and distant parts of the image meet only after many layers. A fully connected layer assumes nothing, and has to learn a separate weight for every input–output pair. This assumption is called the inductive bias.

Attention sits in between. Every input may interact with every other one, and the network decides from the data, at run time, which interactions matter. This post builds attention from the ground up, following the same route the idea took historically: from sentences to the Transformer. The next post applies it to images.

---

1. The Problem: Long Sentences and Recurrent Networks

Before attention, sequences were read by recurrent neural networks (RNNs): one word at a time, each step updating a hidden state $h_t=\tanh (W_x{x}_t+W_h{h}_{t-1}+b)$ that has to carry everything the network knows so far.

Two problems follow from this design:

1. Long paths. A word $k$ steps before the end influences the output only through $k$ repeated multiplications by $W_h$ (and the derivative of $\tanh$). During training, its gradient is multiplied by the same factors on the way back, so it shrinks (or grows) exponentially with the distance. This is the vanishing gradient problem (Bengio et al., 1994; Pascanu et al., 2013).
2. No parallelism. Step $t$ needs $h_{t-1}$, so a 50-word sentence takes 50 sequential steps, however many GPU cores are available.

An experiment you can check

To see the first problem directly, I trained three small models on the same task: decide whether a short course review is positive or negative. The sentences are generated from templates with an 89-word vocabulary, so the right answer is always known:

honestly the lecture was not very helpful in the winter semester according to my friends although the coffee was great

The label depends only on the first adjective and whether it is negated (not helpful → negative). Filler phrases push those words away from the end, and a distractor clause (although the coffee was great) adds a second sentiment word that must be ignored. Training sentences have 4 to 59 words. All three models were trained the same way (8,000 sentences, Adam, 20 epochs, 3 random seeds each; the training script reproduces the exported weights exactly):

Test set (1,000 sentences each)	plain RNN	LSTM	attention (2 layers)
no fillers, no distractor	52–55%	100%	100%
no fillers, with distractor	48–51%	100%	99.2–100%
4 filler phrases	49–53%	99.9–100%	99.8–100%
8 filler phrases (42–60 words)	48–52%	100%	99.7–100%
parameters	3,089	7,889	22,689

The plain RNN stays at chance level on every test set, even the short sentences. It is not that it handles short sentences and fails on long ones: because most training sentences are long, it never finds the signal at all. The chart shows why. It measures, before training, how strongly each input word can affect the output:

For the plain RNN, a word 40 positions back has a gradient about $10^{12}$ times smaller than the last word: training cannot tell that the early words matter. The LSTM (Hochreiter & Schmidhuber, 1997) starts with a decaying signal too, but its gated cell state $c_t=f_t\odot c_{t-1}+i_t\odot {\tilde{c}}_t$ is an additive path that the network can learn to keep open, and it solves the task. The attention model’s curve is flat: every word is one step away from the output.

So the honest summary is: the vanishing gradient is a real and measurable problem of plain RNNs, and LSTMs largely fixed it for accuracy. What attention adds on top is a constant path length and full parallelism, which is what made the much larger models of the last decade trainable.

Attention is older than the Transformer. It was introduced for RNN translation models (Bahdanau, Cho & Bengio, 2015): instead of squeezing the whole source sentence into the last hidden state, the decoder looks back at all encoder states and takes a weighted average. The Transformer (Vaswani et al., 2017) then asked: what if we drop the recurrence and keep only the attention?

---

2. Self-Attention, Step by Step

To understand a word, we often need other words: in the food was not good, the meaning of good depends on not. Self-attention lets every word gather information from the other words of the same sentence. It works like a soft lookup in a database:

Each word’s embedding $x_i$ is projected three times, with three learned matrices:

• a query $q_i=x_i{W}_Q$: what am I looking for?
• a key $k_i=x_i{W}_K$: what do I offer, as a match for a query?
• a value $v_i=x_i{W}_V$: what information do I pass on if I am chosen?

The query of word $i$ is compared with the key of every word $j$ by a dot product, the scores go through a softmax so they are positive and sum to 1, and the output is the weighted sum of the values. For all words at once, with the embeddings stacked as rows of $X$:

$$\text{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{Q{K}^{\top }}{\sqrt{d_k}}\right)V,Q=X{W}_Q,K=X{W}_K,V=X{W}_V$$

$Q{K}^{\top }$ is an $n\times n$ matrix of scores (row $i$: how much word $i$ attends to each word), and multiplying by $V$ ($n\times d_v$) mixes the value rows. Pick a query word below and follow the numbers. The embeddings and weights are small made-up values, so everything fits on screen:

Two interpretations help:

• Soft selection. If one score is much larger than the others, the softmax becomes nearly one-hot and attention simply copies the value of the best-matching key. Usually it is a blend.
• A data-dependent filter. A convolution mixes neighbours with fixed weights, the same for every image. Attention mixes all positions with weights computed from the input itself, different for every sentence or image.

---

3. Why Divide by $\sqrt{d_k}$?

If the entries of $q$ and $k$ have mean 0 and variance 1, their dot product $q\cdot k={\sum }_{m=1}^{d_k}{q}_m{k}_m$ is a sum of $d_k$ such products, so its variance is $d_k$ and its typical size is $\sqrt{d_k}$. With $d_k=64$, the scores have a standard deviation of 8, and the softmax of such numbers is almost one-hot. The gradient of the softmax, $p_i(1-p_i)$, is then close to zero for every key, and training stalls. Dividing by $\sqrt{d_k}$ brings the spread back to about 1:

---

4. Word Order: Positional Encoding

There is nothing in the attention formula that knows where a word is. Shuffle the input words in the calculator above and the output for each word is exactly the same, just in a different row. Mathematically, self-attention is permutation-equivariant: permuting the inputs permutes the outputs in the same way. For a sentence, the food was not good and not the food was good become indistinguishable.

The fix is to add a positional encoding to each embedding before the first layer: $x_i\leftarrow x_i+p_i$. The original Transformer used fixed sinusoids of different frequencies:

$$P{E}_{(pos,2i)}=\sin \left(\frac{pos}{10000^{2i/d}}\right),P{E}_{(pos,2i+1)}=\cos \left(\frac{pos}{10000^{2i/d}}\right)$$

Low dimensions oscillate quickly and tell close positions apart; high dimensions change slowly and encode coarse position, a bit like the digits of a binary counter:

The useful property is in the right-hand plot: $PE(p)\cdot PE(q)$ depends only on the offset $q-p$, so relative position is easy to read off. In practice there are three families:

Kind	How	Used in
Fixed sinusoidal	the formula above, added once	original Transformer
Learned absolute	one trainable vector per position	BERT, GPT-2, ViT
Relative	a bias or rotation that depends on $i-j$, inside every attention layer	Shaw et al. (2018), Swin (relative position bias), RoPE (Su et al., 2021) in most current language models

Vaswani et al. found learned and sinusoidal encodings gave nearly identical results. They chose sinusoids because they might extrapolate to longer sentences than those seen in training; later measurements showed that they do not do this well (Press et al., 2022), which is one reason relative schemes took over.

---

5. Multi-Head Attention

One attention pattern per word is limiting: good may need not (negation) and food (what is good) at the same time. Multi-head attention runs several attentions in parallel, each with its own, smaller projections:

$$\text{MultiHead}(X)=\mathrm{Concat}({\text{head}}_1,\dots ,{\text{head}}_h)W_O,{\text{head}}_i=\text{Attention}(X{W}_Q^{(i)},X{W}_K^{(i)},X{W}_V^{(i)})$$

In the original Transformer, $d_{\text{model}}=512$ is split into $h=8$ heads of $d_k=d_v=64$, so multi-head attention costs about the same as a single 512-dimensional head. Vaswani et al. report that one head is 0.9 BLEU worse than eight, and that too many heads also hurt. Trained models show heads with different jobs (attending to the previous word, to rare words, to syntactic relations), but also a lot of redundancy: a large fraction of heads can be removed after training with little loss, and in some layers a single head is enough (Michel et al., 2019; Voita et al., 2019).

---

6. The Transformer

The Transformer (Vaswani et al., 2017) builds a full translation model from attention alone, plus three ingredients that we have already met in CNNs:

• Residual connections and normalisation. Every sub-layer is wrapped as $\text{LayerNorm}(x+\text{Sublayer}(x))$. LayerNorm (Ba et al., 2016) normalises each token’s vector over its features, not over the batch like BatchNorm.
• A feed-forward network per position. After attention mixes information between words, a two-layer MLP processes each word on its own: $\text{FFN}(x)=\max (0,x{W}_1+b_1)W_2+b_2$, with ReLU and an inner size of 2048 (4 × 512). Most of the parameters live here. BERT, GPT and ViT later switched to GELU (Hendrycks & Gimpel, 2016).
• Three kinds of attention. The encoder uses self-attention over the source sentence. The decoder uses masked self-attention over the words generated so far (right of the diagram), so the prediction for word $i$ cannot peek at the answer, and cross-attention, where the queries come from the decoder and the keys and values from the encoder output: which source words matter for the word I am writing now?

The base model stacks 6 encoder and 6 decoder layers (65 M parameters) and trained in 12 hours on 8 GPUs, reaching a better translation score than previous models that took far longer to train. Because of the causal mask, all positions of a target sentence are trained in parallel, something an RNN decoder cannot do.

Post-norm and pre-norm

The original Transformer applies LayerNorm after the residual addition (post-norm). Most later models, including GPT-2 and the Vision Transformer (but not BERT), apply it to the input of each sub-layer instead (pre-norm). The skip connection is then a clean identity path, as in ResNet, and deep stacks train without the careful learning-rate warm-up post-norm needs (Xiong et al., 2020).

---

7. What Do the Trained Models Look At?

Here are the three sentiment models from Section 1, running in your browser (plain JavaScript, weights exported from PyTorch; outputs match PyTorch to the printed precision). Type a sentence from the vocabulary or pick an example. The words are coloured by how much the attention model’s classification token, [CLS], attends to them:

Some things to look for:

• In layer 1, [CLS] puts 90–100% of its attention on a single word: the negation (not, never) if there is one, otherwise the adjective. Nobody told it that rule; it is the structure of the task, found from data.
• The distractor gets almost nothing. In the mixed example, messy takes 94% of the attention and the distractor’s good 2%, although both are sentiment words: the model has learned that the first adjective decides.
• Layer 2 is sometimes just as sharp and sometimes spread out (in the long example, 23% goes to [CLS] itself). By then the other words have mixed information among themselves in layer 1, so a diffuse pattern can still carry the answer. An attention map shows where one layer reads from, not the whole computation, so be careful when using attention maps as explanations.
• The LSTM curve (probability after each word) moves when the deciding words appear and then stays put across the fillers. The plain RNN hovers around 50% throughout.

---

8. What Does Attention Cost?

Vaswani et al. compared the three layer types for a sequence of $n$ items with $d$ features (and convolution kernel size $k$):

Layer	Operations per layer	Sequential steps	Longest path between two positions
Self-attention	$O(n^2\cdot d)$	$O(1)$	$O(1)$
Recurrent	$O(n\cdot d^2)$	$O(n)$	$O(n)$
Convolution	$O(k\cdot n\cdot d^2)$	$O(1)$	$O({\log }_{k}n)$ (dilated) or $O(n/k)$

The table counts only the attention operation itself. A full attention layer also has the projections $W_Q,W_K,W_V,W_O$, which cost $O(n\cdot d^2)$, and the FFN costs $O(n\cdot d\cdot d_{\text{ff}})$. Which term dominates depends on $n$ versus $d$:

• Sentences usually have $n<d$ (a few dozen words, $d=512$), so the $n^2$ term is small.
• For a ViT-Base layer on a 224 × 224 image ($n=197$ tokens, $d=768$), the two $n\times n$ products are only about 4% of the layer’s multiply-adds; the projections and the MLP dominate.
• At 1024 × 1024 pixels ($n=4,097$) the same products are about 47%, and the $n\times n$ attention matrix per head no longer fits comfortably in GPU memory. Images have many tokens, which is exactly the problem the next post’s Swin Transformer addresses.

---

9. From Words to Pixels

Attention was used in vision before the Vision Transformer. We already met squeeze-and-excitation blocks, which reweight channels, and attention was added to CNNs as non-local blocks (Wang et al., 2018), where every position of a feature map attends to every other. Two reasons it suits images:

• Long-range interactions in one step. A convolution needs many layers before two distant parts of an image meet; attention connects them directly.
• Self-similarity. Images repeat themselves: the same texture, window or leg appears in several places. Attention finds such matches between regions by comparing their features.

How close is attention to convolution? Without positional information, not very: it treats the image as an unordered set of patches. With relative positional encodings, Cordonnier et al. (2020) proved that a multi-head attention layer with enough heads (one per kernel position, e.g. 9 for 3×3) can express any convolution, and found that trained attention layers in vision models often learn local, convolution-like patterns in early layers. Convolution is a special case attention can learn, if it has enough data to do so. That last condition is the main topic of the next post.

---

10. Common Claims, Checked

Claim	Verdict
”RNNs focus on the end of a sentence because the backpropagation path is long.”	Correct for plain RNNs, and measurable (Section 1). But gated RNNs (LSTM, GRU) largely solved the accuracy problem long before Transformers. The lasting advantages of attention are the constant path length and parallel training.
”Attention $=\mathrm{softmax}(Q{K}^{\top }/\sqrt{d_k})V^{\top }$.”	With tokens as rows ($Q=X{W}_Q$), it is $\cdot V$, not $V^{\top }$: the $n\times n$ weights mix the $n$ value rows. (With tokens as columns, the formula is written $V\mathrm{softmax}(\cdot {)}^{\top }$.)
”Self-attention is invariant to the order of the tokens.”	More precisely equivariant: shuffling the input shuffles the output rows the same way. A pooled output, or the [CLS] token without positional encoding, is invariant. Either way, word order is lost without positional information.
”An embedding layer normalises the input.”	An embedding is a learned lookup table: token id → dense vector. It does not normalise anything; normalisation comes from LayerNorm inside the network.
”Sinusoidal encodings generalise to longer sequences than seen in training.”	This was a hope in the original paper (“may allow”). Measured later, sinusoidal models degrade quickly beyond the training length (Press et al., 2022). Relative schemes (ALiBi, RoPE) were developed for this.
”Multi-head attention is a within-model ensemble.”	Only loosely. Heads are trained jointly, and their outputs are concatenated and mixed by $W_O$, not averaged as independent votes. Many heads turn out to be redundant: a large fraction can be pruned after training with little loss (Michel et al., 2019).
”Attention itself only scales linearly with the feature dimension $d$.”	True for the $Q{K}^{\top }$ and $AV$ products ($O(n^{2}d)$). But the Q/K/V/output projections cost $O(n{d}^2)$, the same as an RNN step, and with the FFN they dominate for short sequences: for ViT-Base at 224 px, the $n^2$ part is only ~4% of the work.
”Self-attention can be seen as a data-dependent convolution.”	With relative positional encoding, yes: multi-head attention can provably express any convolution (Cordonnier et al., 2020). Without positional information it is a weighted average over an unordered set.

---

Summary

Idea	What it solves
Query · key → softmax → weighted sum of values	Each word gathers information from the words that are relevant to it, with weights computed from the input
Scaling by $\sqrt{d_k}$	Keeps the softmax out of its flat, near-one-hot region as $d_k$ grows
Positional encoding	Attention has no notion of order; adding position vectors gives it one
Multiple heads	Several attention patterns at once, at the cost of one
Residuals, LayerNorm, FFN	The same tools that made deep CNNs trainable, applied to tokens
Masking and cross-attention	Turn the encoder into a decoder that generates output from a source

Next: Vision Transformers: ViT and Swin: cutting an image into 16 × 16 patches and treating them as words, why that needs a lot of data, and how Swin’s shifted windows bring back some of the structure of a CNN.

---

References

Recurrent networks and the origin of attention

• Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157–166.
• Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780.
• Pascanu, R., Mikolov, T., & Bengio, Y. (2013). On the difficulty of training recurrent neural networks. ICML.
• Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. ICLR.

The Transformer and its parts

• Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. NeurIPS.
• Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). Layer normalization. arXiv:1607.06450.
• Hendrycks, D., & Gimpel, K. (2016). Gaussian error linear units (GELUs). arXiv:1606.08415.
• Xiong, R., Yang, Y., He, D., et al. (2020). On layer normalization in the Transformer architecture. ICML.
• Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional Transformers for language understanding. NAACL.

Positional encoding

• Shaw, P., Uszkoreit, J., & Vaswani, A. (2018). Self-attention with relative position representations. NAACL.
• Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021). RoFormer: Enhanced Transformer with rotary position embedding. arXiv:2104.09864.
• Press, O., Smith, N. A., & Lewis, M. (2022). Train short, test long: Attention with linear biases enables input length extrapolation. ICLR.

What heads do

• Michel, P., Levy, O., & Neubig, G. (2019). Are sixteen heads really better than one? NeurIPS.
• Voita, E., Talbot, D., Moiseev, F., Sennrich, R., & Titov, I. (2019). Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. ACL.

Attention for images

• Wang, X., Girshick, R., Gupta, A., & He, K. (2018). Non-local neural networks. CVPR.
• Cordonnier, J.-B., Loukas, A., & Jaggi, M. (2020). On the relationship between self-attention and convolutional layers. ICLR.