At the end of Modern Hopfield Networks, one update of the memory looked like this:
and in Attention from Scratch, one attention step looked like this:
Ramsauer et al. (2021) titled their paper Hopfield Networks is All You Need to make the point that these are the same operation. This post checks that claim in three ways: the algebra, the code, and the trained models. The short version is that the claim is exact for the weights, holds for one step, and leaves out one thing the reader should know about: where the output comes from.
1. From One Formula to the Other
Take two sets of vectors: , the tokens that are asking (the state patterns), and , the tokens that are being remembered (the stored patterns). Map both into a common “associative space” with learned matrices, and choose β as the usual attention scaling:
One Hopfield update, with the queries as states and the keys as stored patterns, is then (written for all queries at once, one per row)
That is attention with . Attention proper reads out a separately projected value, , with the same weights. Ramsauer et al. reach it by multiplying the retrieved pattern by one more learned matrix. In self-attention, the asking and remembered tokens are the same, , and each token retrieves from the whole sequence:
Multi-head attention is several such Hopfield networks running in parallel, each in its own associative space, with their outputs concatenated.
2. Checked in Code
The authors’ PyTorch package, hopfield-layers, implements this as a Hopfield module. Copy the weights of a PyTorch nn.MultiheadAttention into it, switch off the layer normalisation it applies to its inputs by default, and the two compute the same thing:
| Comparison | Largest difference |
|---|---|
Hopfield (one update) vs. nn.MultiheadAttention, self-attention | 2.8 × 10⁻⁹ |
| the same, cross-attention (queries from one set, memories from another) | 2.8 × 10⁻⁹ |
Hopfield with its default input LayerNorms switched on | 1.6 × 10⁻³ |
The first two are floating-point rounding: the operations are identical. The script is equivalence.py. In the demo below, the left panel is the Hopfield update and the right panel is attention, both with the same weights. Switch the values to “equal to the keys” and the two outputs coincide:
3. Where the Analogy Stretches
The weights are exactly the same; three other things are not.
The output is a read-out, not the retrieved memory. A Hopfield update returns a mix of the stored patterns themselves, so applying it again continues the retrieval. Attention returns the same mix of the values. If were square and invertible, the values would be a fixed linear function of the keys, and attention would be “retrieve, then apply one linear map”. In real transformers it is not: each head works in a 64-dimensional space, while the tokens have 768 dimensions in BERT-base (192 in DeiT-tiny). Attention is best described as retrieve by key, read out the value: a hetero-associative memory, not one that settles to a stored pattern.
One step, then a new memory. A Hopfield network iterates until it reaches a fixed point. A transformer layer applies the update once, then adds the result to the tokens (residual connection), normalises, transforms them with an MLP, and passes them to the next layer. There, a new memory is built from the changed tokens. Are trained heads perhaps already at a fixed point, so that one step is all they need? I took every head of a trained Vision Transformer (DeiT-tiny, the model of the ViT post), applied its own update once (that is its attention), and then once more. The token each query attends to most changed for 38% to 98% of the queries, depending on the layer:
So trained heads are far from fixed points. Each layer performs exactly one retrieval step, and the “fixed point” view does not describe what a single layer computes.
The energy is not minimised by the whole layer. Each attention step lowers the Hopfield energy of its own head. The residual connection, the normalisation and the MLP are not part of that energy, so a transformer as a whole does not descend it. Hoover et al. (2023) built an Energy Transformer in which every part of a block does descend one energy, which shows it is possible but is not what ordinary transformers do.
4. Which Regime Do Trained Heads Use?
The previous post showed three regimes of the Hopfield update: global averaging, metastable averages over groups, and retrieval of single patterns. Ramsauer et al. measured, for every head of BERT, how many tokens hold 90% of each query’s attention. They reported that the first layers mostly average globally, and later layers operate in metastable states. I repeated that measurement on two models:
- BERT-base, the model they analysed, on four paragraphs of English text (86–108 tokens each);
- DeiT-tiny, the Vision Transformer from the ViT post, on the four test images of that post (197 tokens: 196 patches and the class token).
| Layer | BERT-base: globally averaging heads (of 12) | DeiT-tiny: globally averaging heads (of 3) |
|---|---|---|
| 1 | 7 | 0 |
| 2 | 6 | 0 |
| 3 | 2 | 0 |
| 4 | 3 | 1 |
| 5 | 2 | 0 |
| 6 | 0 | 1 |
| 7 | 0 | 1 |
| 8 | 0 | 0 |
| 9 | 0 | 2 |
| 10 | 0 | 3 |
| 11 | 0 | 3 |
| 12 | 0 | 0 |
A head counts as globally averaging when, for the median query, 90% of its attention is spread over at least half the tokens.
For BERT, the claim holds. Globally averaging heads are concentrated in the first layers. From layer 6 on there are none, and heads focus on small groups of tokens.
For the Vision Transformer, it is roughly reversed. The early layers contain the sharpest heads, some putting 90% of their attention on 4 to 6 patches, and globally averaging heads appear later, in all three heads of layers 10 and 11. This agrees with Raghu et al. (2021), who found that higher layers of vision transformers attend globally. Which regime a head uses is learned and depends on the model and the data; it is not a property of attention itself. (Four inputs per model is a small sample; it is enough to see the trend, not to rank individual heads.)
5. What the Hopfield View Gives You
Even with these limits, the correspondence is useful:
- A vocabulary for attention maps. A head that spreads its weight evenly is averaging; one that concentrates on a group is in a metastable state; one that picks a single token is retrieving. β and the norms of the queries and keys, both learned, decide which.
- Design ideas. If the queries need not come from the input, a learned query can pool a whole set into one vector; if the stored patterns need not come from the input, learned patterns become a lookup table. These are the
HopfieldPoolingandHopfieldLayermodules of the next post. - Limits worth knowing. Two near-identical keys cannot be told apart by a single head (the separation condition of the previous post), and a key with a small norm can lose to a longer neighbour. Normalising the inputs, which transformers do, helps with both.
6. Common Claims, Checked
| Claim | Verdict |
|---|---|
| ”Transformer attention is the update rule of a modern Hopfield network.” | For one step and up to the value projection, yes: the weights are identical, and the code matches PyTorch’s attention to 3 × 10⁻⁹. Attention mixes values rather than the stored patterns themselves. |
| ”Transformers retrieve memories until they converge.” | No. Each layer applies one update; a second step would change the most-attended token for 38–98% of queries in DeiT-tiny. |
| ”The stored patterns are what the transformer learned in training.” | In self-attention, the stored patterns are the tokens of the current input, rebuilt in every layer. Knowledge from training lives in the weights, not in this memory. |
| ”Early layers average globally, later layers are metastable.” | True for BERT-base in our test (7 and 6 of 12 heads in layers 1–2, none from layer 6). Not for the Vision Transformer, where global averaging is concentrated in layers 9–11. |
| ”The Hopfield layer is a drop-in replacement for attention.” | It computes attention when its input normalisation is switched off and it takes one step. Its defaults add layer normalisation, and it can take more steps; then it is a different layer. |
| ”Because attention is a Hopfield update, a transformer minimises an energy.” | Each head’s step lowers its own energy, but residual connections, normalisation and MLPs are outside it. |
Summary
| Idea | In one line |
|---|---|
| The correspondence | state → query, stored patterns → keys, β → ; the softmax weights are identical |
| The difference | attention reads out values, takes one step, and builds a new memory in every layer |
| In code | the Hopfield module equals nn.MultiheadAttention to 3 × 10⁻⁹ with its input normalisation off |
| Trained heads | not at fixed points: a second step changes the attended token for 38–98% of queries |
| Regimes | BERT: global averaging early, focused later; DeiT: the reverse; learned, not built in |
Next: Hopfield Pooling: learned queries and learned memories as layers, and multiple-instance learning, where one relevant item has to be found among thousands.
References
The correspondence
- Ramsauer, H., Schäfl, B., Lehner, J., et al. (2021). Hopfield networks is all you need. ICLR. arXiv:2008.02217
- Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention is all you need. NeurIPS.
- Hoover, B., Liang, Y., Pham, B., Panda, R., Strobelt, H., Chau, D. H., Zaki, M. J., & Krotov, D. (2023). Energy transformer. NeurIPS. arXiv:2302.07253
The models and analyses of attention
- Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL.
- Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jégou, H. (2021). Training data-efficient image transformers & distillation through attention. ICML.
- Clark, K., Khandelwal, U., Levy, O., & Manning, C. D. (2019). What does BERT look at? An analysis of BERT’s attention. BlackboxNLP Workshop.
- Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., & Dosovitskiy, A. (2021). Do vision transformers see like convolutional neural networks? NeurIPS.