Modern Hopfield Networks

The classical Hopfield network of the previous post recalls a memory by rolling downhill in an energy landscape, but it runs out of room fast: about $0.138d$ random patterns for $d$ neurons, and only five or six of the simple shapes we tried, because similar patterns interfere. This post follows the fix, in three steps:

1. Dense associative memory (Krotov & Hopfield, 2016): make the energy steeper, so that the closest memory dominates instead of all memories voting.
2. Exponential interactions (Demircigil et al., 2017): make it steeper still; the capacity becomes exponential in $d$.
3. Continuous states (Ramsauer et al., 2021): let the state be any real vector; the update rule becomes a softmax over similarities, the same operation as the attention in Attention from Scratch.

As before, every number comes from experiments, available for download (modern.py) and is reproduced in the demos.

1. Why the Classical Network Runs Out of Room

Write the classical energy in terms of the overlaps $m_{\mu }=x^{\mu }\cdot s$, the similarity of the state with each stored pattern:

$E(s)=-\frac{1}{2}{s}^{\top }Ws=-\frac{1}{2d}{\sum }_{\mu }(x^{\mu }\cdot s{)}^2+\text{const}.$

Each pattern contributes $-m_{\mu }^2$: the pull towards a pattern grows only with the square of its overlap. A state that resembles one pattern strongly and four others weakly is pulled by all five, and the weak ones together can outvote the strong one. That collective pull of the “wrong” patterns is the crosstalk that limits capacity.

2. Dense Associative Memory: a Steeper Energy

Krotov and Hopfield replaced the square by a general interaction function $F$:

$E(s)=-{\sum }_{\mu =1}^{N}F(x^{\mu }\cdot s).$

The update stays one neuron at a time: neuron $i$ takes whichever sign gives the lower energy, $s_i\leftarrow \mathrm{sign}({\sum }_{\mu }F(x^{\mu }\cdot s^{(i+)})-F(x^{\mu }\cdot s^{(i-)}))$, where $s^{(i\pm )}$ is the state with neuron $i$ set to $\pm 1$. With $F(z)=z^2$ this is the classical network. With a steeper $F$, a pattern’s pull grows much faster with its overlap, so the closest pattern wins by a wider margin:

Theory says the capacity grows like $d^{n-1}$ for $F(z)=z^n$ (Krotov & Hopfield, 2016) and exponentially in $d$ for $F(z)=e^z$ (Demircigil et al., 2017). I measured two things: the largest number of random patterns recalled perfectly from a cue with 10% of its bits flipped, at least 90% of the time ($d=64$); and how often the eight shapes that broke the classical network, all stored at once, are recalled from 15% noise:

Interaction $F(z)$	Random patterns recalled, $d=64$	The 8 shapes recalled
$z^2$ (classical)	6	0%
$z^3$	128	38%
$z^4$	1,024	99%
$e^z$	≥ 16,384 (the most I tried)	100%

3. Continuous States: the Softmax Update

Dense associative memories still use binary neurons and update them one at a time. Ramsauer et al. (2021) defined an energy for a real-valued state $\xi \in {\mathbb{R}}^d$, with the stored patterns as the columns of $X=(x^1,\dots ,x^N)$:

$E(\xi )=-\mathrm{lse}(\beta ,X^{\top }\xi )+\frac{1}{2}{\xi }^{\top }\xi +{\beta }^{-1}\log N+\frac{1}{2}{M}^2,$

$\mathrm{lse}(\beta ,z)={\beta }^{-1}\log {\sum }_{\mu =1}^N{e}^{\beta z_{\mu }},$

where $M$ is the largest pattern norm. The first term is the exponential interaction again, written as a smooth maximum (“log-sum-exp”) of the similarities $x^{\mu }\cdot \xi$: it is low when $\xi$ is very similar to some stored pattern. The second term keeps $\xi$ from growing without bound. The last two are constants that make $E$ non-negative.

Minimising this energy with the concave-convex procedure (Yuille & Rangarajan, 2003; the $-\mathrm{lse}$ term is concave and $\frac{1}{2}{\xi }^{\top }\xi$ convex) gives a remarkably simple update for all of $\xi$ at once:

${\xi }^{\text{new}}=X\mathrm{softmax}\left(\beta X^{\top }\xi \right).$

The concave-convex procedure guarantees that each update lowers the energy or leaves it unchanged, and that the state converges. Over 300 random runs, checking every update, I found no increase. The new state is always a weighted average of the stored patterns, with weights given by a softmax of the similarities. That makes the inverse temperature $\beta$ the central knob.

4. Three Kinds of Memory, Set by β

How sharp the softmax is decides what the network retrieves:

• Small β: all weights are nearly equal, and the state goes to the global average of all patterns. There is one minimum.
• Intermediate β: patterns that are similar to each other share their weight. The state goes to the average of a group of similar patterns, a metastable state.
• Large β: one weight is almost 1, and the state goes to a single pattern.

In two dimensions you can see the minima merge as β drops. Click to place the state, then try the three presets:

The same in 64 dimensions, with 4 clusters of 5 similar patterns and a noisy copy of one pattern as the start:

β	Weight on each cluster at the end	Largest single weight	Regime
0.001	0.25, 0.25, 0.25, 0.25	0.05	global average
0.05	0.27, 0.20, 0.19, 0.35	0.08	still close to the average
0.2	0, 1.00, 0, 0	0.23	metastable: the right cluster, no single pattern
1.0	0, 1.00, 0, 0	1.00	the right pattern

Note that metastable states appear only when stored patterns are similar to each other. With random patterns, which are all nearly orthogonal, I saw only the global average and single-pattern retrieval.

5. One Update Is Usually Enough

For well-separated patterns and a large enough β, the very first update already lands on the stored pattern, with an error that shrinks exponentially as β times the separation grows (Ramsauer et al., 2021, Theorem 4). With 100 random patterns in $d=64$ and a cue with 6 bits flipped:

β	Weight on the right pattern	Error after one update, $\Vert \xi -x\Vert /\Vert x\Vert$
0.02	0.03	96% (close to the average)
0.05	0.12	85%
0.1	0.59	38%
0.2	0.99	0.65%
0.5	1.00	0.0%

Away from that regime, convergence takes longer. With Gaussian patterns ($d=32$, $N=50$), the median number of updates to converge was 2.5 at β = 1, but 36 at β = 0.2, where the state settles into a metastable average slowly.

6. Exponential Capacity, Measured

The continuous network is read out by taking the sign of the state after one update. Same protocol as before: random $\pm 1$ patterns, 10% of the bits flipped, β = 1, and capacity counted as the largest $N$ recalled perfectly at least 90% of the time.

Neurons $d$	Classical Hopfield	Modern Hopfield, one step	$2^{d/2}$
10	2	16	32
20	3	512	1,024
30	4	46,341	32,768
40	5	≥ 741,455 (the most I tried)	1,048,576

The classical network gains about one pattern per ten neurons. The modern one follows $2^{d/2}$ closely, the exponential scaling Demircigil et al. derived for the exponential interaction.

7. What the Exponential Capacity Costs

Two points are easy to miss, and they matter when these networks become layers in the next posts.

The patterns are stored explicitly. The classical network keeps everything in the $d\times d$ weight matrix: storing another pattern only changes the existing numbers. The modern network keeps the matrix $X$ of all $N$ patterns, $N\times d$ numbers, and every update compares the state with every one of them. “Exponential capacity” therefore does not mean exponential compression. It means the network can still tell its stored patterns apart when there are exponentially many of them. Krotov & Hopfield (2021) make this concrete: the model is a network with one hidden “memory neuron” per stored pattern, connected by ordinary pairwise synapses.

A pattern must be more similar to itself than to any other. Similarity here is a dot product, so a short pattern next to a longer one can lose to its neighbour even when the state sits exactly on it. Pattern $x^i$ can only be retrieved if its separation ${\Delta }_i=x^{i\top }{x}^i-{\max }_{j\ne i}{x}^{i\top }{x}^j$ is positive, and retrieval is reliable only when ${\Delta }_i$ is large compared to $1/\beta$. In the demo above, switch the stored patterns to “different lengths”: patterns 1 and 3 can then never be recalled on their own, at any β. Giving all patterns the same length avoids this, which is one reason the Hopfield layers of the next posts normalise their inputs.

8. Common Claims, Checked

Claim	Verdict
”Modern Hopfield networks store exponentially many patterns.”	In $d$, yes: measured close to $2^{d/2}$ for random patterns. But all $N$ patterns are kept explicitly ($N\times d$ numbers), unlike the fixed $d\times d$ matrix of the classical network, so this is not compression.
”One update step is enough.”	For well-separated patterns and large enough β, yes: exact after one step at β = 0.5 in the test above. At intermediate β, metastable states took a median of 36 steps.
”Every stored pattern can be retrieved.”	Only if it is more similar, by dot product, to itself than to any other pattern. With unequal lengths, some patterns are never retrieved at any β.
”A larger β is always better.”	No. Too small and everything averages: β = 0.01 recalled none of the shapes. Too large and nothing averages, which is sometimes what you want (next post). The right β depends on how well separated the patterns are.
”The energy guarantees convergence to a stored pattern.”	It guarantees convergence to a fixed point of the energy, which can also be a metastable average or the global average.
”Higher-order interactions make dense memories biologically implausible.”	Krotov & Hopfield (2021) rewrote them as a network of feature and memory neurons with ordinary pairwise synapses.
”The softmax update is transformer attention.”	Very nearly, for one step and up to learned projections. That is the subject of the next post.

---

Summary

Idea	In one line
Why the classical net fails	every pattern pulls in proportion to its overlap squared, so many weak pulls outvote the right one
Dense associative memory	$E=-\sum F(x^{\mu }\cdot s)$ with a steeper $F$: capacity $\sim d^{n-1}$ for $z^n$, exponential for $e^z$
Continuous modern Hopfield	$E=-\mathrm{lse}(\beta ,X^{\top }\xi )+\frac{1}{2}{\xi }^{\top }\xi +\text{const}$, minimised by $\xi \leftarrow X\mathrm{softmax}(\beta X^{\top }\xi )$
β	global average, metastable group average, or single pattern
Retrieval	usually one update, for separated patterns and large enough β
The price	all patterns stored explicitly; a pattern must be more similar to itself than to any other

Next: Attention Is a Hopfield Update: how this update becomes the attention of a transformer, where the equivalence is exact and where it is not.

---

References

Dense and modern Hopfield networks

• Krotov, D., & Hopfield, J. J. (2016). Dense associative memory for pattern recognition. NeurIPS. arXiv:1606.01164
• Demircigil, M., Heusel, J., Löwe, M., Upgang, S., & Vermet, F. (2017). On a model of associative memory with huge storage capacity. Journal of Statistical Physics, 168(2), 288–299. arXiv:1702.01929
• Ramsauer, H., Schäfl, B., Lehner, J., et al. (2021). Hopfield networks is all you need. ICLR. arXiv:2008.02217
• Krotov, D., & Hopfield, J. J. (2021). Large associative memory problem in neurobiology and machine learning. ICLR. arXiv:2008.06996
• Millidge, B., Salvatori, T., Song, Y., Lukasiewicz, T., & Bogacz, R. (2022). Universal Hopfield networks: A general framework for single-shot associative memory models. ICML. arXiv:2202.04557

Optimisation

• Yuille, A. L., & Rangarajan, A. (2003). The concave-convex procedure. Neural Computation, 15(4), 915–936.

Blog and code

• Brandstetter, J., et al. Hopfield Networks is All You Need (blog post). ml-jku.github.io/hopfield-layers, and the hopfield-layers PyTorch package.