Hopfield Networks: Memory as Energy

In Attention from Scratch, a query looked up the values whose keys it resembled. That is a memory you address by content rather than by location: you show it part of what you want, and it returns the rest. Long before attention, physicists and neuroscientists built networks that do exactly this, and the most famous one, the Hopfield network (1982), explains why such a memory works with a single idea: memories are the valleys of an energy landscape, and recall is rolling downhill.

This is the first post of a series that goes from that idea to the layers used in modern deep learning:

Hopfield Networks: Memory as Energy (this post): the classical network, its energy, its capacity and where it fails.
Modern Hopfield Networks: a sharper energy, exponential capacity and one-step retrieval.
Attention Is a Hopfield Update: the connection to transformer attention, and its limits.
Hopfield Pooling: learned queries that summarise a set.

Every number below comes from experiments that can be downloaded (hopfield (zip)) and is reproduced by the interactive demos, which run the same network in your browser.

1. A Memory Addressed by Content

A computer’s RAM is addressed by location: give it address 4096 and it returns whatever is stored there, whether or not you know anything about it. An associative (or content-addressable) memory works the other way round. Give it a corrupted photo, half a melody, or the first words of a sentence, and it returns the stored item that matches best.

Three things are needed for that:

a way to store many items in the same set of numbers,
a way to recall one of them from a partial or noisy cue,
a guarantee that recall stops at a stored item instead of wandering forever.

The Hopfield network provides all three with very little machinery.

2. The Network

The network has $d$ binary neurons, $s_{i} \in {- 1, + 1}$ , every one connected to every other with a symmetric weight $w_{ij} = w_{ji}$ and no connection to itself ( $w_{ii} = 0$ ). The patterns to store, $x^{1}, \dots, x^{N}$ , are vectors of $\pm 1$ : for the demos below, $16 \times 16$ black-and-white images, so $d = 256$ .

Storing uses Hebb’s rule (“neurons that fire together wire together”): the weight between two neurons grows when they have the same sign in a pattern and shrinks when they differ, summed over all patterns:

$W = \frac{1}{d} \sum_{μ = 1}^{N} x^{μ} (x^{μ})^{⊤}, w_{ii} = 0.$

There is no training loop and no gradient: storing a pattern is a single addition. Recalling starts the network in the cue and lets one neuron at a time align with its input field $h_{i}$ :

$h_{i} = \sum_{j} w_{ij} s_{j}, s_{i} \leftarrow sign (h_{i}) .$

Why does this repair a corrupted pattern? Write the field of neuron $i$ when the network sits exactly at a stored pattern $x^{ν}$ :

$h_{i} = signal x_{i}^{ν} \frac{d - 1}{d} + crosstalk from the other patterns \frac{1}{d} μ \neq = ν \sum x_{i}^{μ} j \neq = i \sum x_{j}^{μ} x_{j}^{ν} .$

The signal pushes every neuron towards the stored pattern. The crosstalk is the sum of $N - 1$ random-looking terms; for random patterns it behaves like Gaussian noise with variance $α = N / d$ . A bit flips when the crosstalk outweighs the signal, which happens with probability $P (z < - 1/ α)$ for a standard normal $z$ . My measurement on $d = 500$ neurons agrees with that formula:

Patterns per neuron $α = N / d$	Unstable bits, theory	Unstable bits, measured
0.05	0.0004%	0%
0.10	0.078%	0.076%
0.138	0.36%	0.32%
0.20	1.27%	1.25%

A corrupted cue works the same way: as long as most bits agree with one stored pattern, the signal from that pattern dominates every neuron’s field, and the wrong bits are flipped back.

3. Energy: Why Recall Stops

Hopfield’s key observation was that the network has an energy,

$E (s) = - \frac{1}{2} \sum_{i, j} w_{ij} s_{i} s_{j} = - \frac{1}{2} s^{⊤} W s,$

that never increases when a single neuron updates. The proof takes two lines. If neuron $i$ changes from $s_{i}$ to $s_{i}^{'}$ , and because $W$ is symmetric with a zero diagonal, the energy changes by

$Δ E = - (s_{i}^{'} - s_{i}) h_{i} .$

The update sets $s_{i}^{'} = sign (h_{i})$ , so a change happens only when $s_{i}^{'}$ has the sign of $h_{i}$ , and then $Δ E < 0$ . The energy is bounded below and there are finitely many states, so the network must stop at a state where no neuron wants to change: a local minimum of $E$ . Each stored pattern is (ideally) such a minimum, a valley in the landscape, and recall is rolling downhill into the nearest one:

Over 200 random runs, checking the energy after every single neuron update, I found no increase at all. Try it: store a few shapes, corrupt one, and watch the energy fall step by step while the shape reappears. You can also draw on the state yourself.

The proof needs one neuron at a time

The energy argument assumes asynchronous updates: one neuron changes, then the next one sees the new state. If all neurons update at once (synchronously), each reacts to a state that is changing under it, and the network can oscillate between two states forever. Starting 200 synchronous networks (10 random patterns, $d = 100$ ) from random states, 189 settled and 11 ended in such a 2-cycle; asynchronous updates never do. Hopfield’s network also needs symmetric weights: with $w_{ij} \neq = w_{ji}$ the energy no longer exists, and cycles of any length are possible.

4. How Many Memories Fit?

Each new pattern adds crosstalk to all the others, so at some point the valleys merge and the stored patterns stop being minima. For random patterns and large $d$ , the statistical physics of spin glasses gives a sharp answer (Amit, Gutfreund & Sompolinsky, 1985): recall works up to about

$N \approx 0.138 d,$

with about 1.5% of the bits wrong right at that limit, and it collapses beyond it. Requiring every bit of a pattern to be perfect is much stricter: about $d / (2 ln d)$ patterns (McEliece et al., 1987), only 40 for $d = 500$ .

My measurement, starting the network exactly at a stored pattern and counting the wrong bits after it settles:

Patterns per neuron $α$ ( $d = 500$ )	Wrong bits after recall	Patterns recalled perfectly
0.08	0.05%	89%
0.12	0.26%	56%
0.14	1.0%	30%
0.16	6.4%	7%
0.18	15.8%	0%
0.20	19.1%	1%

The collapse sits close to 0.138; in smaller networks it is smoother, because finite networks fluctuate more. And note the last column: at $α = 0.08$ , well below the famous limit, one stored pattern in nine already comes back with at least one wrong bit.

5. Memories You Did Not Store

The valleys of the energy are not only the stored patterns:

Inverted patterns. $E (- s) = E (s)$ , because the energy is quadratic in $s$ . So every stored pattern $x$ comes with an equally deep valley at $- x$ , the photographic negative.
Mixture states. Combining three stored patterns by majority vote, $sign (x^{1} + x^{2} + x^{3})$ , gives a stable state that overlaps each of the three by about $\frac{1}{2}$ (Amit et al., 1985).
Spin-glass states, unrelated to any pattern, which appear in large numbers as the load grows.

With 6 random patterns stored ( $d = 200$ ) and 500 random starting states, the network ended in a stored pattern 182 times, in an inverted pattern 161 times, and in a spurious state 157 times. The spurious states overlapped their three closest patterns by about 0.5 each, exactly the mixture states of the theory. A Hopfield network always stops; it does not always stop at something you stored.

6. Real Patterns Are Correlated

The 0.138 limit is for random patterns, which overlap each other by only about $1/ d$ (6% for $d = 256$ ). Real images are nothing like that. The eight shapes in the demo share most of their background and overlap by 33% on average, and the crosstalk from such patterns is not small random noise but a systematic pull towards their common parts:

Shapes stored ( $d = 256$ )	Stored shapes that are stable	Recalled from 15% noise
1–3	all	100%
4	4 of 4	96%
5	5 of 5	74%
6	2 of 6	10%
7	0 of 7	0%
8	0 of 8	0%

The theory promises about 35 random patterns for 256 neurons; with these shapes, the network breaks down at five or six, an $α$ of about 0.02. Eight random patterns in the same network are all recalled perfectly. This is the practical weakness of the classical Hopfield network, and the reason the modern version in the next post replaces the quadratic energy with a much steeper one. Classical remedies exist too: storing patterns with the pseudo-inverse rule instead of Hebb’s rule (Personnaz, Guyon & Dreyfus, 1985) removes the correlations at the cost of a non-local learning rule.

7. Where the Idea Came From

Hopfield did not invent associative memory networks. Correlation-matrix memories, threshold networks storing patterns with Hebbian weights, and their recall dynamics were published in 1972 by Shun-ichi Amari, Kaoru Nakano and Teuvo Kohonen, and William Little studied persistent states of such networks in 1974. Hopfield’s 1982 paper made two contributions that changed the field: the energy function that proves recall converges, and the link to spin-glass physics, which brought the tools of statistical mechanics, and the exact capacity results above, to neural networks. In 1984 he extended the model to continuous neurons. The 2024 Nobel Prize in Physics was awarded to John Hopfield and Geoffrey Hinton for foundational discoveries that enable machine learning with artificial neural networks.

8. Common Claims, Checked

Claim	Verdict
”A Hopfield network stores 0.15·d patterns.”	0.15 was Hopfield’s estimate from simulations in 1982. The theory gives 0.138·d for random patterns with about 1.5% wrong bits at the limit; perfect recall of every bit holds only up to about d / (2 ln d), 40 patterns for d = 500.
”So a 16 × 16 image memory holds about 35 images.”	Only if the images look random. Eight simple shapes overlapping 33% broke the network at five or six.
”The energy always decreases.”	Never increases, and only for asynchronous updates with symmetric weights and zero self-connections. Synchronous updates ended in 2-cycles from 11 of 200 random starts.
”Recall always returns one of the stored patterns.”	No. Inverted patterns are always minima too, and mixtures of three patterns are stable: from random starts, 31% of the runs ended in a spurious state.
”Hopfield networks are trained like other neural networks.”	There is no training: Hebb’s rule stores a pattern with one outer product, no loss and no gradient.
”Hopfield invented associative neural memories.”	Amari, Nakano and Kohonen published Hebbian associative memories in 1972. Hopfield’s contributions were the energy function and the spin-glass connection.
”A Hopfield network is a recurrent network like an LSTM.”	It is recurrent, but it does not process a sequence: it is given one input and runs until it reaches a fixed point, which an energy guarantees.

Summary

Idea	In one line
Associative memory	recall by content: a partial or noisy cue returns the closest stored item
Storage	Hebb’s rule, $W = \frac{1}{d} \sum x x^{⊤}$ with zero diagonal: one outer product per pattern
Recall	one neuron at a time, $s_{i} \leftarrow sign (h_{i})$
Energy	$E = - \frac{1}{2} s^{⊤} W s$ never increases under asynchronous updates, so recall stops at a minimum
Capacity	about 0.138·d random patterns; perfect recall far fewer; correlated patterns far fewer still
Failure modes	inverted and mixture states, synchronous cycles, and above all correlated patterns

Next: Modern Hopfield Networks: what happens when the energy is made much steeper, why capacity then grows exponentially with $d$ , and why one update step is enough.

References

The model and its analysis

Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. PNAS, 79(8), 2554–2558.
Hopfield, J. J. (1984). Neurons with graded response have collective computational properties like those of two-state neurons. PNAS, 81(10), 3088–3092.
Amit, D. J., Gutfreund, H., & Sompolinsky, H. (1985). Storing infinite numbers of patterns in a spin-glass model of neural networks. Physical Review Letters, 55(14), 1530–1533.
McEliece, R. J., Posner, E. C., Rodemich, E. R., & Venkatesh, S. S. (1987). The capacity of the Hopfield associative memory. IEEE Transactions on Information Theory, 33(4), 461–482.
Personnaz, L., Guyon, I., & Dreyfus, G. (1985). Information storage and retrieval in spin-glass like neural networks. Journal de Physique Lettres, 46(8), 359–365.

Earlier associative memories

Hebb, D. O. (1949). The Organization of Behavior. Wiley.
Amari, S. (1972). Learning patterns and pattern sequences by self-organizing nets of threshold elements. IEEE Transactions on Computers, C-21(11), 1197–1206.
Nakano, K. (1972). Associatron: A model of associative memory. IEEE Transactions on Systems, Man, and Cybernetics, SMC-2(3), 380–388.
Kohonen, T. (1972). Correlation matrix memories. IEEE Transactions on Computers, C-21(4), 353–359.
Little, W. A. (1974). The existence of persistent states in the brain. Mathematical Biosciences, 19(1–2), 101–120.

Notes

Explorer