Notes

Associative Memory Reference and Resources

2 min read

The material behind the Associative Memory notes, a series that goes from the classical Hopfield network to the Hopfield layers used in deep learning. It builds on Attention from Scratch, and leads on to the spiking versions in the Spiking Neural Networks notes. Each post also lists the original papers behind its content at the end.

Blog and code

  • Hopfield Networks is All You Need (blog post): Johannes Brandstetter and colleagues, Institute for Machine Learning, JKU Linz. ml-jku.github.io/hopfield-layers. A clear walk from classical to modern Hopfield networks and their link to attention.
  • hopfield-layers (PyTorch): github.com/ml-jku/hopfield-layers. The Hopfield, HopfieldPooling and HopfieldLayer modules, with example notebooks on multiple-instance learning.

Papers

  • Neural networks and physical systems with emergent collective computational abilities: John J. Hopfield. PNAS, 1982. The classical network and its energy function.

  • Storing infinite numbers of patterns in a spin-glass model of neural networks: Daniel J. Amit, Hanoch Gutfreund and Haim Sompolinsky. Physical Review Letters, 1985. Where the 0.138·d capacity comes from.

  • Dense Associative Memory for Pattern Recognition: Dmitry Krotov and John J. Hopfield. NeurIPS, 2016. arXiv:1606.01164. Steeper energies and polynomial capacity.

  • On a model of associative memory with huge storage capacity: Mete Demircigil, Judith Heusel, Matthias Löwe, Sven Upgang and Franck Vermet. Journal of Statistical Physics, 2017. arXiv:1702.01929. Exponential capacity.

  • Hopfield Networks is All You Need: Hubert Ramsauer et al. ICLR, 2021. arXiv:2008.02217. Continuous modern Hopfield networks and the equivalence with attention.

  • Attention-based Deep Multiple Instance Learning: Maximilian Ilse, Jakub M. Tomczak and Max Welling. ICML, 2018. arXiv:1802.04712. Learned attention pooling for bags of instances.

  • Set Transformer: Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi and Yee Whye Teh. ICML, 2019. arXiv:1810.00825. Pooling by multi-head attention with learned seed vectors.

  • Modern Hopfield Networks and Attention for Immune Repertoire Classification: Michael Widrich et al. NeurIPS, 2020. arXiv:2007.13505. Hopfield pooling over hundreds of thousands of sequences per sample (DeepRC).

Books

  • Introduction to the Theory of Neural Computation: John Hertz, Anders Krogh and Richard G. Palmer. Addison-Wesley, 1991. The classic textbook treatment of Hopfield networks, their capacity and spurious states.
  • Information Theory, Inference, and Learning Algorithms: David J. C. MacKay. Cambridge University Press, 2003. Free at inference.org.uk/mackay/itila. Chapter 42 covers Hopfield networks clearly and briefly.

Graph View

Backlinks