One Hopfield update and one attention step use the same weights
β (attention uses 1/√d_k)
values
different from the keys
equal to the keys
Apply the Hopfield update again
Key space: stored patterns (keys) and the query
Value space: what attention returns
token: key (left) and its value (right), size = weight p
query / state ξ
output
Which Hopfield regime do trained attention heads use?
model
BERT-base (text)
DeiT-tiny (images)
Trained heads take one step: what would a second Hopfield step change?