Attention over noisy examples: generalization and memorization in the exponential Hopfield network

@misc{alemanno2026attention,
  title = {{Attention over noisy examples: generalization and memorization in the exponential Hopfield network}},
  author = {Francesco Alemanno},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.22191},
  eprint = {2610.22191},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.22191},
}

TY  - UNPB
TI  - Attention over noisy examples: generalization and memorization in the exponential Hopfield network
AU  - Francesco Alemanno
PY  - 2026
DA  - 2026/10/17
PB  - arXiv
DO  - 10.48550/arxiv.2610.22191
UR  - https://arxiv.org/abs/2610.22191
AB  - In Hebbian learning from examples, a Hopfield network retrieves archetypes seen only through noisy examples. We study it in the exponential Hopfield model: one softmax attention step with the unlabelled examples as keys. A signal-to-noise analysis gives the overlap with an unseen archetype in closed form: a majority-rule law in which an attention entropy replaces the dataset entropy. At high temperature the archetype is lost in the crosstalk; at low temperature the softmax freezes on the closest training example, as in a random energy model, and the layer memorizes. In between, in a window widening with the square root of the network size, the layer pools the examples of the correct class and retrieves the unseen archetype as well as their majority vote allows, at any fixed load. On structured datasets, such as MNIST and fashion-MNIST, the best accuracy lies beyond the freezing.
ER  -