Hierarchical Prototype Emergence in Modern Hopfield Models
Abstract
Hierarchical correlations are a universal feature of any realistic model of data, and the question of how associative memory models may learn these correlations and generalize beyond them to construct new sensible images is an important step towards understanding more complex modern architectures such as diffusion models. We consider a hierarchical model for memories which are sampled and stored in a dense Hopfield network with polynomial activation. We analytically derive conditions for each level of this hierarchy to be locally stable - that is they are local energy minima. We use prototype reconstruction as a minimal model of generalization and we find that it takes only a quasi-polynomial amount of information to generalize beyond particular memories and even particular groups in the hierarchy. We observe a qualitatively analogous phase diagram in the number of memories, sharpness of the activation function (polynomial degree) for data from Fashion-MNIST.
Related
- A Bifurcation Theory for the Equilibria of Modern Hopfield Networks
- Hierarchical emergence of network bursting in a four-cell central pattern generator model
- Finiteness and Hierarchical Structure of Open Systems: A Computable Emergence Criterion via the Wuliang Method
- Free energy of the Hopfield model
- Attention over noisy examples: generalization and memorization in the exponential Hopfield network
BibTeX
@misc{cowsik2026hierarchical,
title = {{Hierarchical Prototype Emergence in Modern Hopfield Models}},
author = {Aditya Cowsik and Adithya Sriram},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.12079},
eprint = {2609.12079},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.12079},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Hierarchical Prototype Emergence in Modern Hopfield Models
AU - Aditya Cowsik
AU - Adithya Sriram
PY - 2026
DA - 2026/09/10
PB - arXiv
DO - 10.48550/arxiv.2609.12079
UR - https://arxiv.org/abs/2609.12079
AB - Hierarchical correlations are a universal feature of any realistic model of data, and the question of how associative memory models may learn these correlations and generalize beyond them to construct new sensible images is an important step towards understanding more complex modern architectures such as diffusion models. We consider a hierarchical model for memories which are sampled and stored in a dense Hopfield network with polynomial activation. We analytically derive conditions for each level of this hierarchy to be locally stable - that is they are local energy minima. We use prototype reconstruction as a minimal model of generalization and we find that it takes only a quasi-polynomial amount of information to generalize beyond particular memories and even particular groups in the hierarchy. We observe a qualitatively analogous phase diagram in the number of memories, sharpness of the activation function (polynomial degree) for data from Fashion-MNIST.
ER -