Factual recall in linear associative memories: sharp asymptotics and mechanistic insights
Abstract
Large language models excel at factual recall, yet the fundamental limits of storing input--output associations with neural networks remain unclear. We study these limits in a minimal setting: a linear associative memory that maps p input embeddings in ℝd to their d-dimensional targets via a linear layer, requiring each mapped input to score its own target above all other outputs. Since all inputs compete for the same outputs, these constraints are strongly correlated, which makes the storage capacity pc --- the largest number of associations that can be stored --- hard to characterise. We address this in three steps. First, we introduce a decoupled model, in which each input competes with its own independent set of outputs, and provide numerical and analytical evidence that it is equivalent to the original model, both in capacity and in the structure of the learnt solution. Second, using tools from statistical physics, we show that the decoupled model has capacity pc log pc / d2 = 1/2, extend this computation to linear two-layer networks, and go beyond the capacity by predicting the spectrum of the learnt weights and the storage mechanism. The latter explains how the optimal solution outperforms a naïve Hebbian rule: rather than boosting input--output alignments with broad fluctuations, it raises the correct scores just above the extreme-value threshold set by the competing outputs. Third, we prove the decoupled model's capacity thresholds rigorously, including in the non-convex two-layer case; combined with a recent proof for the original model, this shows that both models have the same capacity for full-rank weights. These findings give a sharp characterisation of factual storage in linear networks, and suggest decoupling as a general tool for memorisation and learning problems in which each input must be matched to one of an extensive number of outputs.
Related
- Phase Transitions and Modulation in Dense Oscillatory Associative Memories
- Phases in a class of associative memories via hidden neurons
- Dense auto-hetero associative memories applied to noisy communication channels
- The Hilbert Basis Theorem for Non-Associative Skew Power Series Rings and Non-Associative Skew Laurent Series Rings over a Non-Associative Coefficient Ring
- On Associative Neural Networks for Sparse Patterns with Large Capacities
BibTeX
@misc{giorlandino2026factual,
title = {{Factual recall in linear associative memories: sharp asymptotics and mechanistic insights}},
author = {Alessio Giorlandino and Sebastian Goldt and Antoine Maillard},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2605.10795},
eprint = {2605.10795},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.10795},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Factual recall in linear associative memories: sharp asymptotics and mechanistic insights
AU - Alessio Giorlandino
AU - Sebastian Goldt
AU - Antoine Maillard
PY - 2026
DA - 2026/11/04
PB - arXiv
DO - 10.48550/arxiv.2605.10795
UR - https://arxiv.org/abs/2605.10795
AB - Large language models excel at factual recall, yet the fundamental limits of storing input--output associations with neural networks remain unclear. We study these limits in a minimal setting: a linear associative memory that maps p input embeddings in ℝ^d to their d-dimensional targets via a linear layer, requiring each mapped input to score its own target above all other outputs. Since all inputs compete for the same outputs, these constraints are strongly correlated, which makes the storage capacity p_c --- the largest number of associations that can be stored --- hard to characterise. We address this in three steps. First, we introduce a decoupled model, in which each input competes with its own independent set of outputs, and provide numerical and analytical evidence that it is equivalent to the original model, both in capacity and in the structure of the learnt solution. Second, using tools from statistical physics, we show that the decoupled model has capacity p_c log p_c / d^2 = 1/2, extend this computation to linear two-layer networks, and go beyond the capacity by predicting the spectrum of the learnt weights and the storage mechanism. The latter explains how the optimal solution outperforms a naïve Hebbian rule: rather than boosting input--output alignments with broad fluctuations, it raises the correct scores just above the extreme-value threshold set by the competing outputs. Third, we prove the decoupled model's capacity thresholds rigorously, including in the non-convex two-layer case; combined with a recent proof for the original model, this shows that both models have the same capacity for full-rank weights. These findings give a sharp characterisation of factual storage in linear networks, and suggest decoupling as a general tool for memorisation and learning problems in which each input must be matched to one of an extensive number of outputs.
ER -