Emergence of the transformer architecture as the score of entropic laws of random measures
Abstract
We show that the transformer architecture emerges as the score of last-token laws in a natural statistical model of language, where sentences are empirical measures of tokens, a language is a law on P(Sd), and our prior belief is that sentences carry no structure at all. By the maximum-entropy principle, we select the law closest to this prior in relative entropy among those reproducing an observed sentence-level statistic. We discuss the relevance of the statistic in the posterior interaction between tokens and we build a statistic, able to capture that interaction, with Luce's choice axiom truncated at the lowest relational order. This procedure gives the softmax choice rule, with the attention cross-moment as its mean parameter. The query--key matrix is then the Lagrange multiplier of a maximum-entropy problem inside each sentence, and the value matrix the multiplier of the constraint over the language, so that the two are untied. Under the optimal entropic law we then show that a transformer head appears in the score of the last-token law. For ordered sentences, causal masking follows from the masked reference choice rule, and with a next-token cross-moment the score equals one causally masked attention head. We also discuss multi-head attention, weight-tied unembedding, and a heuristic reading of residual blocks as projected Langevin steps.
Related
- Khintchine dichotomy for images of self-similar measures and almost random walks
- Absolute continuity of invariant measures for random diffeomorphisms
- Visitation measures for non-linear iterations with random perturbations
- Emergence of Correlations Between Records and Maximum in Random Time Series
- Exact dimensionality of stationary measures for random diffeomorphisms with all Lyapunov exponents equal and negative
BibTeX
@misc{greco2026emergence,
title = {{Emergence of the transformer architecture as the score of entropic laws of random measures}},
author = {Giacomo Greco},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.39614},
eprint = {2610.39614},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.39614},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Emergence of the transformer architecture as the score of entropic laws of random measures
AU - Giacomo Greco
PY - 2026
DA - 2026/10/29
PB - arXiv
DO - 10.48550/arxiv.2610.39614
UR - https://arxiv.org/abs/2610.39614
AB - We show that the transformer architecture emerges as the score of last-token laws in a natural statistical model of language, where sentences are empirical measures of tokens, a language is a law on P(S^d), and our prior belief is that sentences carry no structure at all. By the maximum-entropy principle, we select the law closest to this prior in relative entropy among those reproducing an observed sentence-level statistic. We discuss the relevance of the statistic in the posterior interaction between tokens and we build a statistic, able to capture that interaction, with Luce's choice axiom truncated at the lowest relational order. This procedure gives the softmax choice rule, with the attention cross-moment as its mean parameter. The query--key matrix is then the Lagrange multiplier of a maximum-entropy problem inside each sentence, and the value matrix the multiplier of the constraint over the language, so that the two are untied. Under the optimal entropic law we then show that a transformer head appears in the score of the last-token law. For ordered sentences, causal masking follows from the masked reference choice rule, and with a next-token cross-moment the score equals one causally masked attention head. We also discuss multi-head attention, weight-tied unembedding, and a heuristic reading of residual blocks as projected Langevin steps.
ER -