The Probabilistic Structure of Large Language Models
Abstract
This paper presents a probabilistic perspective on large language models (LLMs), developed with the aim of bringing together, in a single self-contained account, tools that are usually treated separately across the literature. LLMs are described through probability measures on the set of sequences of tokens, specified via their autoregressive conditional distributions. Training is formulated as a maximum-likelihood estimation problem, addressed by stochastic gradient methods, while text generation is viewed as the sequential simulation of the resulting stochastic process. The role of the asymmetry of the Kullback--Leibler divergence in text generation is examined in relation with characteristic phenomena such as hallucination and the distinction between statistical plausibility and truth. As a complementary illustration of the same viewpoint, we also discuss diffusion models, built around the score function, which cast generation not as sequential token prediction but as the simulation of a reverse-time stochastic process transforming noise into data both in discrete and continuous time.
Related
- Probabilistic large deviations approach to obstacle Hamilton-Jacobi equations
- The Markov Accessibility Topology and its Probabilistic Structure
- Probabilistic linkage with graphs: leveraging cluster structure to improve the accuracy of entity resolution at scale
- A Distributional Framework for Behavioral Characterization and Risk Management of Large Language Models
- Sequential Functional Structured Tucker Compression for Large Language Model Attentions
BibTeX
@misc{aboulalaa2026probabilistic,
title = {{The Probabilistic Structure of Large Language Models}},
author = {Adnan Aboulalaâ},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.25134},
eprint = {2609.25134},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.25134},
}
RIS
TY - UNPB
TI - The Probabilistic Structure of Large Language Models
AU - Adnan Aboulalaâ
PY - 2026
DA - 2026/09/20
PB - arXiv
DO - 10.48550/arxiv.2609.25134
UR - https://arxiv.org/abs/2609.25134
AB - This paper presents a probabilistic perspective on large language models (LLMs), developed with the aim of bringing together, in a single self-contained account, tools that are usually treated separately across the literature. LLMs are described through probability measures on the set of sequences of tokens, specified via their autoregressive conditional distributions. Training is formulated as a maximum-likelihood estimation problem, addressed by stochastic gradient methods, while text generation is viewed as the sequential simulation of the resulting stochastic process. The role of the asymmetry of the Kullback--Leibler divergence in text generation is examined in relation with characteristic phenomena such as hallucination and the distinction between statistical plausibility and truth. As a complementary illustration of the same viewpoint, we also discuss diffusion models, built around the score function, which cast generation not as sequential token prediction but as the simulation of a reverse-time stochastic process transforming noise into data both in discrete and continuous time.
ER -