Limits of Reliability and Scaling in Language Models
Abstract
Large language models (LLMs) are trained and evaluated as though perfect reliability is achievable for any task given sufficient scale. We show that this assumption is information-theoretically unjustified. Every generative task has a reliability ceiling that no model can exceed, determined by how much output uncertainty is resolvable from observable context. The gap decomposes into a resolvable component closable with additional context and a subjective component inherent to task ambiguity. Autoregressive generation further degrades this ceiling at a rate governed by the task's dependency kernel, which quantifies inter-token correlations in the output. From these two primitives, we derive a first-principles scaling law where LLM performance is bottlenecked by the scarcer resource: training data or model capacity. This law recovers the Chinchilla scaling law as a special case and provides a structural account of when scaling improves reliability. Beyond scaling, our framework unifies diverse practical phenomena, such as the benefits of retrieval-augmentation and the spectral mechanics of catastrophic forgetting. Our work formalizes the resource-complexity tradeoffs that govern model performance across domains, offering a unified theory of performance limits in generative language models.
Related
- Serrin-type overdetermination in scaling limits: Sharp existence and asymptotic behavior of quasi-linear Schrödinger energy ground states
- Bulk universality of scaling limits for the almost Hermitian Coulomb gas
- Understanding Latent-Dimension Scaling in Dynamical-System Learning through Spectral Reliability
- Scaling limits for peeling explorations of the Brownian half-plane
- Scaling inequalities and limits for clamped plate eigenvalues on geodesic disks
BibTeX
@misc{majumdar2026limits,
title = {{Limits of Reliability and Scaling in Language Models}},
author = {Subhabrata Majumdar},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2607.14112},
eprint = {2607.14112},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2607.14112},
}
RIS
TY - UNPB
TI - Limits of Reliability and Scaling in Language Models
AU - Subhabrata Majumdar
PY - 2026
DA - 2026/09/17
PB - arXiv
DO - 10.48550/arxiv.2607.14112
UR - https://arxiv.org/abs/2607.14112
AB - Large language models (LLMs) are trained and evaluated as though perfect reliability is achievable for any task given sufficient scale. We show that this assumption is information-theoretically unjustified. Every generative task has a reliability ceiling that no model can exceed, determined by how much output uncertainty is resolvable from observable context. The gap decomposes into a resolvable component closable with additional context and a subjective component inherent to task ambiguity. Autoregressive generation further degrades this ceiling at a rate governed by the task's dependency kernel, which quantifies inter-token correlations in the output. From these two primitives, we derive a first-principles scaling law where LLM performance is bottlenecked by the scarcer resource: training data or model capacity. This law recovers the Chinchilla scaling law as a special case and provides a structural account of when scaling improves reliability. Beyond scaling, our framework unifies diverse practical phenomena, such as the benefits of retrieval-augmentation and the spectral mechanics of catastrophic forgetting. Our work formalizes the resource-complexity tradeoffs that govern model performance across domains, offering a unified theory of performance limits in generative language models.
ER -