The Hardness of Estimating Cross-Entropy Risk for Language Models
Abstract
Language models, once pretrained, are routinely assessed by the holdout method, which estimates their per-token cross-entropy risk---a quantity that may be finite or infinite. We prove possibility and impossibility theorems that characterize the hardness of this estimation problem. The estimand depends on two objects: the unknown data distribution p and the distribution q induced by the language model under study. Our ignorance of p is empirical, whereas our ignorance of q, or at least of its tail, is mathematical; so the parameter space {\em should} range over pairs (p,q)---a product space---to fully capture our ignorance. {\em Negative results}: no estimator is pointwise consistent, even when the parameter space is restricted to full-support models and to data distributions whose expected text length is below a fixed bound; worse, every estimator fails on a dense set of pairs. {\em Positive results}: the holdout estimator becomes pointwise consistent when the models are further restricted to Transformers or, architecture-neutrally, to full-support models with a finite context window; and, where no such design is in place, it still consistently estimates the risk capped at any fixed threshold, which suffices for screening candidate models in practice. {\em Lesson}: designs such as the Transformer architecture and finite context windows, although adopted for computational or engineering reasons, pay a {\em statistical dividend}: they suffice to make pointwise consistency possible---a dividend visible only in how they turn a negative result into positive ones.
Related
- Estimating Great Expectations Under Autoregressive Language Models with Potentials
- Estimating Population-Risk Curves Along Nonconvex Gradient Flows from the Training Sample
- A One-Stage Approach for Estimating a Meta-Analytic Continuous-Time Random-Intercept Cross-Lagged Panel Model
- Estimating Group Average Treatment Effects in High-Risk Populations with Unobserved Target Membership
- Estimating heterogeneous treatment effects from randomised trials: a comparison of the risk modelling and treatment effect modelling approaches
BibTeX
@misc{lin2026hardness,
title = {{The Hardness of Estimating Cross-Entropy Risk for Language Models}},
author = {Hanti Lin},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2608.15798},
eprint = {2608.15798},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.15798},
note = {CC BY-NC-SA 4.0},
}
RIS
TY - UNPB
TI - The Hardness of Estimating Cross-Entropy Risk for Language Models
AU - Hanti Lin
PY - 2026
DA - 2026/10/30
PB - arXiv
DO - 10.48550/arxiv.2608.15798
UR - https://arxiv.org/abs/2608.15798
AB - Language models, once pretrained, are routinely assessed by the holdout method, which estimates their per-token cross-entropy risk---a quantity that may be finite or infinite. We prove possibility and impossibility theorems that characterize the hardness of this estimation problem. The estimand depends on two objects: the unknown data distribution p and the distribution q induced by the language model under study. Our ignorance of p is empirical, whereas our ignorance of q, or at least of its tail, is mathematical; so the parameter space {\em should} range over pairs (p,q)---a product space---to fully capture our ignorance. {\em Negative results}: no estimator is pointwise consistent, even when the parameter space is restricted to full-support models and to data distributions whose expected text length is below a fixed bound; worse, every estimator fails on a dense set of pairs. {\em Positive results}: the holdout estimator becomes pointwise consistent when the models are further restricted to Transformers or, architecture-neutrally, to full-support models with a finite context window; and, where no such design is in place, it still consistently estimates the risk capped at any fixed threshold, which suffices for screening candidate models in practice. {\em Lesson}: designs such as the Transformer architecture and finite context windows, although adopted for computational or engineering reasons, pay a {\em statistical dividend}: they suffice to make pointwise consistency possible---a dividend visible only in how they turn a negative result into positive ones.
ER -