The Asymptotics of Language Model Alignment with Memory
Abstract
Language model (LM) alignment broadly aims to perturb a given LM Q into an aligned LM q such that i) the outputs produced by q and Q are 'close' in probability, ii) q has a higher expected reward than Q. Two common techniques for LM alignment are: KL-constrained RL, which requires knowledge of the LM distribution and is computationally expensive, and the best-of-n algorithm, which requires only sampling from the LM. The work of Yang et al. established asymptotic closeness between the distributions produced by the two alignment methods for an m--length i.i.d. token sequence output by the LM, in the limit as m increases to infinity. However, the i.i.d. assumption is not representative of practical LMs, whose output sequences often have memory. In this paper, we extend the asymptotic closeness result to the case when the m--length token sequence outputted by the LM is Markovian. Further, for finite-length output sequences -- particularly, when m=1 -- we provide a complete characterization of LM distributions and reward functions for which the KL-divergence between the distributions produced by the two alignment methods is zero -- a question first posed in Yang et al.
Related
- Long-memory Hermite functional processes in functional time series spectral asymptotics
- ALRA: Adaptive Local Relational Alignment for Logit-Based Pre-training Distillation of Autoregressive Language Models
- Valence-Aware Gradient Memory and Fractional Kernel Comparisons for LoRA-Based Language Model Unlearning
- Boundary-layer asymptotics for Gaussian-smoothed singular measures
- Stretched exponential asymptotics for bases of triangular bootstrap percolation
BibTeX
@misc{balasundaram2026asymptotics,
title = {{The Asymptotics of Language Model Alignment with Memory}},
author = {Haricharan Balasundaram and V. Arvind Rameshwar},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.01828},
eprint = {2610.01828},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.01828},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - The Asymptotics of Language Model Alignment with Memory
AU - Haricharan Balasundaram
AU - V. Arvind Rameshwar
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2610.01828
UR - https://arxiv.org/abs/2610.01828
AB - Language model (LM) alignment broadly aims to perturb a given LM Q into an aligned LM q such that i) the outputs produced by q and Q are 'close' in probability, ii) q has a higher expected reward than Q. Two common techniques for LM alignment are: KL-constrained RL, which requires knowledge of the LM distribution and is computationally expensive, and the best-of-n algorithm, which requires only sampling from the LM. The work of Yang et al. established asymptotic closeness between the distributions produced by the two alignment methods for an m--length i.i.d. token sequence output by the LM, in the limit as m increases to infinity. However, the i.i.d. assumption is not representative of practical LMs, whose output sequences often have memory. In this paper, we extend the asymptotic closeness result to the case when the m--length token sequence outputted by the LM is Markovian. Further, for finite-length output sequences -- particularly, when m=1 -- we provide a complete characterization of LM distributions and reward functions for which the KL-divergence between the distributions produced by the two alignment methods is zero -- a question first posed in Yang et al.
ER -