Vocabulary-size-independent Convergence of Discrete Diffusion Models: adjoint equations induce the right space
Abstract
Discrete diffusion has become a leading framework for generative modeling in various applications including language, vision, and biology. Existing convergence theory, however, exhibits fundamental limitations. KL-based analyses diverge under singular priors such as the masked distribution, while bounds in total variation (TV) depend on the vocabulary size S and become vacuous for modern language tasks, where vocabularies contain hundreds of thousands of tokens. We develop a unified adjoint-equation-based framework that establishes vocabulary-size-independent convergence guarantees in any integral probability metric (IPM). To the best of our knowledge, our bounds are the first to be entirely free of S and applicable to both masked and uniform priors. Importantly, our results can extend existing step complexity guarantees to any IPM. Also, our theory relies only on a single standard rate-matrix regularity assumption and applies to general priors. Five novel techniques drive our improvements: 1. working in the space of observables via adjoint equations rather than directly with probability measures; 2. a regularity analysis that yields bounds on any IPM; 3. a coupling argument that removes S-dependence under uniform transitions; and 4. score-marginal cancellation and 5. exit-routing techniques that remove S-dependence under masked transitions. Our framework thus sharply departs from prior analyses and avoids the shortcomings of pathspace-KL and existing TV-based approaches. Beyond convergence bounds, our framework provides a versatile toolkit for further theoretical study of discrete diffusion models, including principled choices of loss functions and vocabulary-size-independent step complexity.
Related
- Convergence of a fully discrete approximation for the stochastic 2D Euler equations with Kraichnan transport noise
- Strong Convergence of Fully Discrete Tamed Euler Approximations for Time-Changed Stochastic Delay Differential Equations
- CONVERGENCE ANALYSIS OF A DISCRETE DUALITY FINITE-VOLUME METHOD FOR CAHN-HILLIARD EQUATIONS WITH SURFACTANTS
- A Fully Discrete Local Discontinuous Galerkin Method for Quasilinear Stochastic Convection-Diffusion-Type Equations
- Optimization-based Cell Average Limiters as Metric Projections: Constrained Evolution and Convergence Analysis for Convection-diffusion Equations
BibTeX
@misc{kan2026vocabularysizeindependent,
title = {{Vocabulary-size-independent Convergence of Discrete Diffusion Models: adjoint equations induce the right space}},
author = {Kelvin Kan and Xingjian Li and Benjamin J. Zhang and Tuhin Sahai and Stanley Osher and Markos A. Katsoulakis},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2605.17232},
eprint = {2605.17232},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.17232},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Vocabulary-size-independent Convergence of Discrete Diffusion Models: adjoint equations induce the right space
AU - Kelvin Kan
AU - Xingjian Li
AU - Benjamin J. Zhang
AU - Tuhin Sahai
AU - Stanley Osher
AU - Markos A. Katsoulakis
PY - 2026
DA - 2026/09/07
PB - arXiv
DO - 10.48550/arxiv.2605.17232
UR - https://arxiv.org/abs/2605.17232
AB - Discrete diffusion has become a leading framework for generative modeling in various applications including language, vision, and biology. Existing convergence theory, however, exhibits fundamental limitations. KL-based analyses diverge under singular priors such as the masked distribution, while bounds in total variation (TV) depend on the vocabulary size S and become vacuous for modern language tasks, where vocabularies contain hundreds of thousands of tokens. We develop a unified adjoint-equation-based framework that establishes vocabulary-size-independent convergence guarantees in any integral probability metric (IPM). To the best of our knowledge, our bounds are the first to be entirely free of S and applicable to both masked and uniform priors. Importantly, our results can extend existing step complexity guarantees to any IPM. Also, our theory relies only on a single standard rate-matrix regularity assumption and applies to general priors. Five novel techniques drive our improvements: 1. working in the space of observables via adjoint equations rather than directly with probability measures; 2. a regularity analysis that yields bounds on any IPM; 3. a coupling argument that removes S-dependence under uniform transitions; and 4. score-marginal cancellation and 5. exit-routing techniques that remove S-dependence under masked transitions. Our framework thus sharply departs from prior analyses and avoids the shortcomings of pathspace-KL and existing TV-based approaches. Beyond convergence bounds, our framework provides a versatile toolkit for further theoretical study of discrete diffusion models, including principled choices of loss functions and vocabulary-size-independent step complexity.
ER -