Acceleration of Diffusion Language Model through Discrete Average Generator
Abstract
Discrete diffusion models and flow matching have emerged as powerful frameworks for generative modeling over discrete state spaces, yet efficient few-step generation remains a fundamental challenge. In this work, we introduce the Discrete Average Generator, a principled extension of MeanFlow to Continuous-Time Markov Chains (CTMCs). Analogously to how MeanFlow defines an average velocity field over a time interval in continuous spaces, we define an average generator as the normalized increment of the transition kernel over a time interval. We show that this average generator satisfies a self-consistency identity, which provides the foundation for our training objective. We further develop training strategies that align with the standard training paradigm of diffusion language models while keeping the resulting objective tractable. When projected onto per-coordinate marginals, the self-consistency identity admits a closed-form expression, enabling efficient training and inference. In Potts model simulations, our objective reduces the total variation distance of the K-step sampler by up to 67%. On OpenWebText, our method achieves the lowest generative perplexity among the evaluated methods for 8 to 64 sampling steps while enabling a 16× acceleration, and achieves comparable performance to existing methods on ImageNet.
Related
- Continuous Diffusion Scales Competitively with Discrete Diffusion for Language
- Transport-Matched Penalties for Diffusion Synthetic Acceleration of Polytopic Discontinuous Galerkin Discretisations
- Uniform convergence of diffusion synthetic acceleration for heterogeneous slab transport
- Koopman Observers for Diffusion Acceleration: Correcting Feature Forecasts with Shallow Measurements
- A stabilized scheme satisfying the discrete maximum principle for a time fractional convection-diffusion-reaction equation
BibTeX
@misc{ouyang2026acceleration,
title = {{Acceleration of Diffusion Language Model through Discrete Average Generator}},
author = {Yidong Ouyang and Zhengyan Wan and Themis Haris and Tian Tan and Liqian Peng and Henry Li and Ziqian Lin and Jianhang Chen and Maryam Karimzadehgan and Alec Go and George Michailidis},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.38364},
eprint = {2609.38364},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.38364},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Acceleration of Diffusion Language Model through Discrete Average Generator
AU - Yidong Ouyang
AU - Zhengyan Wan
AU - Themis Haris
AU - Tian Tan
AU - Liqian Peng
AU - Henry Li
AU - Ziqian Lin
AU - Jianhang Chen
AU - Maryam Karimzadehgan
AU - Alec Go
AU - George Michailidis
PY - 2026
DA - 2026/09/29
PB - arXiv
DO - 10.48550/arxiv.2609.38364
UR - https://arxiv.org/abs/2609.38364
AB - Discrete diffusion models and flow matching have emerged as powerful frameworks for generative modeling over discrete state spaces, yet efficient few-step generation remains a fundamental challenge. In this work, we introduce the Discrete Average Generator, a principled extension of MeanFlow to Continuous-Time Markov Chains (CTMCs). Analogously to how MeanFlow defines an average velocity field over a time interval in continuous spaces, we define an average generator as the normalized increment of the transition kernel over a time interval. We show that this average generator satisfies a self-consistency identity, which provides the foundation for our training objective. We further develop training strategies that align with the standard training paradigm of diffusion language models while keeping the resulting objective tractable. When projected onto per-coordinate marginals, the self-consistency identity admits a closed-form expression, enabling efficient training and inference. In Potts model simulations, our objective reduces the total variation distance of the K-step sampler by up to 67%. On OpenWebText, our method achieves the lowest generative perplexity among the evaluated methods for 8 to 64 sampling steps while enabling a 16× acceleration, and achieves comparable performance to existing methods on ImageNet.
ER -