Continuous Diffusion Scales Competitively with Discrete Diffusion for Language
Abstract
While diffusion has drawn considerable recent attention from the language modeling community, continuous diffusion has appeared less scalable than discrete approaches. To challenge this belief we revisit Plaid, a likelihood-based continuous diffusion language model (DLM), and construct RePlaid by aligning the architecture of Plaid with modern discrete DLMs. In this unified setting, we establish the first scaling law for continuous DLMs that rivals discrete DLMs: RePlaid exhibits a compute gap of only 20× compared to autoregressive models, outperforms Duo while using fewer parameters, and outperforms MDLM in the over-trained regime. We benchmark RePlaid against recent continuous DLMs: on OpenWebText, RePlaid achieves a new state-of-the-art PPL bound of $22.1$ among continuous DLMs and superior generation quality. These results suggest that continuous diffusion, when trained via likelihood, is a highly competitive and scalable alternative to discrete DLMs. Moreover, we offer theoretical insights to understand the advantage of likelihood-based training. We show that optimizing the noise schedule to minimize the ELBO's variance naturally yields linear cross-entropy (information loss) over time. This evenly distributes denoising difficulty without any case-specific time reparameterization. In addition, we find that optimizing embeddings via likelihood creates structured geometries and drives the most significant likelihood gain.
Related
- Acceleration of Diffusion Language Model through Discrete Average Generator
- A Legendre-Based Modified Extreme Learning Machine for Convection-Diffusion Equations: Steady-State, Continuous-Time, and Discrete-Time Formulations
- Distribution Matching Distillation for Continuous Diffusion Language Models
- CONTINUOUS VERSUS DISCRETE STRUCTURES -DISCRETE EMBEDDING FORMALISMS AND CONTINUALIZATION METHODS
- Recombination in discrete and continuous time from the viewpoint of Markov embedding
BibTeX
@misc{yang2026continuous,
title = {{Continuous Diffusion Scales Competitively with Discrete Diffusion for Language}},
author = {Zhihan Yang and Wei Guo and Shuibai Zhang and Subham Sekhar Sahoo and Yongxin Chen and Arash Vahdat and Morteza Mardani and John Thickstun},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.18530},
eprint = {2605.18530},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.18530},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Continuous Diffusion Scales Competitively with Discrete Diffusion for Language
AU - Zhihan Yang
AU - Wei Guo
AU - Shuibai Zhang
AU - Subham Sekhar Sahoo
AU - Yongxin Chen
AU - Arash Vahdat
AU - Morteza Mardani
AU - John Thickstun
PY - 2026
DA - 2026/10/24
PB - arXiv
DO - 10.48550/arxiv.2605.18530
UR - https://arxiv.org/abs/2605.18530
AB - While diffusion has drawn considerable recent attention from the language modeling community, continuous diffusion has appeared less scalable than discrete approaches. To challenge this belief we revisit Plaid, a likelihood-based continuous diffusion language model (DLM), and construct RePlaid by aligning the architecture of Plaid with modern discrete DLMs. In this unified setting, we establish the first scaling law for continuous DLMs that rivals discrete DLMs: RePlaid exhibits a compute gap of only 20× compared to autoregressive models, outperforms Duo while using fewer parameters, and outperforms MDLM in the over-trained regime. We benchmark RePlaid against recent continuous DLMs: on OpenWebText, RePlaid achieves a new state-of-the-art PPL bound of $22.1$ among continuous DLMs and superior generation quality. These results suggest that continuous diffusion, when trained via likelihood, is a highly competitive and scalable alternative to discrete DLMs. Moreover, we offer theoretical insights to understand the advantage of likelihood-based training. We show that optimizing the noise schedule to minimize the ELBO's variance naturally yields linear cross-entropy (information loss) over time. This evenly distributes denoising difficulty without any case-specific time reparameterization. In addition, we find that optimizing embeddings via likelihood creates structured geometries and drives the most significant likelihood gain.
ER -