Distribution Matching Distillation for Continuous Diffusion Language Models
Abstract
Continuous diffusion language models generate all tokens in parallel, yet high-quality generation can still require hundreds of network evaluations (NFEs). We study how distributional distillation can reduce this cost by exploiting the student's probabilistic token outputs. Our unified formulation connects the student's output parameterization to the resulting gradient estimators and yields two methods with the same student architecture and reverse-KL matching objective: Simplex-DMD uses continuous token relaxations and pathwise gradients, while Reinforce-DMD uses categorical sampling and REINFORCE with a learned density ratio. We develop both methods for multi-step generation and investigate the training and sampling choices associated with each parameterization. On OpenWebText, for sequences of 1,024 tokens, Simplex-DMD achieves a generative perplexity of 45.6 at a unigram entropy of 5.44 nats in just 4 NFEs, a 49% reduction relative to the strongest evaluated diffusion baseline at matched entropy and sampling budget. Reinforce-DMD improves the frontier at larger budgets, reaching a generative perplexity of 14.9 at an entropy of 5.00 nats with 256 NFEs, a 20% reduction under the same comparison protocol.
Related
- Continuous Diffusion Scales Competitively with Discrete Diffusion for Language
- Diffusion Distillation for Efficient Weather Ensembles
- A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensions
- DASH: Dual-Branch Score Distillation for Guidance-Calibrated Compact Diffusion Models
- Beyond Verified Answers: Solver-Informed Self-Distillation for Bootstrapping Operations Research Language Models
BibTeX
@misc{kiem2026distribution,
title = {{Distribution Matching Distillation for Continuous Diffusion Language Models}},
author = {Paul Le Van Kiem and Dario Shariatian and Umut Simsekli and Alain Durmus},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.40235},
eprint = {2609.40235},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.40235},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Distribution Matching Distillation for Continuous Diffusion Language Models
AU - Paul Le Van Kiem
AU - Dario Shariatian
AU - Umut Simsekli
AU - Alain Durmus
PY - 2026
DA - 2026/09/30
PB - arXiv
DO - 10.48550/arxiv.2609.40235
UR - https://arxiv.org/abs/2609.40235
AB - Continuous diffusion language models generate all tokens in parallel, yet high-quality generation can still require hundreds of network evaluations (NFEs). We study how distributional distillation can reduce this cost by exploiting the student's probabilistic token outputs. Our unified formulation connects the student's output parameterization to the resulting gradient estimators and yields two methods with the same student architecture and reverse-KL matching objective: Simplex-DMD uses continuous token relaxations and pathwise gradients, while Reinforce-DMD uses categorical sampling and REINFORCE with a learned density ratio. We develop both methods for multi-step generation and investigate the training and sampling choices associated with each parameterization. On OpenWebText, for sequences of 1,024 tokens, Simplex-DMD achieves a generative perplexity of 45.6 at a unigram entropy of 5.44 nats in just 4 NFEs, a 49% reduction relative to the strongest evaluated diffusion baseline at matched entropy and sampling budget. Reinforce-DMD improves the frontier at larger budgets, reaching a generative perplexity of 14.9 at an entropy of 5.00 nats with 256 NFEs, a 20% reduction under the same comparison protocol.
ER -