Simplex Diffusion Models
Abstract
Diffusion models have revolutionized generative modeling for continuous data through the gradual refinement of a belief state. This iterative refinement has not yet carried over to discrete diffusion models, which discard uncertainty at intermediate steps through categorical sampling (information collapse). We propose Simplex Diffusion Models (SDMs), a framework that lifts the diffusion process to the probability simplex to represent beliefs over categories. SDMs admit probability paths with closed-form reverse transitions and can be trained with a simple cross-entropy loss. Contrary to earlier proposals such as Dirichlet Flow Matching which requires integrating an ordinary differential equation, we introduce a DDIM-like sampler with a tunable level of stochasticity. Because SDMs operate on samples on the simplex, they can carry uncertainty across denoising steps, which mitigates information collapse. On OpenWebText, SDMs are competitive with strong Discrete Diffusion baselines, achieving $17.0$ GenPPL at $5.46$ unigram entropy in 64 sampling steps, close to real validation data. Even without Self-Conditioning (SC), SDMs outperform masked and uniform diffusion (with SC or predictor-corrector sampling) on code generation (TinyGSM, T=0.1; 49.0% vs. 45.8%). Distilled down to 8 steps, SDMs solve 32.1% of GSM8K problems, more than distilled Discrete Diffusion models with 128 steps (21.4%).
Related
BibTeX
@misc{deschenaux2026simplex,
title = {{Simplex Diffusion Models}},
author = {Justin Deschenaux and Alexandre Galashov and Andrew Campbell and Li Kevin Wenliang and James Thornton and Arnaud Doucet and Valentin De Bortoli},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.35553},
eprint = {2609.35553},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.35553},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Simplex Diffusion Models
AU - Justin Deschenaux
AU - Alexandre Galashov
AU - Andrew Campbell
AU - Li Kevin Wenliang
AU - James Thornton
AU - Arnaud Doucet
AU - Valentin De Bortoli
PY - 2026
DA - 2026/09/28
PB - arXiv
DO - 10.48550/arxiv.2609.35553
UR - https://arxiv.org/abs/2609.35553
AB - Diffusion models have revolutionized generative modeling for continuous data through the gradual refinement of a belief state. This iterative refinement has not yet carried over to discrete diffusion models, which discard uncertainty at intermediate steps through categorical sampling (information collapse). We propose Simplex Diffusion Models (SDMs), a framework that lifts the diffusion process to the probability simplex to represent beliefs over categories. SDMs admit probability paths with closed-form reverse transitions and can be trained with a simple cross-entropy loss. Contrary to earlier proposals such as Dirichlet Flow Matching which requires integrating an ordinary differential equation, we introduce a DDIM-like sampler with a tunable level of stochasticity. Because SDMs operate on samples on the simplex, they can carry uncertainty across denoising steps, which mitigates information collapse. On OpenWebText, SDMs are competitive with strong Discrete Diffusion baselines, achieving $17.0$ GenPPL at $5.46$ unigram entropy in 64 sampling steps, close to real validation data. Even without Self-Conditioning (SC), SDMs outperform masked and uniform diffusion (with SC or predictor-corrector sampling) on code generation (TinyGSM, T=0.1; 49.0% vs. 45.8%). Distilled down to 8 steps, SDMs solve 32.1% of GSM8K problems, more than distilled Discrete Diffusion models with 128 steps (21.4%).
ER -