Discrete Diffusion Models are Minimax Optimal Under Sparsity
Abstract
Discrete diffusion models have emerged as a leading alternative to autoregressive models for generating text and other categorical data, yet their theoretical foundations remain less developed than those of their continuous counterparts. For continuous diffusion models, a prominent line of theory suggests that these models can adapt to low-dimensional structure in the data, thereby improving statistical efficiency and generalization. In this work, we investigate the discrete analogue of this phenomenon by studying the statistical rates of discrete diffusion models under sparsity assumptions. By analyzing a nonparametric estimator derived from a regularized score-entropy objective, we establish minimax-optimal rates in the sparse regime. The proposed regularization can be viewed as an adaptation of label smoothing techniques introduced for classification, and does not require the knowledge of the sparsity parameter. This shows that discrete diffusion models can adapt intrinsically to sparsity.
Related
- Uniform Discrete Diffusion Models are Minimax Optimal for Estimating Distributions with Small Effective Support Size
- The information geometry of product-reference discrete diffusion: Interaction growth complexity and optimal scheduling
- Sparsity of Quadratically Regularized Optimal Transport: Scalar Case
- Global regularity and sparsity for regularised optimal transport
- Minimax bounds for watermarked and masked recursive discrete distribution estimation
BibTeX
@misc{blunda2026discrete,
title = {{Discrete Diffusion Models are Minimax Optimal Under Sparsity}},
author = {Luca Blunda and Tyler Farghly and Umut Simsekli and Alain Durmus},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.19020},
eprint = {2610.19020},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.19020},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Discrete Diffusion Models are Minimax Optimal Under Sparsity
AU - Luca Blunda
AU - Tyler Farghly
AU - Umut Simsekli
AU - Alain Durmus
PY - 2026
DA - 2026/10/13
PB - arXiv
DO - 10.48550/arxiv.2610.19020
UR - https://arxiv.org/abs/2610.19020
AB - Discrete diffusion models have emerged as a leading alternative to autoregressive models for generating text and other categorical data, yet their theoretical foundations remain less developed than those of their continuous counterparts. For continuous diffusion models, a prominent line of theory suggests that these models can adapt to low-dimensional structure in the data, thereby improving statistical efficiency and generalization. In this work, we investigate the discrete analogue of this phenomenon by studying the statistical rates of discrete diffusion models under sparsity assumptions. By analyzing a nonparametric estimator derived from a regularized score-entropy objective, we establish minimax-optimal rates in the sparse regime. The proposed regularization can be viewed as an adaptation of label smoothing techniques introduced for classification, and does not require the knowledge of the sparsity parameter. This shows that discrete diffusion models can adapt intrinsically to sparsity.
ER -