Representation-based Masked Diffusion Model
Abstract
Masked Diffusion Models (MDMs) have emerged as a compelling paradigm for language modeling, offering the capability for efficient parallel text generation. However, existing parallel sampling methods typically update multiple masked tokens independently and ignore the complex mutual dependencies among the masked tokens. This independent updating mechanism lacks global coordination and might lead to incoherent outputs. To address this limitation, we propose Representation-based Masked Diffusion Model (RMDM), a framework that leverages the text representation to explicitly encode global semantics and help to parallel update tokens more precisely. Specifically, we first encode text into a continuous semantic space using a pretrained encoder and learn an invertible transformation that normalizes the representation distribution to a Gaussian prior, facilitating efficient sampling during generation. Conditioned on this latent semantic representation, we train a masked diffusion model to learn the conditional text distribution, where the representation serves as global semantic guidance to coordinate parallel token updates and faithfully approximate the target distribution. Empirical results demonstrate that RMDM significantly improves generation quality, particularly in aggressive few-step sampling regimes.
Related
- Structured Masked Diffusion for Shared-Codebook Unsourced Random Access Decoding
- Query-Efficient Rejection Sampling for Parallel Masked Diffusion
- Serial Depth of Adaptive Parallel Sampling: Lower Bounds and Schedules for Masked Diffusion Models
- Schedule optimization for tau-leaping in masked discrete diffusion
- Transformer-Based Masked Diffusion Models Provably Learn Distributions with Latent Structure
BibTeX
@misc{hu2026representationbased,
title = {{Representation-based Masked Diffusion Model}},
author = {Yangrong Hu and Ding Huang and Xueyu Zhou and Jian Huang},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.12382},
eprint = {2609.12382},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.12382},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Representation-based Masked Diffusion Model
AU - Yangrong Hu
AU - Ding Huang
AU - Xueyu Zhou
AU - Jian Huang
PY - 2026
DA - 2026/09/11
PB - arXiv
DO - 10.48550/arxiv.2609.12382
UR - https://arxiv.org/abs/2609.12382
AB - Masked Diffusion Models (MDMs) have emerged as a compelling paradigm for language modeling, offering the capability for efficient parallel text generation. However, existing parallel sampling methods typically update multiple masked tokens independently and ignore the complex mutual dependencies among the masked tokens. This independent updating mechanism lacks global coordination and might lead to incoherent outputs. To address this limitation, we propose Representation-based Masked Diffusion Model (RMDM), a framework that leverages the text representation to explicitly encode global semantics and help to parallel update tokens more precisely. Specifically, we first encode text into a continuous semantic space using a pretrained encoder and learn an invertible transformation that normalizes the representation distribution to a Gaussian prior, facilitating efficient sampling during generation. Conditioned on this latent semantic representation, we train a masked diffusion model to learn the conditional text distribution, where the representation serves as global semantic guidance to coordinate parallel token updates and faithfully approximate the target distribution. Empirical results demonstrate that RMDM significantly improves generation quality, particularly in aggressive few-step sampling regimes.
ER -