dFlowGRPO: Rate-Aware Policy Optimization for Discrete Flow Models
Abstract
Discrete flow models (DFMs) are a class of flexible generative models for generating discrete data, and diffusion large language models (dLLMs) can be viewed as a special case with a specific choice of a mixture path and a masked source distribution. While several recent works have explored reinforcement learning for dLLMs, its application to more general discrete flow models remains underexplored. In this work, we present discrete Flow-GRPO (dFlowGRPO), a unified reinforcement learning framework for discrete flow models that supports a broad family of probability paths and non-masked source distributions. We derive the full trajectory probability for DFMs and formulate the denoising process as a Markov decision process, enabling dFlowGRPO to incorporate information from both the associated conditional transition rates and the posterior model during reinforcement learning. We apply dFlowGRPO to FUDOKI, a recent multimodal discrete flow model, and evaluate it on both image generation and multimodal understanding tasks. Empirical results show that dFlowGRPO outperforms existing GRPO-type methods for dLLMs on text-to-image generation tasks and achieves performance competitive with continuous flow-based models trained using Flow-GRPO, while also demonstrating strong capabilities on understanding tasks.
Related
- Proximal Policy Optimization for Amortized Discrete Sampling
- POETS: Uncertainty-Aware LLM Optimization via Compute-Efficient Policy Ensembles
- ReAPO: Retry-Aware Policy Optimization for Efficient Exploration in LLM Reasoning
- Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms
- A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees
BibTeX
@misc{wan2026dflowgrpo,
title = {{dFlowGRPO: Rate-Aware Policy Optimization for Discrete Flow Models}},
author = {Zhengyan Wan and Yidong Ouyang and Panwen Hu and Qiang Sun},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.09291},
eprint = {2605.09291},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.09291},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - dFlowGRPO: Rate-Aware Policy Optimization for Discrete Flow Models
AU - Zhengyan Wan
AU - Yidong Ouyang
AU - Panwen Hu
AU - Qiang Sun
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2605.09291
UR - https://arxiv.org/abs/2605.09291
AB - Discrete flow models (DFMs) are a class of flexible generative models for generating discrete data, and diffusion large language models (dLLMs) can be viewed as a special case with a specific choice of a mixture path and a masked source distribution. While several recent works have explored reinforcement learning for dLLMs, its application to more general discrete flow models remains underexplored. In this work, we present discrete Flow-GRPO (dFlowGRPO), a unified reinforcement learning framework for discrete flow models that supports a broad family of probability paths and non-masked source distributions. We derive the full trajectory probability for DFMs and formulate the denoising process as a Markov decision process, enabling dFlowGRPO to incorporate information from both the associated conditional transition rates and the posterior model during reinforcement learning. We apply dFlowGRPO to FUDOKI, a recent multimodal discrete flow model, and evaluate it on both image generation and multimodal understanding tasks. Empirical results show that dFlowGRPO outperforms existing GRPO-type methods for dLLMs on text-to-image generation tasks and achieves performance competitive with continuous flow-based models trained using Flow-GRPO, while also demonstrating strong capabilities on understanding tasks.
ER -