Conditional Flow Models for Offline Maximum Entropy Reinforcement Learning
Abstract
Offline maximum entropy reinforcement learning seeks to estimate the optimal soft action-value function Q* from a fixed dataset without further interaction with the environment. Although Q* is characterized by the soft optimal Bellman equation, the corresponding Bellman operator cannot be evaluated directly because both the reward distribution and the transition kernel are unknown. Consequently, estimating Q* requires the joint treatment of environment modeling and value function learning. We propose a framework that addresses these tasks separately. Conditional diffusion models based on probability flow ODEs estimate the conditional reward distribution and transition kernel. The resulting distributional estimators define a data driven approximation of the soft optimal Bellman operator, whose empirical Bellman residual is minimized over a class of deep neural networks to estimate Q*. We establish nonasymptotic guarantees for the proposed method. For operator learning, we derive total variation error bounds for the conditional flow estimators that account explicitly for score estimation, numerical discretization, and terminal smoothing. For soft value learning, we first use the soft Bellman equation to show that Q* inherits Hölder regularity from the conditional reward and transition densities. For any Hölder index β, the value learning stage achieves an excess Bellman residual risk bound of order 𝒪̃(n−2β/(d+2β)), where d is the total dimension of the state and action spaces. In contrast to many existing theoretical analyses of deep reinforcement learning, our results do not require Bellman completeness. Numerical experiments support the theoretical findings and demonstrate the effectiveness of the proposed method.
Related
- Offline Approximate Value Iteration: A Reinforcement Learning Approach for Satellite Constellation Deployment
- Model-based Bootstrap for Offline Policy Evaluation in Tabular Reinforcement Learning
- Confounding Masquerading as Improvement: A Systematic Evaluation of Offline Reinforcement Learning for Stroke Antithrombotic Treatment in a 129,000-Patient Registry
- Provably Efficient Risk-Sensitive Offline Reinforcement Learning via Optimized Certainty Equivalents
- Model-free Reinforcement Learning for Continuous Time and State: A Stochastic Maximum Principle Approach
BibTeX
@misc{cheng2026conditional,
title = {{Conditional Flow Models for Offline Maximum Entropy Reinforcement Learning}},
author = {Hao Cheng and Yuling Jiao and Lican Kang and Jerry Zhijian Yang and Chen Zhong},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.32125},
eprint = {2610.32125},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.32125},
}
RIS
TY - UNPB
TI - Conditional Flow Models for Offline Maximum Entropy Reinforcement Learning
AU - Hao Cheng
AU - Yuling Jiao
AU - Lican Kang
AU - Jerry Zhijian Yang
AU - Chen Zhong
PY - 2026
DA - 2026/10/23
PB - arXiv
DO - 10.48550/arxiv.2610.32125
UR - https://arxiv.org/abs/2610.32125
AB - Offline maximum entropy reinforcement learning seeks to estimate the optimal soft action-value function Q^* from a fixed dataset without further interaction with the environment. Although Q^* is characterized by the soft optimal Bellman equation, the corresponding Bellman operator cannot be evaluated directly because both the reward distribution and the transition kernel are unknown. Consequently, estimating Q^* requires the joint treatment of environment modeling and value function learning. We propose a framework that addresses these tasks separately. Conditional diffusion models based on probability flow ODEs estimate the conditional reward distribution and transition kernel. The resulting distributional estimators define a data driven approximation of the soft optimal Bellman operator, whose empirical Bellman residual is minimized over a class of deep neural networks to estimate Q^*. We establish nonasymptotic guarantees for the proposed method. For operator learning, we derive total variation error bounds for the conditional flow estimators that account explicitly for score estimation, numerical discretization, and terminal smoothing. For soft value learning, we first use the soft Bellman equation to show that Q^* inherits Hölder regularity from the conditional reward and transition densities. For any Hölder index β, the value learning stage achieves an excess Bellman residual risk bound of order 𝒪̃(n^−2β/(d+2β)), where d is the total dimension of the state and action spaces. In contrast to many existing theoretical analyses of deep reinforcement learning, our results do not require Bellman completeness. Numerical experiments support the theoretical findings and demonstrate the effectiveness of the proposed method.
ER -