Offline Deep Q* Estimation with Diffusion Models
Abstract
In offline RL, estimating the optimal action-value function Q* can be formulated as solving the optimal Bellman equation based solely on offline observations. A fundamental challenge is that the reward function and transition kernel are unknown, so the optimal Bellman operator is not directly observable from data. To address this issue, we propose a novel framework that decouples operator estimation from value function learning. In this approach, we first formulate conditional diffusion models to estimate the reward law and transition kernel, which induces a data-driven approximation of the optimal Bellman operator. We then plug these estimators into the Bellman equation and obtain a deep estimator of Q* by minimizing the empirical Bellman residual over a neural network function class. Theoretically, we first establish sharp nonasymptotic convergence rates for learning the optimal Bellman operator. We establish the smoothness of Q* through the Bellman equation using the the Hölder assumptions on the conditional reward and transition densities. With suitable sample sizes, we obtain the oracle value-stage rate 𝒪̃(n−2α/(dx+da+2α)) for the excess Bellman residual risk. We then translate this risk bound into an expected L2 estimation error of order 𝒪̃(n−α/(dx+da+2α)) under a concentrability condition. For the total offline budget ñ=n+nr+nx, we obtain a value estimation rate of 𝒪̃(ñ−α/(2dx+da+2α)) and establish an information-theoretic lower bound of order ñ−α/(dx+da+2α). Importantly, our theoretical analysis does not rely on Bellman completeness assumptions commonly used in deep RL theory.
Related
- Accelerating posterior sampling in Bayesian inverse problems with diffusion models: an online-offline perspective
- Nonparametric Estimation of the Diffusion-Interaction Function in Particle Systems
- Parameter Estimation for the Mixed Fractional Merton Jump Diffusion Model with EM Algorithm
- Two-Stage Score Estimation for Group-Specific Diffusion Models
- Random Jump Intensities and Bernstein Density Estimation in Ergodic Basic Affine Jump-Diffusion Processes
BibTeX
@misc{chen2026offline,
title = {{Offline Deep Q* Estimation with Diffusion Models}},
author = {Xiaohong Chen and Yuling Jiao and Lican Kang and Jerry Zhijian Yang and Chen Zhong},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2608.14401},
eprint = {2608.14401},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.14401},
}
RIS
TY - UNPB
TI - Offline Deep Q* Estimation with Diffusion Models
AU - Xiaohong Chen
AU - Yuling Jiao
AU - Lican Kang
AU - Jerry Zhijian Yang
AU - Chen Zhong
PY - 2026
DA - 2026/10/31
PB - arXiv
DO - 10.48550/arxiv.2608.14401
UR - https://arxiv.org/abs/2608.14401
AB - In offline RL, estimating the optimal action-value function Q^* can be formulated as solving the optimal Bellman equation based solely on offline observations. A fundamental challenge is that the reward function and transition kernel are unknown, so the optimal Bellman operator is not directly observable from data. To address this issue, we propose a novel framework that decouples operator estimation from value function learning. In this approach, we first formulate conditional diffusion models to estimate the reward law and transition kernel, which induces a data-driven approximation of the optimal Bellman operator. We then plug these estimators into the Bellman equation and obtain a deep estimator of Q^* by minimizing the empirical Bellman residual over a neural network function class. Theoretically, we first establish sharp nonasymptotic convergence rates for learning the optimal Bellman operator. We establish the smoothness of Q^* through the Bellman equation using the the Hölder assumptions on the conditional reward and transition densities. With suitable sample sizes, we obtain the oracle value-stage rate 𝒪̃(n^−2α/(d_x+d_a+2α)) for the excess Bellman residual risk. We then translate this risk bound into an expected L^2 estimation error of order 𝒪̃(n^−α/(d_x+d_a+2α)) under a concentrability condition. For the total offline budget ñ=n+n_r+n_x, we obtain a value estimation rate of 𝒪̃(ñ^−α/(2d_x+d_a+2α)) and establish an information-theoretic lower bound of order ñ^−α/(d_x+d_a+2α). Importantly, our theoretical analysis does not rely on Bellman completeness assumptions commonly used in deep RL theory.
ER -