Deep Weighted Bellman Residual Minimization for Q* Estimation
Abstract
Off-policy evaluation is a foundational component of offline reinforcement learning, aiming to assess and optimize policy performance using pre-collected datasets. However, such datasets often suffer from pronounced challenges, including distribution shift, Q-value overestimation, and low sample utilization efficiency. To address these issues, this paper introduces a weighted Bellman residual minimization framework that incorporates density ratio weighting by effectively integrating expert demonstrations with behavioral data. The proposed weighting scheme departs from the conventional completeness assumption commonly imposed in the theoretical analysis of deep reinforcement learning. We establish a sharp convergence rate for density ratio estimation and derive the convergence rate for the excess risk of resulting deep Q* estimator. Extensive empirical evaluations demonstrate that, compared to existing methods, our method achieves significant improvements in numerical performance and policy generalization, providing specific guidance for the rational utilization of expert demonstrations.
Related
- Stable time-stepping via residual minimization: finite-element and neural-network approximations for transient parabolic problems
- Beyond Discrepancy: Oracle Drift in Nonstationary Weighted Risk Minimization
- When does least-squares residual minimization solve a differential equation?
- Weighted Empirical Risk Minimization for Machine Learning under Long-Range Dependence: Exact Pathwise Rates and Learning-Error Geometry
- Robust low-rank tensor completion via factorized weighted tensor schatten-p norm minimization
BibTeX
@misc{kang2026deep,
title = {{Deep Weighted Bellman Residual Minimization for $Q^*$ Estimation}},
author = {Lican Kang and Jerry Zhijian Yang and Cheng Yuan and Chen Zhong},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.34593},
eprint = {2609.34593},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.34593},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - Deep Weighted Bellman Residual Minimization for Q^* Estimation
AU - Lican Kang
AU - Jerry Zhijian Yang
AU - Cheng Yuan
AU - Chen Zhong
PY - 2026
DA - 2026/09/28
PB - arXiv
DO - 10.48550/arxiv.2609.34593
UR - https://arxiv.org/abs/2609.34593
AB - Off-policy evaluation is a foundational component of offline reinforcement learning, aiming to assess and optimize policy performance using pre-collected datasets. However, such datasets often suffer from pronounced challenges, including distribution shift, Q-value overestimation, and low sample utilization efficiency. To address these issues, this paper introduces a weighted Bellman residual minimization framework that incorporates density ratio weighting by effectively integrating expert demonstrations with behavioral data. The proposed weighting scheme departs from the conventional completeness assumption commonly imposed in the theoretical analysis of deep reinforcement learning. We establish a sharp convergence rate for density ratio estimation and derive the convergence rate for the excess risk of resulting deep Q^* estimator. Extensive empirical evaluations demonstrate that, compared to existing methods, our method achieves significant improvements in numerical performance and policy generalization, providing specific guidance for the rational utilization of expert demonstrations.
ER -