Optimal Value Inference for Reinforcement Learning
Abstract
We study offline inference for the optimal value in reinforcement learning under finite state and action spaces. Two new nuisances are derived as fixed points of a self-induced Bellman equation, in which we approximate the maximum Bellman operator by its softmax correspondence. We propose a debiased estimator through the Neyman orthogonality and establish its asymptotic normality under diverging horizons even when the behavior policy changes with time, as long as the nuisances have the statistical rates that can be achieved by many machine learning methods. We provide a concrete estimating procedure for these nuisances and show they can lead to valid inference. Synthetic experiments validate the numerical performance of our inference method, and we implement it in real-life decision-making problems, including bike repositioning and AI agentic tool use.
Related
- Offline Approximate Value Iteration: A Reinforcement Learning Approach for Satellite Constellation Deployment
- Bayesian Neural Networks via Dirichlet Process Priors on the Data-Generating Distribution for Randomized Value Functions in Deep Reinforcement Learning
- Online Inference for Quantile Distributional Reinforcement Learning
- Optimal Transport Meets Reinforcement Learning: A Survey
- Near-Optimal Sample Complexity for Recursive Entropic Risk Reinforcement Learning with a Generative Model
BibTeX
@misc{lu2026optimal,
title = {{Optimal Value Inference for Reinforcement Learning}},
author = {Nan Lu and Ethan Lee and Ruicheng Ao and James M. Robins and David Simchi-Levi and Junwei Lu},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2609.09981},
eprint = {2609.09981},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.09981},
}
RIS
TY - UNPB
TI - Optimal Value Inference for Reinforcement Learning
AU - Nan Lu
AU - Ethan Lee
AU - Ruicheng Ao
AU - James M. Robins
AU - David Simchi-Levi
AU - Junwei Lu
PY - 2026
DA - 2026/10/29
PB - arXiv
DO - 10.48550/arxiv.2609.09981
UR - https://arxiv.org/abs/2609.09981
AB - We study offline inference for the optimal value in reinforcement learning under finite state and action spaces. Two new nuisances are derived as fixed points of a self-induced Bellman equation, in which we approximate the maximum Bellman operator by its softmax correspondence. We propose a debiased estimator through the Neyman orthogonality and establish its asymptotic normality under diverging horizons even when the behavior policy changes with time, as long as the nuisances have the statistical rates that can be achieved by many machine learning methods. We provide a concrete estimating procedure for these nuisances and show they can lead to valid inference. Synthetic experiments validate the numerical performance of our inference method, and we implement it in real-life decision-making problems, including bike repositioning and AI agentic tool use.
ER -