Online Inference for Quantile Distributional Reinforcement Learning
Abstract
In this paper, we study online statistical inference for quantile-based distributional policy evaluation, which aims to characterize the distribution of discounted cumulative rewards under a given policy. To obtain a finite-dimensional representation of the return distribution, we consider the fixed point ηm of the quantile-projected distributional Bellman equation, parameterized by the quantile vector θm. We develop two online inference procedures for θm, covering synchronous and asynchronous sampling from a generative model as well as Markovian data. The first method uses the trajectory of quantile temporal-difference learning (QTD). We establish functional central limit theorems for the partial sums of QTD iterates and construct a recursively computable random-scaling statistic. The resulting confidence intervals have asymptotically exact coverage at deterministic horizons, with length of order T−1/2 in probability. The second method constructs anytime-valid confidence intervals directly from samples. Exploiting the monotonicity of the quantile Bellman map, we develop a computationally explicit coordinate-shaving algorithm with finite-sample simultaneous coverage. Under positive limiting state-visit frequencies, the resulting confidence intervals have lengths of order √(loglog T/T), which we establish as classwise optimal through a matching lower bound. Together, these results provide complementary guarantees for online inference in quantile distributional reinforcement learning, combining asymptotically exact fixed-horizon inference with finite-sample anytime validity.
Related
- Near Minimax Rate for Quantile Distributional Reinforcement Learning
- Online Inference in Distributional Temporal-Difference Learning
- Finite-Sample Distribution Theory and Efficient Large-Scale Inference for Online Quantile Regression
- Deep Reinforcement Learning with Buffered Quantile Objectives
- Risk-Sensitive Mission Planning for Multi-Target On-Orbit Inspection via Distributional Reinforcement Learning
BibTeX
@misc{cheng2026online,
title = {{Online Inference for Quantile Distributional Reinforcement Learning}},
author = {Zijie Cheng and Yang Peng and Zhihua Zhang},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2608.12973},
eprint = {2608.12973},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.12973},
}
RIS
TY - UNPB
TI - Online Inference for Quantile Distributional Reinforcement Learning
AU - Zijie Cheng
AU - Yang Peng
AU - Zhihua Zhang
PY - 2026
DA - 2026/10/16
PB - arXiv
DO - 10.48550/arxiv.2608.12973
UR - https://arxiv.org/abs/2608.12973
AB - In this paper, we study online statistical inference for quantile-based distributional policy evaluation, which aims to characterize the distribution of discounted cumulative rewards under a given policy. To obtain a finite-dimensional representation of the return distribution, we consider the fixed point η_m of the quantile-projected distributional Bellman equation, parameterized by the quantile vector θ_m. We develop two online inference procedures for θ_m, covering synchronous and asynchronous sampling from a generative model as well as Markovian data. The first method uses the trajectory of quantile temporal-difference learning (QTD). We establish functional central limit theorems for the partial sums of QTD iterates and construct a recursively computable random-scaling statistic. The resulting confidence intervals have asymptotically exact coverage at deterministic horizons, with length of order T^−1/2 in probability. The second method constructs anytime-valid confidence intervals directly from samples. Exploiting the monotonicity of the quantile Bellman map, we develop a computationally explicit coordinate-shaving algorithm with finite-sample simultaneous coverage. Under positive limiting state-visit frequencies, the resulting confidence intervals have lengths of order √(loglog T/T), which we establish as classwise optimal through a matching lower bound. Together, these results provide complementary guarantees for online inference in quantile distributional reinforcement learning, combining asymptotically exact fixed-horizon inference with finite-sample anytime validity.
ER -