Near Minimax Rate for Quantile Distributional Reinforcement Learning
Abstract
In this paper, we study quantile distributional policy evaluation, where the return distribution under a given policy is approximated by the quantile Bellman fixed point ηm. We investigate the statistical accuracy of estimating ηm across quantile resolutions under Wasserstein distances. Assuming access to a generative model, we construct a quantile fixed point estimator ηm(n) from an empirical Markov decision process and establish non-asymptotic error bounds under Wasserstein distances. When reward densities are uniformly bounded away from zero, we obtain a root-n high-probability bound simultaneously over all resolutions m and all Wasserstein orders 1≤ p≤∞. When reward densities vanish at the support endpoints, we show that continuity and endpoint monotonicity suffice to preserve the same resolution-independent rate for 1≤ p≤2. For p>2, we establish a root-n bound with coefficient m1/2−1/p, together with a slower, resolution-independent n−1/p bound for every finite p>2. We further obtain resolution-independent W∞ consistency governed by the continuity modulus of reward quantile functions. To investigate the sharpness of these bounds, we characterize the exact local asymptotic minimax risk at each fixed resolution as the expected Wasserstein-coordinate norm of the efficient Gaussian limit and prove that our estimator attains it. We then examine the resolution dependence of this optimal local risk. A regular one-state instance has resolution-independent local risk constants, whereas a single endpoint-vanishing two-state instance exhibits growth of order m1/2−1/p, up to logarithmic factors along a resolution subsequence, for every p>2.
Related
- Online Inference for Quantile Distributional Reinforcement Learning
- Near-Minimax Sample Complexity for Multi-Objective Reinforcement Learning under Generative Models
- Minimax-Optimality of Posterior Sampling for Reinforcement Learning
- Deep Reinforcement Learning with Buffered Quantile Objectives
- Reinforcement Learning for Hierarchical Reasoning Rewards: Minimax-Optimal Rates with Transformers
BibTeX
@misc{cheng2026near,
title = {{Near Minimax Rate for Quantile Distributional Reinforcement Learning}},
author = {Zijie Cheng and Zhihua Zhang},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.29595},
eprint = {2610.29595},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.29595},
}
RIS
TY - UNPB
TI - Near Minimax Rate for Quantile Distributional Reinforcement Learning
AU - Zijie Cheng
AU - Zhihua Zhang
PY - 2026
DA - 2026/10/21
PB - arXiv
DO - 10.48550/arxiv.2610.29595
UR - https://arxiv.org/abs/2610.29595
AB - In this paper, we study quantile distributional policy evaluation, where the return distribution under a given policy is approximated by the quantile Bellman fixed point η_m. We investigate the statistical accuracy of estimating η_m across quantile resolutions under Wasserstein distances. Assuming access to a generative model, we construct a quantile fixed point estimator η_m^(n) from an empirical Markov decision process and establish non-asymptotic error bounds under Wasserstein distances. When reward densities are uniformly bounded away from zero, we obtain a root-n high-probability bound simultaneously over all resolutions m and all Wasserstein orders 1≤ p≤∞. When reward densities vanish at the support endpoints, we show that continuity and endpoint monotonicity suffice to preserve the same resolution-independent rate for 1≤ p≤2. For p>2, we establish a root-n bound with coefficient m^1/2−1/p, together with a slower, resolution-independent n^−1/p bound for every finite p>2. We further obtain resolution-independent W_∞ consistency governed by the continuity modulus of reward quantile functions. To investigate the sharpness of these bounds, we characterize the exact local asymptotic minimax risk at each fixed resolution as the expected Wasserstein-coordinate norm of the efficient Gaussian limit and prove that our estimator attains it. We then examine the resolution dependence of this optimal local risk. A regular one-state instance has resolution-independent local risk constants, whereas a single endpoint-vanishing two-state instance exhibits growth of order m^1/2−1/p, up to logarithmic factors along a resolution subsequence, for every p>2.
ER -