Skip to content
DraconiticResearch
Search Sources API
203,234 studies31 sourcesUpdated 2 min ago
PreprintCC BY 4.0arXiv

Online Inference in Distributional Temporal-Difference Learning

  • Yang Peng
  • Liangyu Zhang
Open at sourcePDFDOICite

Abstract

We study online statistical inference for functionals of the return distribution under a fixed policy. The return distribution is estimated by nonparametric distributional temporal-difference learning from a single Markov trajectory. For the Polyak--Ruppert averaged estimator, we prove that its root-T error converges weakly to a centered Gaussian random element in Cramér space. We also prove that, conditionally on the observed trajectory, the root-T difference between the bootstrap and original averages converges weakly to the same Gaussian limit. These results justify bootstrap inference for smooth statistical functionals, including variance, CVaR, expected shortfall, and expectiles. For nonsmooth statistical functionals, we develop a local asymptotic theory for the estimated return CDF over T−1/2-neighborhoods of finitely many thresholds, together with its bootstrap analogue. This theory allows us to conduct inference for nonsmooth statistical functionals characterized by CDF equations, including return quantiles.

Related

  1. Online Inference for Quantile Distributional Reinforcement Learning
    Zijie Cheng, Yang Peng, Zhihua Zhang
    arXivPreprintarXiv licenceFull textStatisticsarXiv
  2. A Sharp Finite-Iteration Theory for Asynchronous Categorical Distributional Temporal-Difference Learning
    Ege C. Kaya, Abolfazl Hashemi
    arXivPreprintCC BY 4.0Full textOptimisation & controlarXiv
  3. Conditional Tensor Diffusion: Distributional Counterfactual Learning and Inference
    Xinbing Kong, Zeyu Li, Junfan Mao +1
    arXivPreprintarXiv licenceFull textStatisticsarXiv
  4. On Convergence of Gradient Temporal-Difference Learning with Regularized Corrections and Eligibility Traces
    Jingyi Jin, Martha White, Hao Wang
    arXivPreprintCC BY-NC-ND 4.0Full textOptimisation & controlarXiv
  5. A Finite-Sample Analysis of Quantile Temporal-Difference Learning
    Zijie Cheng, Xiang Li, Yang Peng +1
    arXivPreprintarXiv licenceFull textStatisticsarXiv

Cite

BibTeXRIS
BibTeX
@misc{peng2026online,
  title = {{Online Inference in Distributional Temporal-Difference Learning}},
  author = {Yang Peng and Liangyu Zhang},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2608.14408},
  eprint = {2608.14408},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.14408},
  note = {CC BY 4.0},
}
RIS
TY  - UNPB
TI  - Online Inference in Distributional Temporal-Difference Learning
AU  - Yang Peng
AU  - Liangyu Zhang
PY  - 2026
DA  - 2026/10/16
PB  - arXiv
DO  - 10.48550/arxiv.2608.14408
UR  - https://arxiv.org/abs/2608.14408
AB  - We study online statistical inference for functionals of the return distribution under a fixed policy. The return distribution is estimated by nonparametric distributional temporal-difference learning from a single Markov trajectory. For the Polyak--Ruppert averaged estimator, we prove that its root-T error converges weakly to a centered Gaussian random element in Cramér space. We also prove that, conditionally on the observed trajectory, the root-T difference between the bootstrap and original averages converges weakly to the same Gaussian limit. These results justify bootstrap inference for smooth statistical functionals, including variance, CVaR, expected shortfall, and expectiles. For nonsmooth statistical functionals, we develop a local asymptotic theory for the estimated return CDF over T^−1/2-neighborhoods of finitely many thresholds, together with its bootstrap analogue. This theory allows us to conduct inference for nonsmooth statistical functionals characterized by CDF equations, including return quantiles.
ER  -