Deep Reinforcement Learning with Buffered Quantile Objectives

Abstract

Quantile-based reinforcement learning provides an interpretable approach to risk-sensitive decision-making by optimizing a prescribed quantile of the cumulative-return distribution. Despite this appeal, learning under a point quantile objective is challenging: quantiles can change abruptly under small perturbations of the return distribution, and exact quantile-sensitive planning requires computationally demanding distributional optimization. Lower-buffered quantiles alleviate the former difficulty by averaging neighboring quantiles immediately below the target level, providing a smoother surrogate while preserving the underlying point-quantile objective. Existing methods based on this principle, however, remain model-based and rely on explicit return-law planning, limiting their applicability beyond small tabular problems. We develop Deep-BQRL, a model-free distributional reinforcement-learning framework that extends buffered-quantile learning to neural function approximation. The method learns conditional return quantiles directly from sampled transitions, constructs buffered action scores from the relevant region of the learned quantile function, and uses ensemble disagreement to guide exploration. An augmented input representation allows the learned policy to respond to trajectory information without explicitly reproducing the quantile-state recursion required by exact planning. Experiments on an asset-selling optimal-stopping problem and slippery FrozenLake compare Deep-BQRL with model-based UCB-BQRL and tabular PPO and TRPO implementations. In asset selling, Deep-BQRL attains smaller mean cumulative point-quantile policy gaps than PPO and TRPO at the reported target levels, while UCB-BQRL retains the smallest gaps. The learned stopping decisions also vary with the target quantile, providing an interpretable illustration of the method's risk-sensitive behavior.

Online Inference for Quantile Distributional Reinforcement Learning
Zijie Cheng, Yang Peng, Zhihua Zhang
arXiv16 Oct 2026PreprintarXiv licenceFull textStatisticsarXiv
Near Minimax Rate for Quantile Distributional Reinforcement Learning
Zijie Cheng, Zhihua Zhang
arXiv21 Oct 2026PreprintarXiv licenceFull textStatisticsarXiv
Deep Reinforcement Learning for Blind Passive Beamforming in RIS-Assisted MIMO Systems
Oumaïma Bounhar, Thomas Chêne, Ghaya Rekaya-Ben Othman +1
arXiv29 Oct 2026PreprintCC BY-NC-SA 4.0Full textStatisticsarXiv
Bayesian Neural Networks via Dirichlet Process Priors on the Data-Generating Distribution for Randomized Value Functions in Deep Reinforcement Learning
Sumit Vashishtha
arXiv20 Oct 2026PreprintCC BY 4.0Full textStatisticsarXiv
Efficient Joint Trajectory Optimization and Resource Allocation for Secure UAV Networks: A Deep Reinforcement Learning Approach
Abdulmalik Alwarafy, Mohamed Hayajneh, Mustafa Al Juboori +4
arXiv27 Oct 2026PreprintCC BY 4.0Full textGeneral mathematicsarXiv

Cite

BibTeX RIS

BibTeX

@misc{alipourvaezi2026deep,
  title = {{Deep Reinforcement Learning with Buffered Quantile Objectives}},
  author = {Mohammad Alipour-vaezi and Sajad Khodadadian},
  howpublished = {arXiv},
  year = {2026},
  month = sep,
  doi = {10.48550/arxiv.2609.21327},
  eprint = {2609.21327},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.21327},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - Deep Reinforcement Learning with Buffered Quantile Objectives
AU  - Mohammad Alipour-vaezi
AU  - Sajad Khodadadian
PY  - 2026
DA  - 2026/09/18
PB  - arXiv
DO  - 10.48550/arxiv.2609.21327
UR  - https://arxiv.org/abs/2609.21327
AB  - Quantile-based reinforcement learning provides an interpretable approach to risk-sensitive decision-making by optimizing a prescribed quantile of the cumulative-return distribution. Despite this appeal, learning under a point quantile objective is challenging: quantiles can change abruptly under small perturbations of the return distribution, and exact quantile-sensitive planning requires computationally demanding distributional optimization. Lower-buffered quantiles alleviate the former difficulty by averaging neighboring quantiles immediately below the target level, providing a smoother surrogate while preserving the underlying point-quantile objective. Existing methods based on this principle, however, remain model-based and rely on explicit return-law planning, limiting their applicability beyond small tabular problems. We develop Deep-BQRL, a model-free distributional reinforcement-learning framework that extends buffered-quantile learning to neural function approximation. The method learns conditional return quantiles directly from sampled transitions, constructs buffered action scores from the relevant region of the learned quantile function, and uses ensemble disagreement to guide exploration. An augmented input representation allows the learned policy to respond to trajectory information without explicitly reproducing the quantile-state recursion required by exact planning. Experiments on an asset-selling optimal-stopping problem and slippery FrozenLake compare Deep-BQRL with model-based UCB-BQRL and tabular PPO and TRPO implementations. In asset selling, Deep-BQRL attains smaller mean cumulative point-quantile policy gaps than PPO and TRPO at the reported target levels, while UCB-BQRL retains the smallest gaps. The learned stopping decisions also vary with the target quantile, providing an interpretable illustration of the method's risk-sensitive behavior.
ER  -

Abstract

Related

Cite