Near-Optimal Sample Complexity for Recursive Entropic Risk Reinforcement Learning with a Generative Model
Abstract
In this paper, we study the sample complexities of value and policy learning in finite discounted Markov decision processes (MDPs) under recursive entropic risk preferences with risk parameter β≠ 0, assuming access to a generative model of the MDP. We provide a refined analysis of model-based risk-sensitive Q-value iteration (MB-RS-QVI), a plug-in model-based method introduced in prior work, and derive (ε,δ)-PAC guarantees for both learning the optimal Q-value function and an ε-optimal policy. Our bounds improve the exponential dependence on the effective horizon 1/(1−γ) compared with the best existing guarantees for this setting. In particular, they match the existing lower bounds in their exponential dependence on |β|/(1−γ), as well as in S, A, ε, and |β|, up to logarithmic factors. Consequently, our analysis removes the exponential gap between the previously known upper and lower bounds, leaving only a polynomial gap in the effective horizon.
Related
- Near-Minimax Sample Complexity for Multi-Objective Reinforcement Learning under Generative Models
- Sample Complexity of Equivariant Reinforcement Learning
- Policy Complexity, Reaction Time, and Bounded Rationality in Reinforcement Learning
- Tight Sample Complexity Bounds for Entropic Best Policy Identification
- Low-Budget Active Learning through Entropic Optimal Transport
BibTeX
@misc{bahrami2026nearoptimal,
title = {{Near-Optimal Sample Complexity for Recursive Entropic Risk Reinforcement Learning with a Generative Model}},
author = {Amirparsa Bahrami and Oliver Mortensen and Mohammad Sadegh Talebi},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.06931},
eprint = {2610.06931},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.06931},
}
RIS
TY - UNPB
TI - Near-Optimal Sample Complexity for Recursive Entropic Risk Reinforcement Learning with a Generative Model
AU - Amirparsa Bahrami
AU - Oliver Mortensen
AU - Mohammad Sadegh Talebi
PY - 2026
DA - 2026/10/03
PB - arXiv
DO - 10.48550/arxiv.2610.06931
UR - https://arxiv.org/abs/2610.06931
AB - In this paper, we study the sample complexities of value and policy learning in finite discounted Markov decision processes (MDPs) under recursive entropic risk preferences with risk parameter β≠ 0, assuming access to a generative model of the MDP. We provide a refined analysis of model-based risk-sensitive Q-value iteration (MB-RS-QVI), a plug-in model-based method introduced in prior work, and derive (ε,δ)-PAC guarantees for both learning the optimal Q-value function and an ε-optimal policy. Our bounds improve the exponential dependence on the effective horizon 1/(1−γ) compared with the best existing guarantees for this setting. In particular, they match the existing lower bounds in their exponential dependence on |β|/(1−γ), as well as in S, A, ε, and |β|, up to logarithmic factors. Consequently, our analysis removes the exponential gap between the previously known upper and lower bounds, leaving only a polynomial gap in the effective horizon.
ER -