Provably Efficient Risk-Sensitive Offline Reinforcement Learning via Optimized Certainty Equivalents

Abstract

Risk-sensitive reinforcement learning (RL) addresses decision-making under uncertainty and adverse outcomes. Learning risk-sensitive policies from fixed datasets is important when additional interaction with the environment is costly. Existing work on risk-sensitive RL focuses on online settings or specific risk measures, leaving a general theoretical understanding of its offline setting underdeveloped. In this work, we investigate efficient offline risk-sensitive RL using the optimized certainty equivalent (OCE), a general class of risk measures. We consider two formulations: dynamic-OCE RL, which applies OCE recursively at each step, and static-OCE RL, which applies a single OCE to the cumulative rewards. Under the tabular setting, we develop pessimistic value-iteration algorithms for both formulations and establish tight sublinear suboptimality bounds, explicitly capturing how risk sensitivity, encoded by utility functions, affects learning difficulty. Moreover, we establish minimax suboptimality lower bounds to characterize the problem's fundamental limits. To handle large state spaces, we further develop pessimistic least-squares value-iteration algorithms that incorporate the general OCE risk criterion into linear function approximation. We establish utility-dependent sublinear guarantees with explicit dependence on the feature dimension d. To our knowledge, we provide the first offline algorithms along with the first upper and lower suboptimality bounds for both dynamic- and static-OCE RL.

Conformal Risk-Averse Decision Making with Optimized Certainty Equivalent Risk Control
Amirmohammad Farzaneh, Osvaldo Simeone
arXiv28 Aug 2026PreprintCC BY 4.0Full textStatisticsarXiv
Certifying Lower Bounds for Risk-Sensitive Reinforcement Learning under Adversarial State Perturbations
Tong Li, Saunak Kumar Panda, Yisha Xiang
arXiv9 Sep 2026PreprintarXiv licenceFull textOptimisation & controlarXiv
Nonparametric Variance-Penalized Actor-Critic: Statistical Inference for Risk-Sensitive Reinforcement Learning
Saunak Kumar Panda, Tong Li, Yisha Xiang +1
arXiv13 Sep 2026PreprintarXiv licenceFull textStatisticsarXiv
Risk-Sensitive Mission Planning for Multi-Target On-Orbit Inspection via Distributional Reinforcement Learning
Adam Abdin
arXiv30 Oct 2026PreprintarXiv licenceFull textOptimisation & controlarXiv
Parallel Optimized Particle Swarm Optimization: An Efficient Parallel Approach for Neural Network Training
Konstantinos G. Barkas, Ioannis G. Tsoulos, Vasileios Charilogis +1
Mathematics3 Oct 2026Peer-reviewedCC BY 4.0General mathematicsCrossref

Cite

BibTeX RIS

BibTeX

@misc{wang2026provably,
  title = {{Provably Efficient Risk-Sensitive Offline Reinforcement Learning via Optimized Certainty Equivalents}},
  author = {Hanzhen Wang and Boxiang Lyu and Zhongxiang Dai and Siyang Gao and Dake Zhang and Shuang Qiu},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.21870},
  eprint = {2610.21870},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.21870},
}

RIS

TY  - UNPB
TI  - Provably Efficient Risk-Sensitive Offline Reinforcement Learning via Optimized Certainty Equivalents
AU  - Hanzhen Wang
AU  - Boxiang Lyu
AU  - Zhongxiang Dai
AU  - Siyang Gao
AU  - Dake Zhang
AU  - Shuang Qiu
PY  - 2026
DA  - 2026/10/15
PB  - arXiv
DO  - 10.48550/arxiv.2610.21870
UR  - https://arxiv.org/abs/2610.21870
AB  - Risk-sensitive reinforcement learning (RL) addresses decision-making under uncertainty and adverse outcomes. Learning risk-sensitive policies from fixed datasets is important when additional interaction with the environment is costly. Existing work on risk-sensitive RL focuses on online settings or specific risk measures, leaving a general theoretical understanding of its offline setting underdeveloped. In this work, we investigate efficient offline risk-sensitive RL using the optimized certainty equivalent (OCE), a general class of risk measures. We consider two formulations: dynamic-OCE RL, which applies OCE recursively at each step, and static-OCE RL, which applies a single OCE to the cumulative rewards. Under the tabular setting, we develop pessimistic value-iteration algorithms for both formulations and establish tight sublinear suboptimality bounds, explicitly capturing how risk sensitivity, encoded by utility functions, affects learning difficulty. Moreover, we establish minimax suboptimality lower bounds to characterize the problem's fundamental limits. To handle large state spaces, we further develop pessimistic least-squares value-iteration algorithms that incorporate the general OCE risk criterion into linear function approximation. We establish utility-dependent sublinear guarantees with explicit dependence on the feature dimension d. To our knowledge, we provide the first offline algorithms along with the first upper and lower suboptimality bounds for both dynamic- and static-OCE RL.
ER  -

Abstract

Related

Cite