Certifying Lower Bounds for Risk-Sensitive Reinforcement Learning under Adversarial State Perturbations
Abstract
Reinforcement learning (RL) agents deployed in real-world environments are often vulnerable to adversarial perturbations in state observations, creating risks in safety-critical applications. Certification methods can improve robustness against adversarial perturbations by providing lower bounds on expected cumulative rewards. Existing certification methods, however, mainly focus on risk-neutral objectives. In this paper, we extend certification methods to risk-sensitive objectives by establishing lower bounds on the exponential utility of cumulative rewards under lp-norm-bounded state adversarial perturbations (1≤ p <∞). By introducing a ϕ-divergence relaxation of the perturbation set, we formulate the risk-sensitive certification problem as a convex optimization and derive its dual to obtain a tractable approximation of the certified lower bound. We further propose an empirical method that improves certified lower bounds by selecting the training risk-aversion parameter β independently of the risk level used during evaluation. Experiments on both OpenAI Gym environments and a machine replacement problem show that, compared to risk-neutral training, risk-averse training generally yields policies with higher certified lower bounds, particularly under larger perturbation budgets. Moreover, under both risk-neutral and risk-averse evaluation settings, increasing risk aversion during training leads to non-monotonic certification performance, where certified lower bounds initially improve but eventually decrease due to overly conservative policies.
Related
- Provably Efficient Risk-Sensitive Offline Reinforcement Learning via Optimized Certainty Equivalents
- Nonparametric Variance-Penalized Actor-Critic: Statistical Inference for Risk-Sensitive Reinforcement Learning
- Risk-Sensitive Mission Planning for Multi-Target On-Orbit Inspection via Distributional Reinforcement Learning
- Radial–Angular Feature Shaping with an eXtra-Node and Prototype-Based Cosine Similarity Under Adversarial Perturbations
- Certified Spherical MUSIC for 3D Localization under Adversarial Subspace Perturbations
BibTeX
@misc{li2026certifying,
title = {{Certifying Lower Bounds for Risk-Sensitive Reinforcement Learning under Adversarial State Perturbations}},
author = {Tong Li and Saunak Kumar Panda and Yisha Xiang},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.10866},
eprint = {2609.10866},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.10866},
}
RIS
TY - UNPB
TI - Certifying Lower Bounds for Risk-Sensitive Reinforcement Learning under Adversarial State Perturbations
AU - Tong Li
AU - Saunak Kumar Panda
AU - Yisha Xiang
PY - 2026
DA - 2026/09/09
PB - arXiv
DO - 10.48550/arxiv.2609.10866
UR - https://arxiv.org/abs/2609.10866
AB - Reinforcement learning (RL) agents deployed in real-world environments are often vulnerable to adversarial perturbations in state observations, creating risks in safety-critical applications. Certification methods can improve robustness against adversarial perturbations by providing lower bounds on expected cumulative rewards. Existing certification methods, however, mainly focus on risk-neutral objectives. In this paper, we extend certification methods to risk-sensitive objectives by establishing lower bounds on the exponential utility of cumulative rewards under l_p-norm-bounded state adversarial perturbations (1≤ p <∞). By introducing a ϕ-divergence relaxation of the perturbation set, we formulate the risk-sensitive certification problem as a convex optimization and derive its dual to obtain a tractable approximation of the certified lower bound. We further propose an empirical method that improves certified lower bounds by selecting the training risk-aversion parameter β independently of the risk level used during evaluation. Experiments on both OpenAI Gym environments and a machine replacement problem show that, compared to risk-neutral training, risk-averse training generally yields policies with higher certified lower bounds, particularly under larger perturbation budgets. Moreover, under both risk-neutral and risk-averse evaluation settings, increasing risk aversion during training leads to non-monotonic certification performance, where certified lower bounds initially improve but eventually decrease due to overly conservative policies.
ER -