Safe Multi-Objective Reinforcement Learning: Zero Constraint Violation and Primal-Averaging Convergence
Abstract
Many applications of multi-objective reinforcement learning (MORL) require policies that balance conflicting objectives while satisfying safety constraints. We study safe MORL with zero constraint violation throughout learning, covering both cumulative and stepwise constraints. A key challenge is to explore sufficiently under unknown dynamics and constraint costs without violating the constraints. For a given preference over objectives, we develop an algorithm to learn a safe Pareto optimal policy by combining optimistic reward estimates with pessimistic cost bounds. With high probability, every policy executed over T episodes incurs zero constraint violation. The algorithm returns a safe Markov policy induced by the averaged occupancy measure and achieves an Õ(1/√T) suboptimality gap under smooth Tchebycheff scalarization. We further develop a preference-free algorithm that first explores safely without a prespecified preference and then computes a safe policy for any given preference without further interaction. Over T exploration episodes and K planning iterations, it retains the same safety guarantee and achieves an Õ(1/√T+1/√K) suboptimality gap. Both algorithms require knowledge of a policy that strictly satisfies the constraints and its costs. We then relax these requirements, incurring at most constant cumulative violation. To our knowledge, we provide the first sample-efficient MORL algorithms that guarantee safety for every executed policy.
Related
- Blind Random Search with Noisy Loss Measurements: Averaging, Thresholding, and Almost Sure Convergence
- Dynamic Regret and Tightened Constraint Violation for Communication-Efficient Distributed Online Bilevel Optimization
- AlphaPareto: Formulaic Alpha Discovery with LLM-Guided Multi-Objective Reinforcement Learning
- Multi-Objective Reinforcement Learning Using Routed Ensembles and Trajectory Attribution
- Quantitative Soft-to-Hard Terminal Constraint Convergence for the Heat Equation
BibTeX
@misc{li2026safe,
title = {{Safe Multi-Objective Reinforcement Learning: Zero Constraint Violation and Primal-Averaging Convergence}},
author = {Chenxi Li and Yao Shu and Siyang Gao and Shuang Qiu and Guangxin Jiang},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.39409},
eprint = {2610.39409},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.39409},
}
RIS
TY - UNPB
TI - Safe Multi-Objective Reinforcement Learning: Zero Constraint Violation and Primal-Averaging Convergence
AU - Chenxi Li
AU - Yao Shu
AU - Siyang Gao
AU - Shuang Qiu
AU - Guangxin Jiang
PY - 2026
DA - 2026/10/29
PB - arXiv
DO - 10.48550/arxiv.2610.39409
UR - https://arxiv.org/abs/2610.39409
AB - Many applications of multi-objective reinforcement learning (MORL) require policies that balance conflicting objectives while satisfying safety constraints. We study safe MORL with zero constraint violation throughout learning, covering both cumulative and stepwise constraints. A key challenge is to explore sufficiently under unknown dynamics and constraint costs without violating the constraints. For a given preference over objectives, we develop an algorithm to learn a safe Pareto optimal policy by combining optimistic reward estimates with pessimistic cost bounds. With high probability, every policy executed over T episodes incurs zero constraint violation. The algorithm returns a safe Markov policy induced by the averaged occupancy measure and achieves an Õ(1/√T) suboptimality gap under smooth Tchebycheff scalarization. We further develop a preference-free algorithm that first explores safely without a prespecified preference and then computes a safe policy for any given preference without further interaction. Over T exploration episodes and K planning iterations, it retains the same safety guarantee and achieves an Õ(1/√T+1/√K) suboptimality gap. Both algorithms require knowledge of a policy that strictly satisfies the constraints and its costs. We then relax these requirements, incurring at most constant cumulative violation. To our knowledge, we provide the first sample-efficient MORL algorithms that guarantee safety for every executed policy.
ER -