Policy iteration with discounted cost: near-optimality bounds and stability guarantees ⋆
Abstract
We investigate the stability and near-optimality properties of deterministic discrete-time systems whose inputs are generated by policy iteration (PI) considering a discounted infinite-horizon cost function. We leverage the discounting of the cost to initialize PI with policies that may not be stabilizing. Novel bounds are first established on the difference between the cost induced by a policy generated by PI at each iteration and the optimal value function, which are typically tighter than those encountered in the dynamic programming literature for the considered set-up. Under general assumptions, we then prove that the systems whose inputs are given by a policy generated by PI satisfy a semiglobal practical stability for discount factors sufficiently close to 1 and after a finite number of iterations that we can estimate. Global exponential stability is also ensured under strengthened assumptions. The theoretical findings are illustrated via examples and two notable case studies: the discounted linear quadratic regulator and the discounted absolute value bound problem.
Related
- Algorithmic Optimality Guarantees for Nonsmooth H∞ Output-Feedback Policy Search
- Benign Nonconvex Landscape for Policy Optimization: Infinite-Horizon Discounted MDPs with General State and Action Spaces
- Policy iteration for Hamilton-Jacobi-Isaacs equations with control constraints and comparison with Hamilton-Jacobi-Bellman equations
- Policy Iteration for Domain Randomized Linear Quadratic Systems
- Continuous Policy and Value Iteration for Stochastic Control Problems with Inhomogeneous Synchronous Coupling
BibTeX
@article{brusse2026policy,
title = {{Policy iteration with discounted cost: near-optimality bounds and stability guarantees ⋆}},
author = {Jonathan de Brusse and Mathieu Granzotto and Romain Postoyan and Dragan Nešić},
journal = {Automatica},
year = {2026},
month = aug,
doi = {10.1016/j.automatica.2026.113158},
url = {https://hal.science/hal-05731949v1},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Policy iteration with discounted cost: near-optimality bounds and stability guarantees ⋆
AU - Jonathan de Brusse
AU - Mathieu Granzotto
AU - Romain Postoyan
AU - Dragan Nešić
PY - 2026
DA - 2026/08/31
JO - Automatica
DO - 10.1016/j.automatica.2026.113158
UR - https://hal.science/hal-05731949v1
AB - We investigate the stability and near-optimality properties of deterministic discrete-time systems whose inputs are generated by policy iteration (PI) considering a discounted infinite-horizon cost function. We leverage the discounting of the cost to initialize PI with policies that may not be stabilizing. Novel bounds are first established on the difference between the cost induced by a policy generated by PI at each iteration and the optimal value function, which are typically tighter than those encountered in the dynamic programming literature for the considered set-up. Under general assumptions, we then prove that the systems whose inputs are given by a policy generated by PI satisfy a semiglobal practical stability for discount factors sufficiently close to 1 and after a finite number of iterations that we can estimate. Global exponential stability is also ensured under strengthened assumptions. The theoretical findings are illustrated via examples and two notable case studies: the discounted linear quadratic regulator and the discounted absolute value bound problem.
SN - 0005-1098
ER -