Policy Iteration over Terminal Cost Functions: From LQR to MPC
Abstract
Policy iteration is a classical method for successively improving control policies in optimal control and reinforcement learning. In this paper, we propose a policy iteration framework that refines receding-horizon control (RHC) and model predictive control (MPC) policies online by iteratively updating the terminal cost. We first develop the framework for the linear quadratic regulator (LQR) problem. We show that the Riccati map is a contraction in the Thompson metric on an explicitly constructed invariant order interval. Using this property, we prove convergence of approximate policy iteration for finite-horizon RHC, where the terminal cost is given by an approximate value function. From an initial terminal cost, we then construct a robust invariant order interval within which the RHC policy is refined online. We extend the analysis to constrained MPC and present a constructive algorithm for safely updating the terminal ingredients, which guarantees recursive feasibility and asymptotic stability of the closed loop throughout the iterations. These results connect classical Riccati theory with approximate policy iteration and provide a principled pathway toward learning MPC terminal ingredients. A numerical example illustrates the proposed method.
Related
- Policy iteration with discounted cost: near-optimality bounds and stability guarantees ⋆
- The Effect of Quadrature on the Convergence of Policy Iteration for Hamilton-Jacobi-Bellman Equations
- Policy iteration for Hamilton-Jacobi-Isaacs equations with control constraints and comparison with Hamilton-Jacobi-Bellman equations
- Policy Iteration for Domain Randomized Linear Quadratic Systems
- Continuous Policy and Value Iteration for Stochastic Control Problems with Inhomogeneous Synchronous Coupling
BibTeX
@misc{baltussen2026policy,
title = {{Policy Iteration over Terminal Cost Functions: From LQR to MPC}},
author = {Tren M. J. T. Baltussen and Nathan P. Lawrence and Alexander Katriniok and Ali Mesbah and W. P. M. H. and Heemels},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.03401},
eprint = {2611.03401},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.03401},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - Policy Iteration over Terminal Cost Functions: From LQR to MPC
AU - Tren M. J. T. Baltussen
AU - Nathan P. Lawrence
AU - Alexander Katriniok
AU - Ali Mesbah
AU - W. P. M. H.
AU - Heemels
PY - 2026
DA - 2026/11/02
PB - arXiv
DO - 10.48550/arxiv.2611.03401
UR - https://arxiv.org/abs/2611.03401
AB - Policy iteration is a classical method for successively improving control policies in optimal control and reinforcement learning. In this paper, we propose a policy iteration framework that refines receding-horizon control (RHC) and model predictive control (MPC) policies online by iteratively updating the terminal cost. We first develop the framework for the linear quadratic regulator (LQR) problem. We show that the Riccati map is a contraction in the Thompson metric on an explicitly constructed invariant order interval. Using this property, we prove convergence of approximate policy iteration for finite-horizon RHC, where the terminal cost is given by an approximate value function. From an initial terminal cost, we then construct a robust invariant order interval within which the RHC policy is refined online. We extend the analysis to constrained MPC and present a constructive algorithm for safely updating the terminal ingredients, which guarantees recursive feasibility and asymptotic stability of the closed loop throughout the iterations. These results connect classical Riccati theory with approximate policy iteration and provide a principled pathway toward learning MPC terminal ingredients. A numerical example illustrates the proposed method.
ER -