Penalty-Uniform Localization for State-Constrained Policy Iteration
Abstract
Penalizing a state constraint creates a singular localization problem: whole-space values may grow like the inverse penalty parameter, while numerical diffusion allows even an inward feedback to cross the boundary. For deterministic discounted optimal control, we show how the penalty itself supplies confinement that offsets this growth. With mesh size h and penalty parameter ε, an inward barrier bounds the additional cost of numerical leakage by O(h/ε) for a monotone centered-difference scheme with vanishing viscosity. Under h≤ε, an occupation estimate then yields O(h) localization error with a sufficient box margin logarithmic in 1/h and independent of ε. We extend the result to bounded squared-distance penalties and combine it with residual-based evaluation bounds and discounted policy-error propagation. Under coupled refinement with vanishing penalization and discretization errors, explicit conditions on evaluation errors and greedy gaps ensure convergence of the neural value approximations to the constrained value, allowing measurable, nonunique greedy selectors. Reference calculations isolate leakage and localization across mesh and penalty scales, and distinguish evaluation, iteration, and approximation errors. An explicit cylindrical state-constraint solution provides a benchmark in arbitrary dimension, tested up to dimension twenty. Paired obstacle-navigation experiments illustrate why policy-value accuracy must be assessed alongside sampled residuals when comparing raw-residual and finite-grid-assisted neural evaluation.
Related
- Self-Consistent Adjoint Policy Iteration for Constrained Dynamic Portfolio Choice
- Policy iteration for Hamilton-Jacobi-Isaacs equations with control constraints and comparison with Hamilton-Jacobi-Bellman equations
- Policy Iteration for Domain Randomized Linear Quadratic Systems
- Continuous Policy and Value Iteration for Stochastic Control Problems with Inhomogeneous Synchronous Coupling
- Policy Iteration Is Not Strongly Polynomial for Deterministic Markov Decision Processes: The Price of Algorithmic Anarchy
BibTeX
@misc{kim2026penaltyuniform,
title = {{Penalty-Uniform Localization for State-Constrained Policy Iteration}},
author = {Yeongjong Kim and Jiwoong Jang and Yeoneung Kim},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.16829},
eprint = {2609.16829},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.16829},
}
RIS
TY - UNPB
TI - Penalty-Uniform Localization for State-Constrained Policy Iteration
AU - Yeongjong Kim
AU - Jiwoong Jang
AU - Yeoneung Kim
PY - 2026
DA - 2026/09/15
PB - arXiv
DO - 10.48550/arxiv.2609.16829
UR - https://arxiv.org/abs/2609.16829
AB - Penalizing a state constraint creates a singular localization problem: whole-space values may grow like the inverse penalty parameter, while numerical diffusion allows even an inward feedback to cross the boundary. For deterministic discounted optimal control, we show how the penalty itself supplies confinement that offsets this growth. With mesh size h and penalty parameter ε, an inward barrier bounds the additional cost of numerical leakage by O(h/ε) for a monotone centered-difference scheme with vanishing viscosity. Under h≤ε, an occupation estimate then yields O(h) localization error with a sufficient box margin logarithmic in 1/h and independent of ε. We extend the result to bounded squared-distance penalties and combine it with residual-based evaluation bounds and discounted policy-error propagation. Under coupled refinement with vanishing penalization and discretization errors, explicit conditions on evaluation errors and greedy gaps ensure convergence of the neural value approximations to the constrained value, allowing measurable, nonunique greedy selectors. Reference calculations isolate leakage and localization across mesh and penalty scales, and distinguish evaluation, iteration, and approximation errors. An explicit cylindrical state-constraint solution provides a benchmark in arbitrary dimension, tested up to dimension twenty. Paired obstacle-navigation experiments illustrate why policy-value accuracy must be assessed alongside sampled residuals when comparing raw-residual and finite-grid-assisted neural evaluation.
ER -