Parallel Policy-Gradient Methods for Parameter Optimization of Nonlinear Feedback Controllers
Abstract
Structured feedback controllers provide rigorous stability guarantees, but often require manual parameter tuning to achieve good closed-loop performance. Policy-gradient methods offer a systematic approach to parameter optimization; however, conventional gradient evaluation requires sequential forward state rollout and backward costate propagation. This letter develops a time-parallel policy-gradient framework for discrete-time nonlinear control-affine systems. We derive the policy-gradient expression where the state and costate rollouts required for policy-gradient evaluation are formulated as residual-minimization problems and solved using Gauss-Newton (GN) iterations with parallel associative scans. For closed-loop systems that are globally asymptotically stable and locally exponentially stable, we show that the residual-minimization problems satisfy a local Polyak-Lojasiewicz (PL) inequality and that the GN iterates converge locally at a quadratic rate. Moreover, the PL constant, the size of the convergence neighborhood, and the quadratic convergence bound are independent of the rollout horizon T. We also prove that, for any finite horizon T, the state solver recovers the exact trajectory from any initialization in at most T iterations. Finally, an inertia-wheel pendulum example with interconnection and damping assignment passivity-based control (IDA-PBC) demonstrates improved closed-loop performance and the computational benefits of the proposed parallel policy-gradient framework.
Related
- Optimization Strategy of PID-Inspired Metaheuristic Algorithm-Tuned Fractional-Order Controllers Applied to Nonlinear AVR System
- Global optimization by random perturbation of the gradient method with a fixed parameter
- Initial condition recovery in nonlinear damped viscous photoacoustic tomography using a convolutional neural network-guided gradient-free optimization framework
- Nonparametric autoregressive estimation of probability distributions with parallel gradient boosting
- On Parameter Symmetries and Conservation Laws in Gradient Flow
BibTeX
@misc{nguyen2026parallel,
title = {{Parallel Policy-Gradient Methods for Parameter Optimization of Nonlinear Feedback Controllers}},
author = {An Nguyen and Leilei Cui},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.14114},
eprint = {2609.14114},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.14114},
}
RIS
TY - UNPB
TI - Parallel Policy-Gradient Methods for Parameter Optimization of Nonlinear Feedback Controllers
AU - An Nguyen
AU - Leilei Cui
PY - 2026
DA - 2026/09/12
PB - arXiv
DO - 10.48550/arxiv.2609.14114
UR - https://arxiv.org/abs/2609.14114
AB - Structured feedback controllers provide rigorous stability guarantees, but often require manual parameter tuning to achieve good closed-loop performance. Policy-gradient methods offer a systematic approach to parameter optimization; however, conventional gradient evaluation requires sequential forward state rollout and backward costate propagation. This letter develops a time-parallel policy-gradient framework for discrete-time nonlinear control-affine systems. We derive the policy-gradient expression where the state and costate rollouts required for policy-gradient evaluation are formulated as residual-minimization problems and solved using Gauss-Newton (GN) iterations with parallel associative scans. For closed-loop systems that are globally asymptotically stable and locally exponentially stable, we show that the residual-minimization problems satisfy a local Polyak-Lojasiewicz (PL) inequality and that the GN iterates converge locally at a quadratic rate. Moreover, the PL constant, the size of the convergence neighborhood, and the quadratic convergence bound are independent of the rollout horizon T. We also prove that, for any finite horizon T, the state solver recovers the exact trajectory from any initialization in at most T iterations. Finally, an inertia-wheel pendulum example with interconnection and damping assignment passivity-based control (IDA-PBC) demonstrates improved closed-loop performance and the computational benefits of the proposed parallel policy-gradient framework.
ER -