Hamiltonian Primal-Dual Policy Reinforcement Learning for Derivative-Free Model-Based Control
Abstract
Finite-horizon optimal control becomes difficult when a dynamical model can be simulated reliably, but its derivatives are unavailable, discontinuous, or unreliable. Hamiltonian Primal-Dual Policy Optimization (HPDPO) is introduced as a derivative-free, model-based reinforcement learning framework for this setting. The control is represented as a time-indexed finite-horizon policy. Candidate actions are evaluated by deterministic suffix rollouts of the supplied transition model, while a regularized Hamiltonian dual formulation supplies structured future-cost information through block-coordinate multiplier updates. No differentiation of the state dynamics or learned critic is required. The method is validated on nonsmooth problems with exact solutions, an analytical and learned ReLU biomarker cascade, an amyloid Neural ODE trained from longitudinal ADNI data, and three multidimensional-control problems with up to five simultaneous controls. Across these examples, HPDPO closely reproduces exact or reference objective values, trajectories, and control structures. The results show that forward simulation and Hamiltonian dual structure can be combined to solve deterministic finite-horizon control problems involving nonsmooth, learned, or black-box dynamics.
Related
- Reinforcement Learning to Accelerate Primal-Dual Hybrid Gradient for Linear Programming
- Safe Multi-Objective Reinforcement Learning: Zero Constraint Violation and Primal-Averaging Convergence
- Well-Posedness and Strong Convergence of Infinite-Dimensional Port-Hamiltonian Primal-Dual Gradient Dynamics
- Decentralized Strategies for Finite Population LQG Social Control: A Reinforcement Learning Approach
- Boundary Control of a Modified Timoshenko-Ehrenfest Beam via Reduced-Order Reinforcement Learning
BibTeX
@misc{rabiei2026hamiltonian,
title = {{Hamiltonian Primal-Dual Policy Reinforcement Learning for Derivative-Free Model-Based Control}},
author = {K. Rabiei and Z. Rui and X. Lu and W. Hao},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.18524},
eprint = {2610.18524},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.18524},
}
RIS
TY - UNPB
TI - Hamiltonian Primal-Dual Policy Reinforcement Learning for Derivative-Free Model-Based Control
AU - K. Rabiei
AU - Z. Rui
AU - X. Lu
AU - W. Hao
PY - 2026
DA - 2026/10/13
PB - arXiv
DO - 10.48550/arxiv.2610.18524
UR - https://arxiv.org/abs/2610.18524
AB - Finite-horizon optimal control becomes difficult when a dynamical model can be simulated reliably, but its derivatives are unavailable, discontinuous, or unreliable. Hamiltonian Primal-Dual Policy Optimization (HPDPO) is introduced as a derivative-free, model-based reinforcement learning framework for this setting. The control is represented as a time-indexed finite-horizon policy. Candidate actions are evaluated by deterministic suffix rollouts of the supplied transition model, while a regularized Hamiltonian dual formulation supplies structured future-cost information through block-coordinate multiplier updates. No differentiation of the state dynamics or learned critic is required. The method is validated on nonsmooth problems with exact solutions, an analytical and learned ReLU biomarker cascade, an amyloid Neural ODE trained from longitudinal ADNI data, and three multidimensional-control problems with up to five simultaneous controls. Across these examples, HPDPO closely reproduces exact or reference objective values, trajectories, and control structures. The results show that forward simulation and Hamiltonian dual structure can be combined to solve deterministic finite-horizon control problems involving nonsmooth, learned, or black-box dynamics.
ER -