Evolution Strategy-Assisted TD3 for Continuous Control: Policy-Update Analysis and Plant-State Boundedness
- 1Department of Computer Science, College of Computer Sciences and Information Technology, King Faisal University, Al-Ahsa 36362, Saudi Arabia
- 2Faculty of Engineering and Natural Sciences, Sivas University of Science and Technology, Sivas 58100, Turkey
Abstract
Exploration quality and policy parameter updates influence the training of deterministic actor–critic controllers. This study presents ES–TD3, an evolution strategy-assisted Twin Delayed Deep Deterministic Policy Gradient framework that combines population-based actor-parameter search with off-policy learning. Candidate actors undergo Gaussian perturbation, episodic evaluation, elite selection, and fitness-weighted recombination. The mathematical analysis establishes that the recombined parameter vector lies within the convex hull of the elite actors and bounds its displacement from the current population center by the largest elite displacement, with a perturbation-dependent bound for Gaussian candidates. For a mass–spring–damper plant with positive mass, damping, and stiffness, bounded-input/bounded-state and Lyapunov analyses establish uniform ultimate boundedness of the physical state under the bounded deterministic actor output. These guarantees concern individual recombination steps and plant-state boundedness, not asymptotic tracking or global training convergence. MATLAB/Simulink simulations compare ES–TD3 with standard TD3 and PID control, examine reference and initial condition variations, and assess sensitivity to evolutionary settings. The best observed configuration reaches the moving-average reward threshold of 126,500 at generation 203 and attains a final moving-average reward of 126,512. The reported ES–TD3 run exhibits smoother reward progression than the TD3 baseline, while the deployed policy produces bounded tracking responses across the tested scenarios. The results support the feasibility of combining evolutionary policy search with actor–critic learning for the considered control problem; broader performance claims require independent repeated trials.
Related
- Order Boundedness of Order Continuous Orthogonally Additive Operators
- Optimal Reinsurance-Dividend Strategy with Fixed Transaction Costs in a Regime-Switching Brownian Risk Model: A Viscosity Solution to the Impulse Control Problem
- System Modeling and Performance Analysis of Cloud Virtual Machine Allocation Strategy with Entry Control and Reservation Mechanism
- A comparative optimal control strategy for melanoma treatment via radiotherapy, TAMs-targeted therapy, and CAR T-cell administration
- Continuous Policy and Value Iteration for Stochastic Control Problems with Inhomogeneous Synchronous Coupling
BibTeX
@article{hafez2026evolution,
title = {{Evolution Strategy-Assisted TD3 for Continuous Control: Policy-Update Analysis and Plant-State Boundedness}},
author = {A. H. Abdul Hafez and Ufuk Demircioğlu and Halit Bakır and Badar Almarri},
journal = {Mathematics},
year = {2026},
month = nov,
doi = {10.3390/math14214006},
url = {https://www.mdpi.com/2227-7390/14/21/4006},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Evolution Strategy-Assisted TD3 for Continuous Control: Policy-Update Analysis and Plant-State Boundedness
AU - A. H. Abdul Hafez
AU - Ufuk Demircioğlu
AU - Halit Bakır
AU - Badar Almarri
PY - 2026
DA - 2026/11/03
JO - Mathematics
DO - 10.3390/math14214006
UR - https://www.mdpi.com/2227-7390/14/21/4006
AB - Exploration quality and policy parameter updates influence the training of deterministic actor–critic controllers. This study presents ES–TD3, an evolution strategy-assisted Twin Delayed Deep Deterministic Policy Gradient framework that combines population-based actor-parameter search with off-policy learning. Candidate actors undergo Gaussian perturbation, episodic evaluation, elite selection, and fitness-weighted recombination. The mathematical analysis establishes that the recombined parameter vector lies within the convex hull of the elite actors and bounds its displacement from the current population center by the largest elite displacement, with a perturbation-dependent bound for Gaussian candidates. For a mass–spring–damper plant with positive mass, damping, and stiffness, bounded-input/bounded-state and Lyapunov analyses establish uniform ultimate boundedness of the physical state under the bounded deterministic actor output. These guarantees concern individual recombination steps and plant-state boundedness, not asymptotic tracking or global training convergence. MATLAB/Simulink simulations compare ES–TD3 with standard TD3 and PID control, examine reference and initial condition variations, and assess sensitivity to evolutionary settings. The best observed configuration reaches the moving-average reward threshold of 126,500 at generation 203 and attains a final moving-average reward of 126,512. The reported ES–TD3 run exhibits smoother reward progression than the TD3 baseline, while the deployed policy produces bounded tracking responses across the tested scenarios. The results support the feasibility of combining evolutionary policy search with actor–critic learning for the considered control problem; broader performance claims require independent repeated trials.
SN - 2227-7390
ER -