Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms
Abstract
Continuous-time reinforcement learning seeks an optimal policy under unknown dynamics driven by stochastic differential equations using observations collected at discrete times. We develop a policy gradient framework that combines rigorous convergence analysis with learning from discrete data. We first establish a performance difference lemma and derive policy gradient formulas for randomized and parametrized policies. We then prove exponential convergence of an entropy-regularized policy flow to the regularized optimum. For discrete parametrized updates, we derive a convergence bound that explicitly separates the contributions of critic error, policy-regression error, and the learning step size. To enable learning under unknown dynamics, we integrate this framework with PhiBE, which constructs a continuous-time surrogate generator from discrete transition data. The resulting on-policy and off-policy model-free actor-critic algorithms preserve the continuous-time structure of the control problem. Numerical experiments on an entropy-regularized linear-quadratic regulator and a nonlinear wheeled robot demonstrate the performance and stability of the proposed methods with both known and data-driven dynamics.
Related
- Policy Evaluation and Policy Gradient for Continuous-Time Markov Regime-Switching Systems
- A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees
- A note on the convergence guarantees of RLT-based algorithms for polynomial optimization
- Retraction-Based Gradient Projection Algorithms on Manifolds
- Monte Carlo Power Iteration Algorithms: Convergence, Analysis and Application to Reactor Criticality Calculations
BibTeX
@misc{hernandez2026continuoustime,
title = {{Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms}},
author = {Martin Hernandez and Xinyan Qing and Mo Zhou and Yuhua Zhu},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.28580},
eprint = {2610.28580},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.28580},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms
AU - Martin Hernandez
AU - Xinyan Qing
AU - Mo Zhou
AU - Yuhua Zhu
PY - 2026
DA - 2026/10/20
PB - arXiv
DO - 10.48550/arxiv.2610.28580
UR - https://arxiv.org/abs/2610.28580
AB - Continuous-time reinforcement learning seeks an optimal policy under unknown dynamics driven by stochastic differential equations using observations collected at discrete times. We develop a policy gradient framework that combines rigorous convergence analysis with learning from discrete data. We first establish a performance difference lemma and derive policy gradient formulas for randomized and parametrized policies. We then prove exponential convergence of an entropy-regularized policy flow to the regularized optimum. For discrete parametrized updates, we derive a convergence bound that explicitly separates the contributions of critic error, policy-regression error, and the learning step size. To enable learning under unknown dynamics, we integrate this framework with PhiBE, which constructs a continuous-time surrogate generator from discrete transition data. The resulting on-policy and off-policy model-free actor-critic algorithms preserve the continuous-time structure of the control problem. Numerical experiments on an entropy-regularized linear-quadratic regulator and a nonlinear wheeled robot demonstrate the performance and stability of the proposed methods with both known and data-driven dynamics.
ER -