Model-free Reinforcement Learning for Continuous Time and State: A Stochastic Maximum Principle Approach
Abstract
This paper develops a model-free reinforcement learning (RL) algorithm based on the stochastic maximum principle for continuous-time stochastic control problems with continuous state and action spaces. For a parameterized Markovian policy, we establish the existence of the decoupling field for the adjoint backward stochastic differential equation, which allows the Hamiltonian gradient to be represented as a deterministic function of time and state. We then learn this Hamiltonian gradient directly from data and incorporate it into a policy gradient scheme with inexact gradients. We establish the convergence of the resulting policy gradient algorithm and establish the corresponding error estimates. Under suitable conditions on learning rates, exploration parameters, and approximation errors, the objective values converge and the L2-norm of the policy gradient vanishes asymptotically. We further show that the proposed SMP-based policy gradient representation is equivalent to the existing continuous-time deterministic policy gradient representation based on the advantage-rate function. The effectiveness and efficiency of the proposed RL algorithm are demonstrated by numerical experiments.
Related
- Continuous Delayed-Memory Stochastic Gradient Descent and Continuous-Time Reinforcement Learning from History of Astrophysical Time Series Studies
- Continuous-Time Reinforcement Learning for N-Player Stochastic Differential Games with Exploratory Policies
- Reinforcement learning for constrained stochastic linear--quadratic control in continuous time
- A global stochastic maximum principle for forward-backward stochastic control systems with quadratic convex generator and unbounded terminal condition
- On the Pontryagin Maximum Principle for Rough Stochastic Differential Equations
BibTeX
@misc{bo2026modelfree,
title = {{Model-free Reinforcement Learning for Continuous Time and State: A Stochastic Maximum Principle Approach}},
author = {Lijun Bo and Yijie Huang and Jingfei Wang},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2609.38745},
eprint = {2609.38745},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.38745},
}
RIS
TY - UNPB
TI - Model-free Reinforcement Learning for Continuous Time and State: A Stochastic Maximum Principle Approach
AU - Lijun Bo
AU - Yijie Huang
AU - Jingfei Wang
PY - 2026
DA - 2026/10/09
PB - arXiv
DO - 10.48550/arxiv.2609.38745
UR - https://arxiv.org/abs/2609.38745
AB - This paper develops a model-free reinforcement learning (RL) algorithm based on the stochastic maximum principle for continuous-time stochastic control problems with continuous state and action spaces. For a parameterized Markovian policy, we establish the existence of the decoupling field for the adjoint backward stochastic differential equation, which allows the Hamiltonian gradient to be represented as a deterministic function of time and state. We then learn this Hamiltonian gradient directly from data and incorporate it into a policy gradient scheme with inexact gradients. We establish the convergence of the resulting policy gradient algorithm and establish the corresponding error estimates. Under suitable conditions on learning rates, exploration parameters, and approximation errors, the objective values converge and the L^2-norm of the policy gradient vanishes asymptotically. We further show that the proposed SMP-based policy gradient representation is equivalent to the existing continuous-time deterministic policy gradient representation based on the advantage-rate function. The effectiveness and efficiency of the proposed RL algorithm are demonstrated by numerical experiments.
ER -