Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence
Abstract
We study continuous-time and possibly high-dimensional stochastic control problems where drift coefficients and running reward functions are unknown. Due to these missing model primitives, we take the exploratory, reinforcement learning (RL) framework of Wang, Zariphopoulou, and Zhou(2020) with relaxed controls and entropy regularization. The objective is to develop theoretically grounded, efficient and scalable RL algorithms to learn both the optimal value functions (which also solve the exploratory HJB equation) and optimal exploratory feedback control policies. When the diffusion coefficients do not contain control, we employ probabilistic representations of both the optimal value function and its gradient based on an auxiliary state process depending only on the diffusion part of the original dynamics. With a delicate analysis on some properly defined mappings and their fixed points, this leads to the introduction of our policy iteration algorithms and their convergence. We demonstrate the performance of our algorithms through various numerical examples. Finally, we study a special control-dependent diffusion case where probability representation of the Hessian is called for.
Related
- Exponential rate of convergence of relative value iteration algorithms for ergodic controls of diffusions
- Convergence guarantees for stochastic algorithms solving non-unique problems in metric spaces
- Fully Online Decentralized Learning in Stochastic Games with Unknown Independent Chains
- Wong--Zakai Convergence and FNO Learning for Stochastic Volterra Equations
- Convergence of stochastic gradient descent for learning linear neural networks
BibTeX
@misc{ma2026learning,
title = {{Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence}},
author = {Jin Ma and Gaozhan Wang and Jianfeng Zhang and Xunyu Zhou},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.14972},
eprint = {2609.14972},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.14972},
}
RIS
TY - UNPB
TI - Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence
AU - Jin Ma
AU - Gaozhan Wang
AU - Jianfeng Zhang
AU - Xunyu Zhou
PY - 2026
DA - 2026/09/14
PB - arXiv
DO - 10.48550/arxiv.2609.14972
UR - https://arxiv.org/abs/2609.14972
AB - We study continuous-time and possibly high-dimensional stochastic control problems where drift coefficients and running reward functions are unknown. Due to these missing model primitives, we take the exploratory, reinforcement learning (RL) framework of Wang, Zariphopoulou, and Zhou(2020) with relaxed controls and entropy regularization. The objective is to develop theoretically grounded, efficient and scalable RL algorithms to learn both the optimal value functions (which also solve the exploratory HJB equation) and optimal exploratory feedback control policies. When the diffusion coefficients do not contain control, we employ probabilistic representations of both the optimal value function and its gradient based on an auxiliary state process depending only on the diffusion part of the original dynamics. With a delicate analysis on some properly defined mappings and their fixed points, this leads to the introduction of our policy iteration algorithms and their convergence. We demonstrate the performance of our algorithms through various numerical examples. Finally, we study a special control-dependent diffusion case where probability representation of the Hessian is called for.
ER -