Policy Evaluation and Policy Gradient for Continuous-Time Markov Regime-Switching Systems
Abstract
We study policy evaluation (PE) and policy gradient (PG) for continuous-time stochastic control with Markov regime-switching. The main challenges are the coexistence of diffusion and regime jumps, the coupling of regime-dependent value functions, and the construction of model-free learning rules from the coupled generators. We establish a martingale characterization of PE that incorporates both Brownian and switching martingales, and extend martingale loss and orthogonality methods to this setting. We further derive a coupled PG equation and a trajectory-based gradient representation that, under an action-independent switching generator, avoids explicit knowledge of the system coefficients and transition rates. These results lead to offline and online actor-critic algorithms. We also establish convergence and study the finite-grid approximation required for implementation. Numerical experiments for a regime-switching linear-quadratic problem illustrate the proposed methods.
Related
- Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms
- A switching proximal gradient method for constrained difference-of-convex optimization problems
- Continuous-Time Mean-Variance Portfolio Selection Reinforcement Learning Problems for a Markov Regime-Switching System with No-Shorting Constraints
- (Cheap) Stochastic Policy Gradient Converges with High Probability for Linear Quadratic Regulator
- Boundary-aware Reinforcement Learning for Hypercube State Spaces via Deterministic Policy Gradient
BibTeX
@misc{zhang2026policy,
title = {{Policy Evaluation and Policy Gradient for Continuous-Time Markov Regime-Switching Systems}},
author = {Minghui Zhang and Xun Li and Xin Zhang},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.31094},
eprint = {2610.31094},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.31094},
}
RIS
TY - UNPB
TI - Policy Evaluation and Policy Gradient for Continuous-Time Markov Regime-Switching Systems
AU - Minghui Zhang
AU - Xun Li
AU - Xin Zhang
PY - 2026
DA - 2026/10/22
PB - arXiv
DO - 10.48550/arxiv.2610.31094
UR - https://arxiv.org/abs/2610.31094
AB - We study policy evaluation (PE) and policy gradient (PG) for continuous-time stochastic control with Markov regime-switching. The main challenges are the coexistence of diffusion and regime jumps, the coupling of regime-dependent value functions, and the construction of model-free learning rules from the coupled generators. We establish a martingale characterization of PE that incorporates both Brownian and switching martingales, and extend martingale loss and orthogonality methods to this setting. We further derive a coupled PG equation and a trajectory-based gradient representation that, under an action-independent switching generator, avoids explicit knowledge of the system coefficients and transition rates. These results lead to offline and online actor-critic algorithms. We also establish convergence and study the finite-grid approximation required for implementation. Numerical experiments for a regime-switching linear-quadratic problem illustrate the proposed methods.
ER -