PhiBE-Q-Learning: Bridging Off-Policy Reinforcement Learning and Continuous-Time Control
Abstract
In this paper, we develop an off-policy method for continuous-time reinforcement learning (CTRL), where the system dynamics are governed by an unknown stochastic differential equation (SDE) and only discrete-time trajectory data are available. A central challenge is that the classical discrete-time state--action value function loses its action-dependent signal in the continuous-time limit (Baird, 1994; Jia and Zhou, 2023; Tallec et al., 2019). On the other hand, continuous-time control provides local notions such as the instantaneous advantage function q(s,a), but these typically rely on state value function V(s). To obtain a formulation with a single unknown function, we introduce a new definition of the state--action value function Q(s,a) in CTRL and derive its governing equation. Building on the PhiBE approximation (Zhu, 2024; Zhu et al., 2025), we propose iterative algorithms to approximate the optimal Q-function in both model-based and model-free settings using only discrete-time off-policy data. Under linear function approximation, we establish error bounds and explicit convergence rates for the algorithm in the popoluation level. The finite-data algorithm is evaluated numerically.
Related
- Reinforcement learning for constrained stochastic linear--quadratic control in continuous time
- Safety-Certified Hierarchical Control of DC Microgrids via Continuous Deep Reinforcement Learning and Covariance-Weighted Sliding Mode Synthesis
- miniTE: Diagnosing and Bridging the Execution Gap in Native NVFP4 Reinforcement Learning
- Continuous Delayed-Memory Stochastic Gradient Descent and Continuous-Time Reinforcement Learning from History of Astrophysical Time Series Studies
- Continuous-Time Reinforcement Learning for N-Player Stochastic Differential Games with Exploratory Policies
BibTeX
@misc{zhu2026phibeqlearning,
title = {{PhiBE-Q-Learning: Bridging Off-Policy Reinforcement Learning and Continuous-Time Control}},
author = {Yuhua Zhu},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2606.21925},
eprint = {2606.21925},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.21925},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - PhiBE-Q-Learning: Bridging Off-Policy Reinforcement Learning and Continuous-Time Control
AU - Yuhua Zhu
PY - 2026
DA - 2026/10/20
PB - arXiv
DO - 10.48550/arxiv.2606.21925
UR - https://arxiv.org/abs/2606.21925
AB - In this paper, we develop an off-policy method for continuous-time reinforcement learning (CTRL), where the system dynamics are governed by an unknown stochastic differential equation (SDE) and only discrete-time trajectory data are available. A central challenge is that the classical discrete-time state--action value function loses its action-dependent signal in the continuous-time limit (Baird, 1994; Jia and Zhou, 2023; Tallec et al., 2019). On the other hand, continuous-time control provides local notions such as the instantaneous advantage function q(s,a), but these typically rely on state value function V(s). To obtain a formulation with a single unknown function, we introduce a new definition of the state--action value function Q(s,a) in CTRL and derive its governing equation. Building on the PhiBE approximation (Zhu, 2024; Zhu et al., 2025), we propose iterative algorithms to approximate the optimal Q-function in both model-based and model-free settings using only discrete-time off-policy data. Under linear function approximation, we establish error bounds and explicit convergence rates for the algorithm in the popoluation level. The finite-data algorithm is evaluated numerically.
ER -