Continuous-Time q-Learning with Nonsmooth Value Functions: a BSDE Approach
Abstract
We study continuous-time q-learning for entropy-regularized diffusion control without assuming classical differentiability of the value function. Rewards and admissible policies need only be Borel measurable, which allows discontinuous rewards. The diffusion coefficient σ is action independent and locally uniformly elliptic, and the drift is b0 + σψ, with b0 action independent and ψ bounded. In this setting the q-function involves the value only through its gradient. For each admissible policy, we represent its value by a backward stochastic differential equation (BSDE) and prove that this value is continuous before the horizon, although the data are only measurable. Replacing the gradient with a measurable representation of the martingale integrand, we define the q-function up to Lebesgue-null sets and recover the classical definition when the value function is smooth. We then prove policy improvement, when the Gibbs update is admissible, and martingale characterizations of the value and the q-function from on-policy and off-policy observations. Here the candidate value need only be continuous before the horizon, and actions are sampled on a deterministic finite grid and held constant between sampling times. Under an integrability condition on the reward, we show that a second BSDE gives an upper bound for the optimal value, attained when its Gibbs policy is admissible, and we characterize its solution in the same way.
Related
- Space-time finite element interpolation of nonsmooth functions satisfying boundary conditions
- Bayesian Neural Networks via Dirichlet Process Priors on the Data-Generating Distribution for Randomized Value Functions in Deep Reinforcement Learning
- Continuous Delayed-Memory Stochastic Gradient Descent and Continuous-Time Reinforcement Learning from History of Astrophysical Time Series Studies
- Asymptotic Analysis of Gradient Mapping-type Stationarity Measure for the Sum of Nonconvex Nonsmooth Functions and Applications to Proximal Gradient-type Algorithms
- Continuous-Time Reinforcement Learning for N-Player Stochastic Differential Games with Exploratory Policies
BibTeX
@misc{asero2026continuoustime,
title = {{Continuous-Time q-Learning with Nonsmooth Value Functions: a BSDE Approach}},
author = {Manuel Asero and Elena Bandini and Andrea Cosso and Laura Perelli},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.20169},
eprint = {2610.20169},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.20169},
}
RIS
TY - UNPB
TI - Continuous-Time q-Learning with Nonsmooth Value Functions: a BSDE Approach
AU - Manuel Asero
AU - Elena Bandini
AU - Andrea Cosso
AU - Laura Perelli
PY - 2026
DA - 2026/10/14
PB - arXiv
DO - 10.48550/arxiv.2610.20169
UR - https://arxiv.org/abs/2610.20169
AB - We study continuous-time q-learning for entropy-regularized diffusion control without assuming classical differentiability of the value function. Rewards and admissible policies need only be Borel measurable, which allows discontinuous rewards. The diffusion coefficient σ is action independent and locally uniformly elliptic, and the drift is b_0 + σψ, with b_0 action independent and ψ bounded. In this setting the q-function involves the value only through its gradient. For each admissible policy, we represent its value by a backward stochastic differential equation (BSDE) and prove that this value is continuous before the horizon, although the data are only measurable. Replacing the gradient with a measurable representation of the martingale integrand, we define the q-function up to Lebesgue-null sets and recover the classical definition when the value function is smooth. We then prove policy improvement, when the Gibbs update is admissible, and martingale characterizations of the value and the q-function from on-policy and off-policy observations. Here the candidate value need only be continuous before the horizon, and actions are sampled on a deterministic finite grid and held constant between sampling times. Under an integrability condition on the reward, we show that a second BSDE gives an upper bound for the optimal value, attained when its Gibbs policy is admissible, and we characterize its solution in the same way.
ER -