Martingale Deep Reinforcement Learning for High-Dimensional Financial Problems
- 1School of Mathematical Sciences, Tongji University, Shanghai 200092, China
- 2School of Mathematics and Information of Science, Guangzhou University, Guangzhou 510006, China
Abstract
We develop a martingale-residual framework for continuous-time reinforcement learning in high-dimensional financial problems, linking policy evaluation (PE), policy gradient (PG), and entropy-regularized Actor–Critic (AC) learning. First, continuous-time PE and PG are often developed through separate constructions. We formulate a common martingale residual for the Critic and Actor, and show that, as the local time step vanishes, its rescaled policy derivative recovers the classical continuous-time PG. We also quantify how Critic approximation errors affect policy-gradient accuracy. Second, martingale-orthogonality temporal-difference (TD) methods require prescribed test functions, and their performance can depend on that choice. We introduce a weak-martingale adversarial formulation in which the test function is learned rather than prescribed. The resulting minimax problem avoids the sensitivity to hand-chosen test functions and provides a unified computational representation of the associated moment conditions. Third, direct high-dimensional Hamilton–Jacobi–Bellman (HJB) implementations can require explicit Hessians and other higher-order differential operators. Our algorithms work directly with sampled martingale residuals and first-order automatic differentiation, yielding Hessian-free implementations for both offline training from complete trajectories and online updating from sequential transitions. Numerical experiments on high-dimensional basket-option pricing and mean–variance portfolio selection demonstrate accurate PE and effective PG. In high-dimensional mean–variance portfolio-selection experiments, the proposed method achieves competitive or superior performance on several key metrics relative to benchmarks: the offline variant achieves a mean annualized return of 16.223%, the online variant attains a mean Sharpe ratio of 2.337, and fixed-budget training time increases by 19.3% as the asset dimension grows from 10 to 1000.
Related
- Continuous-Time Mean-Variance Portfolio Selection Reinforcement Learning Problems for a Markov Regime-Switching System with No-Shorting Constraints
- An Improved Escape Algorithm with Reinforcement Learning and Multi-Strategy Synergy for Global Optimization and Real Problems
- Deep Reinforcement Learning on Item-Compatibility Graphs for One-Dimensional Bin Packing
- A Pricing and Hedging Framework for Infinite-Dimensional Financial Markets
- Mathematical Characterization of Dynamic Risk Spillovers in Carbon-Energy Financial Networks with Multi-Dimensional Uncertainty: A TVP-VAR-DY Connectedness Approach via Generalized Forecast-Error Variance Decomposition
BibTeX
@article{qi2026martingale,
title = {{Martingale Deep Reinforcement Learning for High-Dimensional Financial Problems}},
author = {Hang Qi and Dingli Xia and Mingxiang Deng and Xiaofei Guan},
journal = {Mathematics},
year = {2026},
month = nov,
doi = {10.3390/math14214012},
url = {https://www.mdpi.com/2227-7390/14/21/4012},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Martingale Deep Reinforcement Learning for High-Dimensional Financial Problems
AU - Hang Qi
AU - Dingli Xia
AU - Mingxiang Deng
AU - Xiaofei Guan
PY - 2026
DA - 2026/11/03
JO - Mathematics
DO - 10.3390/math14214012
UR - https://www.mdpi.com/2227-7390/14/21/4012
AB - We develop a martingale-residual framework for continuous-time reinforcement learning in high-dimensional financial problems, linking policy evaluation (PE), policy gradient (PG), and entropy-regularized Actor–Critic (AC) learning. First, continuous-time PE and PG are often developed through separate constructions. We formulate a common martingale residual for the Critic and Actor, and show that, as the local time step vanishes, its rescaled policy derivative recovers the classical continuous-time PG. We also quantify how Critic approximation errors affect policy-gradient accuracy. Second, martingale-orthogonality temporal-difference (TD) methods require prescribed test functions, and their performance can depend on that choice. We introduce a weak-martingale adversarial formulation in which the test function is learned rather than prescribed. The resulting minimax problem avoids the sensitivity to hand-chosen test functions and provides a unified computational representation of the associated moment conditions. Third, direct high-dimensional Hamilton–Jacobi–Bellman (HJB) implementations can require explicit Hessians and other higher-order differential operators. Our algorithms work directly with sampled martingale residuals and first-order automatic differentiation, yielding Hessian-free implementations for both offline training from complete trajectories and online updating from sequential transitions. Numerical experiments on high-dimensional basket-option pricing and mean–variance portfolio selection demonstrate accurate PE and effective PG. In high-dimensional mean–variance portfolio-selection experiments, the proposed method achieves competitive or superior performance on several key metrics relative to benchmarks: the offline variant achieves a mean annualized return of 16.223%, the online variant attains a mean Sharpe ratio of 2.337, and fixed-budget training time increases by 19.3% as the asset dimension grows from 10 to 1000.
SN - 2227-7390
ER -