On Convergence of Gradient Temporal-Difference Learning with Regularized Corrections and Eligibility Traces
Abstract
Temporal-difference (TD) learning underlies many prediction and control algorithms, but can diverge when combined with off-policy learning, bootstrapping, and function approximation. TD with Regularized Corrections (TDRC) is theoretically sound and practically simple, but existing convergence theory covers only the one-step case under i.i.d. sampling, while the Huber Bellman-error formulation has not been extended to TDRC or eligibility traces. In this work, we establish convergence of constrained TDRC(λ) with diminishing stepsizes and non-i.i.d. off-policy data, for state-dependent and history-dependent λ-parameters. We also propose TDRC(λ)-Huber, which preserves the TDRC(λ) fixed point while reducing sensitivity to large Bellman errors. For both algorithms, we apply an ODE-based weak-convergence method from stochastic approximation theory. For the nonlinear mean dynamics of TDRC(λ)-Huber, we give an integral small-gain condition that guarantees uniqueness and global asymptotic stability of the target equilibrium. We then establish convergence of the constrained algorithm, derive computable sufficient conditions for convergence, and identify a regularization-based convergence mechanism shared by linear TDRC(λ) and TDRC(λ)-Huber. When the mean TD matrix is positive stable, both constrained algorithms converge under sufficiently strong regularization without requiring positive definiteness.
Related
- Global 𝒪(k−2) Convergence of Multilevel Gradient-Regularized Newton Methods
- Geometry and Convergence of Quadratically Regularized Optimal Transport I
- On convergence of formal Dulac series, satisfying an algebraic q-difference equation
- A Stochastic Proximal-Gradient Method for Regularized Generally Constrained Continuous Optimization
- From Tsallis to KL: Convergence and Error Estimates for Tsallis-Regularized Optimal Transport
BibTeX
@misc{jin2026convergence,
title = {{On Convergence of Gradient Temporal-Difference Learning with Regularized Corrections and Eligibility Traces}},
author = {Jingyi Jin and Martha White and Hao Wang},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.05869},
eprint = {2611.05869},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.05869},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - On Convergence of Gradient Temporal-Difference Learning with Regularized Corrections and Eligibility Traces
AU - Jingyi Jin
AU - Martha White
AU - Hao Wang
PY - 2026
DA - 2026/11/04
PB - arXiv
DO - 10.48550/arxiv.2611.05869
UR - https://arxiv.org/abs/2611.05869
AB - Temporal-difference (TD) learning underlies many prediction and control algorithms, but can diverge when combined with off-policy learning, bootstrapping, and function approximation. TD with Regularized Corrections (TDRC) is theoretically sound and practically simple, but existing convergence theory covers only the one-step case under i.i.d. sampling, while the Huber Bellman-error formulation has not been extended to TDRC or eligibility traces. In this work, we establish convergence of constrained TDRC(λ) with diminishing stepsizes and non-i.i.d. off-policy data, for state-dependent and history-dependent λ-parameters. We also propose TDRC(λ)-Huber, which preserves the TDRC(λ) fixed point while reducing sensitivity to large Bellman errors. For both algorithms, we apply an ODE-based weak-convergence method from stochastic approximation theory. For the nonlinear mean dynamics of TDRC(λ)-Huber, we give an integral small-gain condition that guarantees uniqueness and global asymptotic stability of the target equilibrium. We then establish convergence of the constrained algorithm, derive computable sufficient conditions for convergence, and identify a regularization-based convergence mechanism shared by linear TDRC(λ) and TDRC(λ)-Huber. When the mean TD matrix is positive stable, both constrained algorithms converge under sufficiently strong regularization without requiring positive definiteness.
ER -