Limiting-Kernel Q(λ): Bridging Short and Long Horizons
Abstract
In reinforcement learning, more accurate policy evaluation has been frequently shown to improve downstream policy optimization performance. For tractability, most value estimators rely on n-step truncation and thereby omit the long-horizon behavior of the transition dynamics. Methods that instead capture this behavior through the global structure of the dynamics can converge faster, but their computational and memory requirements limit their scalability to large or continuous state spaces. To reconcile these limitations, we introduce Limiting-Kernel Q(λ) (LKQL), an off-policy value estimator that uses the limiting kernel to approximate the long-horizon terms omitted by truncation. LKQL has the same order of complexity as n-step estimators and integrates directly into both on- and off-policy actor-critic algorithms. We prove that, under aperiodicity and in the near-on-policy regime, the operator underlying LKQL improves the policy evaluation convergence rate over its truncated counterpart for sufficiently large n, and that LKQL itself converges almost surely to the optimal values in finite Markov decision processes (MDPs) under a fixed behavior policy. On the MuJoCo continuous-control benchmark, we show that LKQL improves over n-step baselines in most settings, particularly on long-horizon tasks.
Related
- Online Discrepancy of Sparse Random Vectors over Long Horizons
- Choosing the penalty in nonparametric regression: short and long-range dependence
- Limit Theory for Online Expectile Estimation under Short and Long Range Dependence
- Almost primes in short intervals and averages of long Dirichlet polynomial approximations to (ζ'/ζ)k
- A plug-in estimator of the non-iterative conditional expectation g-formula using long short-term memory models
BibTeX
@misc{ok2026limitingkernel,
title = {{Limiting-Kernel Q($λ$): Bridging Short and Long Horizons}},
author = {Tolga Ok and Arman Sharifi Kolarijani and Peyman Mohajerin Esfahani and Mohamad Amin Sharifi Kolarijani},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2609.27741},
eprint = {2609.27741},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.27741},
note = {CC BY-SA 4.0},
}
RIS
TY - UNPB
TI - Limiting-Kernel Q(λ): Bridging Short and Long Horizons
AU - Tolga Ok
AU - Arman Sharifi Kolarijani
AU - Peyman Mohajerin Esfahani
AU - Mohamad Amin Sharifi Kolarijani
PY - 2026
DA - 2026/10/12
PB - arXiv
DO - 10.48550/arxiv.2609.27741
UR - https://arxiv.org/abs/2609.27741
AB - In reinforcement learning, more accurate policy evaluation has been frequently shown to improve downstream policy optimization performance. For tractability, most value estimators rely on n-step truncation and thereby omit the long-horizon behavior of the transition dynamics. Methods that instead capture this behavior through the global structure of the dynamics can converge faster, but their computational and memory requirements limit their scalability to large or continuous state spaces. To reconcile these limitations, we introduce Limiting-Kernel Q(λ) (LKQL), an off-policy value estimator that uses the limiting kernel to approximate the long-horizon terms omitted by truncation. LKQL has the same order of complexity as n-step estimators and integrates directly into both on- and off-policy actor-critic algorithms. We prove that, under aperiodicity and in the near-on-policy regime, the operator underlying LKQL improves the policy evaluation convergence rate over its truncated counterpart for sufficiently large n, and that LKQL itself converges almost surely to the optimal values in finite Markov decision processes (MDPs) under a fixed behavior policy. On the MuJoCo continuous-control benchmark, we show that LKQL improves over n-step baselines in most settings, particularly on long-horizon tasks.
ER -