Learning Multiple Timescales for Goal-Conditioned Reinforcement Learning
Abstract
Existing approaches to offline goal-conditioned reinforcement learning (GCRL) struggle with long-horizon tasks. Discounting shrinks value differences between distant states until they fall below the function approximation error, leaving the agent with no signal for ranking states. Temporal abstraction, which treats k environment steps as a single transition, restores this signal at long range, but no single fixed k suits all state-goal distances: large k preserves value differences across long temporal distances while collapsing distinctions between nearby states, and small k does the reverse. We make this trade-off explicit and introduce Generalized Implicit Temporal Abstraction (GITA), which conditions a single value function on k. GITA trains one policy by aggregating advantage-weighted supervision across multiple k values, so scales assigning larger positive advantages to a state-goal pair contribute more strongly to its update. GITA does not need to choose between local resolution and long-range signal; it retains both without committing to a single k. On OGBench, GITA outperforms a broad range of offline GCRL baselines, raising average success rate across all tasks by 25 percentage points (73% relative improvement) over HIQL. It also improves over the strongest fixed-k method, OTA, by 7 percentage points (14% relative).
Related
- DCRA: Diffusion-Conditioned Representation Alignment for Robust Time-Series Learning
- Reinforcement Learning Under Runtime Confounding: Policy Learning and Off-Policy Evaluation
- Class-Conditioned Diffusion Augmentation and Explainable In-Context Ensemble Learning for Imbalanced Maintenance Action Classification in Aircraft Systems
- Deep Reinforcement Learning for Blind Passive Beamforming in RIS-Assisted MIMO Systems
- PhiBE-Q-Learning: Bridging Off-Policy Reinforcement Learning and Continuous-Time Control
BibTeX
@misc{dutenhefner2026learning,
title = {{Learning Multiple Timescales for Goal-Conditioned Reinforcement Learning}},
author = {Pedro Robles Dutenhefner and Dikshant Shehmar and Wagner Meira Jr. and Marlos C. Machado},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.00849},
eprint = {2610.00849},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.00849},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Learning Multiple Timescales for Goal-Conditioned Reinforcement Learning
AU - Pedro Robles Dutenhefner
AU - Dikshant Shehmar
AU - Wagner Meira Jr.
AU - Marlos C. Machado
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2610.00849
UR - https://arxiv.org/abs/2610.00849
AB - Existing approaches to offline goal-conditioned reinforcement learning (GCRL) struggle with long-horizon tasks. Discounting shrinks value differences between distant states until they fall below the function approximation error, leaving the agent with no signal for ranking states. Temporal abstraction, which treats k environment steps as a single transition, restores this signal at long range, but no single fixed k suits all state-goal distances: large k preserves value differences across long temporal distances while collapsing distinctions between nearby states, and small k does the reverse. We make this trade-off explicit and introduce Generalized Implicit Temporal Abstraction (GITA), which conditions a single value function on k. GITA trains one policy by aggregating advantage-weighted supervision across multiple k values, so scales assigning larger positive advantages to a state-goal pair contribute more strongly to its update. GITA does not need to choose between local resolution and long-range signal; it retains both without committing to a single k. On OGBench, GITA outperforms a broad range of offline GCRL baselines, raising average success rate across all tasks by 25 percentage points (73% relative improvement) over HIQL. It also improves over the strongest fixed-k method, OTA, by 7 percentage points (14% relative).
ER -