Density-Guided Bootstrap Boundaries for Latent World-Model Reinforcement Learning

Abstract

Latent world models allow reinforcement learning agents to learn policies from imagined trajectories, yet determining the optimal depth of model reliance remains a fundamental challenge. Fixed imagination horizons apply uniform prediction depths regardless of trajectory reliability, exposing target values to compounding model error or over-relying on critic approximations. We introduce density-guided bootstrap boundaries, a trajectory-dependent mechanism that adapts the depth of imagined returns using a score-informed relative density proxy. The proxy accumulates the estimated divergence of a score-corrected auxiliary field formed by a feature displacement network and a noise-conditioned score estimator along imagined paths. We propose two gating mechanisms: a hard boundary that halts return expansion at a discrete state, and a soft boundary that continuously shifts the return recursion toward critic bootstrapping. Both variants preserve boundary value estimates and apply corresponding survival weights to actor and critic losses. Formulated independently of specific dynamics models and instantiated within Dreamer V3, we prove that the resulting gated Bellman operator is a strict contraction mapping and that early bootstrapping monotonically reduces conditional target variance. We clarify the theoretical assumptions distinguishing our expansion proxy from exact path likelihoods. Across continuous-control tasks and Atari Pong, empirical evaluations show that dynamic boundary selection enhances sample efficiency and late-training performance over fixed-horizon and auxiliary-enabled baselines.

Model-based Bootstrap for Offline Policy Evaluation in Tabular Reinforcement Learning
Weiwei Wang, Yuqiang Li, Xianyi Wu +1
arXiv17 Sep 2026PreprintCC BY 4.0Full textStatisticsarXiv
Toward Maximizing Algebraic Connectivity via Density-Dependent Backbone Construction and Reinforcement Learning Completion
Zeru Zhu, Marco Gamarra, Ji Liu
arXiv8 Oct 2026PreprintarXiv licenceFull textCombinatorics & discrete mathematics Optimisation & controlarXiv
AlphaPareto: Formulaic Alpha Discovery with LLM-Guided Multi-Objective Reinforcement Learning
Yingbo Zhao, Zeyu Yang, Zhoufan Zhu
arXiv28 Sep 2026PreprintCC BY-SA 4.0Full textStatisticsarXiv
Causal DAG-Guided Reinforcement Learning for Long-Term Physical Activity Maintenance: Using Prior Data to Inform Future Deployment
Xueqing Liu, Young Hyun Cho, Bhanu Teja Gullapalli +2
arXiv26 Oct 2026PreprintarXiv licenceFull textStatisticsarXiv
Data-Driven On/Off Topology Optimization of Electromagnetic Devices Based on Graph Physics-Informed Neural Networks and Physics-Guided Reinforcement Learning
Xiaoyu Liu, Yihan Si, Wei Xiang +1
Mathematics9 Oct 2026Peer-reviewedCC BY 4.0General mathematicsCrossref

Cite

BibTeX RIS

BibTeX

@misc{gupta2026densityguided,
  title = {{Density-Guided Bootstrap Boundaries for Latent World-Model Reinforcement Learning}},
  author = {Shashank Gupta and Pilhwa Lee},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.41022},
  eprint = {2610.41022},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.41022},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - Density-Guided Bootstrap Boundaries for Latent World-Model Reinforcement Learning
AU  - Shashank Gupta
AU  - Pilhwa Lee
PY  - 2026
DA  - 2026/10/29
PB  - arXiv
DO  - 10.48550/arxiv.2610.41022
UR  - https://arxiv.org/abs/2610.41022
AB  - Latent world models allow reinforcement learning agents to learn policies from imagined trajectories, yet determining the optimal depth of model reliance remains a fundamental challenge. Fixed imagination horizons apply uniform prediction depths regardless of trajectory reliability, exposing target values to compounding model error or over-relying on critic approximations. We introduce density-guided bootstrap boundaries, a trajectory-dependent mechanism that adapts the depth of imagined returns using a score-informed relative density proxy. The proxy accumulates the estimated divergence of a score-corrected auxiliary field formed by a feature displacement network and a noise-conditioned score estimator along imagined paths. We propose two gating mechanisms: a hard boundary that halts return expansion at a discrete state, and a soft boundary that continuously shifts the return recursion toward critic bootstrapping. Both variants preserve boundary value estimates and apply corresponding survival weights to actor and critic losses. Formulated independently of specific dynamics models and instantiated within Dreamer V3, we prove that the resulting gated Bellman operator is a strict contraction mapping and that early bootstrapping monotonically reduces conditional target variance. We clarify the theoretical assumptions distinguishing our expansion proxy from exact path likelihoods. Across continuous-control tasks and Atari Pong, empirical evaluations show that dynamic boundary selection enhances sample efficiency and late-training performance over fixed-horizon and auxiliary-enabled baselines.
ER  -

Abstract

Related

Cite