Preserving Unstable Modes Through Inverse Dynamics in JEPA World Models
Abstract
Robotic systems often exhibit unstable modes, along which small perturbations and disturbances can cause unbounded growth unless corrected through feedback. Controlling such systems from high-dimensional visual observations requires representations that preserve these modes. Joint-embedding predictive architectures (JEPAs) provide a natural framework for learning such representations and their dynamics from visual data. However, we demonstrate that next step prediction combined with anti-collapse regularization does not guarantee that controllable unstable modes are preserved: the training loss can be minimized while these modes are collapsed, making stabilization from the learned representation impossible. To address this, we augment world-model training with an action reconstruction objective (i.e., an inverse dynamics loss) that encourages control-aware representations, namely, visual representations that preserve crucial features for control. We prove that exact action reconstruction makes the encoder injective on the finite-horizon reachable subspace. Thus, the encoder cannot discard any state direction reachable by an action sequence within H steps. Moreover, we show that, as H grows, the dominant eigenspace of the finite-horizon controllability Gramian converges to the controllable unstable subspace. We establish our theoretical results for linear systems and demonstrate empirically that our findings extend to nonlinear visual control tasks (CartPole, Walker2D, and PointMaze), highlighting the benefits of control-aware representation learning.
Related
- Structure-Preserving Dimension Reduction and Surrogates for Partially Linear Bayesian Inverse Problems
- Stable and Unstable Potential-Well Dynamics for an Indirectly Damped Wave-MGT System with General Focusing Sources
- Toward Human-Perceptive 6G: Privacy-Preserving Network Intelligence from Human Dynamics
- Involution-Preserving schemes through constrained mesh motion: a novel Arbitrary-Lagrangian-Eulerian framework
- Structure-preserving diffuse-domain accelerated saddle dynamics for wetting transitions
BibTeX
@misc{toso2026preserving,
title = {{Preserving Unstable Modes Through Inverse Dynamics in JEPA World Models}},
author = {Leonardo F. Toso and Yann LeCun and James Anderson and Oumayma Bounou},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.07540},
eprint = {2610.07540},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.07540},
}
RIS
TY - UNPB
TI - Preserving Unstable Modes Through Inverse Dynamics in JEPA World Models
AU - Leonardo F. Toso
AU - Yann LeCun
AU - James Anderson
AU - Oumayma Bounou
PY - 2026
DA - 2026/10/06
PB - arXiv
DO - 10.48550/arxiv.2610.07540
UR - https://arxiv.org/abs/2610.07540
AB - Robotic systems often exhibit unstable modes, along which small perturbations and disturbances can cause unbounded growth unless corrected through feedback. Controlling such systems from high-dimensional visual observations requires representations that preserve these modes. Joint-embedding predictive architectures (JEPAs) provide a natural framework for learning such representations and their dynamics from visual data. However, we demonstrate that next step prediction combined with anti-collapse regularization does not guarantee that controllable unstable modes are preserved: the training loss can be minimized while these modes are collapsed, making stabilization from the learned representation impossible. To address this, we augment world-model training with an action reconstruction objective (i.e., an inverse dynamics loss) that encourages control-aware representations, namely, visual representations that preserve crucial features for control. We prove that exact action reconstruction makes the encoder injective on the finite-horizon reachable subspace. Thus, the encoder cannot discard any state direction reachable by an action sequence within H steps. Moreover, we show that, as H grows, the dominant eigenspace of the finite-horizon controllability Gramian converges to the controllable unstable subspace. We establish our theoretical results for linear systems and demonstrate empirically that our findings extend to nonlinear visual control tasks (CartPole, Walker2D, and PointMaze), highlighting the benefits of control-aware representation learning.
ER -