Theoretical Foundations and Effective Algorithms for Policy-Aware Simulator Learning
Abstract
Model-based reinforcement learning (MBRL) agents typically learn world models by minimizing predictive loss. However, powerful RL optimizers inevitably exploit minor model inaccuracies, leading to simulator exploitation and a reality gap where policies succeed in simulation but fail in the real world. We propose that the objective for learning simulators should be strategic robustness rather than predictive accuracy, and formulate this as a zero-sum minimax game between a model player and an adversarial policy player. We provide a comprehensive theoretical analysis: (1) an online learning guarantee showing the game is learnable with sublinear regret bounds; (2) a tractable critic-based simplification bounding the global policy-value gap by the local critic's loss; and (3) an Error-MDP duality, proving that finding the worst-case policy is formally dual to a standard RL problem where the reward is the one-step critic error. This duality yields a provably convergent active data selection algorithm. Experiments on continuous control tasks demonstrate that our approach reduces prediction error in strategically important regions by $1.5−2.2\times$ and enables policies trained purely in simulation to match near-optimal real-world performance.
Related
- Theoretical Foundations of Computational Islamic Jurisprudence (Fiqh)
- Theoretical Foundations of Ordinal Spherical Multidimensional Scaling
- Theoretical Foundations of Sequential Dynamic Programming
- Distributed Intraday Energy Management for 16.7-Hz Railway Power Systems---Part I: Theoretical Foundations
- Poisson Exchange Beyond Submodularity: Effective Approximation Algorithms for Offline and Online Subset Selection over Matroids
BibTeX
@misc{dann2026theoretical,
title = {{Theoretical Foundations and Effective Algorithms for Policy-Aware Simulator Learning}},
author = {Christoph Dann and Yishay Mansour and Mehryar Mohri},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2605.29032},
eprint = {2605.29032},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.29032},
}
RIS
TY - UNPB
TI - Theoretical Foundations and Effective Algorithms for Policy-Aware Simulator Learning
AU - Christoph Dann
AU - Yishay Mansour
AU - Mehryar Mohri
PY - 2026
DA - 2026/09/02
PB - arXiv
DO - 10.48550/arxiv.2605.29032
UR - https://arxiv.org/abs/2605.29032
AB - Model-based reinforcement learning (MBRL) agents typically learn world models by minimizing predictive loss. However, powerful RL optimizers inevitably exploit minor model inaccuracies, leading to simulator exploitation and a reality gap where policies succeed in simulation but fail in the real world. We propose that the objective for learning simulators should be strategic robustness rather than predictive accuracy, and formulate this as a zero-sum minimax game between a model player and an adversarial policy player. We provide a comprehensive theoretical analysis: (1) an online learning guarantee showing the game is learnable with sublinear regret bounds; (2) a tractable critic-based simplification bounding the global policy-value gap by the local critic's loss; and (3) an Error-MDP duality, proving that finding the worst-case policy is formally dual to a standard RL problem where the reward is the one-step critic error. This duality yields a provably convergent active data selection algorithm. Experiments on continuous control tasks demonstrate that our approach reduces prediction error in strategically important regions by $1.5−2.2\times$ and enables policies trained purely in simulation to match near-optimal real-world performance.
ER -