Reinforcement Learning Under Runtime Confounding: Policy Learning and Off-Policy Evaluation
Abstract
Reinforcement learning (RL) is increasingly used to train robust, adaptable systems across healthcare, e-commerce, and robotics. In these settings, legal, ethical, or logistical concerns may preclude certain key variables that drive decisions from being collected during deployment time. This gives rise to a setting known as runtime confounding: the logged data hold full observability, but deployment time lacks knowledge of certain variables, known as runtime confounders. Unlike single timepoint and non-adaptive settings, runtime confounding under reinforcement learning has the potential to propagate over time, as earlier decisions affect future states, observations, and action assignments, leading to biased learning. We address these problems with three contributions: we (1) develop policy learning algorithms under runtime-confounding, (2) develop semi-parametric efficient off-policy estimation procedures, and (3) provide diagnostics for understanding the price of runtime confounding.
Related
- Confounding Masquerading as Improvement: A Systematic Evaluation of Offline Reinforcement Learning for Stroke Antithrombotic Treatment in a 129,000-Patient Registry
- Model-based Bootstrap for Offline Policy Evaluation in Tabular Reinforcement Learning
- PhiBE-Q-Learning: Bridging Off-Policy Reinforcement Learning and Continuous-Time Control
- Boundary-aware Reinforcement Learning for Hypercube State Spaces via Deterministic Policy Gradient
- Hamiltonian Primal-Dual Policy Reinforcement Learning for Derivative-Free Model-Based Control
BibTeX
@misc{kim2026reinforcement,
title = {{Reinforcement Learning Under Runtime Confounding: Policy Learning and Off-Policy Evaluation}},
author = {Raphael Kim and Michele Santacatterina and Ramin Zabih and Keith Barnatchez},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.40056},
eprint = {2610.40056},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.40056},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Reinforcement Learning Under Runtime Confounding: Policy Learning and Off-Policy Evaluation
AU - Raphael Kim
AU - Michele Santacatterina
AU - Ramin Zabih
AU - Keith Barnatchez
PY - 2026
DA - 2026/10/29
PB - arXiv
DO - 10.48550/arxiv.2610.40056
UR - https://arxiv.org/abs/2610.40056
AB - Reinforcement learning (RL) is increasingly used to train robust, adaptable systems across healthcare, e-commerce, and robotics. In these settings, legal, ethical, or logistical concerns may preclude certain key variables that drive decisions from being collected during deployment time. This gives rise to a setting known as runtime confounding: the logged data hold full observability, but deployment time lacks knowledge of certain variables, known as runtime confounders. Unlike single timepoint and non-adaptive settings, runtime confounding under reinforcement learning has the potential to propagate over time, as earlier decisions affect future states, observations, and action assignments, leading to biased learning. We address these problems with three contributions: we (1) develop policy learning algorithms under runtime-confounding, (2) develop semi-parametric efficient off-policy estimation procedures, and (3) provide diagnostics for understanding the price of runtime confounding.
ER -