Offline Policy Evaluation via Mixed Bellman Residuals and Adaptive Critic Representations
Abstract
Evaluating a target policy using data generated by a different behavior policy remains a fundamental challenge in reinforcement learning. While most existing work relies on the standard one-step Bellman residual, we consider a convex combination of one-step and two-step residuals with a fixed mixing weight. In ideal settings, this mixed Bellman formulation can provide a natural bias--variance trade-off between approximation error under a restricted value-function class and the increased variance arising from multi-step importance weighting. To solve this mixed residual optimization, we adopt a minimax formulation involving a critic function. Unlike standard approaches that rely on a fixed functional class, we construct a data-dependent critic representation using predicted future feature directions which effectively induces a kernel adapted to the underlying transition dynamics. This allows the critic to focus on directions that are most relevant for the estimated Bellman error. To control overfitting, we use sample splitting to construct the critic and estimate the value function on separate data subsets. Simulation studies and MetaWorld tasks illustrate the effect of the mixing parameter and show that intermediate residual combinations can improve value estimation in challenging settings.
Related
- Model Selection for Off-Policy Evaluation via Pairwise Bellman Validation
- Model-based Bootstrap for Offline Policy Evaluation in Tabular Reinforcement Learning
- Confounding Masquerading as Improvement: A Systematic Evaluation of Offline Reinforcement Learning for Stroke Antithrombotic Treatment in a 129,000-Patient Registry
- When Can You Ship on Evals Alone? Trial-Level Surrogacy for Offline Evaluation of LLM Systems
- Offline–Online Decoupled Architecture for Projected Microsecond-Latency and near Machine-Precision Caputo Fractional Derivative Evaluation, with an Edge-Computing Feasibility Study
BibTeX
@misc{biswas2026offline,
title = {{Offline Policy Evaluation via Mixed Bellman Residuals and Adaptive Critic Representations}},
author = {Amitakshar Biswas and Yuhan Li and Ruoqing Zhu},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.33186},
eprint = {2609.33186},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.33186},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Offline Policy Evaluation via Mixed Bellman Residuals and Adaptive Critic Representations
AU - Amitakshar Biswas
AU - Yuhan Li
AU - Ruoqing Zhu
PY - 2026
DA - 2026/09/27
PB - arXiv
DO - 10.48550/arxiv.2609.33186
UR - https://arxiv.org/abs/2609.33186
AB - Evaluating a target policy using data generated by a different behavior policy remains a fundamental challenge in reinforcement learning. While most existing work relies on the standard one-step Bellman residual, we consider a convex combination of one-step and two-step residuals with a fixed mixing weight. In ideal settings, this mixed Bellman formulation can provide a natural bias--variance trade-off between approximation error under a restricted value-function class and the increased variance arising from multi-step importance weighting. To solve this mixed residual optimization, we adopt a minimax formulation involving a critic function. Unlike standard approaches that rely on a fixed functional class, we construct a data-dependent critic representation using predicted future feature directions which effectively induces a kernel adapted to the underlying transition dynamics. This allows the critic to focus on directions that are most relevant for the estimated Bellman error. To control overfitting, we use sample splitting to construct the critic and estimate the value function on separate data subsets. Simulation studies and MetaWorld tasks illustrate the effect of the mixing parameter and show that intermediate residual combinations can improve value estimation in challenging settings.
ER -