Q-MMR: Off-Policy Evaluation via Recursive Reweighting and Moment Matching
Abstract
We present a novel theoretical framework, Q-MMR, for off-policy evaluation in finite-horizon MDPs. Q-MMR learns a set of scalar weights, one for each data point, such that the reweighted rewards approximate the expected return under the target policy. The weights are learned inductively in a top-down manner via a moment matching objective against a value-function discriminator class. Notably, and perhaps surprisingly, a data-dependent finite-sample guarantee for general function approximation can be established under only the realizability of Qπ, with a dimension-free bound -- that is, the error does not depend on the statistical complexity of the function class. We also establish connections to several existing methods, such as importance sampling and linear FQE. Further theoretical analyses shed new light on the nature of coverage, a concept of fundamental importance to offline RL.
Related
- Recursive moment updates for scalable Bayesian nonparametric regression
- Differentially Private Synthetic Data via Generalized Moment Matching
- Matrix-Free Moment-Matching Method for Reduced-Order Modeling of Quadratic-Bilinear Descriptor Systems with Multiple Inputs
- Moment-Matching Probabilistic Data Association for Optimization-Based SLAM
- Feasible closed form moment matching for scenario generation through covariance rotations
BibTeX
@misc{li2026qmmr,
title = {{Q-MMR: Off-Policy Evaluation via Recursive Reweighting and Moment Matching}},
author = {Xiang Li and Nan Jiang},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.06474},
eprint = {2605.06474},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.06474},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Q-MMR: Off-Policy Evaluation via Recursive Reweighting and Moment Matching
AU - Xiang Li
AU - Nan Jiang
PY - 2026
DA - 2026/10/23
PB - arXiv
DO - 10.48550/arxiv.2605.06474
UR - https://arxiv.org/abs/2605.06474
AB - We present a novel theoretical framework, Q-MMR, for off-policy evaluation in finite-horizon MDPs. Q-MMR learns a set of scalar weights, one for each data point, such that the reweighted rewards approximate the expected return under the target policy. The weights are learned inductively in a top-down manner via a moment matching objective against a value-function discriminator class. Notably, and perhaps surprisingly, a data-dependent finite-sample guarantee for general function approximation can be established under only the realizability of Q^π, with a dimension-free bound -- that is, the error does not depend on the statistical complexity of the function class. We also establish connections to several existing methods, such as importance sampling and linear FQE. Further theoretical analyses shed new light on the nature of coverage, a concept of fundamental importance to offline RL.
ER -