Budgeted Multi-Source Counterfactual Annotation for Off-Policy Evaluation
Abstract
Off-policy evaluation (OPE) estimates the value of a target policy from logged data, but limited behavior-policy coverage can force high-variance reweighting or reward-model extrapolation. Counterfactual annotations can add evidence about unobserved actions, yet practical sources, including domain experts and large language models (LLMs), may be costly, biased, or noisy. We study budgeted acquisition of such annotations for contextual-bandit OPE. Given source-specific costs and error profiles, we formulate an integer allocation problem over context-action pairs and annotation sources to minimize the component of estimator variance that depends on the annotation plan. We characterize when annotations are valuable through a first-annotation threshold and local annotation-value regimes. For the coupled multi-source problem, we develop a majorization-minimization algorithm with dynamic-programming subroutines that monotonically improves the objective. Experiments in synthetic clinical and LLM-annotated education bandits show that our allocation method reduces fixed-profile mean squared error (MSE) by 20.58% and 10.77%, respectively, relative to no annotation.
Related
- Are Good Generators Good Decision-Makers? Policy Learning for General Interventions via Retargeted Counterfactual Generation
- Counterfactual Constraint-Conditioned On-Policy Distillation for Multi-Constraint Instruction Following
- Which LLM for Which Work? Budgeted Model Allocation under Uncertain Evaluation
- Which Metrics Save the Most Human Annotation? Prediction-Powered Evaluation and Meta-Evaluation
- Reinforcement Learning Under Runtime Confounding: Policy Learning and Off-Policy Evaluation
BibTeX
@misc{xiang2026budgeted,
title = {{Budgeted Multi-Source Counterfactual Annotation for Off-Policy Evaluation}},
author = {Biao Xiang and Ali Eshragh and Yuexing Li and Kai Wang},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.10974},
eprint = {2610.10974},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.10974},
}
RIS
TY - UNPB
TI - Budgeted Multi-Source Counterfactual Annotation for Off-Policy Evaluation
AU - Biao Xiang
AU - Ali Eshragh
AU - Yuexing Li
AU - Kai Wang
PY - 2026
DA - 2026/10/07
PB - arXiv
DO - 10.48550/arxiv.2610.10974
UR - https://arxiv.org/abs/2610.10974
AB - Off-policy evaluation (OPE) estimates the value of a target policy from logged data, but limited behavior-policy coverage can force high-variance reweighting or reward-model extrapolation. Counterfactual annotations can add evidence about unobserved actions, yet practical sources, including domain experts and large language models (LLMs), may be costly, biased, or noisy. We study budgeted acquisition of such annotations for contextual-bandit OPE. Given source-specific costs and error profiles, we formulate an integer allocation problem over context-action pairs and annotation sources to minimize the component of estimator variance that depends on the annotation plan. We characterize when annotations are valuable through a first-annotation threshold and local annotation-value regimes. For the coupled multi-source problem, we develop a majorization-minimization algorithm with dynamic-programming subroutines that monotonically improves the objective. Experiments in synthetic clinical and LLM-annotated education bandits show that our allocation method reduces fixed-profile mean squared error (MSE) by 20.58% and 10.77%, respectively, relative to no annotation.
ER -