Optimal Expert Annotation for Evaluating Sequential Decision Policies from Text Records
Abstract
Offline reinforcement learning and off-policy evaluation evaluate dynamic decision rules based on historical data, without the ability to deploy online. In recent artificial intelligence (AI) applications, simple state and reward information is recorded as complex text or image, expert annotation may be available but at a higher cost. Recent AI and machine learning (ML) predictors can label data cheaply but with unknown bias. We show how a limited budget for ground-truth data-annotation can safely make use of plentiful ML predictions in a new estimator for doubly-robust off-policy evaluation with missing rewards. Our method directs where data should be collected by optimizing annotation probabilities for the best downstream estimation variance, which we implement in a feasible batch-adaptive approach. Our work is motivated by a collaboration with a homelessness services nonprofit that writes casenotes for individuals over time. Our method can be used to unlock trustworthy inference from casenote data and answer new inferential questions such as: how does expanding outreach effort over time affect progress towards a housing application and improvement in housing placement? In simulations and on two real datasets -- casenotes from the nonprofit and human-preference votes from LMArena -- we see reductions in RMSE of 37-65% for housing placement and 21-67% for progress towards a housing application at budgets of 50% of full annotation and above, and by 55-62% at every budget on LMArena under an equal-exposure policy.
Related
- Optimal sequential decision-making with initiation regimes
- Optimal ratcheting dividend policies with resets
- Optimal Feature-Based Newsvendor Policies under Censored Demand
- Optimal Regret for Online Storage Control via Cumulative Policies
- Optimal long-run control of endemic infections: bang-bang threshold policies in a stochastic SIS model
BibTeX
@misc{chae2026optimal,
title = {{Optimal Expert Annotation for Evaluating Sequential Decision Policies from Text Records}},
author = {Woojin Chae and Ezinne Nwankwo and Haitong Qin and Angela Zhou},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2609.26707},
eprint = {2609.26707},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.26707},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Optimal Expert Annotation for Evaluating Sequential Decision Policies from Text Records
AU - Woojin Chae
AU - Ezinne Nwankwo
AU - Haitong Qin
AU - Angela Zhou
PY - 2026
DA - 2026/10/28
PB - arXiv
DO - 10.48550/arxiv.2609.26707
UR - https://arxiv.org/abs/2609.26707
AB - Offline reinforcement learning and off-policy evaluation evaluate dynamic decision rules based on historical data, without the ability to deploy online. In recent artificial intelligence (AI) applications, simple state and reward information is recorded as complex text or image, expert annotation may be available but at a higher cost. Recent AI and machine learning (ML) predictors can label data cheaply but with unknown bias. We show how a limited budget for ground-truth data-annotation can safely make use of plentiful ML predictions in a new estimator for doubly-robust off-policy evaluation with missing rewards. Our method directs where data should be collected by optimizing annotation probabilities for the best downstream estimation variance, which we implement in a feasible batch-adaptive approach. Our work is motivated by a collaboration with a homelessness services nonprofit that writes casenotes for individuals over time. Our method can be used to unlock trustworthy inference from casenote data and answer new inferential questions such as: how does expanding outreach effort over time affect progress towards a housing application and improvement in housing placement? In simulations and on two real datasets -- casenotes from the nonprofit and human-preference votes from LMArena -- we see reductions in RMSE of 37-65% for housing placement and 21-67% for progress towards a housing application at budgets of 50% of full annotation and above, and by 55-62% at every budget on LMArena under an equal-exposure policy.
ER -