Inference on the Value of Policy Improvement in Adaptive Experiments
Abstract
In this paper, we develop an asymptotically normal estimator for the value of policy improvement (VPI) under adaptively collected data. This value, which is defined as the gap between the value of the best treatment policy in a class and that of a baseline policy, gauges whether or not it is worth deploying a personalized treatment policy. Building an estimator for the VPI is naturally more difficult than estimating the value of a fixed policy. This is because, in addition to contending with the correlated data and decaying propensities associated with adaptive experiments, any estimator must also handle the non-regularity, or lack of pathwise differentiability, of the underlying estimand when the optimal policy is non-unique. Our proposed estimator works by sequentially fitting a weighted empirical welfare maximizing policy on historical data and then evaluating it using inverse propensity weighting on future samples. We neither assume that the optimal policy is unique nor make a margin assumption on the underlying heterogeneous treatment effects. A key difficulty in analyzing our estimator is showing that the regret incurred through policy learning is asymptotically smaller than the standard error of the estimator. We accomplish this by showing faster regret rates for policy learning under decaying propensities, which may be of independent interest. We not only evaluate the coverage of our estimator on a mix of synthetic and semi-synthetic data, but also employ it to estimate the VPI in an adaptive experiment evaluating the efficacy of job assistance programs in Jordan due to Caria et al. Although our estimates are generally more conservative than their hierarchical Bayesian analysis, which assumes that the optimal policy is unique, we still corroborate the benefits of personalization amongst Syrian refugees.
Related
- Schrödinger--Föllmer Actor--Critic: Diffusion Policy Improvement with Finite-Sample Analysis
- Design-Driven Inference for Online Experiments: Jointly Optimal Evidence Collection, Testing, and Estimation
- A Tutorial on Safe Anytime-Valid Inference: Practical Maximally Flexible Sampling Designs for Experiments Based on e-Values
- Studentized inference in randomized experiments when a few units dominate the outcome variation
- Anytime-Valid Inference in Conjoint Experiments
BibTeX
@misc{whitehouse2026inference,
title = {{Inference on the Value of Policy Improvement in Adaptive Experiments}},
author = {Justin Whitehouse and Emma Brunskill and Vasilis Syrgkanis},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.15789},
eprint = {2610.15789},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.15789},
}
RIS
TY - UNPB
TI - Inference on the Value of Policy Improvement in Adaptive Experiments
AU - Justin Whitehouse
AU - Emma Brunskill
AU - Vasilis Syrgkanis
PY - 2026
DA - 2026/10/11
PB - arXiv
DO - 10.48550/arxiv.2610.15789
UR - https://arxiv.org/abs/2610.15789
AB - In this paper, we develop an asymptotically normal estimator for the value of policy improvement (VPI) under adaptively collected data. This value, which is defined as the gap between the value of the best treatment policy in a class and that of a baseline policy, gauges whether or not it is worth deploying a personalized treatment policy. Building an estimator for the VPI is naturally more difficult than estimating the value of a fixed policy. This is because, in addition to contending with the correlated data and decaying propensities associated with adaptive experiments, any estimator must also handle the non-regularity, or lack of pathwise differentiability, of the underlying estimand when the optimal policy is non-unique. Our proposed estimator works by sequentially fitting a weighted empirical welfare maximizing policy on historical data and then evaluating it using inverse propensity weighting on future samples. We neither assume that the optimal policy is unique nor make a margin assumption on the underlying heterogeneous treatment effects. A key difficulty in analyzing our estimator is showing that the regret incurred through policy learning is asymptotically smaller than the standard error of the estimator. We accomplish this by showing faster regret rates for policy learning under decaying propensities, which may be of independent interest. We not only evaluate the coverage of our estimator on a mix of synthetic and semi-synthetic data, but also employ it to estimate the VPI in an adaptive experiment evaluating the efficacy of job assistance programs in Jordan due to Caria et al. Although our estimates are generally more conservative than their hierarchical Bayesian analysis, which assumes that the optimal policy is unique, we still corroborate the benefits of personalization amongst Syrian refugees.
ER -