Many Hands Make Light Work: Long-Term Effect Inference for the Experiment Reservoir
Abstract
The surrogate index method is the modern standard for estimation of long-term effects in a designed experiment using only short-term outcomes. This method relies on the surrogacy and comparability assumptions which may be violated in practice leading to estimation bias. We propose a new method called Experiment Reservoir Surrogate (ERS) which leverages a reservoir of historical long-term experiments that is common in large-scale experimentation platforms. ERS avoids the surrogacy assumption by predicting the long-term average treatment effect directly, and relaxes the comparability assumption by matching to similar historical experiments, without requiring every experiment in the reservoir to be comparable to the target experiment. Moreover, ERS is memory-efficient, as it only requires aggregate data for each time period, instead of raw individual-level data. We theoretically justify the improved efficiency of ERS and demonstrate the improvement empirically on a reservoir of experiments from the online platform DoorDash.
Related
- A charge selection rule fixes what a squeezed-light reservoir computer can compute and afford
- Infinite light rays and infinite clusters with infinitely many pivots
- Goldbach and Alexandroff Shake Hands
- How long is a game of Uno? A counting law for large hands and the Wild Draw Four hoard
- The internal model principle in reservoir computing
BibTeX
@misc{hartog2026many,
title = {{Many Hands Make Light Work: Long-Term Effect Inference for the Experiment Reservoir}},
author = {Will Hartog and Lihua Lei and Mandar Rahurkar and Navdeep Sahni and Yixin Tang},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.36352},
eprint = {2610.36352},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.36352},
}
RIS
TY - UNPB
TI - Many Hands Make Light Work: Long-Term Effect Inference for the Experiment Reservoir
AU - Will Hartog
AU - Lihua Lei
AU - Mandar Rahurkar
AU - Navdeep Sahni
AU - Yixin Tang
PY - 2026
DA - 2026/10/26
PB - arXiv
DO - 10.48550/arxiv.2610.36352
UR - https://arxiv.org/abs/2610.36352
AB - The surrogate index method is the modern standard for estimation of long-term effects in a designed experiment using only short-term outcomes. This method relies on the surrogacy and comparability assumptions which may be violated in practice leading to estimation bias. We propose a new method called Experiment Reservoir Surrogate (ERS) which leverages a reservoir of historical long-term experiments that is common in large-scale experimentation platforms. ERS avoids the surrogacy assumption by predicting the long-term average treatment effect directly, and relaxes the comparability assumption by matching to similar historical experiments, without requiring every experiment in the reservoir to be comparable to the target experiment. Moreover, ERS is memory-efficient, as it only requires aggregate data for each time period, instead of raw individual-level data. We theoretically justify the improved efficiency of ERS and demonstrate the improvement empirically on a reservoir of experiments from the online platform DoorDash.
ER -