PreprintCC BY 4.0arXiv
Testing for trends in online experiments
Abstract
Time-varying treatment effects can be estimated with linear regression paired with jackknife confidence intervals. In experiments at a large online video platform, the test based on this estimator is not much less powerful than modern tests for monotonic trends. If you suspect trends in your online experiments, consider using this estimator.
Related
- Design-Driven Inference for Online Experiments: Jointly Optimal Evidence Collection, Testing, and Estimation
- On the Benefit of Blocking for Online Experiments with Skewed Data
- CAQ: Clustered Asymptotically Anytime-Valid Quantile Treatment Effects for Online Experiments
- Optimal Multi-way Decision Trees for Stratified Sampling in Online Controlled Experiments
- Equivalence testing and run-count design for stochastic machine-learning experiments
BibTeX
@misc{haulk2026testing,
title = {{Testing for trends in online experiments}},
author = {Chris Haulk and Lee Richardson and Jacopo Soriano},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.01973},
eprint = {2609.01973},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.01973},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Testing for trends in online experiments
AU - Chris Haulk
AU - Lee Richardson
AU - Jacopo Soriano
PY - 2026
DA - 2026/09/02
PB - arXiv
DO - 10.48550/arxiv.2609.01973
UR - https://arxiv.org/abs/2609.01973
AB - Time-varying treatment effects can be estimated with linear regression paired with jackknife confidence intervals. In experiments at a large online video platform, the test based on this estimator is not much less powerful than modern tests for monotonic trends. If you suspect trends in your online experiments, consider using this estimator.
ER -