Sequential Probability Ratio Test using Z-Statistics (SPRT-z): A Practical Approach for Online Experimentation
Abstract
Modern online experimentation platforms produce data continuously, yet practitioners often apply Fixed Horizon Testing (FHT) while repeatedly peeking at results, inflating Type I error. Popular always-valid methods permit early efficacy stopping but do not natively combine futility stopping, business-relevant launch thresholds, and Type II error control. We adapt Wald's Sequential Probability Ratio Test (SPRT) to meet these needs. SPRT-z uses a large-sample normal approximation to Hajnal's sequential t-test, reducing computation and enabling two Brownian methods. Scale-Free Horizon Calibration (SFHC) sets a maximum sample size to approximately recover fixed-horizon power under discrete monitoring and futility stopping. A Brownian median unbiased estimator (MUE) and confidence intervals are defined for every stopping outcome. A validity analysis provides exact Gaussian-oracle accounting for the deployed stopping rule and a qualified fixed-look approximation for stationary finite-variance outcomes. In simulations with identically distributed arms, SPRT-z kept Type I error at or below nominal, tracked fixed-horizon power within a few percentage points except under the coarsest monitoring, and reduced expected sample size in all but one underpowered setting. The MUE reduced early-stopping bias, with close-to-nominal interval coverage in most settings; we report exceptions and operating rules. Stress tests identify outcome shapes and monitoring schedules that require separate error and power calibration. Across production experiments at Atlassian, median sample saving relative to fixed-horizon plans was 45.5%. More than 75% of decisions came from the futility boundary, supporting its practical importance.
Related
- Probability and statistics in Greece
- Practical Advantages of Online Conformal Prediction for Data Streams
- Martingalized U-statistics-based sequential inference: a blessing of degeneracy
- Structure-Aware Online Optimization of the Age of Information with Unknown System Statistics
- FOCUS: Fixed-Confidence Online Causal Learning Using Sequential Adaptive Interventions
BibTeX
@misc{ho2026sequential,
title = {{Sequential Probability Ratio Test using Z-Statistics (SPRT-z): A Practical Approach for Online Experimentation}},
author = {Derek L. Ho and Emma G. Thomas},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2606.24871},
eprint = {2606.24871},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.24871},
}
RIS
TY - UNPB
TI - Sequential Probability Ratio Test using Z-Statistics (SPRT-z): A Practical Approach for Online Experimentation
AU - Derek L. Ho
AU - Emma G. Thomas
PY - 2026
DA - 2026/09/29
PB - arXiv
DO - 10.48550/arxiv.2606.24871
UR - https://arxiv.org/abs/2606.24871
AB - Modern online experimentation platforms produce data continuously, yet practitioners often apply Fixed Horizon Testing (FHT) while repeatedly peeking at results, inflating Type I error. Popular always-valid methods permit early efficacy stopping but do not natively combine futility stopping, business-relevant launch thresholds, and Type II error control. We adapt Wald's Sequential Probability Ratio Test (SPRT) to meet these needs. SPRT-z uses a large-sample normal approximation to Hajnal's sequential t-test, reducing computation and enabling two Brownian methods. Scale-Free Horizon Calibration (SFHC) sets a maximum sample size to approximately recover fixed-horizon power under discrete monitoring and futility stopping. A Brownian median unbiased estimator (MUE) and confidence intervals are defined for every stopping outcome. A validity analysis provides exact Gaussian-oracle accounting for the deployed stopping rule and a qualified fixed-look approximation for stationary finite-variance outcomes. In simulations with identically distributed arms, SPRT-z kept Type I error at or below nominal, tracked fixed-horizon power within a few percentage points except under the coarsest monitoring, and reduced expected sample size in all but one underpowered setting. The MUE reduced early-stopping bias, with close-to-nominal interval coverage in most settings; we report exceptions and operating rules. Stress tests identify outcome shapes and monitoring schedules that require separate error and power calibration. Across production experiments at Atlassian, median sample saving relative to fixed-horizon plans was 45.5%. More than 75% of decisions came from the futility boundary, supporting its practical importance.
ER -