Let Them Peek: Group Sequential Designs for A/B Testing
Abstract
Randomized controlled trials, commonly known as A/B tests, are the standard mechanism for inferring causal effects of product innovations. Standard platforms evaluate these tests with fixed-horizon procedures that control the false positive rate at a predetermined time, forcing teams to wait until the end even when early data strongly indicates success or failure. Unfortunately, continuous monitoring ("peeking") invalidates these guarantees, while always-valid inference is overly conservative under a strict deadline, and group sequential designs (GSD) have received limited traction in large-scale experimentation due to operational barriers: terminal-rule preservation, autocorrelation, latent effect-size distributions, calendar-duration optimization. We fill in this gap by proposing a Predictive Probability of Success (PPoS) group sequential framework that is autocorrelation-aware and duration-optimal while preserving the platform's fixed-horizon terminal rule. Our key insight is that PPoS thresholds map to closed-form lift boundaries that converge to the standard test at the horizon by construction, admit an effective-sample-size correction estimated from historical A/A tests, and suffice to attain the likelihood-ratio efficiency floor. Experimentally on online A/A and A/B experiments, we show that the proposed policy reduces expected duration by 43.1% (7.97 days vs. 14-day baseline), increasing effective platform capacity by 76%, while retaining the guarantees on FPR and power.
Related
- Combining Adaptive Seamless Phase II/III Designs and Group Sequential Analysis for Time-to-Event Endpoints in Oncology Trials
- Target-Aware Sequential Inference: Pooled versus Stratified Anytime-Valid Designs
- Amply regular graphs with μ close to half the valency and group divisible designs
- Graphs with three eigenvalues from group divisible designs
- Sequential Convergence in the Testing Function Space of the Two-Sided FKF Transform
BibTeX
@misc{perini2026them,
title = {{Let Them Peek: Group Sequential Designs for A/B Testing}},
author = {Lorenzo Perini and Mia Garrard and Rina Friedberg and Rohit Kumar and Houssam Nassif and Abbas Zaidi and Pablo Alcain},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.27899},
eprint = {2610.27899},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.27899},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Let Them Peek: Group Sequential Designs for A/B Testing
AU - Lorenzo Perini
AU - Mia Garrard
AU - Rina Friedberg
AU - Rohit Kumar
AU - Houssam Nassif
AU - Abbas Zaidi
AU - Pablo Alcain
PY - 2026
DA - 2026/10/20
PB - arXiv
DO - 10.48550/arxiv.2610.27899
UR - https://arxiv.org/abs/2610.27899
AB - Randomized controlled trials, commonly known as A/B tests, are the standard mechanism for inferring causal effects of product innovations. Standard platforms evaluate these tests with fixed-horizon procedures that control the false positive rate at a predetermined time, forcing teams to wait until the end even when early data strongly indicates success or failure. Unfortunately, continuous monitoring ("peeking") invalidates these guarantees, while always-valid inference is overly conservative under a strict deadline, and group sequential designs (GSD) have received limited traction in large-scale experimentation due to operational barriers: terminal-rule preservation, autocorrelation, latent effect-size distributions, calendar-duration optimization. We fill in this gap by proposing a Predictive Probability of Success (PPoS) group sequential framework that is autocorrelation-aware and duration-optimal while preserving the platform's fixed-horizon terminal rule. Our key insight is that PPoS thresholds map to closed-form lift boundaries that converge to the standard test at the horizon by construction, admit an effective-sample-size correction estimated from historical A/A tests, and suffice to attain the likelihood-ratio efficiency floor. Experimentally on online A/A and A/B experiments, we show that the proposed policy reduces expected duration by 43.1% (7.97 days vs. 14-day baseline), increasing effective platform capacity by 76%, while retaining the guarantees on FPR and power.
ER -