PreprintCC BY 4.0arXiv
On the Benefit of Blocking for Online Experiments with Skewed Data
Abstract
This paper advocates for using blocked (stratified) assignment in online A/B tests to handle highly skewed population data. While blocking yields only modest precision gains (5-10%) due to the limits of discretization, the authors demonstrate it offers two crucial structural benefits over post-hoc statistical adjustments. First, fixed-weight blocking correctly targets the true average treatment effect, avoiding the severe bias introduced by efficiency-weighted alternatives. Second, blocking localizes extreme outliers, enabling targeted within-block winsorization that effectively controls noise without destroying the tail-end treatment effect.
Related
- Testing for trends in online experiments
- CAQ: Clustered Asymptotically Anytime-Valid Quantile Treatment Effects for Online Experiments
- Optimal Multi-way Decision Trees for Stratified Sampling in Online Controlled Experiments
- Design-Driven Inference for Online Experiments: Jointly Optimal Evidence Collection, Testing, and Estimation
- A Hajnal-Szemerédi Theorem for Skewed Colorings
BibTeX
@misc{zaidi2026benefit,
title = {{On the Benefit of Blocking for Online Experiments with Skewed Data}},
author = {Abbas Zaidi and Matthew Reimherr and Rina Friedberg and Fabian Martinez and Richard Mudd},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.10435},
eprint = {2610.10435},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.10435},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - On the Benefit of Blocking for Online Experiments with Skewed Data
AU - Abbas Zaidi
AU - Matthew Reimherr
AU - Rina Friedberg
AU - Fabian Martinez
AU - Richard Mudd
PY - 2026
DA - 2026/10/07
PB - arXiv
DO - 10.48550/arxiv.2610.10435
UR - https://arxiv.org/abs/2610.10435
AB - This paper advocates for using blocked (stratified) assignment in online A/B tests to handle highly skewed population data. While blocking yields only modest precision gains (5-10%) due to the limits of discretization, the authors demonstrate it offers two crucial structural benefits over post-hoc statistical adjustments. First, fixed-weight blocking correctly targets the true average treatment effect, avoiding the severe bias introduced by efficiency-weighted alternatives. Second, blocking localizes extreme outliers, enabling targeted within-block winsorization that effectively controls noise without destroying the tail-end treatment effect.
ER -