CAQ: Clustered Asymptotically Anytime-Valid Quantile Treatment Effects for Online Experiments
Abstract
Confidence sequences let an experiment be monitored continuously without inflating error, but the existing constructions for quantiles assume that observations arrive independently. In online experiments this rarely holds: randomization assigns users, while the metric is computed over the many correlated events each user generates, so intervals built for independent data are too narrow and miss the truth more often than their nominal rate allows. We propose CAQ (Clustered, asymptotically Anytime-valid Quantile), a confidence sequence for a quantile that is both robust to this within-user dependence and valid at arbitrary stopping times, together with a sequential test for the difference in a quantile between two treatment arms. The width adapts to the dependence actually present, widening beyond the independent-data benchmark only when correlation within users is strong. In simulations, CAQ is the only method we tested that holds both coverage and false-positive rate at their nominal levels across clustering strengths and heavy-tailed data, while the fixed-horizon baseline fails under continuous monitoring and the independence-assuming sequential baseline fails as within-user correlation grows.
Related
- Asymptotic Anytime-Valid Quantile Inference under Local Differential Privacy
- Anytime-Valid Inference in Linear Models with Clustered Outcomes: Exact Multivariate Tests and Cluster-Robust Confidence Sequences
- Bet on Features: Anytime-Valid and Feature-Aware Auditing of Conditional Quantile Forecasters
- Boot-AVT: Asymptotically Anytime-Valid Testing with Degenerate U-Statistics via Bootstrap
- A Tutorial on Safe Anytime-Valid Inference: Practical Maximally Flexible Sampling Designs for Experiments Based on e-Values
BibTeX
@misc{lu2026clustered,
title = {{CAQ: Clustered Asymptotically Anytime-Valid Quantile Treatment Effects for Online Experiments}},
author = {Diane Lu and Eric J. Oh},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.30019},
eprint = {2610.30019},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.30019},
}
RIS
TY - UNPB
TI - CAQ: Clustered Asymptotically Anytime-Valid Quantile Treatment Effects for Online Experiments
AU - Diane Lu
AU - Eric J. Oh
PY - 2026
DA - 2026/10/23
PB - arXiv
DO - 10.48550/arxiv.2610.30019
UR - https://arxiv.org/abs/2610.30019
AB - Confidence sequences let an experiment be monitored continuously without inflating error, but the existing constructions for quantiles assume that observations arrive independently. In online experiments this rarely holds: randomization assigns users, while the metric is computed over the many correlated events each user generates, so intervals built for independent data are too narrow and miss the truth more often than their nominal rate allows. We propose CAQ (Clustered, asymptotically Anytime-valid Quantile), a confidence sequence for a quantile that is both robust to this within-user dependence and valid at arbitrary stopping times, together with a sequential test for the difference in a quantile between two treatment arms. The width adapts to the dependence actually present, widening beyond the independent-data benchmark only when correlation within users is strong. In simulations, CAQ is the only method we tested that holds both coverage and false-positive rate at their nominal levels across clustering strengths and heavy-tailed data, while the fixed-horizon baseline fails under continuous monitoring and the independence-assuming sequential baseline fails as within-user correlation grows.
ER -