Sample-Efficient Multiple Testing with Adaptive Data Collection
Abstract
This paper studies adaptive experimental design for multiple testing, where an experimenter sequentially chooses which hypothesis to sample. We propose the e-value-based posterior sampling (e-PS) procedure, which uses the empirical average of log e-value increments to guide randomized sampling and applies e-BH to construct rejection sets. Under conditionally valid e-value increments, the procedure controls the false discovery rate at arbitrary stopping times and produces nested rejection sets. We establish high-probability bounds on the number of samples needed to discover all nonnull hypotheses in terms of the growth and concentration of the underlying e-processes. We specialize these bounds to simple-versus-simple, composite-versus-simple, and simple-versus-composite testing. Simulations and experiments using joke ratings and watermarked text illustrate the procedure's power under limited sampling budgets.
Related
- Design-Driven Inference for Online Experiments: Jointly Optimal Evidence Collection, Testing, and Estimation
- Data-Matrix-Based Overall Goodness-of-Fit Testing for Confirmatory Factor Analysis: In-Sample Prediction Error for Observed Variables
- OLS Testing with Predictors Scaled to Have Unit Sample Variance, and Similar Results for a Robust Regression Estimator
- Finite Sample Bounds for Composite Hypothesis Testing
- A Model of Multiple Hypothesis Testing
BibTeX
@misc{lin2026sampleefficient,
title = {{Sample-Efficient Multiple Testing with Adaptive Data Collection}},
author = {Zhanran Lin and Wanteng Ma and Zhimei Ren and Yuting Wei},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.26651},
eprint = {2609.26651},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.26651},
}
RIS
TY - UNPB
TI - Sample-Efficient Multiple Testing with Adaptive Data Collection
AU - Zhanran Lin
AU - Wanteng Ma
AU - Zhimei Ren
AU - Yuting Wei
PY - 2026
DA - 2026/09/22
PB - arXiv
DO - 10.48550/arxiv.2609.26651
UR - https://arxiv.org/abs/2609.26651
AB - This paper studies adaptive experimental design for multiple testing, where an experimenter sequentially chooses which hypothesis to sample. We propose the e-value-based posterior sampling (e-PS) procedure, which uses the empirical average of log e-value increments to guide randomized sampling and applies e-BH to construct rejection sets. Under conditionally valid e-value increments, the procedure controls the false discovery rate at arbitrary stopping times and produces nested rejection sets. We establish high-probability bounds on the number of samples needed to discover all nonnull hypotheses in terms of the growth and concentration of the underlying e-processes. We specialize these bounds to simple-versus-simple, composite-versus-simple, and simple-versus-composite testing. Simulations and experiments using joke ratings and watermarked text illustrate the procedure's power under limited sampling budgets.
ER -