The Exceedance Design Effect: Effective Sample Size for Thresholds under Clustering
Abstract
Suppose we want a cutoff that 90% of a population falls below. We estimate it from a sample, and another sample would give a different cutoff and a different fraction below it. We ask how much that fraction varies when observations come in independent groups, such as pupils in classrooms or sentences in news articles. We prove that grouping multiplies its large-sample variance by 1+(m−1)ρI(p), where m is the group size, p is the target fraction, and ρI(p) measures whether two members of a group fall on the same side of the cutoff. That correlation can differ from the correlation between the scores themselves, and it changes with the target. We give a direct proof, a counterexample to using score correlation, and an extension to unequal group sizes. A dataset therefore does not have one effective sample size. How much information it contains depends on the question you ask. In our document experiment, the same 1,000 rows carried about 217 independent observations' worth of information at the median. At the 95th percentile, they carried about 621. Nothing about the dataset changed. We asked it a different question. The number of rows is a property of the dataset. The effective sample size belongs to the analysis.
Related
- Reliable Sample-Level Quantum Error Mitigation via Dominance-Aware Clustering
- A Note on the Finite-Exceedance Dependence Principle for Sum--Maximum Asymptotic Independence
- Improving power without increasing sample size: Model-based and design-based strategies for controlling time-dependent nuisance variation in experiments
- Finite sample behavior of the maximum likelihood estimator in the Poisson model under Gaussian design
- On Combined Maximal Precedence–Exceedance Test with Application to HIV Clinical Trials
BibTeX
@misc{noonan2026exceedance,
title = {{The Exceedance Design Effect: Effective Sample Size for Thresholds under Clustering}},
author = {Adam Noonan},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2608.21262},
eprint = {2608.21262},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.21262},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - The Exceedance Design Effect: Effective Sample Size for Thresholds under Clustering
AU - Adam Noonan
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2608.21262
UR - https://arxiv.org/abs/2608.21262
AB - Suppose we want a cutoff that 90% of a population falls below. We estimate it from a sample, and another sample would give a different cutoff and a different fraction below it. We ask how much that fraction varies when observations come in independent groups, such as pupils in classrooms or sentences in news articles. We prove that grouping multiplies its large-sample variance by 1+(m−1)ρ_I(p), where m is the group size, p is the target fraction, and ρ_I(p) measures whether two members of a group fall on the same side of the cutoff. That correlation can differ from the correlation between the scores themselves, and it changes with the target. We give a direct proof, a counterexample to using score correlation, and an extension to unequal group sizes. A dataset therefore does not have one effective sample size. How much information it contains depends on the question you ask. In our document experiment, the same 1,000 rows carried about 217 independent observations' worth of information at the median. At the 95th percentile, they carried about 621. Nothing about the dataset changed. We asked it a different question. The number of rows is a property of the dataset. The effective sample size belongs to the analysis.
ER -