Studentized inference in randomized experiments when a few units dominate the outcome variation
Abstract
In completely randomized experiments with heavy-tailed outcomes, a few units can carry a share of the residual variation that does not vanish as the number of units grows. We study the studentized difference in means and Lin's regression-adjusted estimator, with covariates whose leverage vanishes, when the finite population is fixed, and the treatment assignment is the only source of randomness. The studentized statistic then has a nonnormal limit in general. Populations with the same marginal distributions of potential outcomes can have different limits, as the limit depends on the unobserved pairing mechanism of the two potential outcomes of each dominant unit. When the outcome errors are drawn from a heavy-tailed law, the limit is random and, outside a degenerate case, varies with the population. Under balanced allocation, the conventional Wald interval has an explicit asymptotic coverage floor. At the nominal level $0.95$, the floor is about $0.42$ for every population and about $0.84$ when the two potential residuals of each unit are equal. Both floors are close to the best possible, and without balanced allocation, the coverage can fall to the smaller of the proportions of treated and control units.
Related
- Improving power without increasing sample size: Model-based and design-based strategies for controlling time-dependent nuisance variation in experiments
- Regression adjustment in completely randomized experiments with many covariates
- A Primer on the Analysis of Randomized Experiments and a Survey of some Recent Advances
- Design-Driven Inference for Online Experiments: Jointly Optimal Evidence Collection, Testing, and Estimation
- A Tutorial on Safe Anytime-Valid Inference: Practical Maximally Flexible Sampling Designs for Experiments Based on e-Values
BibTeX
@misc{chen2026studentized,
title = {{Studentized inference in randomized experiments when a few units dominate the outcome variation}},
author = {Xinyuan Chen and Jonathan W. Kiesel and Jonathan R. Woody},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.03561},
eprint = {2611.03561},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.03561},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Studentized inference in randomized experiments when a few units dominate the outcome variation
AU - Xinyuan Chen
AU - Jonathan W. Kiesel
AU - Jonathan R. Woody
PY - 2026
DA - 2026/11/02
PB - arXiv
DO - 10.48550/arxiv.2611.03561
UR - https://arxiv.org/abs/2611.03561
AB - In completely randomized experiments with heavy-tailed outcomes, a few units can carry a share of the residual variation that does not vanish as the number of units grows. We study the studentized difference in means and Lin's regression-adjusted estimator, with covariates whose leverage vanishes, when the finite population is fixed, and the treatment assignment is the only source of randomness. The studentized statistic then has a nonnormal limit in general. Populations with the same marginal distributions of potential outcomes can have different limits, as the limit depends on the unobserved pairing mechanism of the two potential outcomes of each dominant unit. When the outcome errors are drawn from a heavy-tailed law, the limit is random and, outside a degenerate case, varies with the population. Under balanced allocation, the conventional Wald interval has an explicit asymptotic coverage floor. At the nominal level $0.95$, the floor is about $0.42$ for every population and about $0.84$ when the two potential residuals of each unit are equal. Both floors are close to the best possible, and without balanced allocation, the coverage can fall to the smaller of the proportions of treated and control units.
ER -