Improving power without increasing sample size: Model-based and design-based strategies for controlling time-dependent nuisance variation in experiments
Abstract
Measurements taken during experiments are perturbed by time-dependent nuisance variation, due to factors such as learning or fatigue. Researchers can improve power by controlling for these effects using either a model-based approach (e.g., a Generalized Additive Mixed Models with Wiggly Intercepts, or WIMs), or a design-based approach (e.g., Permuted Subblock Randomization, or PSR). We used Monte Carlo simulations to compare the power advantage of using WIMs versus PSR across 36 scenarios combining nine different study designs and four types of temporally structured error (learning effect, random walk, pink noise, and mixed). In nearly all scenarios, WIMs and PSR reliably improved power as compared to an uncorrected baseline, within a range of 6% to 77%. Power was especially enhanced for studies with smaller numbers of trials. In 61% of cases, both approaches yielded similar boosts in power. For data with a learning effect, a random walk, or a mixed structure, WIMs often outperformed PSR, usually by a large margin; for pink noise, PSR modestly outperformed WIMs. There was no consistent added benefit of combining the two approaches. We close with practical recommendations for researchers.
Related
BibTeX
@misc{liang2026improving,
title = {{Improving power without increasing sample size: Model-based and design-based strategies for controlling time-dependent nuisance variation in experiments}},
author = {Jinghui Liang and Dale Barr},
howpublished = {PsyArXiv},
year = {2026},
month = sep,
doi = {10.31234/osf.io/m6fg3_v3},
url = {https://osf.io/preprints/psyarxiv/m6fg3_v3/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Improving power without increasing sample size: Model-based and design-based strategies for controlling time-dependent nuisance variation in experiments
AU - Jinghui Liang
AU - Dale Barr
PY - 2026
DA - 2026/09/05
PB - PsyArXiv
DO - 10.31234/osf.io/m6fg3_v3
UR - https://osf.io/preprints/psyarxiv/m6fg3_v3/
AB - Measurements taken during experiments are perturbed by time-dependent nuisance variation, due to factors such as learning or fatigue. Researchers can improve power by controlling for these effects using either a model-based approach (e.g., a Generalized Additive Mixed Models with Wiggly Intercepts, or WIMs), or a design-based approach (e.g., Permuted Subblock Randomization, or PSR). We used Monte Carlo simulations to compare the power advantage of using WIMs versus PSR across 36 scenarios combining nine different study designs and four types of temporally structured error (learning effect, random walk, pink noise, and mixed). In nearly all scenarios, WIMs and PSR reliably improved power as compared to an uncorrected baseline, within a range of 6% to 77%. Power was especially enhanced for studies with smaller numbers of trials. In 61% of cases, both approaches yielded similar boosts in power. For data with a learning effect, a random walk, or a mixed structure, WIMs often outperformed PSR, usually by a large margin; for pink noise, PSR modestly outperformed WIMs. There was no consistent added benefit of combining the two approaches. We close with practical recommendations for researchers.
ER -