Beyond the Illusion of Power: Calibrating Quasi-Experiments in Observational IS
Abstract
Information systems (IS) researchers increasingly use quasi-experimental methods such as difference-in-differences (DiD) and instrumental variables (IV) to recover causal effects from observational panel data. Power calculations that justify these designs assume i.i.d. errors, but the deeper problem is what even a cluster-robust calculator cannot see. We report a Monte Carlo study over 9837 parameter conditions (approx 9.8 million datasets) and decompose the planned-versus-achieved power gap. The serial-correlation component is recoverable by an AR(1)-aware calculator when rho is known, and partially when rho must be estimated from short pre-periods, but panel attrition, staggered-adoption bias, and parallel-trends pretesting are captured by no closed-form formula; exogenous attrition alone costs approx 8 to 11 percentage points at the few-hundred-to-thousand sample sizes IS studies use. Treatment-correlated, outcome-dependent attrition instead induces bias, not just power loss. For IV, holding first-stage F fixed, larger N neither raises power nor curbs exclusion bias, though with a fixed instrument more data does sharpen the first stage, so identification rests on instrument strength, not sample size.
Related
- Improving power without increasing sample size: Model-based and design-based strategies for controlling time-dependent nuisance variation in experiments
- Multi-Power Inference Beyond Semimartingales: Rough Gaussian Activity and Jumps
- Calibrating Ambiguity Set via Diagnostic Transport for Distributionally Robust Optimization
- Calibrating subgrid parametrizations of single-column ocean models via simulation-based inference
- A Sensitivity-based Framework for Calibrating Coupled Ordinary Differential Equations under Model Discrepancy
BibTeX
@misc{chowdhury2026beyond,
title = {{Beyond the Illusion of Power: Calibrating Quasi-Experiments in Observational IS}},
author = {Spandan Ghose Chowdhury},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.27299},
eprint = {2609.27299},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.27299},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Beyond the Illusion of Power: Calibrating Quasi-Experiments in Observational IS
AU - Spandan Ghose Chowdhury
PY - 2026
DA - 2026/09/23
PB - arXiv
DO - 10.48550/arxiv.2609.27299
UR - https://arxiv.org/abs/2609.27299
AB - Information systems (IS) researchers increasingly use quasi-experimental methods such as difference-in-differences (DiD) and instrumental variables (IV) to recover causal effects from observational panel data. Power calculations that justify these designs assume i.i.d. errors, but the deeper problem is what even a cluster-robust calculator cannot see. We report a Monte Carlo study over 9837 parameter conditions (approx 9.8 million datasets) and decompose the planned-versus-achieved power gap. The serial-correlation component is recoverable by an AR(1)-aware calculator when rho is known, and partially when rho must be estimated from short pre-periods, but panel attrition, staggered-adoption bias, and parallel-trends pretesting are captured by no closed-form formula; exogenous attrition alone costs approx 8 to 11 percentage points at the few-hundred-to-thousand sample sizes IS studies use. Treatment-correlated, outcome-dependent attrition instead induces bias, not just power loss. For IV, holding first-stage F fixed, larger N neither raises power nor curbs exclusion bias, though with a fixed instrument more data does sharpen the first stage, so identification rests on instrument strength, not sample size.
ER -