When is statistical evidence strong enough? Using hypothesis tests to value data collection
Abstract
Policy decisions often hinge on conventional p-value thresholds, which ignore economic costs and benefits of further data collection. This paper recasts statistical significance as a choice between making an immediate policy recommendation and deferring it until further evidence is collected. The welfare-optimal decision corresponds, under minimax regret, to a statistical test whose level depends on the cost and precision of additional evidence. Inverting this rule, we introduce and recommend reporting the abstention-value (A-value) to determine where additional data collection is most needed. The A-value defines the break-even welfare cost of abstaining and recommending further experimentation given the initial evidence. Computing the A-value only requires a point estimate and its standard error, imposes no prior assumptions on policy effects, and can be converted to a monetary research budget using inputs already frequently used by regulators and funding agencies. When experimentation capacity is limited, we show that prioritizing additional data collection where A-values are the largest yields strong finite-sample guarantees. Applications to anti-poverty programs illustrate the practical benefits of using A-values.
Related
- Finite-Sample Binary Hypothesis Testing via Rényi Divergences: Strong Converse and Local Privacy
- Design-Driven Inference for Online Experiments: Jointly Optimal Evidence Collection, Testing, and Estimation
- When n is not enough: Assessing the interplay of spatial sampling error and sample selection in abundance estimation
- One internal stabilization is enough for resolving self-referential tubes
- When Is Enough Enough in Self-Evolving LLM Systems?
BibTeX
@misc{epanomeritakis2026when,
title = {{When is statistical evidence strong enough? Using hypothesis tests to value data collection}},
author = {Aristotelis Epanomeritakis and Davide Viviano},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2609.09544},
eprint = {2609.09544},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.09544},
note = {CC0},
}
RIS
TY - UNPB
TI - When is statistical evidence strong enough? Using hypothesis tests to value data collection
AU - Aristotelis Epanomeritakis
AU - Davide Viviano
PY - 2026
DA - 2026/10/18
PB - arXiv
DO - 10.48550/arxiv.2609.09544
UR - https://arxiv.org/abs/2609.09544
AB - Policy decisions often hinge on conventional p-value thresholds, which ignore economic costs and benefits of further data collection. This paper recasts statistical significance as a choice between making an immediate policy recommendation and deferring it until further evidence is collected. The welfare-optimal decision corresponds, under minimax regret, to a statistical test whose level depends on the cost and precision of additional evidence. Inverting this rule, we introduce and recommend reporting the abstention-value (A-value) to determine where additional data collection is most needed. The A-value defines the break-even welfare cost of abstaining and recommending further experimentation given the initial evidence. Computing the A-value only requires a point estimate and its standard error, imposes no prior assumptions on policy effects, and can be converted to a monetary research budget using inputs already frequently used by regulators and funding agencies. When experimentation capacity is limited, we show that prioritizing additional data collection where A-values are the largest yields strong finite-sample guarantees. Applications to anti-poverty programs illustrate the practical benefits of using A-values.
ER -