Beyond χ2: Ensemble null tests for reliable scientific discovery
Abstract
Unrecognized systematic errors can survive standard checks and undermine scientific discoveries. Null tests and experimental controls can expose these failures, but averaging measurements or reducing them to scalar statistics can hide important structure. We propose ensemble null tests that compare multivariate distributions with references from observations or simulations. We demonstrate their benefits across science. In cosmic microwave background simulations, they detect systematic effects missed by conventional χ2 tests. In Gaia data, they detect velocity asymmetry as sensitively as an informed search. In a CRISPR screen, they expose a control mismatch in DNA damage and reveal which cellular profiles drive it. As scientific datasets grow, ensemble testing turns repeated measurements into a broader search for unknown unknowns, strengthening the empirical foundation for scientific discovery.
Related
- Controlled ensemble null tests of dynamical dark energy hints: single-probe recovery of ΛCDM with BAO, compressed-CMB, and SNe mocks
- Physical Law Ecology: mapping multi-mechanism ecologies as the zeroth step of data-driven scientific discovery
- The impact of spin priors on parameterized tests of general relativity
- AI's Capability in Assisting Scientific Research in Physics, Astrophysics, and Cosmology I: Literature Review
- AI's Capability in Assisting Scientific Research in Physics, Astrophysics, and Cosmology II: Project Planning and Proposal Evaluation
BibTeX
@misc{bayer2026beyond,
title = {{Beyond $χ^2$: Ensemble null tests for reliable scientific discovery}},
author = {Adrian E. Bayer and David N. Spergel},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.40167},
eprint = {2610.40167},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.40167},
}
RIS
TY - UNPB
TI - Beyond χ^2: Ensemble null tests for reliable scientific discovery
AU - Adrian E. Bayer
AU - David N. Spergel
PY - 2026
DA - 2026/10/29
PB - arXiv
DO - 10.48550/arxiv.2610.40167
UR - https://arxiv.org/abs/2610.40167
AB - Unrecognized systematic errors can survive standard checks and undermine scientific discoveries. Null tests and experimental controls can expose these failures, but averaging measurements or reducing them to scalar statistics can hide important structure. We propose ensemble null tests that compare multivariate distributions with references from observations or simulations. We demonstrate their benefits across science. In cosmic microwave background simulations, they detect systematic effects missed by conventional χ^2 tests. In Gaia data, they detect velocity asymmetry as sensitively as an informed search. In a CRISPR screen, they expose a control mismatch in DNA damage and reveal which cellular profiles drive it. As scientific datasets grow, ensemble testing turns repeated measurements into a broader search for unknown unknowns, strengthening the empirical foundation for scientific discovery.
ER -