Sparse departures from independence in two-way tables: a heteroscedasticity profile and detection boundary, an adaptive higher-criticism gate, and an assumption-lean exact anchor
Abstract
Two-way contingency tables are tested for independence throughout applied statistics (genomics, network and text co-occurrence, pharmacovigilance, ecology, survey cross-tabulation), and the routine test reads asymptotic Gaussian tail probabilities off the table cell by cell. On the tables people actually analyze this is badly miscalibrated: across 5,543 real public two-way tables, two-thirds have counts small enough or margins heterogeneous enough that the asymptotic maximum-cell independence scan false-positives at a mean 33% against a 0.05 target, while an exact margin-conditional anchor holds at about 1%. The failure is sharpest when the departure is sparse, the association concentrated in a few cells, where the omnibus chi-square is inefficient and the sharp object is the detection boundary. In three parts we answer where a sparse signal can be seen, which combiner attains that limit, and how to calibrate at small counts. Part I specializes the sparse-detection theory of Ingster (1997), Donoho and Jin (2004), and Chhor, Mukherjee and Sen (2024) to independence: the table is a heteroscedastic Gaussian sequence whose variance profile is the expected-count table, fixed by the margins via CVe^2 = (1 + CVr^2)(1 + CVc^2) - 1, giving an exact signal map and a closed-form separation radius valid under a moderate-deviation growing-count condition. Part II shows higher criticism, with its closed-form Jaeschke-Eicker null, attains that boundary adaptively over unknown sparsity. Part III measures the small-count calibration failure (size 0.3 to 0.6 even under uniform margins, so the driver is the per-cell tail), removes it with exact margin-conditional laws (size 0.003 to 0.009), and gives a routing rule sending large-count tables to the asymptotic gate and small-count tables to the exact anchor. Every claim is reproduced from openly deposited, deterministically seeded code.
Related
- lifelines-hc: Higher Criticism testing for sparse non-proportional hazard departures in Python
- On the Relaxation of Conditional Independence Assumption for Image Segmentation
- Precise Asymptotics for Dense High-Dimensional Huber Regression under Heteroscedasticity
- Using Heteroscedasticity for Causal Discovery under Hidden Confounding
- A Copula-Based Regression Framework for Enhanced Prediction under Heteroscedasticity
BibTeX
@misc{dwyer2026sparse,
title = {{Sparse departures from independence in two-way tables: a heteroscedasticity profile and detection boundary, an adaptive higher-criticism gate, and an assumption-lean exact anchor}},
author = {William J. Dwyer},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2608.07979},
eprint = {2608.07979},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.07979},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - Sparse departures from independence in two-way tables: a heteroscedasticity profile and detection boundary, an adaptive higher-criticism gate, and an assumption-lean exact anchor
AU - William J. Dwyer
PY - 2026
DA - 2026/09/08
PB - arXiv
DO - 10.48550/arxiv.2608.07979
UR - https://arxiv.org/abs/2608.07979
AB - Two-way contingency tables are tested for independence throughout applied statistics (genomics, network and text co-occurrence, pharmacovigilance, ecology, survey cross-tabulation), and the routine test reads asymptotic Gaussian tail probabilities off the table cell by cell. On the tables people actually analyze this is badly miscalibrated: across 5,543 real public two-way tables, two-thirds have counts small enough or margins heterogeneous enough that the asymptotic maximum-cell independence scan false-positives at a mean 33% against a 0.05 target, while an exact margin-conditional anchor holds at about 1%. The failure is sharpest when the departure is sparse, the association concentrated in a few cells, where the omnibus chi-square is inefficient and the sharp object is the detection boundary. In three parts we answer where a sparse signal can be seen, which combiner attains that limit, and how to calibrate at small counts. Part I specializes the sparse-detection theory of Ingster (1997), Donoho and Jin (2004), and Chhor, Mukherjee and Sen (2024) to independence: the table is a heteroscedastic Gaussian sequence whose variance profile is the expected-count table, fixed by the margins via CVe^2 = (1 + CVr^2)(1 + CVc^2) - 1, giving an exact signal map and a closed-form separation radius valid under a moderate-deviation growing-count condition. Part II shows higher criticism, with its closed-form Jaeschke-Eicker null, attains that boundary adaptively over unknown sparsity. Part III measures the small-count calibration failure (size 0.3 to 0.6 even under uniform margins, so the driver is the per-cell tail), removes it with exact margin-conditional laws (size 0.003 to 0.009), and gives a routing rule sending large-count tables to the asymptotic gate and small-count tables to the exact anchor. Every claim is reproduced from openly deposited, deterministically seeded code.
ER -