The Price of Correlated Tests: How Strict Should a Model Release Gate Be?
Abstract
Before a machine learning model ships, it often has to pass a suite of automated tests. Requiring every test to pass looks safe, yet it can reject many models that would have served users well, and it does not say how trustworthy a passing model actually is. We treat the release gate as a design problem: choose how many tests a model must pass so that cleared models meet a stated reliability target, while keeping as many good models as possible. A two-class latent-factor model makes both costs explicit and reduces each calculation to a one-dimensional integral. We prove that when both classes share the same latent correlation, a stricter gate always raises reliability, so the gate that keeps the most good models is the most lenient one that still meets the target. Under pass-all gating, any reliability target short of perfection is attainable within the model, but the share of good models kept tends to zero as the suite grows. Correlation between tests sets the price. In one configuration, a 99 percent target needs 8 independent tests, but 74 tests at a latent correlation of 0.3 and 5,182 at 0.5, where the gate keeps fewer than one good model in ten. We also give a validation procedure, built on exact binomial bounds, that certifies a gate from labelled data even when the gate is chosen from a fixed shortlist.
Related
- Where should the null live? Surrogate tests for clustering of threshold-derived air-quality indices, with application to PM2.5 exceedances in Chile
- The Benjamini-Hochberg procedure can fail to control the FDR for correlated two-sided Gaussian tests.
- Strict weak Hörmander conditions for hypoellipticity and structure equations
- A Categorical Realization of the (2-)Category of Monoids via Sch{ü}tzenberger Categories and Strict Factorization Systems
- Equivalence between Strict Viscosity Solution and Viscosity Solution in the Wasserstein Space and Regular Extension of the Hamiltonian in L 2 ℙ
BibTeX
@misc{pollanen2026price,
title = {{The Price of Correlated Tests: How Strict Should a Model Release Gate Be?}},
author = {Marco Pollanen},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.00993},
eprint = {2610.00993},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.00993},
}
RIS
TY - UNPB
TI - The Price of Correlated Tests: How Strict Should a Model Release Gate Be?
AU - Marco Pollanen
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2610.00993
UR - https://arxiv.org/abs/2610.00993
AB - Before a machine learning model ships, it often has to pass a suite of automated tests. Requiring every test to pass looks safe, yet it can reject many models that would have served users well, and it does not say how trustworthy a passing model actually is. We treat the release gate as a design problem: choose how many tests a model must pass so that cleared models meet a stated reliability target, while keeping as many good models as possible. A two-class latent-factor model makes both costs explicit and reduces each calculation to a one-dimensional integral. We prove that when both classes share the same latent correlation, a stricter gate always raises reliability, so the gate that keeps the most good models is the most lenient one that still meets the target. Under pass-all gating, any reliability target short of perfection is attainable within the model, but the share of good models kept tends to zero as the suite grows. Correlation between tests sets the price. In one configuration, a 99 percent target needs 8 independent tests, but 74 tests at a latent correlation of 0.3 and 5,182 at 0.5, where the gate keeps fewer than one good model in ten. We also give a validation procedure, built on exact binomial bounds, that certifies a gate from labelled data even when the gate is chosen from a fixed shortlist.
ER -