Neyman-Pearson Classification with Indecisions: Simultaneous Control of Selective Type I and Type II Errors
Abstract
Binary classification often requires controlling both type I and type II error rates at prespecified levels. When the class distributions overlap substantially, the two error limits may be incompatible if every observation must be classified. We study Neyman-Pearson classification with indecisions (NPI), which permits uncertain observations to remain unclassified and assesses selective type I and type II errors among definitive classifications within each class. For an independently trained score, we develop a model-free calibration procedure that combines all candidate threshold pairs passing calibration into a rule whose decision region contains that of every passing candidate. We couple the entire threshold search to a common martingale within each class and compute its finite-horizon crossing probability exactly, avoiding a multiplicity adjustment for the number of candidate pairs. The resulting classifier satisfies both population selective error limits simultaneously with user-specified high probability in finite samples, without assumptions on the accuracy of the fitted score. We bound the additional indecision due to calibration and evaluate the procedure in simulations and applications to recidivism and credit-default prediction.
Related
- Simultaneous Latent Budget Trees for Stratified Classification
- Estimation of the Label-Noise Transition Matrix with Performance Guarantees via Selective Classification
- Epanechnikov Pearson Type V distribution for reliability and predictive maintenance modeling
- Analytical and Numerical Analysis of Time-Delayed NDF Control for Nonlinear Rotating Pendulums with Stiffness Modulation Under Simultaneous Resonance Conditions
- Selective Posterior Margin Regularization for Forward-Corrected Classification
BibTeX
@misc{rava2026neymanpearson,
title = {{Neyman-Pearson Classification with Indecisions: Simultaneous Control of Selective Type I and Type II Errors}},
author = {Bradley Rava and Bowen Gang},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.04233},
eprint = {2610.04233},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.04233},
}
RIS
TY - UNPB
TI - Neyman-Pearson Classification with Indecisions: Simultaneous Control of Selective Type I and Type II Errors
AU - Bradley Rava
AU - Bowen Gang
PY - 2026
DA - 2026/10/03
PB - arXiv
DO - 10.48550/arxiv.2610.04233
UR - https://arxiv.org/abs/2610.04233
AB - Binary classification often requires controlling both type I and type II error rates at prespecified levels. When the class distributions overlap substantially, the two error limits may be incompatible if every observation must be classified. We study Neyman-Pearson classification with indecisions (NPI), which permits uncertain observations to remain unclassified and assesses selective type I and type II errors among definitive classifications within each class. For an independently trained score, we develop a model-free calibration procedure that combines all candidate threshold pairs passing calibration into a rule whose decision region contains that of every passing candidate. We couple the entire threshold search to a common martingale within each class and compute its finite-horizon crossing probability exactly, avoiding a multiplicity adjustment for the number of candidate pairs. The resulting classifier satisfies both population selective error limits simultaneously with user-specified high probability in finite samples, without assumptions on the accuracy of the fitted score. We bound the additional indecision due to calibration and evaluate the procedure in simulations and applications to recidivism and credit-default prediction.
ER -