Knowing When Not to Predict: Selective Classification under Feature Missingness in Tabular Machine Learning
- 1Instituto de Tecnologia e Liderança
Abstract
Tabular classifiers are often trained under complete feature availability but deployed in pipelines where fields can disappear because of sensor failures, unavailable services, incomplete records, or upstream schema changes. This study asks whether model confidence remains useful for identifying errors after such feature loss. A compact, fully reproducible benchmark evaluates Logistic Regression, Random Forest, and XGBoost on three public scikit-learn classification datasets using five repeated stratified 60/20/20 train/calibration/test splits. Models are trained only on complete data and evaluated under entry-wise missing completely at random (MCAR) corruption and structured feature outage at 20% and 40% requested missingness, with five corruption realizations per nonzero condition. Across equal-weight dataset-model-seed cells, mean balanced accuracy decreased from 0.964 on clean inputs to 0.811 under 40% MCAR and 0.805 under 40% structured outage. Mean AURC increased from 0.007 to 0.083 and 0.105, respectively, indicating worse confidence ordering. A confidence threshold selected on clean calibration data for approximately 80% coverage achieved 77.9% coverage on clean tests but only 50.9% under 40% MCAR and 47.8% under 40% structured outage. At 40%, all 18 dataset-model-mechanism cells worsened in balanced accuracy, AURC, frozen-threshold coverage, and frozen-threshold selective risk. The results show that deployment-time feature loss can degrade both prediction and self-assessment, and that abstention thresholds validated on complete inputs may not transfer reliably after feature availability changes.
Related
- Informative Label Missingness in Multiclass Classification Information Geometry and Excess Risk
- Uncertainty-Aware Selective Prediction in Brain-Tumour MRI Classification: Selective Risk Reduction and Component-Wise Reliability Findings
- Estimation of the Label-Noise Transition Matrix with Performance Guarantees via Selective Classification
- Neyman-Pearson Classification with Indecisions: Simultaneous Control of Selective Type I and Type II Errors
- An effective and deep learning model to predict Alzheimer’s disease at an early stage from audio data
BibTeX
@misc{rego2026knowing,
title = {{Knowing When Not to Predict: Selective Classification under Feature Missingness in Tabular Machine Learning}},
author = {Lucas Bittencourt Moraes Rego},
howpublished = {Zenodo},
year = {2026},
month = oct,
doi = {10.5281/zenodo.23579514},
url = {https://zenodo.org/records/23579514},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Knowing When Not to Predict: Selective Classification under Feature Missingness in Tabular Machine Learning
AU - Lucas Bittencourt Moraes Rego
PY - 2026
DA - 2026/10/22
PB - Zenodo
DO - 10.5281/zenodo.23579514
UR - https://zenodo.org/records/23579514
AB - Tabular classifiers are often trained under complete feature availability but deployed in pipelines where fields can disappear because of sensor failures, unavailable services, incomplete records, or upstream schema changes. This study asks whether model confidence remains useful for identifying errors after such feature loss. A compact, fully reproducible benchmark evaluates Logistic Regression, Random Forest, and XGBoost on three public scikit-learn classification datasets using five repeated stratified 60/20/20 train/calibration/test splits. Models are trained only on complete data and evaluated under entry-wise missing completely at random (MCAR) corruption and structured feature outage at 20% and 40% requested missingness, with five corruption realizations per nonzero condition. Across equal-weight dataset-model-seed cells, mean balanced accuracy decreased from 0.964 on clean inputs to 0.811 under 40% MCAR and 0.805 under 40% structured outage. Mean AURC increased from 0.007 to 0.083 and 0.105, respectively, indicating worse confidence ordering. A confidence threshold selected on clean calibration data for approximately 80% coverage achieved 77.9% coverage on clean tests but only 50.9% under 40% MCAR and 47.8% under 40% structured outage. At 40%, all 18 dataset-model-mechanism cells worsened in balanced accuracy, AURC, frozen-threshold coverage, and frozen-threshold selective risk. The results show that deployment-time feature loss can degrade both prediction and self-assessment, and that abstention thresholds validated on complete inputs may not transfer reliably after feature availability changes.
ER -