Fisher-Glass Learning: Classification Bounds and Subpopulation Audits
Abstract
Accurate classifiers can fail when background cues change without altering the correct labels. Predictive performance alone does not quantify how much task information remains after accounting for such nuisance variation. Here we extend the Fisher-glass framework to connect nuisance-adjusted information with local classification difficulty and subpopulation auditing. We derive lower bounds on local minimax excess classification risk under explicit distinguishability and decision-separation conditions, alongside statistical audits accounting for estimation error and regularization. On Waterbirds, a feature-based diagnostic of task-background alignment selects an accurately classified group, with selection frozen before held-out evaluation. Across five training seeds, background replacements preserving the extracted bird foreground and labels yield 93.2% accuracy when background type is preserved and 40.6% when it changes, a controlled difference of 52.6 percentage points. Separately, search using task labels and a fixed nuisance direction identifies an independently evaluated population with 98.8% membership-weighted accuracy and low estimated projected predictive Fisher information. Training and acquisition experiments show that information gains and predictive gains can diverge. These findings support joint assessment of predictive performance, information under declared nuisance models and controlled sensitivity when diagnosing vulnerability and evaluating model improvements.
Related
- CleanScore: Black-Box Benchmark Audits with Negative Controls and Sensitivity Bounds
- Hermite-Fisher bounds and stability for min-entropy power inequalities
- Asymptotic Behavior and Error Bounds for Fisher-KPP Equations on the Real Half-Line
- Bayesian Optimization with Fisher Information Geometry: Gradient Bounds and Trust-Region Methods
- Asymptotics and finite sample bounds for prediction and smoothing in Wright-Fisher hidden Markov models
BibTeX
@misc{mansouri2026fisherglass,
title = {{Fisher-Glass Learning: Classification Bounds and Subpopulation Audits}},
author = {El Mustapha Mansouri},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.12702},
eprint = {2611.12702},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.12702},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Fisher-Glass Learning: Classification Bounds and Subpopulation Audits
AU - El Mustapha Mansouri
PY - 2026
DA - 2026/11/09
PB - arXiv
DO - 10.48550/arxiv.2611.12702
UR - https://arxiv.org/abs/2611.12702
AB - Accurate classifiers can fail when background cues change without altering the correct labels. Predictive performance alone does not quantify how much task information remains after accounting for such nuisance variation. Here we extend the Fisher-glass framework to connect nuisance-adjusted information with local classification difficulty and subpopulation auditing. We derive lower bounds on local minimax excess classification risk under explicit distinguishability and decision-separation conditions, alongside statistical audits accounting for estimation error and regularization. On Waterbirds, a feature-based diagnostic of task-background alignment selects an accurately classified group, with selection frozen before held-out evaluation. Across five training seeds, background replacements preserving the extracted bird foreground and labels yield 93.2% accuracy when background type is preserved and 40.6% when it changes, a controlled difference of 52.6 percentage points. Separately, search using task labels and a fixed nuisance direction identifies an independently evaluated population with 98.8% membership-weighted accuracy and low estimated projected predictive Fisher information. Training and acquisition experiments show that information gains and predictive gains can diverge. These findings support joint assessment of predictive performance, information under declared nuisance models and controlled sensitivity when diagnosing vulnerability and evaluating model improvements.
ER -