Not All Mislabels Are Equal: Efficient Task-Aware Auditing with Confidence Sequences
Abstract
We study efficient, adaptive auditing of categorical labels in a fixed dataset, motivated by biodiversity-credit programs in which expert verification of species records is costly and downstream scores (e.g., payment) are more sensitive to errors on certain labels than on others. Our proposed ATLAS framework couples anytime-valid inference with a task-aware adaptive auditing policy. Representing label corrections through a confusion matrix, we prioritize which data points to inspect next based on both label uncertainty and the influence of potential label corrections on the downstream score function. For linear scores such as the mislabel rate, our priority-aware approach certifies the overall label error rate to ± 2 percentage points with 24% and 11% fewer audits than random sampling on biodiversity data from Kenya and a widely used CIFAR-10N benchmark, respectively. For nonlinear scores such as the Hill-Shannon diversity, we exploit a linear-envelope representation that reduces inference to a family of linear-score audits; priority-aware sampling reaches the target precision with 36% fewer audits. When distribution-free guarantees are prohibitively costly, we develop a model-based Bayesian alternative that trades assumption-free validity for substantially fewer audits. Our biodiversity case study highlights a gap between statistical validity and practical feasibility: generic methods such as Serfling's bound and existing confidence-sequence procedures are rigorous but can require prohibitively many expert reviews in practice. Together, these results show task-aware auditing can significantly reduce the number of expert audits needed while providing finite-sample, anytime-valid guarantees.
Related
- Efficient computation of mixture confidence sequences in generalized linear models
- Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Panel Experiments
- Regret Lower Bound for Online Learning of Quantum States with Logarithmic Loss, with Applications to Quantum State Tomography and Confidence Sequences for Matrix Means
- On the Tightness and Computational Tractability of Higher-Dimensional Confidence Sequences
- Rank Confidence Sequences:Anytime-valid Leaderboards
BibTeX
@misc{lin2026mislabels,
title = {{Not All Mislabels Are Equal: Efficient Task-Aware Auditing with Confidence Sequences}},
author = {Fangyuan Lin and Noah Krever and Jakub Černý and Lily Xu},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.33262},
eprint = {2610.33262},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.33262},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Not All Mislabels Are Equal: Efficient Task-Aware Auditing with Confidence Sequences
AU - Fangyuan Lin
AU - Noah Krever
AU - Jakub Černý
AU - Lily Xu
PY - 2026
DA - 2026/10/23
PB - arXiv
DO - 10.48550/arxiv.2610.33262
UR - https://arxiv.org/abs/2610.33262
AB - We study efficient, adaptive auditing of categorical labels in a fixed dataset, motivated by biodiversity-credit programs in which expert verification of species records is costly and downstream scores (e.g., payment) are more sensitive to errors on certain labels than on others. Our proposed ATLAS framework couples anytime-valid inference with a task-aware adaptive auditing policy. Representing label corrections through a confusion matrix, we prioritize which data points to inspect next based on both label uncertainty and the influence of potential label corrections on the downstream score function. For linear scores such as the mislabel rate, our priority-aware approach certifies the overall label error rate to ± 2 percentage points with 24% and 11% fewer audits than random sampling on biodiversity data from Kenya and a widely used CIFAR-10N benchmark, respectively. For nonlinear scores such as the Hill-Shannon diversity, we exploit a linear-envelope representation that reduces inference to a family of linear-score audits; priority-aware sampling reaches the target precision with 36% fewer audits. When distribution-free guarantees are prohibitively costly, we develop a model-based Bayesian alternative that trades assumption-free validity for substantially fewer audits. Our biodiversity case study highlights a gap between statistical validity and practical feasibility: generic methods such as Serfling's bound and existing confidence-sequence procedures are rigorous but can require prohibitively many expert reviews in practice. Together, these results show task-aware auditing can significantly reduce the number of expert audits needed while providing finite-sample, anytime-valid guarantees.
ER -