Conformal Annotation with Randomized Expert Review: Application to Online Cell-Type Labeling
Abstract
Artificial intelligence (AI) models can accelerate cell-type annotation in digital pathology, but safely scaling their use requires determining, with error control, which model-generated labels can be accepted without expert review. This is challenging in evolving workflows because cell morphology and type composition may change across tissue regions and slides, making one-time calibration unreliable. We propose Conformal Annotation with Randomized Expert review under Distribution shift (CARED), a distribution-free framework that leverages randomized expert audit to achieve long-run error control for AI-assisted annotation under distribution shift. For each incoming batch, CARED uniformly selects a subset of cases for expert review and uses the acquired labels to calibrate later decisions. Among unreviewed cases, CARED determines a subset of them to safely skip expert labeling, while suggesting candidate labels for the others for downstream resolution. We prove that this workflow provides long-run control of cumulative error over both the full stream and the cells outside of the audit sample, without stationarity or exchangeability assumptions. We further enhance the framework by introducing class-conditional guarantees to ensure error control within important but rare cell types, together with a utility module to target more automatic resolutions, smaller candidate sets, and greater biological coherence, without compromising validity. Experiments on pancreas, lung, and ovary annotation streams, together with a stream constructed from real expert annotations, show that CARED controls the cumulative errors while a substantial fraction of cells (up to 70%) bypass expert review.
Related
- Optimal Expert Annotation for Evaluating Sequential Decision Policies from Text Records
- GoCo: structure-guided conformal calibration for error-controlled Gene Ontology annotation release
- Efficient conformal prediction intervals for time series: Online PID-Expert aggregation
- "Massive" Conformal Field Theories and application to Nambu-Goto string
- Adaptive Conformal Prediction for Image Regression Models with Application to an Inertial Confinement Fusion Emulator
BibTeX
@misc{zhang2026conformal,
title = {{Conformal Annotation with Randomized Expert Review: Application to Online Cell-Type Labeling}},
author = {Xuhe Zhang and Zhi Huang and Ying Jin},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.40252},
eprint = {2610.40252},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.40252},
}
RIS
TY - UNPB
TI - Conformal Annotation with Randomized Expert Review: Application to Online Cell-Type Labeling
AU - Xuhe Zhang
AU - Zhi Huang
AU - Ying Jin
PY - 2026
DA - 2026/10/29
PB - arXiv
DO - 10.48550/arxiv.2610.40252
UR - https://arxiv.org/abs/2610.40252
AB - Artificial intelligence (AI) models can accelerate cell-type annotation in digital pathology, but safely scaling their use requires determining, with error control, which model-generated labels can be accepted without expert review. This is challenging in evolving workflows because cell morphology and type composition may change across tissue regions and slides, making one-time calibration unreliable. We propose Conformal Annotation with Randomized Expert review under Distribution shift (CARED), a distribution-free framework that leverages randomized expert audit to achieve long-run error control for AI-assisted annotation under distribution shift. For each incoming batch, CARED uniformly selects a subset of cases for expert review and uses the acquired labels to calibrate later decisions. Among unreviewed cases, CARED determines a subset of them to safely skip expert labeling, while suggesting candidate labels for the others for downstream resolution. We prove that this workflow provides long-run control of cumulative error over both the full stream and the cells outside of the audit sample, without stationarity or exchangeability assumptions. We further enhance the framework by introducing class-conditional guarantees to ensure error control within important but rare cell types, together with a utility module to target more automatic resolutions, smaller candidate sets, and greater biological coherence, without compromising validity. Experiments on pancreas, lung, and ovary annotation streams, together with a stream constructed from real expert annotations, show that CARED controls the cumulative errors while a substantial fraction of cells (up to 70%) bypass expert review.
ER -