Quantifying the reliability gap in cross-domain plant disease classification: benchmarking the limited efficacy of standard mitigation techniques under controlled-to-field shift
- 1Research Center of Machine Learning and Environment Science, China Three Gorges University
- 2Plant Protection Research Center, Shenzhen Polytechnic University
Abstract
Introduction Confidence calibration, selective prediction, out-of-distribution scoring, and deep ensembles are mature techniques in machine learning, yet their efficacy under the severe domain shift encountered when plant disease classifiers move from controlled laboratory imagery to heterogeneous field photographs has not been systematically benchmarked. Methods Models trained on PlantVillage were evaluated on PlantDoc leaf-level crop images under a parent-image-aware split protocol, and a suite of standard mitigation techniques was applied to characterize the reliability gap. Analyses included temperature scaling and selective prediction for a fine-tuned ResNet-50, quantitative image-level shift analysis, Grad-CAM visualization, simple target-aware adaptation baselines, frozen-feature backbone comparisons, and ensemble baselines. Results In the primary case study, a fine-tuned ResNet-50 suffered a 67.7-percentage-point accuracy collapse upon cross-domain transfer, while mean predicted confidence remained at 79.76%. Post-hoc temperature scaling reduced calibrated ECE to 0.3645 but left selective risk at 80% coverage at 64.30%. Quantitative image-level shift analysis confirmed large-effect-size differences in saturation ( d = 3.90), border edge density ( d = 3.33), and foreground-occupancy proxy ( d = 2.48) between the two domains, while Grad-CAM visualizations showed that the model shifts attention from lesion-centered regions in PlantVillage to background-dominated areas in PlantDoc. Simple target-aware mitigations, including adaptive batch normalization and feature moment matching, improved accuracy from 0.321 to 0.343 and 0.366, respectively, whereas DANN-style adversarial adaptation degraded performance to 0.252. A frozen-feature backbone comparison across five backbones showed that, within the energy-scoring frozen-backbone comparison, DINOv2-S/14 achieved the highest unknown-detection AUROC (0.764) and the lowest selective risk at 80% coverage (0.520), with paired Wilcoxon tests confirming statistically significant accuracy and macro-F1 differences across backbones. Two ensemble baselines were evaluated: a warm-start end-to-end ResNet-50 ensemble reduced calibrated ECE to 0.063 but achieved only 0.666 AUROC, while a lightweight DINOv2 linear-probe ensemble achieved 0.779 AUROC after calibration but under limited epistemic diversity. Discussion Neither ensemble established deployment-grade reliability: the best selective risk at 80% coverage across all configurations remained above 0.51. The principal contribution is a reproducible, deployment-oriented reliability characterization showing that standard post-hoc and lightweight adaptation techniques reduce but do not eliminate the severe reliability gap under controlled-to-field transfer in agricultural computer vision.
Related
- Benchmarking of cell type deconvolution techniques for microarray data
- TasTY: Development and evaluation of a taste test for quantifying taste sensitivity among preschool children-test-retest reliability, application, and comparison between children with and without prior SARS-CoV-2 infection.
- Airway clearance techniques – a proposed classification based on definitions?
- Quantifying performance inflation from hidden sample dependence in biomedical image classification benchmarks
- Machine learning classification of fibromyalgia from blood transcriptomes: confounder benchmarking and implications for endometriosis-associated pain
BibTeX
@article{xiang2026quantifying,
title = {{Quantifying the reliability gap in cross-domain plant disease classification: benchmarking the limited efficacy of standard mitigation techniques under controlled-to-field shift}},
author = {Kun Xiang and Danxi Shi and Xiangbo Zhu},
journal = {Frontiers in Plant Science},
year = {2026},
month = may,
doi = {10.3389/fpls.2026.1826962},
url = {https://doi.org/10.3389/fpls.2026.1826962},
note = {CC BY},
}
RIS
TY - JOUR
TI - Quantifying the reliability gap in cross-domain plant disease classification: benchmarking the limited efficacy of standard mitigation techniques under controlled-to-field shift
AU - Kun Xiang
AU - Danxi Shi
AU - Xiangbo Zhu
PY - 2026
DA - 2026/05/22
JO - Frontiers in Plant Science
DO - 10.3389/fpls.2026.1826962
UR - https://doi.org/10.3389/fpls.2026.1826962
AB - Introduction Confidence calibration, selective prediction, out-of-distribution scoring, and deep ensembles are mature techniques in machine learning, yet their efficacy under the severe domain shift encountered when plant disease classifiers move from controlled laboratory imagery to heterogeneous field photographs has not been systematically benchmarked. Methods Models trained on PlantVillage were evaluated on PlantDoc leaf-level crop images under a parent-image-aware split protocol, and a suite of standard mitigation techniques was applied to characterize the reliability gap. Analyses included temperature scaling and selective prediction for a fine-tuned ResNet-50, quantitative image-level shift analysis, Grad-CAM visualization, simple target-aware adaptation baselines, frozen-feature backbone comparisons, and ensemble baselines. Results In the primary case study, a fine-tuned ResNet-50 suffered a 67.7-percentage-point accuracy collapse upon cross-domain transfer, while mean predicted confidence remained at 79.76%. Post-hoc temperature scaling reduced calibrated ECE to 0.3645 but left selective risk at 80% coverage at 64.30%. Quantitative image-level shift analysis confirmed large-effect-size differences in saturation ( d = 3.90), border edge density ( d = 3.33), and foreground-occupancy proxy ( d = 2.48) between the two domains, while Grad-CAM visualizations showed that the model shifts attention from lesion-centered regions in PlantVillage to background-dominated areas in PlantDoc. Simple target-aware mitigations, including adaptive batch normalization and feature moment matching, improved accuracy from 0.321 to 0.343 and 0.366, respectively, whereas DANN-style adversarial adaptation degraded performance to 0.252. A frozen-feature backbone comparison across five backbones showed that, within the energy-scoring frozen-backbone comparison, DINOv2-S/14 achieved the highest unknown-detection AUROC (0.764) and the lowest selective risk at 80% coverage (0.520), with paired Wilcoxon tests confirming statistically significant accuracy and macro-F1 differences across backbones. Two ensemble baselines were evaluated: a warm-start end-to-end ResNet-50 ensemble reduced calibrated ECE to 0.063 but achieved only 0.666 AUROC, while a lightweight DINOv2 linear-probe ensemble achieved 0.779 AUROC after calibration but under limited epistemic diversity. Discussion Neither ensemble established deployment-grade reliability: the best selective risk at 80% coverage across all configurations remained above 0.51. The principal contribution is a reproducible, deployment-oriented reliability characterization showing that standard post-hoc and lightweight adaptation techniques reduce but do not eliminate the severe reliability gap under controlled-to-field transfer in agricultural computer vision.
SN - 1664-462X
ER -