Knowing When to Defer: Trustworthy Multimodal AI for BI-RADS-Derived Management Using Paired Mammography and Ultrasound
- 1Graduate School of Information Science, University of Hyogo, Kobe 650-0047, Japan
Abstract
Background: Breast imaging ends in a management decision, from routine return to biopsy, yet most models are evaluated by accuracy alone and say nothing about when they may be wrong. We present a trust-aware multimodal model that reads paired mammography and ultrasound, recommends one of three BI-RADS-derived management actions, and returns to the radiologist the cases it cannot call. Methods: Two foundation encoders are adapted with low-rank adapters and combined by a mask-aware fusion head. A trustworthiness layer adds calibration, conformal prediction sets, selective deferral, and an atypicality flag. We evaluated our method on the Breast Cancer Multimodal Imaging Dataset (BCMID), comprising 332 cases from 323 patients at a single centre. The reference standard is a three-class management grouping we derive from the reporting radiologist’s BI-RADS assessment, so performance is in concordance with that derived label rather than with pathology, observed patient management or longitudinal clinical outcome. Results: Macro AUROC was 0.759 and balanced accuracy was 0.556. Isotonic calibration reduced calibration error from 0.088 to 0.052, and prediction sets reached an empirical coverage of 0.934 at a mean set size of 2.32. Deferring the least confident 30% by a retrospective ranking of the pooled cohort raised balanced accuracy to 0.631. Two of 63 positive-management cases were under-triaged and 15 routed to additional imaging, with a recall of 0.730; freezing the encoders left macro AUROC at 0.756 but raised the under-triage count to twelve. Conclusions: Error rate and error direction are separable properties, and neither accuracy nor macro AUROC records the direction. The system therefore pairs each recommendation with calibrated probabilities, a conformal set of plausible actions and an explicit defer option, so uncertain cases return to the radiologist. These results establish an internally validated operating profile for radiologist-facing support; clinical safety, deployment readiness and benefit to patients still require external and prospective evaluation.
Related
- Ultrasound-based multimodal diagnosis and endovascular therapy for phlegmasia cerulea dolens secondary to May–Thurner syndrome in a middle-aged male patient : A case report
- Multimodal Multiple Instance Learning for Fibroepithelial Tumor Diagnosis in Breast Ultrasound
- A comparative study of diaphragm ultrasound using multimodal parameters in healthy adults
- Successful Management of a Completely Disconnected Bile Leak After Hepatectomy Using Endoscopic Ultrasound‐Guided Transluminal Drainage of a Biloma: A Case Report
- Multimodal Management of Isolated Pulmonary Metastasis From Choroidal Melanoma: A Case Report
BibTeX
@article{nouman2026knowing,
title = {{Knowing When to Defer: Trustworthy Multimodal AI for BI-RADS-Derived Management Using Paired Mammography and Ultrasound}},
author = {Muhammad Nouman and Ryo Haraguchi},
journal = {BioMedInformatics},
year = {2026},
month = sep,
doi = {10.3390/biomedinformatics6050074},
url = {https://doi.org/10.3390/biomedinformatics6050074},
note = {CC BY},
}
RIS
TY - JOUR
TI - Knowing When to Defer: Trustworthy Multimodal AI for BI-RADS-Derived Management Using Paired Mammography and Ultrasound
AU - Muhammad Nouman
AU - Ryo Haraguchi
PY - 2026
DA - 2026/09/01
JO - BioMedInformatics
DO - 10.3390/biomedinformatics6050074
UR - https://doi.org/10.3390/biomedinformatics6050074
AB - Background: Breast imaging ends in a management decision, from routine return to biopsy, yet most models are evaluated by accuracy alone and say nothing about when they may be wrong. We present a trust-aware multimodal model that reads paired mammography and ultrasound, recommends one of three BI-RADS-derived management actions, and returns to the radiologist the cases it cannot call. Methods: Two foundation encoders are adapted with low-rank adapters and combined by a mask-aware fusion head. A trustworthiness layer adds calibration, conformal prediction sets, selective deferral, and an atypicality flag. We evaluated our method on the Breast Cancer Multimodal Imaging Dataset (BCMID), comprising 332 cases from 323 patients at a single centre. The reference standard is a three-class management grouping we derive from the reporting radiologist’s BI-RADS assessment, so performance is in concordance with that derived label rather than with pathology, observed patient management or longitudinal clinical outcome. Results: Macro AUROC was 0.759 and balanced accuracy was 0.556. Isotonic calibration reduced calibration error from 0.088 to 0.052, and prediction sets reached an empirical coverage of 0.934 at a mean set size of 2.32. Deferring the least confident 30% by a retrospective ranking of the pooled cohort raised balanced accuracy to 0.631. Two of 63 positive-management cases were under-triaged and 15 routed to additional imaging, with a recall of 0.730; freezing the encoders left macro AUROC at 0.756 but raised the under-triage count to twelve. Conclusions: Error rate and error direction are separable properties, and neither accuracy nor macro AUROC records the direction. The system therefore pairs each recommendation with calibrated probabilities, a conformal set of plausible actions and an explicit defer option, so uncertain cases return to the radiologist. These results establish an internally validated operating profile for radiologist-facing support; clinical safety, deployment readiness and benefit to patients still require external and prospective evaluation.
SN - 2673-7426
ER -