Clinical referral text overrides MRI evidence in multimodal large language models: glioma versus brain abscess
- 1Institute for Diagnostic and Interventional Radiology, University Hospital Cologne
Abstract
ObjectivesTo determine how strongly clinical referral text, rather than image content, drives the decisions of multimodal large language models (LLMs) differentiating glioma from brain abscess on MRI.
Materials and methodsThis retrospective single-center study included 60 contrast-enhanced T1-weighted brain MRI examinations (30 gliomas, 30 abscesses; reference-standard diagnosis). One axial slice per case was paired with five referral texts (explicitly inflammatory to explicitly tumor-suggestive) under three prompt instructions and submitted 20 times each to GPT-4o, Claude Sonnet 5, Gemini 3.5 Flash, and Qwen 3.6 (72,000 answers). The primary outcome was the text-driven span (difference in tumor-call frequency between explicit inflammatory and explicit tumor text). Two blinded radiologists classified the images without text; sensitivity, specificity, and accuracy were compared descriptively.
ResultsPatients (29 women) had a mean age of 55.8 {+/-} 16.4 years. With neutral text, models answered tumor in 45-67% of queries (specificity, 0.38-0.78), versus 52% and 60% for radiologists (specificity, 0.77 and 0.63). The text-driven span was 0.79-1.00 (p < 0.001 for all models), whereas confirmed gliomas and abscesses differed by only 0.01-0.10 in three models and 0.25 in Gemini 3.5 Flash. Subtle wording produced at least 83% of the span in three models. An instruction to ignore the text reduced the span to 0.48-0.70 in three models and to 0.04 in Gemini 3.5 Flash, partly through reversion to a tumor default. Model accuracy was 0.52-0.73 versus 0.78 and 0.73 for radiologists.
ConclusionMultimodal LLMs deferred to referral text over imaging evidence in differentiating glioma from brain abscess, and prompt instructions did not reliably correct this.
Key PointsO_ST_ABSQuestionC_ST_ABSMultimodal large language models receive images together with clinical referral text, but how strongly the text rather than the image determines their diagnosis is unknown.
FindingsChanging only the referral text largely reversed the glioma-versus-abscess decision for identical MRI images in four models; instructions to ignore the text reduced this inconsistently.
Clinical relevanceBefore clinical use, multimodal large language models require model- and task-specific testing of their susceptibility to referral text, because instructing them to prioritize the image is not a dependable safeguard against text-driven misdiagnosis.
Related
- Evaluating a Large Language Model for Emergency Triage Using Synthetic Clinical Dialogues: Toward a Multimodal Clinical Safety Framework
- Reply to: "Evaluating the Clinical Readiness of General-Purpose Large Language Models in Evidence Synthesis".
- Image-Based Dietary Assessment Using Multimodal Large Language Models: Performance, Limitations, and Clinical Applicability in Diabetes
- Artificial intelligence across the gastric cancer care continuum: Clinical evidence, multimodal integration, and translation
- Errors, Hallucinations, and Clinical Impact of General-Purpose Multimodal Large Language Models in Histopathology
BibTeX
@misc{beste2026clinical,
title = {{Clinical referral text overrides MRI evidence in multimodal large language models: glioma versus brain abscess}},
author = {N. C. Beste and F. Bauer and N. Lopez and T. Schoemig and M. Schoenfeld and D. Zopfs and L. Goertz and T. Persigehl and C. Kabbasch and M. Schlamann and D. Maintz and R. J. Gertz and N. Weitzer and J. Kottlors},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.17.26365809},
url = {https://www.medrxiv.org/content/10.64898/2026.10.17.26365809v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Clinical referral text overrides MRI evidence in multimodal large language models: glioma versus brain abscess
AU - N. C. Beste
AU - F. Bauer
AU - N. Lopez
AU - T. Schoemig
AU - M. Schoenfeld
AU - D. Zopfs
AU - L. Goertz
AU - T. Persigehl
AU - C. Kabbasch
AU - M. Schlamann
AU - D. Maintz
AU - R. J. Gertz
AU - N. Weitzer
AU - J. Kottlors
PY - 2026
DA - 2026/10/17
PB - medRxiv
DO - 10.64898/2026.10.17.26365809
UR - https://www.medrxiv.org/content/10.64898/2026.10.17.26365809v1
AB - ObjectivesTo determine how strongly clinical referral text, rather than image content, drives the decisions of multimodal large language models (LLMs) differentiating glioma from brain abscess on MRI. Materials and methodsThis retrospective single-center study included 60 contrast-enhanced T1-weighted brain MRI examinations (30 gliomas, 30 abscesses; reference-standard diagnosis). One axial slice per case was paired with five referral texts (explicitly inflammatory to explicitly tumor-suggestive) under three prompt instructions and submitted 20 times each to GPT-4o, Claude Sonnet 5, Gemini 3.5 Flash, and Qwen 3.6 (72,000 answers). The primary outcome was the text-driven span (difference in tumor-call frequency between explicit inflammatory and explicit tumor text). Two blinded radiologists classified the images without text; sensitivity, specificity, and accuracy were compared descriptively. ResultsPatients (29 women) had a mean age of 55.8 {+/-} 16.4 years. With neutral text, models answered tumor in 45-67% of queries (specificity, 0.38-0.78), versus 52% and 60% for radiologists (specificity, 0.77 and 0.63). The text-driven span was 0.79-1.00 (p < 0.001 for all models), whereas confirmed gliomas and abscesses differed by only 0.01-0.10 in three models and 0.25 in Gemini 3.5 Flash. Subtle wording produced at least 83% of the span in three models. An instruction to ignore the text reduced the span to 0.48-0.70 in three models and to 0.04 in Gemini 3.5 Flash, partly through reversion to a tumor default. Model accuracy was 0.52-0.73 versus 0.78 and 0.73 for radiologists. ConclusionMultimodal LLMs deferred to referral text over imaging evidence in differentiating glioma from brain abscess, and prompt instructions did not reliably correct this. Key PointsO_ST_ABSQuestionC_ST_ABSMultimodal large language models receive images together with clinical referral text, but how strongly the text rather than the image determines their diagnosis is unknown. FindingsChanging only the referral text largely reversed the glioma-versus-abscess decision for identical MRI images in four models; instructions to ignore the text reduced this inconsistently. Clinical relevanceBefore clinical use, multimodal large language models require model- and task-specific testing of their susceptibility to referral text, because instructing them to prioritize the image is not a dependable safeguard against text-driven misdiagnosis.
ER -