Errors, Hallucinations, and Clinical Impact of General-Purpose Multimodal Large Language Models in Histopathology
- 1Department of Pathology Informatics, Nagasaki University Graduate School of Biomedical Sciences, Nagasaki, Japan; Department of Pathology, Kameda Medical Center
Abstract
BackgroundGeneral-purpose large language models (LLMs) are increasingly evaluated in diagnostic pathology, but prior studies have largely emphasized diagnostic accuracy rather than how models fail. We evaluated four LLMs for diagnostic performance, pathology-relevant errors and hallucinations, their burden, and potential clinical impact across multi-organ pathology cases.
DesignIn this retrospective multicenter study, 153 pathology cases from two institutions spanning 20 organs were evaluated using ChatGPT-5.3 (LLM1), Gemini 3 (LLM2), Grok 4.20 (LLM3), and Claude Opus 4.6 (LLM4). Each LLM received multi-magnification histologic images with clinical context and generated a microscopic description and diagnosis. No data splitting, training, or fine-tuning was performed. Twenty-one pathologists assessed 612 outputs for diagnostic correctness, error and hallucination type and burden, clinical impact (0-4), and overall performance (1-5).
ResultsStrict diagnostic accuracy was 48.9% overall (60.1% including partially correct diagnoses) and ranged from 36.6% to 58.2% across LLMs. Errors occurred in 82.0% of outputs and hallucinations in 76.6%; 90.2% contained at least one error or hallucination. Misinterpretation was the most frequent error (72.1%), while fabricated histologic features were the dominant hallucination type (74.5%). LLM2 had significantly lower misinterpretation rates than the other three LLMs, while LLM3 had significantly higher fabricated-feature hallucination rates than LLM1 and LLM2. Errors were more frequent in incorrect than correct diagnoses (97.5% vs 66.2%), as were hallucinations (95.1% vs 59.9%; both p < 0.001). Every strictly incorrect diagnosis contained an error and/or hallucination, while 79.9% of strictly correct diagnoses also contained at least one. In multivariable analysis, LLM3 was independently associated with higher odds of an incorrect diagnosis. Diagnostic correctness was the strongest determinant of high clinical impact; each one-point increase in hallucination burden increased the odds by 63% (adjusted OR, 1.63).
ConclusionDiagnostic accuracy alone substantially underestimates the safety limitations of general-purpose LLMs in histopathology. Errors and hallucinations were common even when the final diagnosis was correct, and their burden was independently associated with clinical impact. Evaluation frameworks should therefore assess both diagnostic correctness and the reliability and potential consequences of accompanying generated content.
Related
- Evaluating a Large Language Model for Emergency Triage Using Synthetic Clinical Dialogues: Toward a Multimodal Clinical Safety Framework
- Extracting smoking history from clinical notes for lung cancer screening decision support: comparing a structured-judgment model with general-purpose large language models
- When the Algorithm Becomes the Dietitian: Large Language Model Hallucinations, Algorithmic Food Environments, and the Erosion of Nutritional Autonomy
- Reply to: "Evaluating the Clinical Readiness of General-Purpose Large Language Models in Evidence Synthesis".
- General-purpose large language models outperform specialized clinical AI tools on medical benchmarks
BibTeX
@misc{lami2026errors,
title = {{Errors, Hallucinations, and Clinical Impact of General-Purpose Multimodal Large Language Models in Histopathology}},
author = {K. Lami and S. Agarwal and A. Asaturova and S. Balci and A. Harahap and H. Kang and J. Kim and M. Komuta and T. Laohawetwanit and S. Menon and J. Munkhdelger and H. H. N. Pham and D. G. Pinto and A. Sahay and S. Satturwar and K. Seki and M. Sughayer and Y. Tachibana and I. Turkmen and Q. Wu and E. Ullah and A. Bychkov and J. Fukuoka},
howpublished = {medRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.18.26363369},
url = {https://www.medrxiv.org/content/10.64898/2026.09.18.26363369v1},
note = {CC BY-ND},
}
RIS
TY - UNPB
TI - Errors, Hallucinations, and Clinical Impact of General-Purpose Multimodal Large Language Models in Histopathology
AU - K. Lami
AU - S. Agarwal
AU - A. Asaturova
AU - S. Balci
AU - A. Harahap
AU - H. Kang
AU - J. Kim
AU - M. Komuta
AU - T. Laohawetwanit
AU - S. Menon
AU - J. Munkhdelger
AU - H. H. N. Pham
AU - D. G. Pinto
AU - A. Sahay
AU - S. Satturwar
AU - K. Seki
AU - M. Sughayer
AU - Y. Tachibana
AU - I. Turkmen
AU - Q. Wu
AU - E. Ullah
AU - A. Bychkov
AU - J. Fukuoka
PY - 2026
DA - 2026/09/18
PB - medRxiv
DO - 10.64898/2026.09.18.26363369
UR - https://www.medrxiv.org/content/10.64898/2026.09.18.26363369v1
AB - BackgroundGeneral-purpose large language models (LLMs) are increasingly evaluated in diagnostic pathology, but prior studies have largely emphasized diagnostic accuracy rather than how models fail. We evaluated four LLMs for diagnostic performance, pathology-relevant errors and hallucinations, their burden, and potential clinical impact across multi-organ pathology cases. DesignIn this retrospective multicenter study, 153 pathology cases from two institutions spanning 20 organs were evaluated using ChatGPT-5.3 (LLM1), Gemini 3 (LLM2), Grok 4.20 (LLM3), and Claude Opus 4.6 (LLM4). Each LLM received multi-magnification histologic images with clinical context and generated a microscopic description and diagnosis. No data splitting, training, or fine-tuning was performed. Twenty-one pathologists assessed 612 outputs for diagnostic correctness, error and hallucination type and burden, clinical impact (0-4), and overall performance (1-5). ResultsStrict diagnostic accuracy was 48.9% overall (60.1% including partially correct diagnoses) and ranged from 36.6% to 58.2% across LLMs. Errors occurred in 82.0% of outputs and hallucinations in 76.6%; 90.2% contained at least one error or hallucination. Misinterpretation was the most frequent error (72.1%), while fabricated histologic features were the dominant hallucination type (74.5%). LLM2 had significantly lower misinterpretation rates than the other three LLMs, while LLM3 had significantly higher fabricated-feature hallucination rates than LLM1 and LLM2. Errors were more frequent in incorrect than correct diagnoses (97.5% vs 66.2%), as were hallucinations (95.1% vs 59.9%; both p < 0.001). Every strictly incorrect diagnosis contained an error and/or hallucination, while 79.9% of strictly correct diagnoses also contained at least one. In multivariable analysis, LLM3 was independently associated with higher odds of an incorrect diagnosis. Diagnostic correctness was the strongest determinant of high clinical impact; each one-point increase in hallucination burden increased the odds by 63% (adjusted OR, 1.63). ConclusionDiagnostic accuracy alone substantially underestimates the safety limitations of general-purpose LLMs in histopathology. Errors and hallucinations were common even when the final diagnosis was correct, and their burden was independently associated with clinical impact. Evaluation frameworks should therefore assess both diagnostic correctness and the reliability and potential consequences of accompanying generated content.
ER -