Benchmarking ten frontier large language models on 1,477 board style multiple choice questions in hematology
- 1Department of Internal Medicine I University Hospital Carl Gustav Carus TUD Dresden University of Technology, Dresden, Germany
Abstract
Large Language Models (LLMs) are increasingly used by clinicians and patients for medical queries, yet their accuracy and safety at the specialist level in hematology remain insufficiently characterised. We benchmarked ten frontier proprietary and open-weight LLMs across two generations on 1,477 board-style hematology multiple-choice questions (MCQs) derived from five educational datasets spanning nine disease areas and six clinical skill domains, including text-only and multimodal case vignettes. Claude Opus 5 had the highest mean accuracy (92.7% text, 76.9% multimodal), followed closely by Gemini-3.1 Pro (91.4% and 78.7%), Gemini-3.6 Flash (91.0% and 74.8%) and GPT-5.6 Sol (89.9% and 76.7%). Accuracy significantly correlated with model size both for text-only and multimodal MCQs. Between model generations, the largest improvements in accuracy were seen for open-weight models whereas proprietary models showed only marginal gains. In error analysis, top-performing models exhibited highly concordant failure patterns, suggesting shared limitations on challenging cases. Frontier LLMs exhibit substantial specialist hematology knowledge across diverse subspecialist domains and clinical skill sets. Yet, despite high accuracy on board-style questions in hematology, continuous expert-on-the-loop output monitoring is paramount.
Related
- Benchmarking the performance of large language models on medical genetics questions: a comparative evaluation of GPT-5.1, Gemini Pro 3, and Claude Opus 4.5.
- Construct Validity of Frontier Large Language Models on 4 Spanish Health Professions Specialty-Access Examinations (MIR, EIR, PIR, and FIR): Cross-Sectional Benchmarking and Psychometric Study.
- Leveraging large language models to generate multiple-choice questions for epilepsy education
- How well do large language models answer postoperative lumbar fusion questions? A blinded comparative analysis of accuracy, quality, readability, and safety-related content
- Safety and guideline concordance of public large language model responses to patient-facing questions on active charcot neuro-osteoarthropathy
BibTeX
@misc{radoynova2026benchmarking,
title = {{Benchmarking ten frontier large language models on 1,477 board style multiple choice questions in hematology}},
author = {M. Radoynova and M. Benouis and f. schulze and S. Winter and M. Bornhauser and J. M. Middeke and J.-N. Eckardt},
howpublished = {medRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.01.26361881},
url = {https://www.medrxiv.org/content/10.64898/2026.09.01.26361881v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Benchmarking ten frontier large language models on 1,477 board style multiple choice questions in hematology
AU - M. Radoynova
AU - M. Benouis
AU - f. schulze
AU - S. Winter
AU - M. Bornhauser
AU - J. M. Middeke
AU - J.-N. Eckardt
PY - 2026
DA - 2026/09/01
PB - medRxiv
DO - 10.64898/2026.09.01.26361881
UR - https://www.medrxiv.org/content/10.64898/2026.09.01.26361881v1
AB - Large Language Models (LLMs) are increasingly used by clinicians and patients for medical queries, yet their accuracy and safety at the specialist level in hematology remain insufficiently characterised. We benchmarked ten frontier proprietary and open-weight LLMs across two generations on 1,477 board-style hematology multiple-choice questions (MCQs) derived from five educational datasets spanning nine disease areas and six clinical skill domains, including text-only and multimodal case vignettes. Claude Opus 5 had the highest mean accuracy (92.7% text, 76.9% multimodal), followed closely by Gemini-3.1 Pro (91.4% and 78.7%), Gemini-3.6 Flash (91.0% and 74.8%) and GPT-5.6 Sol (89.9% and 76.7%). Accuracy significantly correlated with model size both for text-only and multimodal MCQs. Between model generations, the largest improvements in accuracy were seen for open-weight models whereas proprietary models showed only marginal gains. In error analysis, top-performing models exhibited highly concordant failure patterns, suggesting shared limitations on challenging cases. Frontier LLMs exhibit substantial specialist hematology knowledge across diverse subspecialist domains and clinical skill sets. Yet, despite high accuracy on board-style questions in hematology, continuous expert-on-the-loop output monitoring is paramount.
ER -