Diagnosis, triage and response quality of five large language models on emergency medicine vignettes: a blinded comparison
- 1University of British Columbia & Simon Fraser University
Abstract
BackgroundEmergency care requires appropriate triage and management as well as diagnosis. We compared five large language models on these tasks using the same clinical vignettes.
MethodsSeventeen synthetic emergency department vignettes were submitted to Claude Fable 5, GPT-5.6 Sol, Grok 4.5, Gemini 3.6 Flash and Kimi K3 through consumer interfaces on 11-12 August 2026. Two emergency physicians independently assessed the 85 responses with model identities concealed. Outcomes were agreement with reference diagnoses and Canadian Triage and Acuity Scale (CTAS) levels, appropriateness and overall quality (1-5 scales), potential harm and preferred response.
ResultsEvery model correctly named the reference diagnosis in all 17 cases. CTAS exact agreement ranged from 14/17 to 15/17 per model (p=0.938). Of 13 discordant assignments, nine were undertriage and four overtriage. Appropriateness and quality differed across models (both p<0.001): Kimi K3 had the highest mean scores (4.82 and 4.74) and Grok 4.5 the lowest (3.91 and 3.71); Grok 4.5 scored significantly lower than every other model on both scales after Holm adjustment. Nine responses were flagged for potential harm: four from Grok 4.5, two each from Gemini 3.6 Flash and GPT-5.6 Sol, one from Claude Fable 5 and none from Kimi K3 (p=0.119).
ConclusionIn this selected set of synthetic vignettes, all five models correctly named the reference diagnosis, yet undertriage and potentially harmful omissions occurred and response quality differed between models. These findings do not establish any models clinical safety.
Key messagesO_ST_ABSWhat is already known on this topicC_ST_ABSLarge language models have been evaluated on emergency diagnosis, triage and management, including multi-model comparisons using the Canadian Triage and Acuity Scale.
What this study addsIn 17 synthetic vignettes, five models all named the reference diagnosis, yet physician ratings differed and triage errors and potential-harm flags remained.
How this study might affect research, practice or policyLLMs, once clinically validated, have great promise to improve emergency care. Evaluations of these tools should assess management and urgency, not diagnosis alone. These findings do not establish that any model is safe for clinical use.
Related
- Cost-Effectiveness of a Host-Response Point-of-Care Assay for Early Sepsis Diagnosis in UK Emergency Departments
- Evaluating large language models as clinical decision support tools in primary healthcare settings: Protocol for a multi-country comparative validation study on expert-adjudicated hypothetical vignettes (hypMOOVE-PHC)
- Point-of-Care Ultrasound Diagnosis of Retrobulbar Hematoma: A Sight-Threatening Emergency
- Automated Calculation of the Pediatric Emergency Care Applied Research Network (PECARN) Intra-Abdominal Injury Rule from Unstructured Emergency Department Notes Using a Large Language Model
- Evaluating a Large Language Model for Emergency Triage Using Synthetic Clinical Dialogues: Toward a Multimodal Clinical Safety Framework
BibTeX
@misc{johal2026diagnosis,
title = {{Diagnosis, triage and response quality of five large language models on emergency medicine vignettes: a blinded comparison}},
author = {G. Johal and L. Liang},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.21.26366076},
url = {https://www.medrxiv.org/content/10.64898/2026.10.21.26366076v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Diagnosis, triage and response quality of five large language models on emergency medicine vignettes: a blinded comparison
AU - G. Johal
AU - L. Liang
PY - 2026
DA - 2026/10/21
PB - medRxiv
DO - 10.64898/2026.10.21.26366076
UR - https://www.medrxiv.org/content/10.64898/2026.10.21.26366076v1
AB - BackgroundEmergency care requires appropriate triage and management as well as diagnosis. We compared five large language models on these tasks using the same clinical vignettes. MethodsSeventeen synthetic emergency department vignettes were submitted to Claude Fable 5, GPT-5.6 Sol, Grok 4.5, Gemini 3.6 Flash and Kimi K3 through consumer interfaces on 11-12 August 2026. Two emergency physicians independently assessed the 85 responses with model identities concealed. Outcomes were agreement with reference diagnoses and Canadian Triage and Acuity Scale (CTAS) levels, appropriateness and overall quality (1-5 scales), potential harm and preferred response. ResultsEvery model correctly named the reference diagnosis in all 17 cases. CTAS exact agreement ranged from 14/17 to 15/17 per model (p=0.938). Of 13 discordant assignments, nine were undertriage and four overtriage. Appropriateness and quality differed across models (both p<0.001): Kimi K3 had the highest mean scores (4.82 and 4.74) and Grok 4.5 the lowest (3.91 and 3.71); Grok 4.5 scored significantly lower than every other model on both scales after Holm adjustment. Nine responses were flagged for potential harm: four from Grok 4.5, two each from Gemini 3.6 Flash and GPT-5.6 Sol, one from Claude Fable 5 and none from Kimi K3 (p=0.119). ConclusionIn this selected set of synthetic vignettes, all five models correctly named the reference diagnosis, yet undertriage and potentially harmful omissions occurred and response quality differed between models. These findings do not establish any models clinical safety. Key messagesO_ST_ABSWhat is already known on this topicC_ST_ABSLarge language models have been evaluated on emergency diagnosis, triage and management, including multi-model comparisons using the Canadian Triage and Acuity Scale. What this study addsIn 17 synthetic vignettes, five models all named the reference diagnosis, yet physician ratings differed and triage errors and potential-harm flags remained. How this study might affect research, practice or policyLLMs, once clinically validated, have great promise to improve emergency care. Evaluations of these tools should assess management and urgency, not diagnosis alone. These findings do not establish that any model is safe for clinical use.
ER -