Performance evaluation and benchmarking across 16 large language model configurations on a large real-world emergency department triage data set
- 1IKIM/DHZC/Charite
Abstract
BackgroundEmergency department (ED) triage is a decision process that determines patient prioritization and resource allocation. Large language models (LLMs) have been proposed as decision-support tools for triage, yet most evaluations rely on curated datasets. Evidence from real-world clinical environments remains limited. The objective of this project was to evaluate the performance and reproducibility of multiple contemporary large language models for Emergency Severity Index (ESI) classification and sectoral allocation (ED vs. urgent care practice, UCP) using a real-world triage dataset.
Material and MethodsRetrospective cross-sectional benchmarking study conducted at a tertiary academic ED in Germany with an integrated central point of assessment (CPA). The study included all 16,107 walk-in encounters between October 18, 2023, and February 27, 2024. Sixteen configurations were compared on the first 2,000 encounters; GPT 5.1 with high reasoning effort was additionally evaluated on all 16,107 encounters. Data were collected from a structured clinical decision support system capturing presenting complaints, vital signs, and triage decisions recorded by specialized nursing staff.
ResultsThe primary outcome was the agreement between LLM-predicted and nurse-assigned ESI levels measured using quadratic-weighted Cohens {kappa}. Secondary outcomes included sectoral assignment agreement, misclassification patterns (over- and under-triage), confidence-agreement comparisons, and output reproducibility. In the primary analysis of all 16,107 encounters, GPT 5.1 with high reasoning effort achieved quadratic-weighted {kappa} = 0.5031 (95% CI 0.4904-0.5158). In the exploratory comparison of 16 configurations on the shared 2,000-encounter subset, quadratic-weighted {kappa} ranged from 0.180 to 0.860. The Stepwise approach, combining structured model assessments with deterministic ESI mapping, achieved {kappa}_qw = 0.860 (95% CI 0.841-0.878) and 82.70% exact agreement. Most models demonstrated moderate or lower agreement. Sectoral assignment agreement (i.e. ED vs. urgent care practice, UCP) was uniformly low ({kappa} [≤] 0.304). Reproducibility testing revealed substantial variability in 23% of cases, indicating non-deterministic output behavior for clinically relevant decisions.
ConclusionsCurrent large language models demonstrate heterogeneous and generally limited performance in real-world emergency triage tasks. The Stepwise approach achieved the highest agreement. Before clinical implementation, confidence estimates, reliability, and workflow integration require prospective validation in regulatory-compliant clinical settings.
Related
- Comparative performance of pre- and post-endoscopic risk scores for outcome prediction in endoscopy-confirmed upper gastrointestinal bleeding in the emergency department : A retrospective single-center study
- On-Device ECG and Vital-Sign Prediction of 24-Hour Mortality and Unplanned ICU Admission in Emergency Department Patients: A Retrospective Evaluation of MEDI-AI
- A Mixed Methods Evaluation of a Model of Care for Domestic Violence Within the Emergency Department
- Performance of the Heatwave ED Forecast Tool in the Emergency Department Network of Ticino Canton, Switzerland
- Implementation and Performance of a Clinical Debriefing Programme in a Paediatric Emergency Department: A Descriptive Qualitative Review
BibTeX
@misc{benning2026performance,
title = {{Performance evaluation and benchmarking across 16 large language model configurations on a large real-world emergency department triage data set}},
author = {L. Benning and A. Hirsch and M. Groeschel and T. Roeschl and M. Spott and F. P. Hans and T. Urban and H.-J. Busch and A. Meyer and J. Madrid},
howpublished = {medRxiv},
year = {2026},
month = may,
doi = {10.64898/2026.05.28.26353935},
url = {https://www.medrxiv.org/content/10.64898/2026.05.28.26353935v2},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Performance evaluation and benchmarking across 16 large language model configurations on a large real-world emergency department triage data set
AU - L. Benning
AU - A. Hirsch
AU - M. Groeschel
AU - T. Roeschl
AU - M. Spott
AU - F. P. Hans
AU - T. Urban
AU - H.-J. Busch
AU - A. Meyer
AU - J. Madrid
PY - 2026
DA - 2026/05/28
PB - medRxiv
DO - 10.64898/2026.05.28.26353935
UR - https://www.medrxiv.org/content/10.64898/2026.05.28.26353935v2
AB - BackgroundEmergency department (ED) triage is a decision process that determines patient prioritization and resource allocation. Large language models (LLMs) have been proposed as decision-support tools for triage, yet most evaluations rely on curated datasets. Evidence from real-world clinical environments remains limited. The objective of this project was to evaluate the performance and reproducibility of multiple contemporary large language models for Emergency Severity Index (ESI) classification and sectoral allocation (ED vs. urgent care practice, UCP) using a real-world triage dataset. Material and MethodsRetrospective cross-sectional benchmarking study conducted at a tertiary academic ED in Germany with an integrated central point of assessment (CPA). The study included all 16,107 walk-in encounters between October 18, 2023, and February 27, 2024. Sixteen configurations were compared on the first 2,000 encounters; GPT 5.1 with high reasoning effort was additionally evaluated on all 16,107 encounters. Data were collected from a structured clinical decision support system capturing presenting complaints, vital signs, and triage decisions recorded by specialized nursing staff. ResultsThe primary outcome was the agreement between LLM-predicted and nurse-assigned ESI levels measured using quadratic-weighted Cohens {kappa}. Secondary outcomes included sectoral assignment agreement, misclassification patterns (over- and under-triage), confidence-agreement comparisons, and output reproducibility. In the primary analysis of all 16,107 encounters, GPT 5.1 with high reasoning effort achieved quadratic-weighted {kappa} = 0.5031 (95% CI 0.4904-0.5158). In the exploratory comparison of 16 configurations on the shared 2,000-encounter subset, quadratic-weighted {kappa} ranged from 0.180 to 0.860. The Stepwise approach, combining structured model assessments with deterministic ESI mapping, achieved {kappa}_qw = 0.860 (95% CI 0.841-0.878) and 82.70% exact agreement. Most models demonstrated moderate or lower agreement. Sectoral assignment agreement (i.e. ED vs. urgent care practice, UCP) was uniformly low ({kappa} [≤] 0.304). Reproducibility testing revealed substantial variability in 23% of cases, indicating non-deterministic output behavior for clinically relevant decisions. ConclusionsCurrent large language models demonstrate heterogeneous and generally limited performance in real-world emergency triage tasks. The Stepwise approach achieved the highest agreement. Before clinical implementation, confidence estimates, reliability, and workflow integration require prospective validation in regulatory-compliant clinical settings.
ER -