{
  "abstract": "BackgroundEmergency department (ED) triage is a decision process that determines patient prioritization and resource allocation. Large language models (LLMs) have been proposed as decision-support tools for triage, yet most evaluations rely on curated datasets. Evidence from real-world clinical environments remains limited. The objective of this project was to evaluate the performance and reproducibility of multiple contemporary large language models for Emergency Severity Index (ESI) classification and sectoral allocation (ED vs. urgent care practice, UCP) using a real-world triage dataset.\n\nMaterial and MethodsRetrospective cross-sectional benchmarking study conducted at a tertiary academic ED in Germany with an integrated central point of assessment (CPA). The study included all 16,107 walk-in encounters between October 18, 2023, and February 27, 2024. Sixteen configurations were compared on the first 2,000 encounters; GPT 5.1 with high reasoning effort was additionally evaluated on all 16,107 encounters. Data were collected from a structured clinical decision support system capturing presenting complaints, vital signs, and triage decisions recorded by specialized nursing staff.\n\nResultsThe primary outcome was the agreement between LLM-predicted and nurse-assigned ESI levels measured using quadratic-weighted Cohens {kappa}. Secondary outcomes included sectoral assignment agreement, misclassification patterns (over- and under-triage), confidence-agreement comparisons, and output reproducibility. In the primary analysis of all 16,107 encounters, GPT 5.1 with high reasoning effort achieved quadratic-weighted {kappa} = 0.5031 (95% CI 0.4904-0.5158). In the exploratory comparison of 16 configurations on the shared 2,000-encounter subset, quadratic-weighted {kappa} ranged from 0.180 to 0.860. The Stepwise approach, combining structured model assessments with deterministic ESI mapping, achieved {kappa}_qw = 0.860 (95% CI 0.841-0.878) and 82.70% exact agreement. Most models demonstrated moderate or lower agreement. Sectoral assignment agreement (i.e. ED vs. urgent care practice, UCP) was uniformly low ({kappa} [≤] 0.304). Reproducibility testing revealed substantial variability in 23% of cases, indicating non-deterministic output behavior for clinically relevant decisions.\n\nConclusionsCurrent large language models demonstrate heterogeneous and generally limited performance in real-world emergency triage tasks. The Stepwise approach achieved the highest agreement. Before clinical implementation, confidence estimates, reliability, and workflow integration require prospective validation in regulatory-compliant clinical settings.",
  "abstract_shown": true,
  "affiliations": [
    "IKIM/DHZC/Charite"
  ],
  "article_type": "PUBLISHAHEADOFPRINT",
  "arxiv": null,
  "author_total": 10,
  "authors": [
    {
      "name": "L. Benning"
    },
    {
      "name": "A. Hirsch"
    },
    {
      "name": "M. Groeschel"
    },
    {
      "name": "T. Roeschl"
    },
    {
      "name": "M. Spott"
    },
    {
      "name": "F. P. Hans"
    },
    {
      "name": "T. Urban"
    },
    {
      "name": "H.-J. Busch"
    },
    {
      "name": "A. Meyer"
    },
    {
      "affs": [
        0
      ],
      "name": "J. Madrid"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/17yxgdzph0qea.bib",
  "cited_by": 1,
  "doi": "10.64898/2026.05.28.26353935",
  "fulltext_url": "https://www.medrxiv.org/content/10.64898/2026.05.28.26353935v2.full",
  "id": "17yxgdzph0qea",
  "issns": [],
  "landing_url": "https://www.medrxiv.org/content/10.64898/2026.05.28.26353935v2",
  "language": null,
  "licence": "CC-BY-NC-ND",
  "licence_label": "CC BY-NC-ND",
  "licence_source": "article",
  "licence_url": "https://creativecommons.org/licenses/by-nc-nd/4.0/",
  "metadata_licence": null,
  "native_subjects": [
    [
      "medrxiv",
      "health informatics"
    ],
    [
      "openalex",
      "2718|Artificial Intelligence in Healthcare and Education"
    ]
  ],
  "openalex": "W7163648063",
  "pdf_url": "https://www.medrxiv.org/content/10.64898/2026.05.28.26353935v2.full.pdf",
  "pmcid": null,
  "pmid": null,
  "published": "2026-05-28",
  "publisher": null,
  "ris": "https://research.draconitic.com/paper/17yxgdzph0qea.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:21:01Z",
      "native_id": "10.64898/2026.05.28.26353935",
      "source": "medrxiv",
      "url": "https://www.medrxiv.org/content/10.64898/2026.05.28.26353935v2"
    }
  ],
  "subjects": [
    {
      "field": "medicine",
      "origin": "medrxiv:health informatics",
      "subfield": "epidemiology-public-health"
    }
  ],
  "title": "Performance evaluation and benchmarking across 16 large language model configurations on a large real-world emergency department triage data set",
  "title_text": "Performance evaluation and benchmarking across 16 large language model configurations on a large real-world emergency department triage data set",
  "type": "preprint",
  "updated_at": "2026-10-30T12:21:01Z",
  "url": "https://research.draconitic.com/paper/17yxgdzph0qea",
  "venue": "medRxiv",
  "versions": []
}