EM-BENCH: A Reproducible Framework for Benchmarking Guideline-Concordant Clinical Decision Making in Large Language Models
- 1Department of Emergency Medicine, Stanford University School of Medicine
Abstract
Background Physicians increasingly use artificial intelligence tools to support clinical work, yet there is no accepted standard for measuring the competency of large language models (LLMs) in clinical decision support. We developed EM-BENCH, a physician-validated, deterministically scored benchmark for evaluating whether models reliably apply the American College of Emergency Physicians (ACEP) guidelines to clinical decisions. Methods EM-BENCH comprises 210 binary clinical vignettes covering all 70 critical questions across 25 ACEP clinical policies. We evaluated 149 LLM configurations representing 79 base models using prespecified answers and deterministic scoring. Each vignette had a prespecified correct answer to a binary yes-or-no question based on an ACEP recommendation. We examined associations between accuracy and model access tier (closed-access vs. open-weight models), model size, release date, clinical policy, recommendation evidence level, and reasoning settings. Results No configuration achieved complete guideline concordance. The highest performing configuration answered 207 of 210 vignettes correctly (98.6%). Closed-access models outperformed open-weight models (median accuracy, 91% vs. 63%; Hodges-Lehmann location shift, 26 percentage points; 99% CI, 20 to 33; adjusted P<0.001). Among open-weight model families, a 10-fold increase in parameter count was associated with a 14-percentage-point increase in accuracy (99% CI, 7.7 to 20; adjusted P<0.001). Across all models, accuracy increased by 8.9 percentage points per release year (99% CI, 3.6 to 14; adjusted P<0.001). Neither enabling a thinking mode (pseudomedian change, 0.95 percentage points; 99% CI, -2.6 to 3.1; adjusted P=1.00) nor high versus low reasoning effort (0.91 percentage points; 99% CI, -0.69 to 3.1; adjusted P=0.56) consistently improved accuracy. Conclusions Contemporary LLMs showed improving but incomplete ability to apply ACEP guidelines. Concordance varied markedly across models and was associated with access tier, size, and release recency, while additional inference-time reasoning offered inconsistent benefit. Specialty-specific, physician-validated benchmarks may provide a reproducible assessment of LLMs' ability to make guideline-concordant clinical decisions.
Related
- Receipt of guideline-concordant prenatal care among Medicaid-insured women with disabilities in Texas
- Comparative evaluation of large language models for guideline-concordant decision support in colorectal cancer multidisciplinary team care: A source-masked retrospective study
- Predictors of guideline-concordant treatment intensification in metastatic castration-sensitive prostate cancer: an explainable machine learning study
- Guideline-Concordant Answer Quality of Five Large Language Models on Indication Criteria for Total Hip Arthroplasty in Hip Osteoarthritis: A Single-Run Benchmark Against the 2021 German S3 Guideline (EKIT-Hip)
- Myocardial Stiffness Tracking & Assessment Toolkit Driven by Artificial Intelligence For Reproducible Benchmarking of Temporal Segmentation and Shear-Wave Velocity Stabilization on Synthetic Data
BibTeX
@misc{nguyen2026embench,
title = {{EM-BENCH: A Reproducible Framework for Benchmarking Guideline-Concordant Clinical Decision Making in Large Language Models}},
author = {N. Nguyen and L. Perice and R. Sangal and V. Kumar and A. Chu and D. Dash},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.27.26366650},
url = {https://www.medrxiv.org/content/10.64898/2026.10.27.26366650v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - EM-BENCH: A Reproducible Framework for Benchmarking Guideline-Concordant Clinical Decision Making in Large Language Models
AU - N. Nguyen
AU - L. Perice
AU - R. Sangal
AU - V. Kumar
AU - A. Chu
AU - D. Dash
PY - 2026
DA - 2026/10/27
PB - medRxiv
DO - 10.64898/2026.10.27.26366650
UR - https://www.medrxiv.org/content/10.64898/2026.10.27.26366650v1
AB - Background Physicians increasingly use artificial intelligence tools to support clinical work, yet there is no accepted standard for measuring the competency of large language models (LLMs) in clinical decision support. We developed EM-BENCH, a physician-validated, deterministically scored benchmark for evaluating whether models reliably apply the American College of Emergency Physicians (ACEP) guidelines to clinical decisions. Methods EM-BENCH comprises 210 binary clinical vignettes covering all 70 critical questions across 25 ACEP clinical policies. We evaluated 149 LLM configurations representing 79 base models using prespecified answers and deterministic scoring. Each vignette had a prespecified correct answer to a binary yes-or-no question based on an ACEP recommendation. We examined associations between accuracy and model access tier (closed-access vs. open-weight models), model size, release date, clinical policy, recommendation evidence level, and reasoning settings. Results No configuration achieved complete guideline concordance. The highest performing configuration answered 207 of 210 vignettes correctly (98.6%). Closed-access models outperformed open-weight models (median accuracy, 91% vs. 63%; Hodges-Lehmann location shift, 26 percentage points; 99% CI, 20 to 33; adjusted P<0.001). Among open-weight model families, a 10-fold increase in parameter count was associated with a 14-percentage-point increase in accuracy (99% CI, 7.7 to 20; adjusted P<0.001). Across all models, accuracy increased by 8.9 percentage points per release year (99% CI, 3.6 to 14; adjusted P<0.001). Neither enabling a thinking mode (pseudomedian change, 0.95 percentage points; 99% CI, -2.6 to 3.1; adjusted P=1.00) nor high versus low reasoning effort (0.91 percentage points; 99% CI, -0.69 to 3.1; adjusted P=0.56) consistently improved accuracy. Conclusions Contemporary LLMs showed improving but incomplete ability to apply ACEP guidelines. Concordance varied markedly across models and was associated with access tier, size, and release recency, while additional inference-time reasoning offered inconsistent benefit. Specialty-specific, physician-validated benchmarks may provide a reproducible assessment of LLMs' ability to make guideline-concordant clinical decisions.
ER -