Benchmarking Large Language AI Models Against Human Doctors Using an Emergency Medicine Simulation Approach
- 1Emergency Medicine Department, School of Medicine, AJA University of Medical Sciences, Tehran, Iran
Abstract
Background: Proper adoption of AI in emergency medicine necessitate thorough and rigorous appraisal. Classic benchmarks, which rely heavily on basic question-and-answer formats, though dominating the benchmarking field of LLM usage in medicine, have proven inadequate in assessing the full spectrum of skills incumbent upon an agent in healthcare. Methods: We propose an intricate benchmarking scheme in which the skill set required of an emergency physician is tested and quantified. Our benchmarking structure is 3-fold: skills such as history taking, diagnostic accuracy, treatment plan, and empathy are evaluated by an emergency medicine expert, while the same expertise is scored by an intern from the perspective of a patient, through what we call a simulated emergency room visit. A comprehensive written examination is also utilized to measure general emergency medicine knowledge. Results: Eight LLMs were benchmarked against each other and in contrast to nine emergency medical doctors. Our findings indicate that the AI models significantly outperformed human physicians across several key domains, including history taking, investigation, diagnostic accuracy, and treatment planning. Furthermore, the models demonstrated superior performance in comprehensibility, credibility, and empathy. When comparing AI models against each other, performance advantages were marginal during simulated emergency room visits and were mostly pronounced during the written examination. Conclusion: AI models demonstrate a clear advantage over human physicians across a diverse range of emergency medicine competencies. The proposed benchmarking scheme provides a comprehensive metric for a thorough evaluation of medical skill set, furthering validation, adaptation, and utilization of AI in emergency medicine.
Related
- LEAF: benchmarking multiobjective evolutionary feature selection over large language model–extracted symptoms for interpretable Traditional Chinese Medicine respiratory syndrome differentiation
- Unifying Vision and Language: Benchmarking End to End Transformer Model Against the ClipCap Framework
- Addressing benchmarking gaps in large language models for health and medicine with dynamic red-teaming
- Simulation-based evaluation of emergency medicine residents' preparedness for toxic terrorism.
- Performance evaluation and benchmarking across 16 large language model configurations on a large real-world emergency department triage data set
BibTeX
@misc{shahabian2026benchmarking,
title = {{Benchmarking Large Language AI Models Against Human Doctors Using an Emergency Medicine Simulation Approach}},
author = {M. Shahabian and A. Hadizadeh Esfahani and M. Ahmadi and S. Rajabzadeh and M. H. Givechin Ghamsari and A. Tabaraee and S. Z. Hejripour Rafsanjani},
howpublished = {medRxiv},
year = {2026},
month = nov,
doi = {10.64898/2026.11.09.26367590},
url = {https://www.medrxiv.org/content/10.64898/2026.11.09.26367590v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Benchmarking Large Language AI Models Against Human Doctors Using an Emergency Medicine Simulation Approach
AU - M. Shahabian
AU - A. Hadizadeh Esfahani
AU - M. Ahmadi
AU - S. Rajabzadeh
AU - M. H. Givechin Ghamsari
AU - A. Tabaraee
AU - S. Z. Hejripour Rafsanjani
PY - 2026
DA - 2026/11/09
PB - medRxiv
DO - 10.64898/2026.11.09.26367590
UR - https://www.medrxiv.org/content/10.64898/2026.11.09.26367590v1
AB - Background: Proper adoption of AI in emergency medicine necessitate thorough and rigorous appraisal. Classic benchmarks, which rely heavily on basic question-and-answer formats, though dominating the benchmarking field of LLM usage in medicine, have proven inadequate in assessing the full spectrum of skills incumbent upon an agent in healthcare. Methods: We propose an intricate benchmarking scheme in which the skill set required of an emergency physician is tested and quantified. Our benchmarking structure is 3-fold: skills such as history taking, diagnostic accuracy, treatment plan, and empathy are evaluated by an emergency medicine expert, while the same expertise is scored by an intern from the perspective of a patient, through what we call a simulated emergency room visit. A comprehensive written examination is also utilized to measure general emergency medicine knowledge. Results: Eight LLMs were benchmarked against each other and in contrast to nine emergency medical doctors. Our findings indicate that the AI models significantly outperformed human physicians across several key domains, including history taking, investigation, diagnostic accuracy, and treatment planning. Furthermore, the models demonstrated superior performance in comprehensibility, credibility, and empathy. When comparing AI models against each other, performance advantages were marginal during simulated emergency room visits and were mostly pronounced during the written examination. Conclusion: AI models demonstrate a clear advantage over human physicians across a diverse range of emergency medicine competencies. The proposed benchmarking scheme provides a comprehensive metric for a thorough evaluation of medical skill set, furthering validation, adaptation, and utilization of AI in emergency medicine.
ER -