{
  "abstract": "Background: Proper adoption of AI in emergency medicine necessitate thorough and rigorous appraisal. Classic benchmarks, which rely heavily on basic question-and-answer formats, though dominating the benchmarking field of LLM usage in medicine, have proven inadequate in assessing the full spectrum of skills incumbent upon an agent in healthcare. Methods: We propose an intricate benchmarking scheme in which the skill set required of an emergency physician is tested and quantified. Our benchmarking structure is 3-fold: skills such as history taking, diagnostic accuracy, treatment plan, and empathy are evaluated by an emergency medicine expert, while the same expertise is scored by an intern from the perspective of a patient, through what we call a simulated emergency room visit. A comprehensive written examination is also utilized to measure general emergency medicine knowledge. Results: Eight LLMs were benchmarked against each other and in contrast to nine emergency medical doctors. Our findings indicate that the AI models significantly outperformed human physicians across several key domains, including history taking, investigation, diagnostic accuracy, and treatment planning. Furthermore, the models demonstrated superior performance in comprehensibility, credibility, and empathy. When comparing AI models against each other, performance advantages were marginal during simulated emergency room visits and were mostly pronounced during the written examination. Conclusion: AI models demonstrate a clear advantage over human physicians across a diverse range of emergency medicine competencies. The proposed benchmarking scheme provides a comprehensive metric for a thorough evaluation of medical skill set, furthering validation, adaptation, and utilization of AI in emergency medicine.",
  "abstract_shown": true,
  "affiliations": [
    "Emergency Medicine Department, School of Medicine, AJA University of Medical Sciences, Tehran, Iran"
  ],
  "article_type": "PUBLISHAHEADOFPRINT",
  "arxiv": null,
  "author_total": 7,
  "authors": [
    {
      "name": "M. Shahabian"
    },
    {
      "name": "A. Hadizadeh Esfahani"
    },
    {
      "name": "M. Ahmadi"
    },
    {
      "name": "S. Rajabzadeh"
    },
    {
      "name": "M. H. Givechin Ghamsari"
    },
    {
      "name": "A. Tabaraee"
    },
    {
      "affs": [
        0
      ],
      "name": "S. Z. Hejripour Rafsanjani"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/0mcb7wyzdqkhz.bib",
  "cited_by": 0,
  "doi": "10.64898/2026.11.09.26367590",
  "fulltext_url": "https://www.medrxiv.org/content/10.64898/2026.11.09.26367590v1.full",
  "id": "0mcb7wyzdqkhz",
  "issns": [],
  "landing_url": "https://www.medrxiv.org/content/10.64898/2026.11.09.26367590v1",
  "language": null,
  "licence": "CC-BY",
  "licence_label": "CC BY",
  "licence_source": "article",
  "licence_url": "https://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": null,
  "native_subjects": [
    [
      "medrxiv",
      "emergency medicine"
    ]
  ],
  "openalex": null,
  "pdf_url": "https://www.medrxiv.org/content/10.64898/2026.11.09.26367590v1.full.pdf",
  "pmcid": null,
  "pmid": null,
  "published": "2026-11-09",
  "publisher": null,
  "ris": "https://research.draconitic.com/paper/0mcb7wyzdqkhz.ris",
  "sources": [
    {
      "first_seen": "2026-11-11T00:21:36Z",
      "native_id": "10.64898/2026.11.09.26367590",
      "source": "medrxiv",
      "url": "https://www.medrxiv.org/content/10.64898/2026.11.09.26367590v1"
    }
  ],
  "subjects": [
    {
      "field": "medicine",
      "origin": "medrxiv:emergency medicine",
      "subfield": "general"
    }
  ],
  "title": "Benchmarking Large Language AI Models Against Human Doctors Using an Emergency Medicine Simulation Approach",
  "title_text": "Benchmarking Large Language AI Models Against Human Doctors Using an Emergency Medicine Simulation Approach",
  "type": "preprint",
  "updated_at": "2026-11-12T00:20:32Z",
  "url": "https://research.draconitic.com/paper/0mcb7wyzdqkhz",
  "venue": "medRxiv",
  "versions": []
}