Assessing the Performance of Artificial Intelligence on Anesthesiology In-Training Examinations and Applicability in Medical Education
- 1School of Medicine, Texas Tech University Health Sciences Center
Abstract
INTRODUCTIONLarge language models (LLMs) have demonstrated substantial performance on medical licensing and board examinations, but their application to anesthesiology remains less well studied. Prior evaluations have not compared current-generation models from different developers or assessed performance on figure-dependent questions that earlier models lacked the capability to interpret. This study evaluates the performance of two current-generation LLMs on a comprehensive anesthesiology in-training examination (ITE) review question bank, including figure-dependent items.
METHODSA total of 1,001 single-best-answer questions from an anesthesiology ITE review text, spanning 11 content chapters, were administered to Claude Opus 5 and GPT-5.6. No questions were excluded. Each item was presented once in a fresh, stateless context with no tool access, retrieval, answer key, or explanation provided in the prompt. Performance was analyzed by question format, and all 27 figure-dependent items were administered with their published figures supplied. Accuracy between models was compared using McNemars test.
RESULTSClaude Opus 5 answered 947/1,001 items correctly (94.6%; 95% CI, 93.0-95.8), and GPT-5.6 answered 946/1,001 correctly (94.5%; 95% CI, 92.9-95.8); the difference was not significant (McNemar p=1.00). Accuracy was similar across most question formats. On standard questions, accuracy was 94.3% and 94.6% for Claude Opus 5 and GPT-5.6, respectively; on the 18 figure-based questions with figures supplied, accuracy was 94.4% and 83.3%, respectively; and both models achieved 100% accuracy on image-option items. Withholding figures from the same 18 figure-based questions reduced pooled accuracy from 88.9% to 52.8% (exact McNemar p=0.03 and p=0.04 for Claude Opus 5 and GPT-5.6, respectively). The models agreed on 957/1,001 items (95.6%). Of the 33 items both models answered incorrectly, they selected the same incorrect option on 32 (97%).
DISCUSSIONCurrent-generation LLMs achieved approximately 95% accuracy, substantially improving on our prior results with earlier-generation models and placing both models within the highest band of the ITE normative framework, corresponding approximately to the 99th percentile across training levels. By comparison, our prior work with previous models placed ChatGPT-3.5 at the 52nd, 3rd, and 1st percentiles and ChatGPT-4.0 at the 99th, 95th, and 84th percentiles across increasing levels of training. Multimodal capability now permits successful interpretation of many figure-dependent questions, although performance declines markedly when required visual information is unavailable, and highly concordant errors remain. These findings support an increasingly promising role for LLMs as accessible, on-demand educational adjuncts for anesthesiology trainees, provided complete source material is supplied and outputs are critically reviewed.
Related
- Artificial intelligence and machine learning for prediction of postoperative complications: A systematic review focused on anesthesiology, perioperative risk stratification, and clinical applicability.
- Performance and Hallucination Analysis of Large Language Models on European Anesthesiology Examinations: Cross-Sectional Comparative Study
- CT-Referenced Diagnostic Performance and Complementary Error Patterns of Artificial Intelligence Versus a Single Emergency Physician in Chest Radiograph Interpretation: A Retrospective Paired Diagnostic Accuracy Study
- Evaluating the impact of social determinants of health on model performance and fairness in artificial intelligence for glaucoma progression prediction
- From underdiagnosis to scalable screening: Applications of artificial intelligence for vertebral compression fracture detection – a systematic review and meta-analysis of diagnostic performance
BibTeX
@misc{ibrahim2026assessing,
title = {{Assessing the Performance of Artificial Intelligence on Anesthesiology In-Training Examinations and Applicability in Medical Education}},
author = {A. F. Ibrahim and J. F. Zaki},
howpublished = {medRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.21.26363591},
url = {https://www.medrxiv.org/content/10.64898/2026.09.21.26363591v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Assessing the Performance of Artificial Intelligence on Anesthesiology In-Training Examinations and Applicability in Medical Education
AU - A. F. Ibrahim
AU - J. F. Zaki
PY - 2026
DA - 2026/09/21
PB - medRxiv
DO - 10.64898/2026.09.21.26363591
UR - https://www.medrxiv.org/content/10.64898/2026.09.21.26363591v1
AB - INTRODUCTIONLarge language models (LLMs) have demonstrated substantial performance on medical licensing and board examinations, but their application to anesthesiology remains less well studied. Prior evaluations have not compared current-generation models from different developers or assessed performance on figure-dependent questions that earlier models lacked the capability to interpret. This study evaluates the performance of two current-generation LLMs on a comprehensive anesthesiology in-training examination (ITE) review question bank, including figure-dependent items. METHODSA total of 1,001 single-best-answer questions from an anesthesiology ITE review text, spanning 11 content chapters, were administered to Claude Opus 5 and GPT-5.6. No questions were excluded. Each item was presented once in a fresh, stateless context with no tool access, retrieval, answer key, or explanation provided in the prompt. Performance was analyzed by question format, and all 27 figure-dependent items were administered with their published figures supplied. Accuracy between models was compared using McNemars test. RESULTSClaude Opus 5 answered 947/1,001 items correctly (94.6%; 95% CI, 93.0-95.8), and GPT-5.6 answered 946/1,001 correctly (94.5%; 95% CI, 92.9-95.8); the difference was not significant (McNemar p=1.00). Accuracy was similar across most question formats. On standard questions, accuracy was 94.3% and 94.6% for Claude Opus 5 and GPT-5.6, respectively; on the 18 figure-based questions with figures supplied, accuracy was 94.4% and 83.3%, respectively; and both models achieved 100% accuracy on image-option items. Withholding figures from the same 18 figure-based questions reduced pooled accuracy from 88.9% to 52.8% (exact McNemar p=0.03 and p=0.04 for Claude Opus 5 and GPT-5.6, respectively). The models agreed on 957/1,001 items (95.6%). Of the 33 items both models answered incorrectly, they selected the same incorrect option on 32 (97%). DISCUSSIONCurrent-generation LLMs achieved approximately 95% accuracy, substantially improving on our prior results with earlier-generation models and placing both models within the highest band of the ITE normative framework, corresponding approximately to the 99th percentile across training levels. By comparison, our prior work with previous models placed ChatGPT-3.5 at the 52nd, 3rd, and 1st percentiles and ChatGPT-4.0 at the 99th, 95th, and 84th percentiles across increasing levels of training. Multimodal capability now permits successful interpretation of many figure-dependent questions, although performance declines markedly when required visual information is unavailable, and highly concordant errors remain. These findings support an increasingly promising role for LLMs as accessible, on-demand educational adjuncts for anesthesiology trainees, provided complete source material is supplied and outputs are critically reviewed.
ER -