Mechanism-aware benchmarking of large language models as learning aids in pediatric difficult airway training: true hallucinations vs. false contraindications.
- 1Department of Anesthesiology and Reanimation, University of Health Sciences Turkey, Kartal Dr.Lütfi Kırdar City Hospital, D-100 Guney Yanyol No:47 Cevizli Mevkii, Kartal, Istanbul, 34865, Turkey. [email protected].
- 2Department of Anesthesiology and Reanimation, University of Health Sciences Turkey, Kartal Dr.Lütfi Kırdar City Hospital, D-100 Guney Yanyol No:47 Cevizli Mevkii, Kartal, Istanbul, 34865, Turkey.
Abstract
Background
Large language models (LLMs) are being investigated for their potential applications in perioperative decision-making and as informal "just-in-time" learning tools for trainees. This study evaluated five LLMs in pediatric difficult-airway management using a mechanism-aware scoring approach that distinguishes true hallucinations from false contraindications and examines their suitability for supervised learning.Methods
In a simulation study involving a comparative blinded evaluation, 50 cases of pediatric difficult airway management were presented to five LLMs using a standardized prompt based on the 2022 American Society of Anesthesiologists Difficult Airway Guidelines. Two anesthesiologists evaluated the responses on a modified 0-5 scale.Results
The inter-rater agreement was deemed to be of the highest caliber, as evidenced by a quadratic weighted κ value of 0.982. A statistically significant variation in overall performance was observed among the models (Friedman χ²(4) = 100.12, p < 0.001). In this single-pass evaluation, GPT-5.2 Thinking had the highest mean score and proportion of acceptable responses, whereas Claude Opus 4.5 had the highest combined critical-error rate (23.0% vs. 6.5% for both GPT models). True hallucinations were most frequent with Claude Opus 4.5 (11.8%; other models 0.5-2.2%), while false contraindications occurred across all models (5.0-13.0%) and clustered around rocuronium. Scenario inadequacy (score ≤ 2) increased with complexity: 32-98% for difficult intubation, 46-94% for difficult ventilation, and 52-92% for CICO.Conclusion
The contemporary generation of LLMs exhibits model-specific performance characteristics and error patterns in pediatric difficult airway management. Dosing-related failures manifest through different mechanisms of true hallucination vs. false contraindication. The findings reveal a marked inadequacy of the scenarios employed in these models, particularly as complexity and critical error rates increase. Current LLMs may have value as supervised supplementary learning tools, but not as autonomous sources of clinical or educational guidance.
Related
- Benchmarking Gemini 1.5 Pro in Pediatric Emergency Radiology: Sensitivity Gap and Physiological Bias in the Immature Skeleton.
- Benchmarking freely accessible large language models for pediatric epilepsy care: a prospective mixed-methods study.
- Assessing the Concordance Between LLM-Based ICD-10 Coding and a Drug Database for Antihypertensive Drug Contraindications: A Weighted Inter-Rater Agreement Study
- Inguinal Hernia Repair Under Local Anesthesia: Benefits, Contraindications, Barriers to Use; A Narrative Review
- High-velocity trauma by social media: a new mechanism of injury in pediatric traumatology?
BibTeX
@article{yediyıldız2026mechanismaware,
title = {{Mechanism-aware benchmarking of large language models as learning aids in pediatric difficult airway training: true hallucinations vs. false contraindications.}},
author = {Merve Bulun Yediyıldız and İrem Durmuş},
journal = {BMC medical education},
year = {2026},
month = oct,
doi = {10.1186/s12909-026-10509-y},
url = {https://europepmc.org/article/MED/42907955},
note = {CC BY-NC-ND},
}
RIS
TY - JOUR
TI - Mechanism-aware benchmarking of large language models as learning aids in pediatric difficult airway training: true hallucinations vs. false contraindications.
AU - Merve Bulun Yediyıldız
AU - İrem Durmuş
PY - 2026
DA - 2026/10/01
JO - BMC medical education
DO - 10.1186/s12909-026-10509-y
UR - https://europepmc.org/article/MED/42907955
AB - Background Large language models (LLMs) are being investigated for their potential applications in perioperative decision-making and as informal "just-in-time" learning tools for trainees. This study evaluated five LLMs in pediatric difficult-airway management using a mechanism-aware scoring approach that distinguishes true hallucinations from false contraindications and examines their suitability for supervised learning.Methods In a simulation study involving a comparative blinded evaluation, 50 cases of pediatric difficult airway management were presented to five LLMs using a standardized prompt based on the 2022 American Society of Anesthesiologists Difficult Airway Guidelines. Two anesthesiologists evaluated the responses on a modified 0-5 scale.Results The inter-rater agreement was deemed to be of the highest caliber, as evidenced by a quadratic weighted κ value of 0.982. A statistically significant variation in overall performance was observed among the models (Friedman χ²(4) = 100.12, p < 0.001). In this single-pass evaluation, GPT-5.2 Thinking had the highest mean score and proportion of acceptable responses, whereas Claude Opus 4.5 had the highest combined critical-error rate (23.0% vs. 6.5% for both GPT models). True hallucinations were most frequent with Claude Opus 4.5 (11.8%; other models 0.5-2.2%), while false contraindications occurred across all models (5.0-13.0%) and clustered around rocuronium. Scenario inadequacy (score ≤ 2) increased with complexity: 32-98% for difficult intubation, 46-94% for difficult ventilation, and 52-92% for CICO.Conclusion The contemporary generation of LLMs exhibits model-specific performance characteristics and error patterns in pediatric difficult airway management. Dosing-related failures manifest through different mechanisms of true hallucination vs. false contraindication. The findings reveal a marked inadequacy of the scenarios employed in these models, particularly as complexity and critical error rates increase. Current LLMs may have value as supervised supplementary learning tools, but not as autonomous sources of clinical or educational guidance.
SN - 1472-6920
ER -