Multimodal artificial intelligence assessment of simulated cardiac arrest performance: a pilot study
6 affiliations
- 1Acute Care Research Program, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
- 2Department of Family Medicine, McGill University, Montreal, Quebec, Canada
- 3Department of Anesthesiology and Pain Medicine, University of Ottawa, Ottawa, Ontario, Canada
- 4Department of Medicine, University of Ottawa, Ottawa, Ontario, Canada
- 5Centre for Innovation in Medical Education, University of Ottawa, Ottawa, Ontario, Canada
- 6Department of Innovation in Medical Education, University of Ottawa, Ottawa, Ontario, Canada
Abstract
Objectives: Assessment of simulated cardiac arrest performance is resource-intensive and subject to inter-rater variability. We evaluated whether a multimodal artificial intelligence (AI) system could score video-recorded cardiac arrest megacode simulations with agreement comparable to expert raters.
Methods: Six scripted megacode simulations (three ventricular fibrillation, three pulseless electrical activity) representing exemplar, guideline-deviation and non-technical-deficiency performance were scored by two ACLS instructors using a validated team-leader checklist; consensus ratings served as the reference. Two operators analysed each video with Gemini 1.5 Pro at three temperature settings (504 item-level ratings). We calculated Cohen’s kappa, Gwet’s AC1, and sensitivity and specificity for identifying performed and omitted actions, and compared AI self-reported confidence between correct and incorrect ratings.
Results: Expert inter-rater agreement was κ = 0.78 (95% CI 0.64–0.93). AI–expert agreement was κ = 0.62 (95% CI 0.49–0.75; range 0.40–0.86 across configurations) and AC1 = 0.77. The AI identified performed actions with high sensitivity (0.98, 95% CI 0.96–0.99) but detected omitted actions poorly (specificity 0.58, 95% CI 0.44–0.72), particularly omissions defined by timing or sequence. Temperature had no consistent effect. Incorrect ratings of actions as performed carried lower mean self-reported confidence than correct ones (93.6 vs 98.1), but 48% were made with maximal confidence.
Conclusions: A multimodal AI system reliably confirmed completed actions but frequently failed to detect omitted actions, yielding moderate overall agreement that was below expert–expert agreement. Current systems may support, but not replace, expert assessment of cardiac arrest simulation.
Related
- Multimodal Artificial Intelligence for the Clinical Assessment of Otitis Media
- Generative Artificial Intelligence Use and Applied Higher-Order Cognitive Performance Among Health Informatics Students: A Multicenter Cross-Sectional Case-Based Assessment Study.
- Performance Assessment of an Artificial Intelligence System Used for Case Detection in a Large Community-based Diabetic Retinopathy Screening Program: A Retrospective Diagnostic Accuracy Validation Study.
- Toward integrated multimodal artificial intelligence in bone marrow examination: current state and open challenges
- Artificial Intelligence, Multi-Omics, and Multimodal Integration in Precision Radiotherapy for Head and Neck Cancer: From Biological Insights to Treatment Decisions
BibTeX
@article{dion2026multimodal,
title = {{Multimodal artificial intelligence assessment of simulated cardiac arrest performance: a pilot study}},
author = {Pierre-Marc Dion and Jennifer Cardenas-Mancera and Claire Touchie and Vicki LeBlanc and Arnaud Mbadjeu and Curtis Nickel and Richard Waldolf and YeonJung Yoo and Lucas Bergamascki and Pooyan Sekhavati and Simone Crooks and Yuqi Gu},
journal = {Resuscitation Plus},
year = {2026},
month = oct,
doi = {10.1016/j.resplu.2026.101517},
url = {https://doi.org/10.1016/j.resplu.2026.101517},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Multimodal artificial intelligence assessment of simulated cardiac arrest performance: a pilot study
AU - Pierre-Marc Dion
AU - Jennifer Cardenas-Mancera
AU - Claire Touchie
AU - Vicki LeBlanc
AU - Arnaud Mbadjeu
AU - Curtis Nickel
AU - Richard Waldolf
AU - YeonJung Yoo
AU - Lucas Bergamascki
AU - Pooyan Sekhavati
AU - Simone Crooks
AU - Yuqi Gu
PY - 2026
DA - 2026/10/06
JO - Resuscitation Plus
DO - 10.1016/j.resplu.2026.101517
UR - https://doi.org/10.1016/j.resplu.2026.101517
AB - Objectives: Assessment of simulated cardiac arrest performance is resource-intensive and subject to inter-rater variability. We evaluated whether a multimodal artificial intelligence (AI) system could score video-recorded cardiac arrest megacode simulations with agreement comparable to expert raters. Methods: Six scripted megacode simulations (three ventricular fibrillation, three pulseless electrical activity) representing exemplar, guideline-deviation and non-technical-deficiency performance were scored by two ACLS instructors using a validated team-leader checklist; consensus ratings served as the reference. Two operators analysed each video with Gemini 1.5 Pro at three temperature settings (504 item-level ratings). We calculated Cohen’s kappa, Gwet’s AC1, and sensitivity and specificity for identifying performed and omitted actions, and compared AI self-reported confidence between correct and incorrect ratings. Results: Expert inter-rater agreement was κ = 0.78 (95% CI 0.64–0.93). AI–expert agreement was κ = 0.62 (95% CI 0.49–0.75; range 0.40–0.86 across configurations) and AC1 = 0.77. The AI identified performed actions with high sensitivity (0.98, 95% CI 0.96–0.99) but detected omitted actions poorly (specificity 0.58, 95% CI 0.44–0.72), particularly omissions defined by timing or sequence. Temperature had no consistent effect. Incorrect ratings of actions as performed carried lower mean self-reported confidence than correct ones (93.6 vs 98.1), but 48% were made with maximal confidence. Conclusions: A multimodal AI system reliably confirmed completed actions but frequently failed to detect omitted actions, yielding moderate overall agreement that was below expert–expert agreement. Current systems may support, but not replace, expert assessment of cardiac arrest simulation.
SN - 2666-5204
ER -