AI Video Analysis of Psychomotor Performance in EMS Education: Agreement With Human Evaluators Across Three Skills
- 1Des Moines Area Community College
Abstract
ObjectivesPsychomotor competency verification for emergency medical technician (EMT) candidates is set by each state and, where states approve program-verified competency, rests on programs and their local evaluators. Commercial artificial intelligence (AI) video-analysis platforms are marketed to programs, but none has been evaluated against human evaluators on EMT skills. We measured the inter-rater reliability of certified evaluators scoring identical video-recorded performances, then evaluated one platform against that baseline.
MethodsFive human evaluators and one AI platform independently scored every video of three skills in a fully crossed design: cervical collar application (15 performances), bag-valve-mask ventilation (14), and medical patient assessment (15), using identical dichotomous checklists with critical-failure criteria. We computed item-level Fleiss {kappa}, Krippendorff , and Gwet AC1; absolute-agreement intraclass correlation coefficients (ICC) for percentage scores; and {kappa} for critical-failure and pass/fail decisions, for the human panel alone and with the platform as a sixth rater.
ResultsHuman item-level agreement was moderate on all three skills ({kappa} 0.41-0.47). Single-rater score reliability was moderate (ICC 0.54-0.69) against 0.85-0.92 for the five-rater mean. Agreement on an objective 75% threshold was moderate ({kappa} 0.54-0.57), but agreement on the recorded pass/fail decision was fair ({kappa} 0.30-0.39) and on the critical-failure determination near zero for cervical collar (0.03) and medical assessment (0.12). Platform alignment tracked rubric observability, not task complexity: correlation with the five-human mean was 0.857 for cervical collar, exceeding every human evaluator, -0.173 for bag-valve-mask, and 0.664 for medical assessment. Adding the platform improved panel reliability for cervical collar and reduced it for the others.
ConclusionsThe platform rated a discrete, visually observable skill as well as the panel it joined and degraded where credit depended on rates, volumes, or verbalization. Formative practice feedback and adjunct rating are defensible now; autonomous summative use is not. The human baseline is least reliable precisely where it decides outcomes.
Related
- Improving prostate MRI quality assessment across diverse evaluators and institutional practice through a PI-QUAL v2 lecture: a pilot study
- Analytical Agreement and Diagnostic Performance of a Commercial Point-of-Care Lipid Testing System for Lipid Profile and Atherogenic Ratios
- How much of the incremental protocol does a lactate threshold need? Agreement between three-stage and five-stage testing across seven lactate-derived indices in swimmers
- Technical Performance and Automated-manual Scoring Agreement in Type III Home Sleep Apnea Testing: A Single-center Study from Saudi Arabia.
- Performance of GPT-5.5 thinking for Manchester Triage System-based emergency department triage: agreement with an expert reference standard and prediction of short-term outcomes.
BibTeX
@misc{otte2026video,
title = {{AI Video Analysis of Psychomotor Performance in EMS Education: Agreement With Human Evaluators Across Three Skills}},
author = {J. H. Otte and A. Cartagena},
howpublished = {medRxiv},
year = {2026},
month = aug,
doi = {10.64898/2026.08.26.26361437},
url = {https://www.medrxiv.org/content/10.64898/2026.08.26.26361437v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - AI Video Analysis of Psychomotor Performance in EMS Education: Agreement With Human Evaluators Across Three Skills
AU - J. H. Otte
AU - A. Cartagena
PY - 2026
DA - 2026/08/26
PB - medRxiv
DO - 10.64898/2026.08.26.26361437
UR - https://www.medrxiv.org/content/10.64898/2026.08.26.26361437v1
AB - ObjectivesPsychomotor competency verification for emergency medical technician (EMT) candidates is set by each state and, where states approve program-verified competency, rests on programs and their local evaluators. Commercial artificial intelligence (AI) video-analysis platforms are marketed to programs, but none has been evaluated against human evaluators on EMT skills. We measured the inter-rater reliability of certified evaluators scoring identical video-recorded performances, then evaluated one platform against that baseline. MethodsFive human evaluators and one AI platform independently scored every video of three skills in a fully crossed design: cervical collar application (15 performances), bag-valve-mask ventilation (14), and medical patient assessment (15), using identical dichotomous checklists with critical-failure criteria. We computed item-level Fleiss {kappa}, Krippendorff , and Gwet AC1; absolute-agreement intraclass correlation coefficients (ICC) for percentage scores; and {kappa} for critical-failure and pass/fail decisions, for the human panel alone and with the platform as a sixth rater. ResultsHuman item-level agreement was moderate on all three skills ({kappa} 0.41-0.47). Single-rater score reliability was moderate (ICC 0.54-0.69) against 0.85-0.92 for the five-rater mean. Agreement on an objective 75% threshold was moderate ({kappa} 0.54-0.57), but agreement on the recorded pass/fail decision was fair ({kappa} 0.30-0.39) and on the critical-failure determination near zero for cervical collar (0.03) and medical assessment (0.12). Platform alignment tracked rubric observability, not task complexity: correlation with the five-human mean was 0.857 for cervical collar, exceeding every human evaluator, -0.173 for bag-valve-mask, and 0.664 for medical assessment. Adding the platform improved panel reliability for cervical collar and reduced it for the others. ConclusionsThe platform rated a discrete, visually observable skill as well as the panel it joined and degraded where credit depended on rates, volumes, or verbalization. Formative practice feedback and adjunct rating are defensible now; autonomous summative use is not. The human baseline is least reliable precisely where it decides outcomes.
ER -