Development and Pilot Evaluation of the ECALM Program: A Bedside Clinical Assessment Tool for First-Year Residents in the Emergency Department
Abstract
Objectives: To describe the development and pilot evaluation of the ECALM program ("bedside clinical evaluation”, in French), a formative bedside assessment framework designed for first-year residents during emergency department (ED) rotations. Methods: The ECALM program was developed prospectively in three stages, in accordance with Stufflebeam’s "context, input, process, product” framework: (i) a baseline needs assessment survey, (ii) the development of standardized clinical scenarios via a structured expert consensus process, and (iii) a single-center pilot implementation. Six core clinical themes were selected (chest pain, abdominal pain, limb trauma, syncope, fever, and confusion), on the basis of epidemiological data from French EDs. A pilot study involving 12 first-year residents was conducted at Amiens University Medical Center’s ED (Amiens, France). Each resident was evaluated with a single scenario, and each session included a structured briefing, passive bedside observation by paired, independent, senior evaluators, and a formative debriefing. Psychometric evaluation of the ECALM tool was based on inter-rater reliability, using the intraclass correlation coefficient (ICC(2,1)) calculated using a two-way random-effects model for absolute agreement based on a single measurement, the weighted kappa, and a Bland–Altman analysis of inter-observer differences. Results: In the needs assessment (physician response rate: 27.1%), 97% of the participating senior practitioners reported that residents were insufficiently prepared for independent clinical tasks. During the pilot phase, the mean ± standard deviation total duration of an ECALM session was 43 ± 12 minutes. A preliminary analysis of the scores’ consistency yielded an ICC(2,1) of 0.76, indicating good inter-rater agreement between independent evaluators. The weighted kappa (0.74) was consistent with the ICC results. A Bland–Altman analysis demonstrated a mean inter-observer difference of only 1.25%, indicating negligible systematic measurement bias between evaluators. Qualitative feedback demonstrated high acceptability: all 12 residents expressed their satisfaction, reported an increased sense of psychological safety, and perceived the session to be a "one-on-one tutorial”. Furthermore, ten residents (83.3%) spontaneously reported acquiring new knowledge during the debriefing phase. Conclusion: The results of this pilot study demonstrated the operational feasibility, high acceptability, and good preliminary inter-rater reliability of the ECALM program for the formative assessment of first-year residents. Although ECALM constitutes a promising, non-punitive approach for encouraging early clinical reasoning and self-assessment, further well-powered, multicenter validation is required to demonstrate the tool’s external validity and long-term impact.
Related
- The experience of cross-setting care transitions of long-term care residents to the emergency department: a qualitative study.
- Development and internal validation of a clinical risk score for early delirium detection in older emergency department patients
- On-Device ECG and Vital-Sign Prediction of 24-Hour Mortality and Unplanned ICU Admission in Emergency Department Patients: A Retrospective Evaluation of MEDI-AI
- A Mixed Methods Evaluation of a Model of Care for Domestic Violence Within the Emergency Department
- Simulation-based evaluation of emergency medicine residents' preparedness for toxic terrorism.
BibTeX
@article{kunnas2026development,
title = {{Development and Pilot Evaluation of the ECALM Program: A Bedside Clinical Assessment Tool for First-Year Residents in the Emergency Department}},
author = {Siiri Astudillo Kunnas and Lilia Rotari and Farès Moustafa and Daniel Aiham Ghazali},
journal = {Advances in Medical Education and Practice},
year = {2026},
month = oct,
doi = {10.2147/amep.s629117},
url = {https://u-picardie.hal.science/hal-05785722v1},
note = {CC BY-NC 4.0},
}
RIS
TY - JOUR
TI - Development and Pilot Evaluation of the ECALM Program: A Bedside Clinical Assessment Tool for First-Year Residents in the Emergency Department
AU - Siiri Astudillo Kunnas
AU - Lilia Rotari
AU - Farès Moustafa
AU - Daniel Aiham Ghazali
PY - 2026
DA - 2026/10/06
JO - Advances in Medical Education and Practice
DO - 10.2147/amep.s629117
UR - https://u-picardie.hal.science/hal-05785722v1
AB - Objectives: To describe the development and pilot evaluation of the ECALM program ("bedside clinical evaluation”, in French), a formative bedside assessment framework designed for first-year residents during emergency department (ED) rotations. Methods: The ECALM program was developed prospectively in three stages, in accordance with Stufflebeam’s "context, input, process, product” framework: (i) a baseline needs assessment survey, (ii) the development of standardized clinical scenarios via a structured expert consensus process, and (iii) a single-center pilot implementation. Six core clinical themes were selected (chest pain, abdominal pain, limb trauma, syncope, fever, and confusion), on the basis of epidemiological data from French EDs. A pilot study involving 12 first-year residents was conducted at Amiens University Medical Center’s ED (Amiens, France). Each resident was evaluated with a single scenario, and each session included a structured briefing, passive bedside observation by paired, independent, senior evaluators, and a formative debriefing. Psychometric evaluation of the ECALM tool was based on inter-rater reliability, using the intraclass correlation coefficient (ICC(2,1)) calculated using a two-way random-effects model for absolute agreement based on a single measurement, the weighted kappa, and a Bland–Altman analysis of inter-observer differences. Results: In the needs assessment (physician response rate: 27.1%), 97% of the participating senior practitioners reported that residents were insufficiently prepared for independent clinical tasks. During the pilot phase, the mean ± standard deviation total duration of an ECALM session was 43 ± 12 minutes. A preliminary analysis of the scores’ consistency yielded an ICC(2,1) of 0.76, indicating good inter-rater agreement between independent evaluators. The weighted kappa (0.74) was consistent with the ICC results. A Bland–Altman analysis demonstrated a mean inter-observer difference of only 1.25%, indicating negligible systematic measurement bias between evaluators. Qualitative feedback demonstrated high acceptability: all 12 residents expressed their satisfaction, reported an increased sense of psychological safety, and perceived the session to be a "one-on-one tutorial”. Furthermore, ten residents (83.3%) spontaneously reported acquiring new knowledge during the debriefing phase. Conclusion: The results of this pilot study demonstrated the operational feasibility, high acceptability, and good preliminary inter-rater reliability of the ECALM program for the formative assessment of first-year residents. Although ECALM constitutes a promising, non-punitive approach for encouraging early clinical reasoning and self-assessment, further well-powered, multicenter validation is required to demonstrate the tool’s external validity and long-term impact.
SN - 1179-7258
SN - 1179-7258
ER -