Inter-rater reliability during live pediatric formative Objective Structured Clinical Examination (OSCE) in a French University
Abstract
Background and objective In France, OSCE (Objective Structured Clinical Examination) have recently been incorporated as an official assessment method for the final National Ranking Examination of undergraduate medical students. Given these high stakes, it is of paramount importance to provide high-quality OSCE training in every discipline. Our pediatrics department has developed an OSCE training program covering all areas of pediatric specialties. Students are evaluated by two independent assessors for each station. As part of a quality assessment process, this study investigated inter-rater reliability in OSCE grading between the two assessors. Materials and methods This prospective observational study was conducted from March to June 2023 in the Department of Pediatrics of Sorbonne University (Paris, France). Undergraduate medical students were evaluated during OSCE training sessions. Each student had to complete two consecutive stations. Each station was evaluated by two independent assessors. The scoring checklist was divided into two parts: "clinical skills" and "general skills," assessing the student's behavior and relational aptitudes. Evaluation scales were dichotomous for all items. Results About 206 fifth-year medical students participated in four distinct OSCE sessions. A total of 411 stations were evaluated by 20 pairs of assessors, resulting in 822 different scores. The intraclass correlation coefficients of “total score,” “clinical skills,” and “general skills” scores between both assessors were 0.83 [0.64–0.93], 0.85 [0.68–0.94], and 0.50 [0.14–0.75], respectively. Inter-rater reliability was influenced by overall OSCE score and varied significantly between OSCE scenarios. Variability was greater in pediatric tuberculosis and hemolytic uremic syndrome scenarios, compared to bronchiolitis. Conclusions During live pediatric training OSCE, inter-rater reliability was good regarding clinical skills, and moderate for general skills. Inter-rater variability might be more pronounced for students with lower overall OSCE score. Our study emphasizes the importance of a careful drafting of OSCE cases with precise and clear-cut scoring checklist, especially in stations focusing on very specialized clinical situations.
Related
- Clock Drawing Test Performance in Rural Vietnam: Reliability and Concurrent Validity Relative to the Mini-Mental State Examination in a Community-Based Sample.
- Exploring the association between Zhejiang University index and osteoporosis risk in middle-aged and older adults based on the National Health and Nutrition Examination Survey database
- Reliability-constrained cost optimization of standalone photovoltaic water pumping systems under time-structured irrigation demand
- Comparative evaluation of ChatGPT, DeepSeek, and Doubao for pediatric IgA vasculitis patient education: content quality, reliability, readability, understandability, and actionability
- Clinical Presentation of, Treatment Approaches for, and the Impact of Social Determinants of Health on Pediatric Catatonia: A Structured Qualitative Synthesis and Targeted Literature Review.
BibTeX
@article{pham2026interrater,
title = {{Inter-rater reliability during live pediatric formative Objective Structured Clinical Examination (OSCE) in a French University}},
author = {Aurélie Pham and Florian Blanchard and Antoine Monsel and Manon Allaire and Alain Carrié and Jean-Michel Davaine and Maxens Decavele and Jean-Philippe Foy and Kamila Kolanska and Sohela Moussaoui and Nadia Nathan and Marie-Christine Renaud and Laure Serresse and Yanis Tamzali and Jessica Taytard and Nada Sabourdin},
journal = {Archives de Pédiatrie},
year = {2026},
month = sep,
doi = {10.1016/j.arcped.2026.105614},
url = {https://hal.sorbonne-universite.fr/hal-05771910v1},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Inter-rater reliability during live pediatric formative Objective Structured Clinical Examination (OSCE) in a French University
AU - Aurélie Pham
AU - Florian Blanchard
AU - Antoine Monsel
AU - Manon Allaire
AU - Alain Carrié
AU - Jean-Michel Davaine
AU - Maxens Decavele
AU - Jean-Philippe Foy
AU - Kamila Kolanska
AU - Sohela Moussaoui
AU - Nadia Nathan
AU - Marie-Christine Renaud
AU - Laure Serresse
AU - Yanis Tamzali
AU - Jessica Taytard
AU - Nada Sabourdin
PY - 2026
DA - 2026/09/30
JO - Archives de Pédiatrie
DO - 10.1016/j.arcped.2026.105614
UR - https://hal.sorbonne-universite.fr/hal-05771910v1
AB - Background and objective In France, OSCE (Objective Structured Clinical Examination) have recently been incorporated as an official assessment method for the final National Ranking Examination of undergraduate medical students. Given these high stakes, it is of paramount importance to provide high-quality OSCE training in every discipline. Our pediatrics department has developed an OSCE training program covering all areas of pediatric specialties. Students are evaluated by two independent assessors for each station. As part of a quality assessment process, this study investigated inter-rater reliability in OSCE grading between the two assessors. Materials and methods This prospective observational study was conducted from March to June 2023 in the Department of Pediatrics of Sorbonne University (Paris, France). Undergraduate medical students were evaluated during OSCE training sessions. Each student had to complete two consecutive stations. Each station was evaluated by two independent assessors. The scoring checklist was divided into two parts: "clinical skills" and "general skills," assessing the student's behavior and relational aptitudes. Evaluation scales were dichotomous for all items. Results About 206 fifth-year medical students participated in four distinct OSCE sessions. A total of 411 stations were evaluated by 20 pairs of assessors, resulting in 822 different scores. The intraclass correlation coefficients of “total score,” “clinical skills,” and “general skills” scores between both assessors were 0.83 [0.64–0.93], 0.85 [0.68–0.94], and 0.50 [0.14–0.75], respectively. Inter-rater reliability was influenced by overall OSCE score and varied significantly between OSCE scenarios. Variability was greater in pediatric tuberculosis and hemolytic uremic syndrome scenarios, compared to bronchiolitis. Conclusions During live pediatric training OSCE, inter-rater reliability was good regarding clinical skills, and moderate for general skills. Inter-rater variability might be more pronounced for students with lower overall OSCE score. Our study emphasizes the importance of a careful drafting of OSCE cases with precise and clear-cut scoring checklist, especially in stations focusing on very specialized clinical situations.
SN - 0929-693X
SN - 1769-664X
ER -