Assessment of Clinical Competence Using AI-Standardized Clinical Examination
- 1University Department of General Practice, Montpellier-Nimes Faculty of Medicine, University of Montpellier, Montpellier, France
Abstract
BACKGROUNDAs artificial intelligence (AI) enters clinical practice, concerns about deskilling and never-skilling make the independent assessment of clinical competence increasingly important. The AI-Standardized Clinical Examination (ASCE) uses AI to simulate and score standardized clinical encounters, potentially overcoming some of the logistical and standardization constraints of conventional objective structured clinical examinations (OSCEs). Whether it provides valid and reliable assessment at scale is unknown.
METHODSIn this prospective multicenter cohort study, final-year medical students from six French medical schools completed a 10-station ASCE six weeks before the national OSCE session. GPT-4o simulated standardized clinical partners; transcripts were subsequently scored by GPT-4.1 against faculty-defined rubrics. The primary outcome was the correlation between ASCE and national OSCE scores. Secondary outcomes were inter-rater reliability of expert transcript scoring, alignment of ASCE scores with expert ratings, and internal consistency.
RESULTSAmong 948 enrolled students, 588 had paired ASCE and national OSCE scores. ASCE scores were correlated with national OSCE performance (Spearmans {rho}, 0.57; 95% CI, 0.51 to 0.63), with less than 1% of residual variance attributable to center (intraclass correlation coefficient, 0.01). Eight experts scored 20 randomly selected ASCE transcripts; Krippendorffs was 0.77 overall, 0.80 for clinical skills, and 0.70 for communication and attitude skills. Adding GPT-4.1 as an evaluator reduced by 0.02, and its scores did not differ significantly from those of individual experts. Internal consistency was 0.67 across 10 stations and was projected to approach 0.80 with 20 stations.
CONCLUSIONSScores from an examination delivered and scored by AI were associated with performance on a national high-stakes OSCE, with minimal center-level variation, tentative agreement among expert raters, and automated scores that aligned with expert ratings. These results support AI delivery and scoring of clinical examinations, with expert review of automated scores.
DescriptionIn what is, to our knowledge, the first prospective multicenter study to evaluate a clinical examination delivered and scored by AI against an independent national OSCE, 588 final-year medical students across six French medical schools completed an AI-Standardized Clinical Examination before the independent national OSCE. Scores correlated with national OSCE performance (Spearmans {rho}, 0.57), with negligible variation between centers, supporting AI-standardized assessment at scale.
Related
- Ability of clinical assessment methods to differentiate levels of competence in undergraduate nursing education: a scoping review protocol.
- Outcome Stratification in Non-ST-Segment Elevation Myocardial Infarction: Standardized Assessment of Left Ventricular Global Longitudinal Strain
- Clinical Officers' Competence in Case Management: An Outcome Evaluation of the Competence‐Based Curriculum in Tanzanian Primary Health Facilities
- An Idiographic Examination of Body-Focused Repetitive Behaviors Using Ecological Momentary Assessment and Network Analysis
- Assessment for learning: a realist evaluation of the portfolio in the transfusion medicine area of focused competence program
BibTeX
@misc{lopez2026assessment,
title = {{Assessment of Clinical Competence Using AI-Standardized Clinical Examination}},
author = {A. Lopez and L. Gabellier and C. Peytavi and Z. Loi and E. Poupin-Lavigne and C. Plotton and V. Dupont and L. Jahdauti and P. de Vries and P. Thuillier and M. Compagnat and P. Vergne-Salle and M. Kuchenbuch and E. Feigerlova and P. Berthelot and N. Bednarek and B. Cochener-Lamard and P.-Y. Robert and M. Braun and D. Morin and P. Deruelle and I. Laffont and Y.-M. Pers and K. Yauy},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.01.26364500},
url = {https://www.medrxiv.org/content/10.64898/2026.10.01.26364500v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Assessment of Clinical Competence Using AI-Standardized Clinical Examination
AU - A. Lopez
AU - L. Gabellier
AU - C. Peytavi
AU - Z. Loi
AU - E. Poupin-Lavigne
AU - C. Plotton
AU - V. Dupont
AU - L. Jahdauti
AU - P. de Vries
AU - P. Thuillier
AU - M. Compagnat
AU - P. Vergne-Salle
AU - M. Kuchenbuch
AU - E. Feigerlova
AU - P. Berthelot
AU - N. Bednarek
AU - B. Cochener-Lamard
AU - P.-Y. Robert
AU - M. Braun
AU - D. Morin
AU - P. Deruelle
AU - I. Laffont
AU - Y.-M. Pers
AU - K. Yauy
PY - 2026
DA - 2026/10/01
PB - medRxiv
DO - 10.64898/2026.10.01.26364500
UR - https://www.medrxiv.org/content/10.64898/2026.10.01.26364500v1
AB - BACKGROUNDAs artificial intelligence (AI) enters clinical practice, concerns about deskilling and never-skilling make the independent assessment of clinical competence increasingly important. The AI-Standardized Clinical Examination (ASCE) uses AI to simulate and score standardized clinical encounters, potentially overcoming some of the logistical and standardization constraints of conventional objective structured clinical examinations (OSCEs). Whether it provides valid and reliable assessment at scale is unknown. METHODSIn this prospective multicenter cohort study, final-year medical students from six French medical schools completed a 10-station ASCE six weeks before the national OSCE session. GPT-4o simulated standardized clinical partners; transcripts were subsequently scored by GPT-4.1 against faculty-defined rubrics. The primary outcome was the correlation between ASCE and national OSCE scores. Secondary outcomes were inter-rater reliability of expert transcript scoring, alignment of ASCE scores with expert ratings, and internal consistency. RESULTSAmong 948 enrolled students, 588 had paired ASCE and national OSCE scores. ASCE scores were correlated with national OSCE performance (Spearmans {rho}, 0.57; 95% CI, 0.51 to 0.63), with less than 1% of residual variance attributable to center (intraclass correlation coefficient, 0.01). Eight experts scored 20 randomly selected ASCE transcripts; Krippendorffs was 0.77 overall, 0.80 for clinical skills, and 0.70 for communication and attitude skills. Adding GPT-4.1 as an evaluator reduced by 0.02, and its scores did not differ significantly from those of individual experts. Internal consistency was 0.67 across 10 stations and was projected to approach 0.80 with 20 stations. CONCLUSIONSScores from an examination delivered and scored by AI were associated with performance on a national high-stakes OSCE, with minimal center-level variation, tentative agreement among expert raters, and automated scores that aligned with expert ratings. These results support AI delivery and scoring of clinical examinations, with expert review of automated scores. DescriptionIn what is, to our knowledge, the first prospective multicenter study to evaluate a clinical examination delivered and scored by AI against an independent national OSCE, 588 final-year medical students across six French medical schools completed an AI-Standardized Clinical Examination before the independent national OSCE. Scores correlated with national OSCE performance (Spearmans {rho}, 0.57), with negligible variation between centers, supporting AI-standardized assessment at scale.
ER -