Calibrated large language models demonstrate a consistent advantage in a randomized longitudinal medical education intervention
- 1Warren Alpert Medical School of Brown University
Abstract
Medical trainees increasingly use large language models to review lecture material, but the educational value of that output depends on how it is generated, with prompt and model choice rarely measured. We tested a calibrated pipeline converting Neurology Grand Rounds recordings into summaries and quizzes in a year-long randomized study. In Phase 1, six summary and six question prompts across two models were scored with 2,692 rubric annotations. Claude Prompt 0 produced the best summaries (composite 76.9%) and GPT Prompt 1 the best questions (79.6%); quality varied widely by prompt, and neither model won both tasks. In Phase 2, students in 11 clerkship cohorts over a year were randomized within cohort to receive a summary before each quiz. Summary recipients reported reviewing more and studying faster and scored higher in 9 of 11 cohorts (p = 0.026), although the between-student difference was small (+0.28 of 10 points; P = 0.28).
Related
- A pilot evaluation of a preventive intervention to support mental health among dementia caregivers: a controlled longitudinal study
- A supportive intervention on infertile couples for continuing infertility treatments: a randomized trial
- Efficacy of a weighted-vest aerobic dance intervention for enhancing lower extremity strength and mobility in older adults: a randomized controlled trial
- Short-term choroidal thickness modulation with SAT-001 digital software intervention in pediatric myopia: a randomized pilot study
- Culturally adapted home-based intervention to improve child’s development in Arab toddlers in Israel: study protocol for a randomized controlled trial
BibTeX
@misc{khurana2026calibrated,
title = {{Calibrated large language models demonstrate a consistent advantage in a randomized longitudinal medical education intervention}},
author = {J. Khurana and H. A. Zaki and A. Dawer and E. Pavlick and S. Gupta and S. S. P. Datla and S. Eldeeb and S. Chaudhry and S. A. Rizvi},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.30.26366878},
url = {https://www.medrxiv.org/content/10.64898/2026.10.30.26366878v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Calibrated large language models demonstrate a consistent advantage in a randomized longitudinal medical education intervention
AU - J. Khurana
AU - H. A. Zaki
AU - A. Dawer
AU - E. Pavlick
AU - S. Gupta
AU - S. S. P. Datla
AU - S. Eldeeb
AU - S. Chaudhry
AU - S. A. Rizvi
PY - 2026
DA - 2026/10/30
PB - medRxiv
DO - 10.64898/2026.10.30.26366878
UR - https://www.medrxiv.org/content/10.64898/2026.10.30.26366878v1
AB - Medical trainees increasingly use large language models to review lecture material, but the educational value of that output depends on how it is generated, with prompt and model choice rarely measured. We tested a calibrated pipeline converting Neurology Grand Rounds recordings into summaries and quizzes in a year-long randomized study. In Phase 1, six summary and six question prompts across two models were scored with 2,692 rubric annotations. Claude Prompt 0 produced the best summaries (composite 76.9%) and GPT Prompt 1 the best questions (79.6%); quality varied widely by prompt, and neither model won both tasks. In Phase 2, students in 11 clerkship cohorts over a year were randomized within cohort to receive a summary before each quiz. Summary recipients reported reviewing more and studying faster and scored higher in 9 of 11 cohorts (p = 0.026), although the between-student difference was small (+0.28 of 10 points; P = 0.28).
ER -