Human-in-the-loop validation of a sequential multi-LLM medical education pipeline
13 affiliations
- 1Department of Radiology, Samsung Changwon Hospital, Sungkyunkwan University School of Medicine, Changwon, Republic of Korea
- 2Department of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea
- 3Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea
- 4Department of Radiology, Hallym University Dongtan Sacred Heart Hospital, Hwaseong, Republic of Korea
- 5Department of Radiology, Seoul National University Bundang Hospital, Seongnam, Republic of Korea
- 6Department of Radiology, Yeouido St. Mary’s Hospital, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea
- 7Department of Radiology, Kyung Hee University Hospital, Kyung Hee University College of Medicine, Seoul, Republic of Korea
- 8Department of Radiology, Ansan Hospital, Korea University School of Medicine, Ansan, Republic of Korea
- 9Centum Breast and Thyroid Clinic, Busan, Republic of Korea
- 10Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea
- 11Department of Radiology and the Research Institute of Radiological Science, Gangnam Severance Hospital, Yonsei University College of Medicine, Seoul, Republic of Korea
- 12Department of Nuclear Medicine, Samsung Changwon Hospital, Sungkyunkwan University School of Medicine, Changwon, Republic of Korea
- 13Department of Radiology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea
Abstract
Large language models (LLMs) can generate medical educational content at scale, but multi-dimensional human validation of multi-LLM pipelines has not been performed at scale. We evaluated a 7-stage sequential multi-LLM pipeline implemented exclusively with the Gemini family (Gemini 3 Flash and Pro for text; Gemini 3 Pro Image, the Nano Banana Pro preview, for images), producing 6000 flashcards and 833 infographics for radiology board preparation. Nine residents and eleven attending radiologists evaluated 1284 flashcards across 11 subspecialties in a two-phase design around fifth-stage (S5) feedback. Among 1100 evaluations of 980 text-level PASS cards, the evaluation-level false-negative rate (FNR) for blocking errors was 1.00% (95% CI 0.50–1.78%), exceeding the pre-specified 0.3% safety threshold; card-level (1.12%) and majority-vote (0.82%) analyses converged. S5 feedback was associated with increased blocking-error flags (39–54 evaluation-level observations, McNemar p = 0.003) and decreased technical accuracy and educational quality scores (both p < 0.001). In unadjusted analyses, attending radiologists identified more errors than residents (odds ratio 4.52, 95% CI 2.38–8.57), although workload-matched sensitivity analyses attenuated this gap. A preliminary image pilot of 19 items showed critical errors in both tables and infographics, with between-format comparisons underpowered and not significant after Holm correction; this image-related observation should be interpreted as exploratory. The conditional safety estimates, low-reliability reference standard, and rater- and workload-dependent patterns do not support fully automated deployment; automated feedback was associated with sharpened evaluator scrutiny, though the single-arm design cannot isolate useful cueing from anchoring or re-review effects.
Related
- Education Research: Strengthening the Neurology Pipeline : The American Academy of Neurology Medical Student Symposium (2018–2025)
- Validation of a two-stage automated screening pipeline for medical systematic reviews: a stratified concordance study with three independent human reviewers, using PABAK and Gwet's AC1
- Translation, Cultural Adaptation, and Preliminary Validation of the Persian Version of the Assessment of Medical Education Environment by Teachers (AMEET) Questionnaire
- Understanding Black Women’s Lived Experiences of Medical Mistrust and Trust in the Reproductive Care Setting: a nested sequential explanatory mixed methods study
- Nutritional Literacy in Early Medical Education: A Cross-Sectional Comparison of First and Third-Year Medical Students in Turkey
BibTeX
@article{nam2026humanintheloop,
title = {{Human-in-the-loop validation of a sequential multi-LLM medical education pipeline}},
author = {Yoojin Nam and Taein An and Sung Il Hwang and Chawoong Jeon and Ji Won Jeong and Juhyun Jeong and Dong Yeong Kim and Pyeong Hwa Kim and So Yeon Kim and Sungkwan Kim and Yangwon Kim and Kwang Hwi Lee and Taehee Lee and Hee Sang Oh and Joon Hyung Park and Seunghyeon Shin and Yongsik Sim and Jeong Min Song and Seunghyun Song and Pa Hong and Namkug Kim},
journal = {NPJ Digital Medicine},
year = {2026},
month = jul,
doi = {10.1038/s41746-026-02978-8},
url = {https://doi.org/10.1038/s41746-026-02978-8},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - JOUR
TI - Human-in-the-loop validation of a sequential multi-LLM medical education pipeline
AU - Yoojin Nam
AU - Taein An
AU - Sung Il Hwang
AU - Chawoong Jeon
AU - Ji Won Jeong
AU - Juhyun Jeong
AU - Dong Yeong Kim
AU - Pyeong Hwa Kim
AU - So Yeon Kim
AU - Sungkwan Kim
AU - Yangwon Kim
AU - Kwang Hwi Lee
AU - Taehee Lee
AU - Hee Sang Oh
AU - Joon Hyung Park
AU - Seunghyeon Shin
AU - Yongsik Sim
AU - Jeong Min Song
AU - Seunghyun Song
AU - Pa Hong
AU - Namkug Kim
PY - 2026
DA - 2026/07/07
JO - NPJ Digital Medicine
DO - 10.1038/s41746-026-02978-8
UR - https://doi.org/10.1038/s41746-026-02978-8
AB - Large language models (LLMs) can generate medical educational content at scale, but multi-dimensional human validation of multi-LLM pipelines has not been performed at scale. We evaluated a 7-stage sequential multi-LLM pipeline implemented exclusively with the Gemini family (Gemini 3 Flash and Pro for text; Gemini 3 Pro Image, the Nano Banana Pro preview, for images), producing 6000 flashcards and 833 infographics for radiology board preparation. Nine residents and eleven attending radiologists evaluated 1284 flashcards across 11 subspecialties in a two-phase design around fifth-stage (S5) feedback. Among 1100 evaluations of 980 text-level PASS cards, the evaluation-level false-negative rate (FNR) for blocking errors was 1.00% (95% CI 0.50–1.78%), exceeding the pre-specified 0.3% safety threshold; card-level (1.12%) and majority-vote (0.82%) analyses converged. S5 feedback was associated with increased blocking-error flags (39–54 evaluation-level observations, McNemar p = 0.003) and decreased technical accuracy and educational quality scores (both p < 0.001). In unadjusted analyses, attending radiologists identified more errors than residents (odds ratio 4.52, 95% CI 2.38–8.57), although workload-matched sensitivity analyses attenuated this gap. A preliminary image pilot of 19 items showed critical errors in both tables and infographics, with between-format comparisons underpowered and not significant after Holm correction; this image-related observation should be interpreted as exploratory. The conditional safety estimates, low-reliability reference standard, and rater- and workload-dependent patterns do not support fully automated deployment; automated feedback was associated with sharpened evaluator scrutiny, though the single-arm design cannot isolate useful cueing from anchoring or re-review effects.
SN - 2398-6352
ER -