Leveraging large language models to generate multiple-choice questions for epilepsy education
7 affiliations
- 1School of Computer Science and Cyber Engineering, Guangzhou University, Guangzhou, China
- 2School of Stomatology, Lanzhou University, Lanzhou, China
- 3The First School of Clinical Medicine, Lanzhou University, Lanzhou, China
- 4School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
- 5Department of Orthopedics, Jiedong District People’s Hospital, Jieyang, Guangdong, China
- 6Department of Orthopedics, Shenzhen Children’s Hospital, Shenzhen, China
- 7Department of Emergency, The Brain Hospital of Guangxi Zhuang Autonomous Region, Liuzhou, China
Abstract
Background: Large language models (LLMs) are increasingly used to generate medical assessment items, but their reliability for disease-specific question writing remains uncertain. We benchmarked seven LLMs for epilepsy single-best-answer multiple-choice question generation under a standardized blueprint, focusing on structural compliance, clinical accuracy, expert-rated quality and educational usability.
Methods: Each model completed the same 60 prespecified tasks spanning foundational knowledge, clinical cases, treatment decision-making, and diagnostic evaluation and longitudinal management, yielding 420 questions. Tasks were delivered through an identical single-turn prompt without model-specific optimization, web search or external tools. Objective evaluation assessed structural compliance, keyed-answer accuracy, textual characteristics and response time. Two blinded clinical reviewers independently rated content rigor, clarity, distractor quality, educational usefulness, cognitive alignment and overall usability on five-point scales.
Results: After one standardized corrective prompt, all 420 questions met structural requirements, although first-pass compliance ranged from 83.33% to 100%. Keyed-answer accuracy varied substantially, from 61.67% to 98.33%, with Claude-5 achieving the highest accuracy. GPT-5.6 obtained the highest composite expert score (median, 3.75), followed by Claude-5 (3.67) and Kimi-K3 (3.50). Distractor quality was a common limitation, with median scores of only 2–3 across models. Only 2 of 420 questions (0.48%) were judged directly usable without revision, both generated by GPT-5.6; 119 (28.33%) were usable after minor revision and 299 (71.19%) required substantial revision. Longer outputs were associated with better cognitive alignment but not with content rigor or clarity, while response time was unrelated to overall usability.
Conclusion: Contemporary LLMs can reliably reproduce the structure of epilepsy MCQs, but structural compliance and answer accuracy do not ensure educational readiness. Model strengths were multidimensional, and weak distractor construction remained a major barrier to direct use. LLMs may support expert-supervised item drafting, but independent clinical review and subsequent learner-based psychometric validation remain necessary before deployment in medical education.
Related
- Evaluating large language models in patient education: a comparative analysis addressing frequently asked questions in peri-acetabular osteotomy
- Benchmarking ten frontier large language models on 1,477 board style multiple choice questions in hematology
- Performance of a foundation model to generate thousands of predicted outcomes across multiple use cases
- The collective turing test: large language models can generate realistic multi-user discussions
- How well do large language models answer postoperative lumbar fusion questions? A blinded comparative analysis of accuracy, quality, readability, and safety-related content
BibTeX
@article{li2026leveraging,
title = {{Leveraging large language models to generate multiple-choice questions for epilepsy education}},
author = {Chao Li and Weifan Wang and Chenchen Li and Xuebin Wang and Xinyan Ma and Weibo Zhao and Yuchen Xie and Weitao Wang and Rubing Lin and Zhaoyin Su and Min Wei},
journal = {Frontiers in Medicine},
year = {2026},
month = oct,
doi = {10.3389/fmed.2026.1978708},
url = {https://doi.org/10.3389/fmed.2026.1978708},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Leveraging large language models to generate multiple-choice questions for epilepsy education
AU - Chao Li
AU - Weifan Wang
AU - Chenchen Li
AU - Xuebin Wang
AU - Xinyan Ma
AU - Weibo Zhao
AU - Yuchen Xie
AU - Weitao Wang
AU - Rubing Lin
AU - Zhaoyin Su
AU - Min Wei
PY - 2026
DA - 2026/10/13
JO - Frontiers in Medicine
DO - 10.3389/fmed.2026.1978708
UR - https://doi.org/10.3389/fmed.2026.1978708
AB - Background: Large language models (LLMs) are increasingly used to generate medical assessment items, but their reliability for disease-specific question writing remains uncertain. We benchmarked seven LLMs for epilepsy single-best-answer multiple-choice question generation under a standardized blueprint, focusing on structural compliance, clinical accuracy, expert-rated quality and educational usability. Methods: Each model completed the same 60 prespecified tasks spanning foundational knowledge, clinical cases, treatment decision-making, and diagnostic evaluation and longitudinal management, yielding 420 questions. Tasks were delivered through an identical single-turn prompt without model-specific optimization, web search or external tools. Objective evaluation assessed structural compliance, keyed-answer accuracy, textual characteristics and response time. Two blinded clinical reviewers independently rated content rigor, clarity, distractor quality, educational usefulness, cognitive alignment and overall usability on five-point scales. Results: After one standardized corrective prompt, all 420 questions met structural requirements, although first-pass compliance ranged from 83.33% to 100%. Keyed-answer accuracy varied substantially, from 61.67% to 98.33%, with Claude-5 achieving the highest accuracy. GPT-5.6 obtained the highest composite expert score (median, 3.75), followed by Claude-5 (3.67) and Kimi-K3 (3.50). Distractor quality was a common limitation, with median scores of only 2–3 across models. Only 2 of 420 questions (0.48%) were judged directly usable without revision, both generated by GPT-5.6; 119 (28.33%) were usable after minor revision and 299 (71.19%) required substantial revision. Longer outputs were associated with better cognitive alignment but not with content rigor or clarity, while response time was unrelated to overall usability. Conclusion: Contemporary LLMs can reliably reproduce the structure of epilepsy MCQs, but structural compliance and answer accuracy do not ensure educational readiness. Model strengths were multidimensional, and weak distractor construction remained a major barrier to direct use. LLMs may support expert-supervised item drafting, but independent clinical review and subsequent learner-based psychometric validation remain necessary before deployment in medical education.
SN - 2296-858X
ER -