AI-augmented versus expert-authored multiple-choice questions: a psychometric comparison in a high-stakes specialty examination
5 affiliations
- 1Department for Assessment and Evaluation, Institute for Medical Education (IML), University of Bern, Bern, Switzerland
- 2Master of Medical Education Program (MME), Institute for Medical Education (IML), University of Bern, Bern, Switzerland
- 3Department of Obstetrics and Gynecology, Regional Hospital of Bellinzona, EOC (Ente Ospedaliero Cantonale), Lugano, Switzerland
- 4Department of Gynecology and Gynecological Oncology, University Hospital Basel, University of Basel, Basel, Switzerland
- 5Reproductive Medicine and Gynecological Endocrinology (RME), University Hospital Basel, University of Basel, Basel, Switzerland
Abstract
Background: Multiple-choice questions (MCQs) are widely used in written assessments, particularly in high-stakes medical examinations. Developing high-quality MCQs is time-consuming and requires subject matter expertise. Large language models (LLMs), such as ChatGPT-4o, have therefore been proposed as tools to support item generation. Prior studies have examined AI-generated MCQs in formative or simulated contexts; evidence from authentic high-stakes examinations remains limited. This study compared the psychometric performance of AI-augmented and expert-authored MCQs within a national specialty examination.
Methods: For a national high-stakes examination, 90 AI-augmented items (generated with ChatGPT-4o) and 25 expert-authored items were developed and underwent expert screening, revision, and blinded review board evaluation. The examination was administered to 304 candidates (166 basic-level, 138 specialist-level). Item analysis included 12 AI-augmented items, 14 expert-authored items, and 89 previously used bank items. Psychometric performance was assessed using classical test theory (item difficulty, discrimination, internal consistency, distractor functioning). Differences across item sources and candidate groups were analyzed using ANOVA with post-hoc testing.
Results: AI-augmented items were easier than both newly expert-authored items and previously used bank items, as reflected by a significantly higher proportion of correct responses (mean P-value: 84,7% vs. 73,5% and 69,4%, respectively; (F(2,604)= 297.187, p<0.001)). Discrimination indices were comparable across item sources. Distractor analysis showed a higher proportion of AI-augmented items with no functioning distractor (p < 0.05). Overall test reliability did not significantly change after inclusion of AI-augmented items.
Conclusions: In this study, AI-augmented MCQs could be integrated into a high-stakes examination following rigorous multi-stage review, without evidence of detrimental effects on overall test performance. However, even after expert revision, such items tended to be easier and showed weaker distractor functioning. The substantial exclusion rate and systematic differences in item difficulty highlight the need for structured quality assurance and difficulty calibration when integrating AI-augmented item development into specialty-level assessments.
Related
- Specialty-associated differences in intensive care unit rotation exit examination performance: a single-center retrospective cross-sectional study.
- Construct Validity of Frontier Large Language Models on 4 Spanish Health Professions Specialty-Access Examinations (MIR, EIR, PIR, and FIR): Cross-Sectional Benchmarking and Psychometric Study.
- Psychometric characterization of human and artificial intelligence performance on cardiology residency in-service examination items
- Evaluation of retrieval-augmented generation (RAG) with GPT-4o for maxillofacial prosthodontics clinical questions.
- Comparison of the performance of large language models (ChatGPT-5.5, Gemini 3.1, and Claude 4.6) in answering pharmacology questions
BibTeX
@article{anschuetz2026aiaugmented,
title = {{AI-augmented versus expert-authored multiple-choice questions: a psychometric comparison in a high-stakes specialty examination}},
author = {Wilma Anschuetz and Daniel Stricker and Claudia Canonica and Céline Montavon and Christian De Geyter and Sören Huwendiek},
journal = {BMC Medical Education},
year = {2026},
month = aug,
doi = {10.1186/s12909-026-10082-4},
url = {https://doi.org/10.1186/s12909-026-10082-4},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - AI-augmented versus expert-authored multiple-choice questions: a psychometric comparison in a high-stakes specialty examination
AU - Wilma Anschuetz
AU - Daniel Stricker
AU - Claudia Canonica
AU - Céline Montavon
AU - Christian De Geyter
AU - Sören Huwendiek
PY - 2026
DA - 2026/08/07
JO - BMC Medical Education
DO - 10.1186/s12909-026-10082-4
UR - https://doi.org/10.1186/s12909-026-10082-4
AB - Background: Multiple-choice questions (MCQs) are widely used in written assessments, particularly in high-stakes medical examinations. Developing high-quality MCQs is time-consuming and requires subject matter expertise. Large language models (LLMs), such as ChatGPT-4o, have therefore been proposed as tools to support item generation. Prior studies have examined AI-generated MCQs in formative or simulated contexts; evidence from authentic high-stakes examinations remains limited. This study compared the psychometric performance of AI-augmented and expert-authored MCQs within a national specialty examination. Methods: For a national high-stakes examination, 90 AI-augmented items (generated with ChatGPT-4o) and 25 expert-authored items were developed and underwent expert screening, revision, and blinded review board evaluation. The examination was administered to 304 candidates (166 basic-level, 138 specialist-level). Item analysis included 12 AI-augmented items, 14 expert-authored items, and 89 previously used bank items. Psychometric performance was assessed using classical test theory (item difficulty, discrimination, internal consistency, distractor functioning). Differences across item sources and candidate groups were analyzed using ANOVA with post-hoc testing. Results: AI-augmented items were easier than both newly expert-authored items and previously used bank items, as reflected by a significantly higher proportion of correct responses (mean P-value: 84,7% vs. 73,5% and 69,4%, respectively; (F(2,604)= 297.187, p<0.001)). Discrimination indices were comparable across item sources. Distractor analysis showed a higher proportion of AI-augmented items with no functioning distractor (p < 0.05). Overall test reliability did not significantly change after inclusion of AI-augmented items. Conclusions: In this study, AI-augmented MCQs could be integrated into a high-stakes examination following rigorous multi-stage review, without evidence of detrimental effects on overall test performance. However, even after expert revision, such items tended to be easier and showed weaker distractor functioning. The substantial exclusion rate and systematic differences in item difficulty highlight the need for structured quality assurance and difficulty calibration when integrating AI-augmented item development into specialty-level assessments.
SN - 1472-6920
ER -