Ambiguity Detection in Medical Exams via Large Language Models: Retrospective Cross-Sectional Pilot Study
Abstract
Background: Large language models (LLMs) have emerged as promising tools in medical education due to their ability to understand, generate, and reason with natural language. Their ability to simulate expert reasoning suggests a potential for supporting quality control in assessment design. In this study, the use of LLMs in identifying ambiguous or poorly constructed exam items in critical care academic assessments was evaluated. Objective: The study aimed to develop automated ambiguity and quality scores to objectively assess individual questions and entire exam components. Methods: We analyzed 264 questions from academic exams conducted over 3 academic years (2023-2025) at the Medical School of Université Côte d'Azur. Questions were drawn from 4 docimological formats: progressive clinical cases (PCC), mini-PCC, key feature problems, and isolated question sequences (IQS). Each element was submitted to 4 LLMs (ChatGPT, Gemini Pro, Le Chat, and DeepSeek) without prompt engineering. Performance was evaluated using the official correction key. We applied 4 binary diagnostic tags based on model agreement and self-reported ambiguity: ambiguity, low performance, incoherence, and subjective ambiguity. These tags generated a composite ambiguity score and contributed to a weighted quality score for each exam component. Results: LLMs achieved mean scores in the same range as students, with no significant differences across academic years and significantly higher performance on the mini-PCC and IQS formats (P=.049 and P=.04, respectively). IQS items had the highest ambiguity scores (54 items received a score of 2 in both 2023 and 2024, and 53 items retained the same score). Tag patterns revealed frequent issues with ambiguity and inconsistency. Quality scores varied across academic years. IQS predominantly showed moderate ambiguity (score 2), with occasional instances of strong signals. There was no significant difference in quality based on author specialty or seniority (P=.08 and P=.44, respectively). Conclusions: In this pilot study, LLMs may offer a preliminary framework to proactively detect ambiguous exam questions and estimate the overall quality of an exam. Integrating these tools into the assessment design process could potentially reduce the need for postexam corrections and may help improve fairness and clarity in medical evaluations.
Related
- Enhancing medical education with AI: A comparative study of various large language models in radiology
- Relation-aware dynamic knowledge subgraph enhanced large language models for medical multiple-choice question answering
- A Vision-Language Model as a Teacher for Bird Vocalization Detection
- Utilization of a HIPAA-compliant large language model chatbot in an academic pediatric medical center
- Core open science practices in major medical journals: development of an automatized tool based on a Large Language Model
BibTeX
@article{lombardi2026ambiguity,
title = {{Ambiguity Detection in Medical Exams via Large Language Models: Retrospective Cross-Sectional Pilot Study}},
author = {Romain Lombardi and Alexandre Destere and Jean Dellamonica and Alexandre Gérard and Mathieu Jozwiak},
journal = {JMIR Medical Education},
year = {2026},
month = may,
doi = {10.2196/82702},
url = {https://hal.science/hal-05804947v1},
note = {CC BY},
}
RIS
TY - JOUR
TI - Ambiguity Detection in Medical Exams via Large Language Models: Retrospective Cross-Sectional Pilot Study
AU - Romain Lombardi
AU - Alexandre Destere
AU - Jean Dellamonica
AU - Alexandre Gérard
AU - Mathieu Jozwiak
PY - 2026
DA - 2026/05/26
JO - JMIR Medical Education
DO - 10.2196/82702
UR - https://hal.science/hal-05804947v1
AB - Background: Large language models (LLMs) have emerged as promising tools in medical education due to their ability to understand, generate, and reason with natural language. Their ability to simulate expert reasoning suggests a potential for supporting quality control in assessment design. In this study, the use of LLMs in identifying ambiguous or poorly constructed exam items in critical care academic assessments was evaluated. Objective: The study aimed to develop automated ambiguity and quality scores to objectively assess individual questions and entire exam components. Methods: We analyzed 264 questions from academic exams conducted over 3 academic years (2023-2025) at the Medical School of Université Côte d'Azur. Questions were drawn from 4 docimological formats: progressive clinical cases (PCC), mini-PCC, key feature problems, and isolated question sequences (IQS). Each element was submitted to 4 LLMs (ChatGPT, Gemini Pro, Le Chat, and DeepSeek) without prompt engineering. Performance was evaluated using the official correction key. We applied 4 binary diagnostic tags based on model agreement and self-reported ambiguity: ambiguity, low performance, incoherence, and subjective ambiguity. These tags generated a composite ambiguity score and contributed to a weighted quality score for each exam component. Results: LLMs achieved mean scores in the same range as students, with no significant differences across academic years and significantly higher performance on the mini-PCC and IQS formats (P=.049 and P=.04, respectively). IQS items had the highest ambiguity scores (54 items received a score of 2 in both 2023 and 2024, and 53 items retained the same score). Tag patterns revealed frequent issues with ambiguity and inconsistency. Quality scores varied across academic years. IQS predominantly showed moderate ambiguity (score 2), with occasional instances of strong signals. There was no significant difference in quality based on author specialty or seniority (P=.08 and P=.44, respectively). Conclusions: In this pilot study, LLMs may offer a preliminary framework to proactively detect ambiguous exam questions and estimate the overall quality of an exam. Integrating these tools into the assessment design process could potentially reduce the need for postexam corrections and may help improve fairness and clarity in medical evaluations.
SN - 2369-3762
ER -