Evaluating and leveraging large language models in clinical pharmacology and therapeutics assessment: From exam takers to exam shapers
Abstract
Aims: In medical education, the ability of large language models (LLMs) to match human performance raises questions about their potential as educational tools. This study evaluates LLMs' performance on Clinical Pharmacology and Therapeutics (CPT) exams, comparing their results to medical students and exploring their ability to identify poorly formulated multiple‐choice questions (MCQs). Methods: ChatGPT‐4 Omni, Gemini Advanced, Le Chat and DeepSeek R1 were tested on local CPT exams (third year of bachelor's degree, first/second year of master's degree) and the European Prescribing Exam (EuroPE + ). The exams included MCQs and open‐ended questions assessing knowledge and prescribing skills. LLM results were analysed using the same scoring system as students. A confusion matrix was used to evaluate the ability of ChatGPT and Gemini to identify ambiguous/erroneous MCQs. Results: LLMs achieved comparable or superior results to medical students across all levels. For local exams, LLMs outperformed M1 students and matched L3 and M2 students. In EuroPE + , LLMs significantly outperformed students in both the knowledge and prescribing skills sections. All LLM errors in EuroPE + were genuine (100%), whereas local exam errors were frequently due to ambiguities or correction flaws (24.3%). When both ChatGPT and Gemini provided the same incorrect answer to an MCQ, the specificity for detecting ambiguous questions was 92.9%, with a negative predictive value of 85.5%. Conclusion: LLMs demonstrate capabilities comparable to or exceeding medical students in CPT exams. Their ability to flag potentially flawed MCQs highlights their value not only as educational tools but also as quality control instruments in exam preparation.
Related
- Informed forecasting: Leveraging auxiliary knowledge to boost large language model performance on time series forecasting
- Evaluating Modern Tools and Large Language Models for Digitising Sea-Level Data from Marigrams
- Sino-US-DrugQA: A benchmark for evaluating large language models in cross-jurisdictional pharmaceutical regulation
- Simultaneous multi-task assessment of five neurological language disorders via cross-corpus multimodal learning
- National assessment of trauma care system capacity in Ecuador: Evaluating institutional resources and practitioner KAPS domains
BibTeX
@article{gerard2026evaluating,
title = {{Evaluating and leveraging large language models in clinical pharmacology and therapeutics assessment: From exam takers to exam shapers}},
author = {Alexandre Gérard and Diane Merino and Marc Labriffe and Fanny Rocher and Delphine Viard and Laurence Zemori and Thibaud Lavrut and Erik Donker and Joost Piët and Jean‐paul Fournier and Milou‐daniel Drici and Alexandre Destere},
journal = {British Journal of Clinical Pharmacology},
year = {2026},
month = oct,
doi = {10.1002/bcp.70137},
url = {https://hal.science/hal-05155988v1},
note = {CC BY-NC 4.0},
}
RIS
TY - JOUR
TI - Evaluating and leveraging large language models in clinical pharmacology and therapeutics assessment: From exam takers to exam shapers
AU - Alexandre Gérard
AU - Diane Merino
AU - Marc Labriffe
AU - Fanny Rocher
AU - Delphine Viard
AU - Laurence Zemori
AU - Thibaud Lavrut
AU - Erik Donker
AU - Joost Piët
AU - Jean‐paul Fournier
AU - Milou‐daniel Drici
AU - Alexandre Destere
PY - 2026
DA - 2026/10/19
JO - British Journal of Clinical Pharmacology
DO - 10.1002/bcp.70137
UR - https://hal.science/hal-05155988v1
AB - Aims: In medical education, the ability of large language models (LLMs) to match human performance raises questions about their potential as educational tools. This study evaluates LLMs' performance on Clinical Pharmacology and Therapeutics (CPT) exams, comparing their results to medical students and exploring their ability to identify poorly formulated multiple‐choice questions (MCQs). Methods: ChatGPT‐4 Omni, Gemini Advanced, Le Chat and DeepSeek R1 were tested on local CPT exams (third year of bachelor's degree, first/second year of master's degree) and the European Prescribing Exam (EuroPE + ). The exams included MCQs and open‐ended questions assessing knowledge and prescribing skills. LLM results were analysed using the same scoring system as students. A confusion matrix was used to evaluate the ability of ChatGPT and Gemini to identify ambiguous/erroneous MCQs. Results: LLMs achieved comparable or superior results to medical students across all levels. For local exams, LLMs outperformed M1 students and matched L3 and M2 students. In EuroPE + , LLMs significantly outperformed students in both the knowledge and prescribing skills sections. All LLM errors in EuroPE + were genuine (100%), whereas local exam errors were frequently due to ambiguities or correction flaws (24.3%). When both ChatGPT and Gemini provided the same incorrect answer to an MCQ, the specificity for detecting ambiguous questions was 92.9%, with a negative predictive value of 85.5%. Conclusion: LLMs demonstrate capabilities comparable to or exceeding medical students in CPT exams. Their ability to flag potentially flawed MCQs highlights their value not only as educational tools but also as quality control instruments in exam preparation.
SN - 0306-5251
SN - 1365-2125
ER -