Psychometric characterization of human and artificial intelligence performance on cardiology residency in-service examination items
- 1Department of Cardiology, Katip Celebi University, Medicine Faculty, Izmir, Türkiye
Abstract
Large language models (LLMs) are increasingly evaluated using medical examination datasets, yet most studies emphasize overall accuracy rather than the psychometric structure of test items. We evaluated five LLMs on 199 text-only cardiology residency in-service examination items previously characterized using resident-derived psychometric metrics. Three frontier models were compared with two open-source comparators using a standardized zero-shot, repeated-query protocol and strict-majority scoring. Frontier models achieved substantially higher accuracy than open-source comparators, with Claude Opus 4.6, Gemini 3.1 Flash-Lite, and GPT-5.4 reaching 86.4%, 82.9%, and 81.9%, respectively, compared with 53.3% for MedQwen and 18.6% for Qwen-3.5-35B. Across frontier models, performance increased progressively from hard to easy resident-derived item strata. In multivariable analyses, item difficulty was the only classical psychometric factor consistently associated with AI correctness. IRT-based re-analysis confirmed that higher latent item difficulty was independently associated with lower frontier-model accuracy. Human–AI item-level correlations were modest but exceeded permutation-based null expectations, and frontier-model errors were concentrated among highly ranked human distractors. These findings show that item-level psychometric analysis helps explain variation in frontier LLM performance beyond overall accuracy alone. However, expert-rated rationale assessment and none-of-the-above perturbation testing revealed that strong examination accuracy did not guarantee high-quality explanatory support or reliable recognition of answer absence, indicating that examination performance and answer-selection robustness represent related but distinct dimensions of model behavior.
Related
- Toward integrated multimodal artificial intelligence in bone marrow examination: current state and open challenges
- Psychometric applications of generative artificial intelligence: Lifecycle, risks, and research agenda
- Artificial Intelligence-Enhanced Cardiac Magnetic Resonance in Hypertrophic Cardiomyopathy: Diagnosis, Phenotypic Characterization, and Risk Prediction
- Artificial Intelligence in Acute Coronary Syndromes: From Early Diagnosis to Culprit Lesion Characterization, Coronary Physiology, and Risk Stratification
- CT-Referenced Diagnostic Performance and Complementary Error Patterns of Artificial Intelligence Versus a Single Emergency Physician in Chest Radiograph Interpretation: A Retrospective Paired Diagnostic Accuracy Study
BibTeX
@article{celik2026psychometric,
title = {{Psychometric characterization of human and artificial intelligence performance on cardiology residency in-service examination items}},
author = {Aykan Çelik and Tuncay Kırış and Uğur Kocabaş and Emre Özdemir and Mustafa Karaca},
journal = {NPJ Digital Medicine},
year = {2026},
month = jul,
doi = {10.1038/s41746-026-03002-9},
url = {https://doi.org/10.1038/s41746-026-03002-9},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - JOUR
TI - Psychometric characterization of human and artificial intelligence performance on cardiology residency in-service examination items
AU - Aykan Çelik
AU - Tuncay Kırış
AU - Uğur Kocabaş
AU - Emre Özdemir
AU - Mustafa Karaca
PY - 2026
DA - 2026/07/10
JO - NPJ Digital Medicine
DO - 10.1038/s41746-026-03002-9
UR - https://doi.org/10.1038/s41746-026-03002-9
AB - Large language models (LLMs) are increasingly evaluated using medical examination datasets, yet most studies emphasize overall accuracy rather than the psychometric structure of test items. We evaluated five LLMs on 199 text-only cardiology residency in-service examination items previously characterized using resident-derived psychometric metrics. Three frontier models were compared with two open-source comparators using a standardized zero-shot, repeated-query protocol and strict-majority scoring. Frontier models achieved substantially higher accuracy than open-source comparators, with Claude Opus 4.6, Gemini 3.1 Flash-Lite, and GPT-5.4 reaching 86.4%, 82.9%, and 81.9%, respectively, compared with 53.3% for MedQwen and 18.6% for Qwen-3.5-35B. Across frontier models, performance increased progressively from hard to easy resident-derived item strata. In multivariable analyses, item difficulty was the only classical psychometric factor consistently associated with AI correctness. IRT-based re-analysis confirmed that higher latent item difficulty was independently associated with lower frontier-model accuracy. Human–AI item-level correlations were modest but exceeded permutation-based null expectations, and frontier-model errors were concentrated among highly ranked human distractors. These findings show that item-level psychometric analysis helps explain variation in frontier LLM performance beyond overall accuracy alone. However, expert-rated rationale assessment and none-of-the-above perturbation testing revealed that strong examination accuracy did not guarantee high-quality explanatory support or reliable recognition of answer absence, indicating that examination performance and answer-selection robustness represent related but distinct dimensions of model behavior.
SN - 2398-6352
ER -