Benchmarking the performance of large language models on medical genetics questions: a comparative evaluation of GPT-5.1, Gemini Pro 3, and Claude Opus 4.5.
- 1Department of Medical Genetics, Konya City Hospital, Konya, Türkiye.
- 2Department of Dentomaxillofacial Radiology, Faculty of Dentistry, Necmettin Erbakan University, Konya, Türkiye.
Abstract
Objective
This study benchmarks the performance of three state of the art large reasoning models comprising GPT 5.1, Gemini Pro 3, and Claude Opus 4.5 by evaluating their accuracy, response consistency, processing speed, and output length on medical genetics questions from the Turkish Medical Specialty Examination (TUS).Methods
A total of 233 medical genetics questions were selected based on predefined inclusion criteria covering molecular mechanisms, inheritance patterns, genetic disorders, and diagnostic techniques. Each question was independently submitted to GPT 5.1, Gemini Pro 3, and Claude Opus 4.5 using a standardized prompt. Model accuracy was compared using McNemar's test. Response consistency was assessed using Cohen's Kappa. Response time and word count were analyzed using the Friedman test with Dunn Bonferroni post hoc comparisons.Results
No statistically significant differences were observed in overall accuracy among the models (p > 0.05). GPT 5.1 achieved the highest accuracy at 97.9 percent followed by Claude Opus 4.5 at 96.6 percent and Gemini Pro 3 at 96.1 percent. Cohen's Kappa indicated moderate qualitative divergence between Gemini Pro 3 and Claude Opus 4.5 with kappa equal to 0.451. Significant differences were found in response time and output length (p < 0.001).Conclusion
Although all models demonstrate similarly high accuracy, they differ in operational behavior, suggesting distinct internal knowledge representations and response strategies.
Related
- Do question type and topic affect the performance of large language models in answering periodontology questions? A comparative study
- Performance evaluation and benchmarking across 16 large language model configurations on a large real-world emergency department triage data set
- Benchmarking ten frontier large language models on 1,477 board style multiple choice questions in hematology
- How well do large language models answer postoperative lumbar fusion questions? A blinded comparative analysis of accuracy, quality, readability, and safety-related content
- Comparison of the performance of large language models (ChatGPT-5.5, Gemini 3.1, and Claude 4.6) in answering pharmacology questions
BibTeX
@article{basdemirci2026benchmarking,
title = {{Benchmarking the performance of large language models on medical genetics questions: a comparative evaluation of GPT-5.1, Gemini Pro 3, and Claude Opus 4.5.}},
author = {Muserref Basdemirci and Melek Tassoker},
journal = {Frontiers in medicine},
year = {2026},
month = oct,
doi = {10.3389/fmed.2026.1946188},
url = {https://europepmc.org/article/MED/42947984},
note = {CC BY},
}
RIS
TY - JOUR
TI - Benchmarking the performance of large language models on medical genetics questions: a comparative evaluation of GPT-5.1, Gemini Pro 3, and Claude Opus 4.5.
AU - Muserref Basdemirci
AU - Melek Tassoker
PY - 2026
DA - 2026/10/13
JO - Frontiers in medicine
DO - 10.3389/fmed.2026.1946188
UR - https://europepmc.org/article/MED/42947984
AB - Objective This study benchmarks the performance of three state of the art large reasoning models comprising GPT 5.1, Gemini Pro 3, and Claude Opus 4.5 by evaluating their accuracy, response consistency, processing speed, and output length on medical genetics questions from the Turkish Medical Specialty Examination (TUS).Methods A total of 233 medical genetics questions were selected based on predefined inclusion criteria covering molecular mechanisms, inheritance patterns, genetic disorders, and diagnostic techniques. Each question was independently submitted to GPT 5.1, Gemini Pro 3, and Claude Opus 4.5 using a standardized prompt. Model accuracy was compared using McNemar's test. Response consistency was assessed using Cohen's Kappa. Response time and word count were analyzed using the Friedman test with Dunn Bonferroni post hoc comparisons.Results No statistically significant differences were observed in overall accuracy among the models (p > 0.05). GPT 5.1 achieved the highest accuracy at 97.9 percent followed by Claude Opus 4.5 at 96.6 percent and Gemini Pro 3 at 96.1 percent. Cohen's Kappa indicated moderate qualitative divergence between Gemini Pro 3 and Claude Opus 4.5 with kappa equal to 0.451. Significant differences were found in response time and output length (p < 0.001).Conclusion Although all models demonstrate similarly high accuracy, they differ in operational behavior, suggesting distinct internal knowledge representations and response strategies.
SN - 2296-858X
ER -