Benchmarking Generative AI Chatbots in Preventive Medicine: A Comparative Analysis of Accuracy, Completeness, and Inter-Rater Reliability.
6 affiliations
- 1Preventive Medicine Residency Program, Saudi Commission for Health Specialties, Aseer Health Cluster, Abha 62512, Saudi Arabia.
- 2Department of Family and Community Medicine, College of Medicine, King Khalid University, Abha 61421, Saudi Arabia.
- 3Aseer Health Cluster, Abha 62523, Saudi Arabia.
- 4Preventive Medicine Department, Armed Forces Hospital Southern Region, Ministry of Defense Health Services, Khamis Mushait 62413, Saudi Arabia.
- 5Preventive Medicine Department, Public Health Authority, Abha 62523, Saudi Arabia.
- 6Department of Physical Therapy, Saudi German Hospital, Khamis Mushait 62451, Saudi Arabia.
Abstract
Background/Objectives: Artificial intelligence (AI) chatbots are increasingly used for healthcare information retrieval, education, and clinical support; however, concerns remain regarding the accuracy and completeness of AI-generated information in preventive medicine, where evidence-based recommendations, epidemiological interpretation, risk assessment, and public health guidance require reliable responses. Comparative evidence across multiple preventive medicine domains remains limited, particularly using standardized expert-validated reference answers. This study aimed to evaluate and compare the accuracy and completeness of responses generated by five AI chatbots across major preventive medicine domains. Methods: A comparative cross-sectional benchmarking study evaluated ChatGPT, DeepSeek, Perplexity AI, Claude AI, and Gemini Advanced using 15 expert-validated questions covering General Preventive Medicine, Clinical Preventive Medicine, Epidemiology and Biostatistics, Occupational Medicine, and Social and Behavioral Science. Responses were independently evaluated by two investigators using predefined accuracy and completeness scoring frameworks. Between-model comparisons were conducted using mixed-effects models accounting for clustering by question, and inter-rater reliability was assessed using agreement statistics. Results: Claude AI achieved the highest overall accuracy (5.67 ± 0.41) and completeness (2.87 ± 0.35), followed by ChatGPT (accuracy, 5.47 ± 0.66; completeness, 2.67 ± 0.49), whereas Gemini Advanced demonstrated the lowest overall performance. Epidemiology and Biostatistics showed the highest pooled completeness score (2.60; 95% CI: 2.32-2.88). Mixed-effects analysis accounting for clustering by question demonstrated a significant overall difference in accuracy among chatbot models (p < 0.001), while inter-rater agreement was high, supporting the consistency of the scoring procedure. Conclusions: Substantial variability in accuracy and completeness was observed among the evaluated AI chatbots, with Claude AI and ChatGPT demonstrating comparatively stronger performance within this specific question set. Given the limited number of unique questions and the continuously evolving nature of AI models, these findings should be considered exploratory and should not be generalized as definitive evidence of model superiority or clinical utility. Larger studies using more diverse question sets and standardized evaluation frameworks are required to confirm these findings.
Related
- Reliability, Readability, and Safety of Generative AI Chatbots for Osteoporosis Health Counseling: A Multidimensional Cross‐Sectional Comparative Study
- Quality indicators, reliability, transparency and content completeness of short-video information on traditional Chinese medicine for insomnia across three major Chinese platforms: a cross-sectional study
- Multidimensional evaluation of generative AI chatbots for public consultation on sudden cardiac death: safety, accuracy, empathy, reliability, quality, and readability across six models
- Comparative effectiveness of preventive strategies against medically-attended respiratory syncytial virus in U.S. infants during the first six months of life, 2023-2025
- Benchmarking the performance of large language models on medical genetics questions: a comparative evaluation of GPT-5.1, Gemini Pro 3, and Claude Opus 4.5.
BibTeX
@article{alshahrani2026benchmarking,
title = {{Benchmarking Generative AI Chatbots in Preventive Medicine: A Comparative Analysis of Accuracy, Completeness, and Inter-Rater Reliability.}},
author = {Nojoud Zoraib Alshahrani and Abdullah Alsabaani and Ali Hassan Almaqsudi and Ali Alshahrani and Naif Moshabub Mosfer Alqahtani and Mazen Abdullah N Alshahrani},
journal = {Healthcare (Basel, Switzerland)},
year = {2026},
month = oct,
doi = {10.3390/healthcare14193330},
url = {https://europepmc.org/article/MED/42881769},
note = {CC BY},
}
RIS
TY - JOUR
TI - Benchmarking Generative AI Chatbots in Preventive Medicine: A Comparative Analysis of Accuracy, Completeness, and Inter-Rater Reliability.
AU - Nojoud Zoraib Alshahrani
AU - Abdullah Alsabaani
AU - Ali Hassan Almaqsudi
AU - Ali Alshahrani
AU - Naif Moshabub Mosfer Alqahtani
AU - Mazen Abdullah N Alshahrani
PY - 2026
DA - 2026/10/06
JO - Healthcare (Basel, Switzerland)
DO - 10.3390/healthcare14193330
UR - https://europepmc.org/article/MED/42881769
AB - Background/Objectives: Artificial intelligence (AI) chatbots are increasingly used for healthcare information retrieval, education, and clinical support; however, concerns remain regarding the accuracy and completeness of AI-generated information in preventive medicine, where evidence-based recommendations, epidemiological interpretation, risk assessment, and public health guidance require reliable responses. Comparative evidence across multiple preventive medicine domains remains limited, particularly using standardized expert-validated reference answers. This study aimed to evaluate and compare the accuracy and completeness of responses generated by five AI chatbots across major preventive medicine domains. Methods: A comparative cross-sectional benchmarking study evaluated ChatGPT, DeepSeek, Perplexity AI, Claude AI, and Gemini Advanced using 15 expert-validated questions covering General Preventive Medicine, Clinical Preventive Medicine, Epidemiology and Biostatistics, Occupational Medicine, and Social and Behavioral Science. Responses were independently evaluated by two investigators using predefined accuracy and completeness scoring frameworks. Between-model comparisons were conducted using mixed-effects models accounting for clustering by question, and inter-rater reliability was assessed using agreement statistics. Results: Claude AI achieved the highest overall accuracy (5.67 ± 0.41) and completeness (2.87 ± 0.35), followed by ChatGPT (accuracy, 5.47 ± 0.66; completeness, 2.67 ± 0.49), whereas Gemini Advanced demonstrated the lowest overall performance. Epidemiology and Biostatistics showed the highest pooled completeness score (2.60; 95% CI: 2.32-2.88). Mixed-effects analysis accounting for clustering by question demonstrated a significant overall difference in accuracy among chatbot models (p < 0.001), while inter-rater agreement was high, supporting the consistency of the scoring procedure. Conclusions: Substantial variability in accuracy and completeness was observed among the evaluated AI chatbots, with Claude AI and ChatGPT demonstrating comparatively stronger performance within this specific question set. Given the limited number of unique questions and the continuously evolving nature of AI models, these findings should be considered exploratory and should not be generalized as definitive evidence of model superiority or clinical utility. Larger studies using more diverse question sets and standardized evaluation frameworks are required to confirm these findings.
SN - 2227-9032
ER -