Comparative Evaluation of AI Chatbots for Medical Research: Cross-Sectional Observational Study
- 1Public Health Laboratory for the Analysis of Community Health Needs, Department of Medicine and Surgery, University of Salerno, Baronissi, Campania, Italy
- 2Department of Medicine and Surgery, University of Salerno, Baronissi Campus, Baronissi, Campania, Italy, 39 3385430632
- 3Department of Political and Social Studies, University of Salerno, Fisciano, Campania, Italy
Abstract
Background: The use of generative AI in research and clinical practice is rapidly expanding, offering unprecedented capabilities in pattern extraction and data analysis. However, its application raises critical concerns regarding the accuracy and reliability of the information provided.
Objective: The aim of this study is to analyze and compare the performance of 4 generative AI platforms (ChatGPT, Grok, Meta AI, and Perplexity) in retrieving bibliographic references within the medical and health care field and to assess factual accuracy and information retrieval strategies.
Methods: A cross-sectional observational study was conducted based on the simulation of 3 bibliographic queries with varying levels of specificity (aflatoxin toxicity, hospital sanitization, and human papillomavirus vaccination). Each prompt was administered in triple iteration on each platform operating in “Guest” mode. Factual accuracy was verified through a manual cross-check against the Scopus and Google Scholar databases using a hierarchical query strategy prioritized by DOI. The incidence of hallucinations was quantified via a standardized scoring system (range 0‐5) based on metadata correctness. Categorical associations were analyzed using the Pearson chi-square test, while nonparametric continuous variables (hallucination score, publication year, and citation count) were compared using the Kruskal-Wallis test.
Results: Of the 360 requested references, 273 were generated (task completion rate: 75.8%). Among these, 190 adhered to the requested article format (instruction adherence rate: 69.6% of generated outputs). Fact-checking revealed that 128 of these 190 validly formatted articles (metadata accuracy rate: 67.4%) showed an exact match with real evidence, resulting in an overall strict yield of 35.6% (128/360) from the initial requested references. The ability to provide real references depended significantly on the AI model used (χ6²=192.5; P<.001). Kruskal-Wallis tests revealed statistically significant differences among the models across all analyzed dimensions (P<.001). Grok emerged as the most reliable model, achieving 71 out of 90 (78.9%) exact matches, exhibiting minimal hallucinations (mean score 0.2, SD 0.5), and showing a marked propensity toward recent sources (median year 2022, IQR 2019.0-2024.8). ChatGPT demonstrated robust stability, with 46 out of 90 (51.1%) exact matches, and anchored itself to older (median year 2020, IQR 2011.0-2023.0), highly cited literature (median 73.5, IQR 10.8‐718.0 citations). Perplexity exhibited critical rates of total hallucination and a structural inability to distinguish scientific articles from gray literature. Meta AI systematically failed the task, producing entirely incorrect data on a single occasion (0/10, 0% exact matches).
Conclusions: Under the specific testing conditions of this study, the evaluated large language models demonstrated a lack of the necessary reliability to replace validated bibliographic databases, primarily due to response instability and a high incidence of metadata fabrication. The results highlight the urgency of introducing structured AI literacy programs across medical education, promoting the use of such technologies exclusively as a complementary tool and always subject to rigorous human fact-checking.
Related
- Suicide-Related Responses From AI Chatbots Through Application Programming Interfaces and Consumer-Facing Apps: Observational Cross-Sectional Comparative Study.
- Quality, Readability, Understandability, and Actionability of AI Chatbots’ Responses to Dental Patient Questions: A Comparative Evaluation
- Reliability, Readability, and Safety of Generative AI Chatbots for Osteoporosis Health Counseling: A Multidimensional Cross‐Sectional Comparative Study
- Evaluating AI chatbots as sources of public-facing encephalitis health information: a cross-sectional comparative study of quality, transparency, and readability
- Psychological support for depression and anxiety across chronic illness contexts: a comparative evaluation of three general-purpose AI chatbots
BibTeX
@article{caro2026comparative,
title = {{Comparative Evaluation of AI Chatbots for Medical Research: Cross-Sectional Observational Study}},
author = {Francesco De Caro and Federico Della Rocca and Alessandra Mea and Matteo Tomeo and Simona Caruccio and Myriam Baldi and Giuseppe D'Arienzo and Raffaele De Caro and Emanuela Santoro and Giuseppina Cersosimo and Giovanni Boccia and Mario Capunzo and Giuseppina Moccia},
journal = {JMIR Formative Research},
year = {2026},
month = nov,
doi = {10.2196/103592},
url = {https://doi.org/10.2196/103592},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Comparative Evaluation of AI Chatbots for Medical Research: Cross-Sectional Observational Study
AU - Francesco De Caro
AU - Federico Della Rocca
AU - Alessandra Mea
AU - Matteo Tomeo
AU - Simona Caruccio
AU - Myriam Baldi
AU - Giuseppe D'Arienzo
AU - Raffaele De Caro
AU - Emanuela Santoro
AU - Giuseppina Cersosimo
AU - Giovanni Boccia
AU - Mario Capunzo
AU - Giuseppina Moccia
PY - 2026
DA - 2026/11/03
JO - JMIR Formative Research
DO - 10.2196/103592
UR - https://doi.org/10.2196/103592
AB - Background: The use of generative AI in research and clinical practice is rapidly expanding, offering unprecedented capabilities in pattern extraction and data analysis. However, its application raises critical concerns regarding the accuracy and reliability of the information provided. Objective: The aim of this study is to analyze and compare the performance of 4 generative AI platforms (ChatGPT, Grok, Meta AI, and Perplexity) in retrieving bibliographic references within the medical and health care field and to assess factual accuracy and information retrieval strategies. Methods: A cross-sectional observational study was conducted based on the simulation of 3 bibliographic queries with varying levels of specificity (aflatoxin toxicity, hospital sanitization, and human papillomavirus vaccination). Each prompt was administered in triple iteration on each platform operating in “Guest” mode. Factual accuracy was verified through a manual cross-check against the Scopus and Google Scholar databases using a hierarchical query strategy prioritized by DOI. The incidence of hallucinations was quantified via a standardized scoring system (range 0‐5) based on metadata correctness. Categorical associations were analyzed using the Pearson chi-square test, while nonparametric continuous variables (hallucination score, publication year, and citation count) were compared using the Kruskal-Wallis test. Results: Of the 360 requested references, 273 were generated (task completion rate: 75.8%). Among these, 190 adhered to the requested article format (instruction adherence rate: 69.6% of generated outputs). Fact-checking revealed that 128 of these 190 validly formatted articles (metadata accuracy rate: 67.4%) showed an exact match with real evidence, resulting in an overall strict yield of 35.6% (128/360) from the initial requested references. The ability to provide real references depended significantly on the AI model used (χ6²=192.5; P<.001). Kruskal-Wallis tests revealed statistically significant differences among the models across all analyzed dimensions (P<.001). Grok emerged as the most reliable model, achieving 71 out of 90 (78.9%) exact matches, exhibiting minimal hallucinations (mean score 0.2, SD 0.5), and showing a marked propensity toward recent sources (median year 2022, IQR 2019.0-2024.8). ChatGPT demonstrated robust stability, with 46 out of 90 (51.1%) exact matches, and anchored itself to older (median year 2020, IQR 2011.0-2023.0), highly cited literature (median 73.5, IQR 10.8‐718.0 citations). Perplexity exhibited critical rates of total hallucination and a structural inability to distinguish scientific articles from gray literature. Meta AI systematically failed the task, producing entirely incorrect data on a single occasion (0/10, 0% exact matches). Conclusions: Under the specific testing conditions of this study, the evaluated large language models demonstrated a lack of the necessary reliability to replace validated bibliographic databases, primarily due to response instability and a high incidence of metadata fabrication. The results highlight the urgency of introducing structured AI literacy programs across medical education, promoting the use of such technologies exclusively as a complementary tool and always subject to rigorous human fact-checking.
SN - 2561-326X
ER -