Comparative benchmarking of three artificial intelligence chatbots (ChatGPT-5.1, Qwen- 3 Max, and Perplexity AI) on faculty validated undergraduate respiratory physiology multiple-choice examinations
- 1Laboratory of Physiology, Faculty of Medicine of Tunis, University of Tunis El Manar, Tunis, Tunisia
Abstract
Background: Artificial intelligence (AI) chatbots are increasingly used by medical students for learning and examination preparation. However, their reliability in mechanistically demanding disciplines such as respiratory physiology remains insufficiently evaluated using authentic examination material. This study aimed to assess and compare the performance of three AI chatbots on validated undergraduate respiratory physiology examination questions. Methods: We conducted a cross-sectional analytical study including all respiratory physiology multiple-choice questions (MCQs), each comprising five propositions, used in official undergraduate examinations at the Faculty of Medicine of Tunis during the academic years 2023–2024 and 2024–2025 (101 questions). Three AI chatbots (ChatGPT-5.1, Qwen-3 Max, and Perplexity AI) were evaluated using standardized French-language prompts. Exact question-level concordance with the faculty-validated answer key, established by the teaching staff responsible for respiratory physiology examinations, was compared using Cochran’s Q test and pairwise McNemar tests. At the proposition level, responses were analyzed as binary outcomes (true/false) to compute accuracy, sensitivity, and specificity. Results: Question-level exact concordance differed significantly across models (Cochran’s Q = 10.18, p = 0.006). ChatGPT showed the highest concordance rate (79.2%), followed by Qwen (74.3%) and Perplexity (62.4%). At the proposition level (505 propositions), ChatGPT achieved the highest overall accuracy (94.1%) and sensitivity for true propositions (96.5%), whereas Qwen demonstrated the highest specificity for false propositions (94.2%). Conclusion: Although all evaluated AI chatbots performed well on respiratory physiology MCQs, substantial variability was observed across models. These findings ..( abstract truncated at 250 words).
Related
- Virology-aware artificial intelligence for respiratory antiviral discovery: from molecular docking biases to RdRp-centered benchmarking
- Construct Validity of Frontier Large Language Models on 4 Spanish Health Professions Specialty-Access Examinations (MIR, EIR, PIR, and FIR): Cross-Sectional Benchmarking and Psychometric Study.
- Performance and Hallucination Analysis of Large Language Models on European Anesthesiology Examinations: Cross-Sectional Comparative Study
- The hidden cost of illness: exploring the relationship between seasonal respiratory illness, self-efficacy, and academic achievement of undergraduate science students.
- Assessing the Performance of Artificial Intelligence on Anesthesiology In-Training Examinations and Applicability in Medical Education
BibTeX
@article{kchaou2026comparative,
title = {{Comparative benchmarking of three artificial intelligence chatbots (ChatGPT-5.1, Qwen- 3 Max, and Perplexity AI) on faculty validated undergraduate respiratory physiology multiple-choice examinations}},
author = {Khouloud Kchaou and Salma Mokaddem and Soumaya Khaldi and Yacine Ouahchi and Khadija Ayed and Rym Baati},
journal = {La Tunisie Médicale},
year = {2026},
month = may,
doi = {10.62438/tunismed.v104i05.6620},
url = {https://doi.org/10.62438/tunismed.v104i05.6620},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - JOUR
TI - Comparative benchmarking of three artificial intelligence chatbots (ChatGPT-5.1, Qwen- 3 Max, and Perplexity AI) on faculty validated undergraduate respiratory physiology multiple-choice examinations
AU - Khouloud Kchaou
AU - Salma Mokaddem
AU - Soumaya Khaldi
AU - Yacine Ouahchi
AU - Khadija Ayed
AU - Rym Baati
PY - 2026
DA - 2026/05/24
JO - La Tunisie Médicale
DO - 10.62438/tunismed.v104i05.6620
UR - https://doi.org/10.62438/tunismed.v104i05.6620
AB - Background: Artificial intelligence (AI) chatbots are increasingly used by medical students for learning and examination preparation. However, their reliability in mechanistically demanding disciplines such as respiratory physiology remains insufficiently evaluated using authentic examination material. This study aimed to assess and compare the performance of three AI chatbots on validated undergraduate respiratory physiology examination questions. Methods: We conducted a cross-sectional analytical study including all respiratory physiology multiple-choice questions (MCQs), each comprising five propositions, used in official undergraduate examinations at the Faculty of Medicine of Tunis during the academic years 2023–2024 and 2024–2025 (101 questions). Three AI chatbots (ChatGPT-5.1, Qwen-3 Max, and Perplexity AI) were evaluated using standardized French-language prompts. Exact question-level concordance with the faculty-validated answer key, established by the teaching staff responsible for respiratory physiology examinations, was compared using Cochran’s Q test and pairwise McNemar tests. At the proposition level, responses were analyzed as binary outcomes (true/false) to compute accuracy, sensitivity, and specificity. Results: Question-level exact concordance differed significantly across models (Cochran’s Q = 10.18, p = 0.006). ChatGPT showed the highest concordance rate (79.2%), followed by Qwen (74.3%) and Perplexity (62.4%). At the proposition level (505 propositions), ChatGPT achieved the highest overall accuracy (94.1%) and sensitivity for true propositions (96.5%), whereas Qwen demonstrated the highest specificity for false propositions (94.2%). Conclusion: Although all evaluated AI chatbots performed well on respiratory physiology MCQs, substantial variability was observed across models. These findings ..( abstract truncated at 250 words).
SN - 0041-4131
SN - 2724-7031
ER -