Benchmarking freely accessible large language models for pediatric epilepsy care: a prospective mixed-methods study.
- 1Pediatric Neurology Section, Department of Pediatrics, Faculty of Medicine, King Abdulaziz University, Jeddah, Saudi Arabia.
- 2Department of Paediatric Neurology, Royal London Hospital, Barts Health NHS Trust, London, UK.
Abstract
Large language models (LLMs) may support patient education and clinical documentation, but their accuracy, safety, and usability in pediatric epilepsy remain insufficiently characterized. This study aimed to compare four freely accessible LLMs using standardized pediatric epilepsy scenarios and to evaluate the top-performing model that met the prespecified safety criterion in a supervised clinician workflow assessment. We conducted a prospective, two-phase mixed-methods study at the University Hospital at King Abdulaziz University in Jeddah. In phase 1, four freely accessible LLMs (ChatGPT, Gemini, DeepSeek, and Perplexity) were evaluated using 10 standardized pediatric epilepsy scenarios. Outputs were assessed by pediatric neurologists across factual accuracy, comprehensiveness and relevance, safety, clinical appropriateness, readability and caregiver tailoring, and transparency and sources. Critical safety flags were recorded separately. In phase 2, the model with the highest observed composite score among systems meeting the prespecified safety criterion underwent a supervised clinician workflow assessment using the System Usability Scale (SUS), a caregiver-facing educational-material assessment, and semi-structured interviews. Forty LLM responses were evaluated. Observed composite scores differed significantly across the four model-specific rater groups (P<0.001); however, because each model was evaluated exclusively by a single rater, the observed differences cannot be attributed to model effects independently of evaluator-specific scoring tendencies. DeepSeek and Gemini generated no critical safety flags, whereas ChatGPT had flags in 3/10 responses (30%) and Perplexity in 8/10 (80%). DeepSeek had the highest observed composite score (4.94/5), compared with 4.42/5 for Gemini, 3.59/5 for ChatGPT, and 2.05/5 for Perplexity, and was therefore selected for phase 2. Clinician approval for caregiver use was higher for responses without safety flags than for flagged responses (82.8% vs. 9.1%). In this specialty-specific benchmark, the evaluated LLMs showed substantial differences in observed accuracy and safety, with DeepSeek achieving the highest observed composite score and no critical safety flags. However, the single-rater-per-model design, small sample, model updates, and limited workflow assessment preclude definitive conclusions about comparative model superiority. These findings support specialty-specific evaluation of LLMs that prioritizes safety and clinical accuracy alongside usability. Larger, independently rated, multi-center studies with repeated model sampling and objective workflow and patient-centered outcomes are needed before routine clinical implementation can be considered.
Related
- Mechanism-aware benchmarking of large language models as learning aids in pediatric difficult airway training: true hallucinations vs. false contraindications.
- Epilepsy Surgery vs Medical Management for Pediatric Drug-Resistant Focal Epilepsy
- Editorial: Pediatric epilepsy in the screen society
- Quantitative MRI Characterization of Fluid-Related Phenotypes in Pediatric Epilepsy: Deep Learning-Assisted Assessment of Enlarged Perivascular Space Burden, Regional Distribution, and Clinical Associations
- Serum metabolomics identifies ganglioside dysregulation and pyruvate dehydrogenase complex–related metabolic disturbances in pediatric drug-resistant epilepsy
BibTeX
@article{muthaffar2026benchmarking,
title = {{Benchmarking freely accessible large language models for pediatric epilepsy care: a prospective mixed-methods study.}},
author = {Osama Muthaffar and Anas Alyazidi},
journal = {Translational pediatrics},
year = {2026},
month = sep,
doi = {10.21037/tp-2026-0731},
url = {https://europepmc.org/article/MED/42996738},
note = {CC BY-NC-ND},
}
RIS
TY - JOUR
TI - Benchmarking freely accessible large language models for pediatric epilepsy care: a prospective mixed-methods study.
AU - Osama Muthaffar
AU - Anas Alyazidi
PY - 2026
DA - 2026/09/25
JO - Translational pediatrics
DO - 10.21037/tp-2026-0731
UR - https://europepmc.org/article/MED/42996738
AB - Large language models (LLMs) may support patient education and clinical documentation, but their accuracy, safety, and usability in pediatric epilepsy remain insufficiently characterized. This study aimed to compare four freely accessible LLMs using standardized pediatric epilepsy scenarios and to evaluate the top-performing model that met the prespecified safety criterion in a supervised clinician workflow assessment. We conducted a prospective, two-phase mixed-methods study at the University Hospital at King Abdulaziz University in Jeddah. In phase 1, four freely accessible LLMs (ChatGPT, Gemini, DeepSeek, and Perplexity) were evaluated using 10 standardized pediatric epilepsy scenarios. Outputs were assessed by pediatric neurologists across factual accuracy, comprehensiveness and relevance, safety, clinical appropriateness, readability and caregiver tailoring, and transparency and sources. Critical safety flags were recorded separately. In phase 2, the model with the highest observed composite score among systems meeting the prespecified safety criterion underwent a supervised clinician workflow assessment using the System Usability Scale (SUS), a caregiver-facing educational-material assessment, and semi-structured interviews. Forty LLM responses were evaluated. Observed composite scores differed significantly across the four model-specific rater groups (P<0.001); however, because each model was evaluated exclusively by a single rater, the observed differences cannot be attributed to model effects independently of evaluator-specific scoring tendencies. DeepSeek and Gemini generated no critical safety flags, whereas ChatGPT had flags in 3/10 responses (30%) and Perplexity in 8/10 (80%). DeepSeek had the highest observed composite score (4.94/5), compared with 4.42/5 for Gemini, 3.59/5 for ChatGPT, and 2.05/5 for Perplexity, and was therefore selected for phase 2. Clinician approval for caregiver use was higher for responses without safety flags than for flagged responses (82.8% vs. 9.1%). In this specialty-specific benchmark, the evaluated LLMs showed substantial differences in observed accuracy and safety, with DeepSeek achieving the highest observed composite score and no critical safety flags. However, the single-rater-per-model design, small sample, model updates, and limited workflow assessment preclude definitive conclusions about comparative model superiority. These findings support specialty-specific evaluation of LLMs that prioritizes safety and clinical accuracy alongside usability. Larger, independently rated, multi-center studies with repeated model sampling and objective workflow and patient-centered outcomes are needed before routine clinical implementation can be considered.
SN - 2224-4336
SN - 2224-4344
ER -