Evaluating Large Language Model Platforms for Medical Nutrition Therapy in Autism Spectrum Disorder: Quality, Guideline Concordance, Readability, and Hallucinations

@article{ozkalaycı2026evaluating,
  title = {{Evaluating Large Language Model Platforms for Medical Nutrition Therapy in Autism Spectrum Disorder: Quality, Guideline Concordance, Readability, and Hallucinations}},
  author = {Rüveyda Esra Özkalaycı and Gözde Ede İleri and Şeyma Nur Ercan and Yeliz Serin and Uğur Binzat and Sedat Arslan},
  journal = {Journal of Clinical Medicine},
  year = {2026},
  month = nov,
  doi = {10.3390/jcm15228770},
  url = {https://www.mdpi.com/2077-0383/15/22/8770},
  note = {CC BY 4.0},
}

TY  - JOUR
TI  - Evaluating Large Language Model Platforms for Medical Nutrition Therapy in Autism Spectrum Disorder: Quality, Guideline Concordance, Readability, and Hallucinations
AU  - Rüveyda Esra Özkalaycı
AU  - Gözde Ede İleri
AU  - Şeyma Nur Ercan
AU  - Yeliz Serin
AU  - Uğur Binzat
AU  - Sedat Arslan
PY  - 2026
DA  - 2026/11/10
JO  - Journal of Clinical Medicine
DO  - 10.3390/jcm15228770
UR  - https://www.mdpi.com/2077-0383/15/22/8770
AB  - Background/Objectives: Children with autism spectrum disorder (ASD) commonly experience feeding- and nutrition-related problems, while families increasingly encounter large language model (LLM)-based tools as sources of health information. Evidence on the reliability of these tools for ASD-specific medical nutrition therapy (MNT) remains limited. The aim of this study is to compare the responses of four publicly available LLM-based platform configurations to nutrition questions related to ASD in terms of quality, guideline compliance, readability, and hallucination. Methods: In this cross-sectional comparative benchmark study, 60 standardized, unbiased informational questions across five MNT domains (12 per domain) were each submitted once in separate sessions to four LLM-based platform configurations: ChatGPT, ChatGPT Plus, Microsoft Copilot, and Google Gemini, yielding 240 responses. Only one response was evaluated for each question-platform combination, and variability between repeated productions was not measured. Four nutrition and dietetics experts, blinded to platform identity, assessed response quality using Medical Artificial Intelligence Quality Analysis (QAMAI) and Global Quality Score (GQS), and reviewed compliance with NICE guidelines and the American Academy of Pediatrics clinical report, and evaluated readability and binary hallucination classifications. QAMAI, GQS, and guideline-concordance scores were compared using generalized estimating equations with a Gamma distribution and a log link. Results: Platform, domain (D), and platform × domain effects were significant for QAMAI, GQS, and guideline concordance (all omnibus p < 0.05). ChatGPT Plus had the highest overall mean QAMAI, GQS, and guideline-concordance scores. Gemini showed relatively high quality ratings but the lowest overall concordance with guidelines. In the analyzed response set, no hallucinations were detected in either the ChatGPT or ChatGPT Plus configurations (0/60 each; 0%). Hallucinations were detected in 21/60 (35.0%) of the Copilot responses and in 38/60 (63.3%) of the Gemini responses. Overall platform comparisons of hallucination classifications across the five domains were significant in all five domains after false discovery rate correction; pairwise differences remained significant in D1, D2, and D5, but not in D3 or D4. Mean Flesch–Kincaid Grade Level scores ranged from 14.62 to 18.87 across platforms. Conclusions: In this standardized single-generation comparison, overall response quality, guideline compliance, readability, and findings on hallucinations did not always coincide. Observed hallucination rates are limited to the response set examined. The observation of zero events in two ChatGPT configurations does not indicate the absence of hallucination risk, nor does it provide a basis for differentiating the configurations or concluding that they are equivalent in this respect. The binary hallucination rates did not reflect error severity and therefore should not be used to compare the clinical risks of the platforms. The findings are limited to a controlled set of questions and cannot be generalized to emotional, directive, or multi-round caregiver interactions. The ASD nutritional information generated by the LLM should be evaluated multidimensionally and subjected to expert validation. Studies involving repeated independent response generation in each question-platform combination are necessary for comparing hallucination tendencies.
SN  - 2077-0383
ER  -