Clinical evaluation of AI Chatbots' performance and safety for total parenteral nutrition planning in critically ill patients: a simulation-based benchmarking study.
- 1Department of Pharmacy Practice, College of Clinical Pharmacy, King Faisal University, Al-Ahsaa, Saudi Arabia.
Abstract
Introduction
Total parenteral nutrition (TPN) is essential for critically ill adults unable to meet their nutritional needs via the gastrointestinal tract. Recent advancements in large language models (LLMs) have sparked interest in their potential to support healthcare providers in complex clinical decisions. Nonetheless, no prior research has systematically compared publicly accessible chatbots to pharmacist-developed reference TPN plans across a diverse patient cohort, using a predefined, guideline-based rubric assessing accuracy, completeness, safety, error types, and numerical precision. This study presents a simulation-based benchmarking to evaluate (i) the accuracy and comprehensiveness of chatbot-generated TPN plans, (ii) their safety, acceptability, and error profiles, and (iii) the numerical agreement and concordance with expert pharmacist references.Methods
This benchmarking study, based on simulations and cross-sectional analysis, evaluated the performance of four publicly accessible AI chatbots-ChatGPT, Gemini, Copilot, and Perplexity-in formulating TPN plans for critically ill adults. Copilot declined to generate patient-specific TPN plans; this refusal was retained as a performance outcome, while Copilot was excluded from analyses requiring an analyzable TPN prescription. Each chatbot produced one plan for each of 10 standardized clinical cases, resulting in a total of 30 TPN plans. Friedman tests, Kendall's W, and Holm-adjusted Wilcoxon signed-rank tests were used for comparative analyses.Results
ChatGPT produced comprehensive plans for all 10 cases, compared to 6 plans from Gemini and 4 from Perplexity. The median accuracy was 57.7% for ChatGPT, 43.9% for Gemini, and 38.5% for Perplexity, with no statistically significant overall difference (p = 0.124). However, ChatGPT performed better than Perplexity in pairwise comparison (adjusted p = 0.035). There was a significant difference in completeness among the chatbots (p = 0.003; Kendall's W = 0.57), with median scores of 64.6, 41.9, and 32.3%, respectively. All 30 plans had at least one major or critical safety issue, 7 (23.3%) contained a critical concern, and 19 (63.3%) were rejected. The most common errors were related to electrolytes (90.0%), micronutrients (70.0%), and monitoring (66.7%). Recommendations from chatbots often exceeded pharmacist-derived values, with median relative differences of +47% for energy, +120% for lipids, and +100% for potassium. Bland-Altman analysis revealed considerable variability in the agreement.Conclusion
ChatGPT produced the most comprehensive TPN plans among chatbots. However, none showed enough accuracy, safety, or concordance for independent use. All plans had critical issues, especially with electrolytes, micronutrients, monitoring, and refeeding precautions. AI chatbots can support education or planning but require review by a well-trained clinician before clinical use.
Related
- A novel meropenem dosing strategy for Outpatient Parenteral Antimicrobial Therapy (OPAT): a pilot pharmacokinetic/pharmacodynamic study in non-critically ill patients
- Benchmarking the performance of large language models on medical genetics questions: a comparative evaluation of GPT-5.1, Gemini Pro 3, and Claude Opus 4.5.
- Performance evaluation and benchmarking across 16 large language model configurations on a large real-world emergency department triage data set
- External validation and comparative performance of four risk scores for acute kidney injury in critically ill adults
- LDAR shows more consistent prognostic performance than other albumin-derived indices for 28-day ICU mortality in critically ill patients with urosepsis: evidence from a dual-cohort retrospective study
BibTeX
@article{alomair2026clinical,
title = {{Clinical evaluation of AI Chatbots' performance and safety for total parenteral nutrition planning in critically ill patients: a simulation-based benchmarking study.}},
author = {Sufyan Alomair},
journal = {Frontiers in nutrition},
year = {2026},
month = oct,
doi = {10.3389/fnut.2026.1969789},
url = {https://europepmc.org/article/MED/42903103},
note = {CC BY},
}
RIS
TY - JOUR
TI - Clinical evaluation of AI Chatbots' performance and safety for total parenteral nutrition planning in critically ill patients: a simulation-based benchmarking study.
AU - Sufyan Alomair
PY - 2026
DA - 2026/10/06
JO - Frontiers in nutrition
DO - 10.3389/fnut.2026.1969789
UR - https://europepmc.org/article/MED/42903103
AB - Introduction Total parenteral nutrition (TPN) is essential for critically ill adults unable to meet their nutritional needs via the gastrointestinal tract. Recent advancements in large language models (LLMs) have sparked interest in their potential to support healthcare providers in complex clinical decisions. Nonetheless, no prior research has systematically compared publicly accessible chatbots to pharmacist-developed reference TPN plans across a diverse patient cohort, using a predefined, guideline-based rubric assessing accuracy, completeness, safety, error types, and numerical precision. This study presents a simulation-based benchmarking to evaluate (i) the accuracy and comprehensiveness of chatbot-generated TPN plans, (ii) their safety, acceptability, and error profiles, and (iii) the numerical agreement and concordance with expert pharmacist references.Methods This benchmarking study, based on simulations and cross-sectional analysis, evaluated the performance of four publicly accessible AI chatbots-ChatGPT, Gemini, Copilot, and Perplexity-in formulating TPN plans for critically ill adults. Copilot declined to generate patient-specific TPN plans; this refusal was retained as a performance outcome, while Copilot was excluded from analyses requiring an analyzable TPN prescription. Each chatbot produced one plan for each of 10 standardized clinical cases, resulting in a total of 30 TPN plans. Friedman tests, Kendall's W, and Holm-adjusted Wilcoxon signed-rank tests were used for comparative analyses.Results ChatGPT produced comprehensive plans for all 10 cases, compared to 6 plans from Gemini and 4 from Perplexity. The median accuracy was 57.7% for ChatGPT, 43.9% for Gemini, and 38.5% for Perplexity, with no statistically significant overall difference (p = 0.124). However, ChatGPT performed better than Perplexity in pairwise comparison (adjusted p = 0.035). There was a significant difference in completeness among the chatbots (p = 0.003; Kendall's W = 0.57), with median scores of 64.6, 41.9, and 32.3%, respectively. All 30 plans had at least one major or critical safety issue, 7 (23.3%) contained a critical concern, and 19 (63.3%) were rejected. The most common errors were related to electrolytes (90.0%), micronutrients (70.0%), and monitoring (66.7%). Recommendations from chatbots often exceeded pharmacist-derived values, with median relative differences of +47% for energy, +120% for lipids, and +100% for potassium. Bland-Altman analysis revealed considerable variability in the agreement.Conclusion ChatGPT produced the most comprehensive TPN plans among chatbots. However, none showed enough accuracy, safety, or concordance for independent use. All plans had critical issues, especially with electrolytes, micronutrients, monitoring, and refeeding precautions. AI chatbots can support education or planning but require review by a well-trained clinician before clinical use.
SN - 2296-861X
ER -