Artificial intelligence versus physician decision-making in acute heart failure: a comparative evaluation of four contemporary AI systems
Abstract
Aims: Large language models (LLMs) are increasingly used by the public and healthcare professionals for clinical reasoning. However, they were developed as general-purpose language models rather than dedicated clinical decision-support tools, and their performance in real-world acute heart failure (AHF) management remains insufficiently evaluated. We compared four contemporary AI models with physician decision-making.
Methods and results: This single-centre retrospective observational study incorporated a cross-sectional AI evaluation of consecutive patients treated at Sotiria General Hospital between July and November 2025, excluding acute coronary syndrome. ChatGPT, ChatGPT Plus, DeepSeek, and an ESC guideline-based retrieval-augmented generation (RAG) model independently assessed identical standardized cases. Recommendations were compared with physician decisions on diagnosis, treatment, guideline-directed medical therapy, timing, and prognosis using Cohen’s kappa, McNemar’s test, Cochran’s Q, Wilcoxon signed-rank testing, and multivariable logistic regression. Diagnostic performance differed significantly across systems (Cochran’s Q = 8.477, P = 0.037). In the balanced diagnostic cohort, ChatGPT Plus achieved the highest heart failure (HF) accuracy (accuracy 86.5%, sensitivity 87.0%, and specificity 86.0%), followed by RAG (accuracy 77.5%), ChatGPT (accuracy 77.0%), and DeepSeek (accuracy 69.0%). For HF within the top three differential diagnoses, sensitivity was 90.0–97.0%; specificity was 70.0% for ChatGPT Plus, 68.0% for RAG, 19.0% for DeepSeek, and 15.0% for ChatGPT. Physician agreement was highest for guideline-directed therapies, with greater variability in prognosis and escalation of care. RAG more closely matched physician therapeutic timing but did not consistently outperform general-purpose models.
Conclusion: Contemporary AI models show promise in AHF decision support but substantial task-specific variability, supporting use as complements rather than replacements for physician judgement.
Related
- OP–249 A comparative evaluation of artificial intelligence models for adolescent-centered scoliosis questions
- Evaluation of Tooth Shade Selection With Conventional Methods and Artificial Intelligence-Integrated Methods: A Comparative Study
- Beyond Artificial Intelligence: Vocation and the Sacred Relationship Between Physician and Patient
- CT-Referenced Diagnostic Performance and Complementary Error Patterns of Artificial Intelligence Versus a Single Emergency Physician in Chest Radiograph Interpretation: A Retrospective Paired Diagnostic Accuracy Study
- Citation Fabrication and Accuracy in AI-Generated Psychiatric Research: A Comparative Evaluation of Four Contemporary Large Language Models
BibTeX
@article{zakynthinos2026artificial,
title = {{Artificial intelligence versus physician decision-making in acute heart failure: a comparative evaluation of four contemporary AI systems}},
author = {Georgios E Zakynthinos and Evangelos Oikonomou and Iason- Daniil Zakynthinos and George Dimeas and Panteleimon Pantelidis and Theodore G Papaioannou and Gerasimos Siasos},
journal = {European Heart Journal. Digital Health},
year = {2026},
month = oct,
doi = {10.1093/ehjdh/ztag169},
url = {https://doi.org/10.1093/ehjdh/ztag169},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Artificial intelligence versus physician decision-making in acute heart failure: a comparative evaluation of four contemporary AI systems
AU - Georgios E Zakynthinos
AU - Evangelos Oikonomou
AU - Iason- Daniil Zakynthinos
AU - George Dimeas
AU - Panteleimon Pantelidis
AU - Theodore G Papaioannou
AU - Gerasimos Siasos
PY - 2026
DA - 2026/10/24
JO - European Heart Journal. Digital Health
DO - 10.1093/ehjdh/ztag169
UR - https://doi.org/10.1093/ehjdh/ztag169
AB - Aims: Large language models (LLMs) are increasingly used by the public and healthcare professionals for clinical reasoning. However, they were developed as general-purpose language models rather than dedicated clinical decision-support tools, and their performance in real-world acute heart failure (AHF) management remains insufficiently evaluated. We compared four contemporary AI models with physician decision-making. Methods and results: This single-centre retrospective observational study incorporated a cross-sectional AI evaluation of consecutive patients treated at Sotiria General Hospital between July and November 2025, excluding acute coronary syndrome. ChatGPT, ChatGPT Plus, DeepSeek, and an ESC guideline-based retrieval-augmented generation (RAG) model independently assessed identical standardized cases. Recommendations were compared with physician decisions on diagnosis, treatment, guideline-directed medical therapy, timing, and prognosis using Cohen’s kappa, McNemar’s test, Cochran’s Q, Wilcoxon signed-rank testing, and multivariable logistic regression. Diagnostic performance differed significantly across systems (Cochran’s Q = 8.477, P = 0.037). In the balanced diagnostic cohort, ChatGPT Plus achieved the highest heart failure (HF) accuracy (accuracy 86.5%, sensitivity 87.0%, and specificity 86.0%), followed by RAG (accuracy 77.5%), ChatGPT (accuracy 77.0%), and DeepSeek (accuracy 69.0%). For HF within the top three differential diagnoses, sensitivity was 90.0–97.0%; specificity was 70.0% for ChatGPT Plus, 68.0% for RAG, 19.0% for DeepSeek, and 15.0% for ChatGPT. Physician agreement was highest for guideline-directed therapies, with greater variability in prognosis and escalation of care. RAG more closely matched physician therapeutic timing but did not consistently outperform general-purpose models. Conclusion: Contemporary AI models show promise in AHF decision support but substantial task-specific variability, supporting use as complements rather than replacements for physician judgement.
SN - 2634-3916
ER -