Development and evaluation of Retrieval-Augmented Generation (RAG) and GraphRAG for guideline-based clinical decision support in chronic kidney disease
- 1Artificial Intelligence & Mathematics Research Lab, London, UK
- 2King’s College London, London, UK
- 3Renal Department, Guy's and St Thomas’ Hospitals NHS Trust, London, UK
Abstract
Objective: Chronic kidney disease (CKD) is a progressive condition requiring evidence-based management, but adherence to complex guidelines remains challenging. Large Language Models (LLMs) could support clinical decision-making, yet their unreliability limits direct use. This study aimed to evaluate whether Retrieval-Augmented Generation (RAG), particularly a knowledge graph-enhanced pipeline (GraphRAG), improves guideline-based clinical decision support (CDS) in CKD management.
Methods and analysis: We compared three approaches: a baseline LLM (GPT-4o), a vector-indexed RAG pipeline and a GraphRAG pipeline. Each model answered nine clinically relevant questions for a synthetic cohort of 70 CKD patients. Outputs were assessed for clinical correctness, patient specificity and clarity using both clinician-led evaluations and an LLM-as-Judge framework.
Results: RAG achieved higher clinician-rated correctness than the baseline LLM (76.6% vs 70.8%), with GraphRAG showing a similar improvement (75.6%). GraphRAG produced the most patient-specific responses but was associated with substantially longer outputs and reduced clarity. Response length was strongly correlated with both specificity and clarity, indicating that this trade-off is largely length-driven. Agreement between clinician and LLM-based evaluation varied by task type (Cohen’s kappa 0.13–0.97), with lower agreement for open-ended tasks. All retrieval-based methods were constrained by the scope of the indexed guideline.
Conclusions: RAG and GraphRAG provide a scalable, auditable foundation for guideline-aligned CDS in CKD. RAG showed higher clinician-rated correctness, while GraphRAG improved patient specificity without a corresponding gain in correctness. Trade-offs between specificity and clarity were largely driven by response length. Effective deployment will require robust content management, calibrated evaluation frameworks and integration within clinical governance structures.
Related
- Using iterative agentic retrieval-augmented generation to refine the risk of infection from social contexts during the COVID-19 pandemic: Development and Evaluation Study
- BanglaDiaHallu: Hallucinations of large language models and the effect of guideline-grounded retrieval-augmented generation in Bangla diabetes question answering
- Evaluation of retrieval-augmented generation (RAG) with GPT-4o for maxillofacial prosthodontics clinical questions.
- Applications, Technical Characteristics, and Evaluation Approaches of Retrieval-Augmented Generation in Nursing: Scoping Review
- Real-world evaluation of a multilingual agentic retrieval-augmented clinical AI framework for evidence-grounded medical decision support
BibTeX
@article{sesen2026development,
title = {{Development and evaluation of Retrieval-Augmented Generation (RAG) and GraphRAG for guideline-based clinical decision support in chronic kidney disease}},
author = {M Berkan Sesen and Joshua A Yeung and Elham Asgari},
journal = {BMJ Digital Health \& AI},
year = {2026},
month = nov,
doi = {10.1136/bmjdh-2026-000041},
url = {https://doi.org/10.1136/bmjdh-2026-000041},
note = {CC BY-NC 4.0},
}
RIS
TY - JOUR
TI - Development and evaluation of Retrieval-Augmented Generation (RAG) and GraphRAG for guideline-based clinical decision support in chronic kidney disease
AU - M Berkan Sesen
AU - Joshua A Yeung
AU - Elham Asgari
PY - 2026
DA - 2026/11/02
JO - BMJ Digital Health & AI
DO - 10.1136/bmjdh-2026-000041
UR - https://doi.org/10.1136/bmjdh-2026-000041
AB - Objective: Chronic kidney disease (CKD) is a progressive condition requiring evidence-based management, but adherence to complex guidelines remains challenging. Large Language Models (LLMs) could support clinical decision-making, yet their unreliability limits direct use. This study aimed to evaluate whether Retrieval-Augmented Generation (RAG), particularly a knowledge graph-enhanced pipeline (GraphRAG), improves guideline-based clinical decision support (CDS) in CKD management. Methods and analysis: We compared three approaches: a baseline LLM (GPT-4o), a vector-indexed RAG pipeline and a GraphRAG pipeline. Each model answered nine clinically relevant questions for a synthetic cohort of 70 CKD patients. Outputs were assessed for clinical correctness, patient specificity and clarity using both clinician-led evaluations and an LLM-as-Judge framework. Results: RAG achieved higher clinician-rated correctness than the baseline LLM (76.6% vs 70.8%), with GraphRAG showing a similar improvement (75.6%). GraphRAG produced the most patient-specific responses but was associated with substantially longer outputs and reduced clarity. Response length was strongly correlated with both specificity and clarity, indicating that this trade-off is largely length-driven. Agreement between clinician and LLM-based evaluation varied by task type (Cohen’s kappa 0.13–0.97), with lower agreement for open-ended tasks. All retrieval-based methods were constrained by the scope of the indexed guideline. Conclusions: RAG and GraphRAG provide a scalable, auditable foundation for guideline-aligned CDS in CKD. RAG showed higher clinician-rated correctness, while GraphRAG improved patient specificity without a corresponding gain in correctness. Trade-offs between specificity and clarity were largely driven by response length. Effective deployment will require robust content management, calibrated evaluation frameworks and integration within clinical governance structures.
SN - 3049-575X
ER -