Evidence-based clinical question answering using retrieval-augmented generation (RAG) and open-source large language models
Abstract
Large language models (LLMs) are being investigated for clinical decision support tools, but can produce unreliable outputs, including hallucinated information, overgeneralized answers, and responses not clearly derived from authoritative medical sources. To ground responses in high-quality medical texts, researchers developed Retrieval-Augmented Generation (RAG). In this work, we show that, within a structured, single-textbook corpus, embedding model choice has significantly greater impact on RAG system reliability and performance than chunking strategy. The system compares twelve embedding models—general-purpose and biomedical-specific—under token-level and character-level chunking, prompting the LLM to answer from retrieved snippets. We evaluate retrieval effectiveness using rank-based metrics and generation quality using standard measures across 232 questions. General-purpose embeddings, particularly Jina-Embeddings-v3 and BGE-base-en-v1.5, outperformed biomedical-specific encoders in both retrieval and generation. Across both strategies, higher retrieval precision translated directly into improved lexical and semantic alignment with gold references. Under our operational definition requiring explicit evidence spans, 35.6% of answered questions lacked supporting evidence; our best configuration (BGE-base-en-v1.5, token-level) achieved the lowest rate at 26.1%—an evidence-grounding measure, not a guarantee of correctness. These findings highlight the central role of retrieval quality in clinical RAG systems, showing that open-source, evidence-driven architectures offer a transparent alternative for textbook-based clinical QA settings.
Related
- OncoAgentic-RAG: An evidence-grounded multi-agent hybrid retrieval-augmented system for patient-centric clinical oncology question answering
- BanglaDiaHallu: Hallucinations of large language models and the effect of guideline-grounded retrieval-augmented generation in Bangla diabetes question answering
- Real-world evaluation of a multilingual agentic retrieval-augmented clinical AI framework for evidence-grounded medical decision support
- Retrieval-augmented generation in biomedicine: a survey of technologies, datasets, and clinical applications
- Retrieval-augmented generation in medicine: A scoping review of technical implementations, clinical applications, and ethical considerations
BibTeX
@article{qasim2026evidencebased,
title = {{Evidence-based clinical question answering using retrieval-augmented generation (RAG) and open-source large language models}},
author = {Ayesha Qasim and Jesus Jaime Moreno Escobar and Christian E. Maldonado-Sifuentes and Zia-ud-din Akhtar and Shazia Qasim and Hiram Calvo},
journal = {Scientific Reports},
year = {2026},
month = oct,
doi = {10.1038/s41598-026-74252-6},
url = {https://www.nature.com/articles/s41598-026-74252-6},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Evidence-based clinical question answering using retrieval-augmented generation (RAG) and open-source large language models
AU - Ayesha Qasim
AU - Jesus Jaime Moreno Escobar
AU - Christian E. Maldonado-Sifuentes
AU - Zia-ud-din Akhtar
AU - Shazia Qasim
AU - Hiram Calvo
PY - 2026
DA - 2026/10/13
JO - Scientific Reports
DO - 10.1038/s41598-026-74252-6
UR - https://www.nature.com/articles/s41598-026-74252-6
AB - Large language models (LLMs) are being investigated for clinical decision support tools, but can produce unreliable outputs, including hallucinated information, overgeneralized answers, and responses not clearly derived from authoritative medical sources. To ground responses in high-quality medical texts, researchers developed Retrieval-Augmented Generation (RAG). In this work, we show that, within a structured, single-textbook corpus, embedding model choice has significantly greater impact on RAG system reliability and performance than chunking strategy. The system compares twelve embedding models—general-purpose and biomedical-specific—under token-level and character-level chunking, prompting the LLM to answer from retrieved snippets. We evaluate retrieval effectiveness using rank-based metrics and generation quality using standard measures across 232 questions. General-purpose embeddings, particularly Jina-Embeddings-v3 and BGE-base-en-v1.5, outperformed biomedical-specific encoders in both retrieval and generation. Across both strategies, higher retrieval precision translated directly into improved lexical and semantic alignment with gold references. Under our operational definition requiring explicit evidence spans, 35.6% of answered questions lacked supporting evidence; our best configuration (BGE-base-en-v1.5, token-level) achieved the lowest rate at 26.1%—an evidence-grounding measure, not a guarantee of correctness. These findings highlight the central role of retrieval quality in clinical RAG systems, showing that open-source, evidence-driven architectures offer a transparent alternative for textbook-based clinical QA settings.
SN - 2045-2322
ER -