Evaluation of retrieval-augmented generation (RAG) with GPT-4o for maxillofacial prosthodontics clinical questions.
13 affiliations
- 1Graduate Institute of Dental Science, School of Dentistry, China Medical University, Taichung, Taiwan.
- 2Department of Prosthodontics, Faculty of Dentistry, Hasanuddin University, Makassar, Indonesia.
- 3Department of Prosthodontics, Faculty of Dentistry, University of Sakarya, Sakarya, Turkey.
- 4Department of Prosthodontics, Faculty of Dentistry, University of Ataturk, Erzurum, Turkey.
- 5Monash University, Jakarta, Indonesia.
- 6Department of Oral and Maxillofacial Surgery, School of Dentistry, University of Alabama at Birmingham, Birmingham, USA.
- 7Department of Dentistry, Division of Oral Oncology and Maxillofacial Prosthetics, Erie County Medical Center, Buffalo, USA.
- 8Department of Prosthodontics, University of Pittsburgh, School of Dental Medicine, Pittsburgh, USA.
- 9Department of Oral and Maxillofacial Surgery, Faculty of Dentistry, Hasanuddin University, Makassar, Indonesia.
- 10Department of Restorative Dentistry, University of Washington School of Dentistry, Seattle, USA.
- 11School of Dentistry, China Medical University, Taichung, Taiwan.
- 12Department of Bioinformatics and Medical Engineering, Asia University, Taichung, Taiwan.
- 13Department of Dentistry, China Medical University Hospital, Taichung, Taiwan.
Abstract
Background/purpose
Large language models (LLMs) offer promising education and clinical decision support to help address the global shortage of maxillofacial prosthodontists. To overcome the lack of source traceability in base LLMs, this study evaluated a retrieval-augmented generation (RAG) architecture for answering complex clinical questions.Materials and methods
An expert panel consisting of five maxillofacial prosthodontists selected 15 high-complexity clinical questions. Responses were generated using base GPT-4o and a naive RAG-GPT-4o model built with domain-specific literatures. Outputs were evaluated across five parameters using a 5-point Likert scale and analyzed via Mann-Whitney U tests.Results
No statistically significant differences were observed between GPT-4o and RAG-GPT-4o across all parameters (P > 0.05). The median scores (IQR) for GPT-4o and RAG-GPT-4o were as follows: relevance (4.40 ± 0.40 vs. 4.40 ± 0.50, P = 0.815), clarity (4.40 ± 0.40 vs. 4.20 ± 0.60, P = 0.232), depth (4.20 ± 0.50 vs. 4.00 ± 0.50, P = 0.378), focus (4.40 ± 0.60 vs. 4.00 ± 0.50, P = 0.258), and coherence (4.40 ± 0.30 vs. 4.20 ± 0.50, P = 0.234).Conclusion
This study demonstrates that GPT-4o possesses significant latent expertise in maxillofacial prosthodontics for clinical questions. While the base LLM and a naive RAG architecture performed comparably across predefined clinical parameters, RAG integration provided an essential advantage: verifiable factual grounding. By anchoring AI outputs in transparent, evidence-based literature, RAG addresses the critical need for clinical safety and clinician trust. Ultimately, while base LLMs offer substantial clinical reasoning capabilities, pairing them with RAG architectures remains necessary to ensure the accountability required for safe integration into specialized dental care, establishing a foundation for future development of more advanced RAG systems.
Related
- Development and evaluation of Retrieval-Augmented Generation (RAG) and GraphRAG for guideline-based clinical decision support in chronic kidney disease
- Using iterative agentic retrieval-augmented generation to refine the risk of infection from social contexts during the COVID-19 pandemic: Development and Evaluation Study
- Applications, Technical Characteristics, and Evaluation Approaches of Retrieval-Augmented Generation in Nursing: Scoping Review
- Real-world evaluation of a multilingual agentic retrieval-augmented clinical AI framework for evidence-grounded medical decision support
- Retrieval-augmented generation in biomedicine: a survey of technologies, datasets, and clinical applications
BibTeX
@article{launardo2026evaluation,
title = {{Evaluation of retrieval-augmented generation (RAG) with GPT-4o for maxillofacial prosthodontics clinical questions.}},
author = {Vinsensia Launardo and Merve Koseoglu and Lucky Susanto and Michael Kase and Amanda Colebeck and Heidi Huber and Maribeth Krzesinski and Derry Wijaya and Muhammad Ruslin and Alvin Gerard Wee and Heng-Li Huang and Lih-Jyh Fuh and Cortino Sukotjo},
journal = {Journal of dental sciences},
year = {2026},
month = oct,
doi = {10.1016/j.jds.2025.07.3504},
url = {https://europepmc.org/article/MED/42923637},
note = {CC BY-NC-ND},
}
RIS
TY - JOUR
TI - Evaluation of retrieval-augmented generation (RAG) with GPT-4o for maxillofacial prosthodontics clinical questions.
AU - Vinsensia Launardo
AU - Merve Koseoglu
AU - Lucky Susanto
AU - Michael Kase
AU - Amanda Colebeck
AU - Heidi Huber
AU - Maribeth Krzesinski
AU - Derry Wijaya
AU - Muhammad Ruslin
AU - Alvin Gerard Wee
AU - Heng-Li Huang
AU - Lih-Jyh Fuh
AU - Cortino Sukotjo
PY - 2026
DA - 2026/10/01
JO - Journal of dental sciences
DO - 10.1016/j.jds.2025.07.3504
UR - https://europepmc.org/article/MED/42923637
AB - Background/purpose Large language models (LLMs) offer promising education and clinical decision support to help address the global shortage of maxillofacial prosthodontists. To overcome the lack of source traceability in base LLMs, this study evaluated a retrieval-augmented generation (RAG) architecture for answering complex clinical questions.Materials and methods An expert panel consisting of five maxillofacial prosthodontists selected 15 high-complexity clinical questions. Responses were generated using base GPT-4o and a naive RAG-GPT-4o model built with domain-specific literatures. Outputs were evaluated across five parameters using a 5-point Likert scale and analyzed via Mann-Whitney U tests.Results No statistically significant differences were observed between GPT-4o and RAG-GPT-4o across all parameters (P > 0.05). The median scores (IQR) for GPT-4o and RAG-GPT-4o were as follows: relevance (4.40 ± 0.40 vs. 4.40 ± 0.50, P = 0.815), clarity (4.40 ± 0.40 vs. 4.20 ± 0.60, P = 0.232), depth (4.20 ± 0.50 vs. 4.00 ± 0.50, P = 0.378), focus (4.40 ± 0.60 vs. 4.00 ± 0.50, P = 0.258), and coherence (4.40 ± 0.30 vs. 4.20 ± 0.50, P = 0.234).Conclusion This study demonstrates that GPT-4o possesses significant latent expertise in maxillofacial prosthodontics for clinical questions. While the base LLM and a naive RAG architecture performed comparably across predefined clinical parameters, RAG integration provided an essential advantage: verifiable factual grounding. By anchoring AI outputs in transparent, evidence-based literature, RAG addresses the critical need for clinical safety and clinician trust. Ultimately, while base LLMs offer substantial clinical reasoning capabilities, pairing them with RAG architectures remains necessary to ensure the accountability required for safe integration into specialized dental care, establishing a foundation for future development of more advanced RAG systems.
SN - 1991-7902
SN - 2213-8862
ER -