Evaluating different large language model performances in evidence-based dental implantology: a blinded controlled experimental evaluation using expert panel assessment.
5 affiliations
- 1Research Center for Digital Technologies in Dentistry and CAD/CAM, Department of Dentistry, Faculty of Medicine and Dentistry, Danube Private University, Steiner Landstrasse 124, Krems an der Donau, 3500, Austria. [email protected].
- 2Department of Prosthodontics, Geriatric Dentistry and Craniomandibular Disorders, Charité-Universitätsmedizin Berlin, Freie Universität Berlin and Humboldt-Universität zu Berlin, Aßmannshauser Str. 4- 6, Berlin, 14197, Germany. [email protected].
- 3Department of Oral and Maxillofacial Surgery, Charité - Universitätsmedizin Berlin, Freie Universität Berlin and Humboldt-Universität zu Berlin, Augustenburger Platz 1, Berlin, 13353, Germany.
- 4Research Center for Digital Technologies in Dentistry and CAD/CAM, Department of Dentistry, Faculty of Medicine and Dentistry, Danube Private University, Steiner Landstrasse 124, Krems an der Donau, 3500, Austria.
- 5Department of Periodontology, Endodontology and Cariology, University Center for Dental Medicine UZB, University of Base, Mattenstrasse 40, Basel, CH-4058, Switzerland.
Abstract
Background
Artificial intelligence (AI) and large language models (LLMs) are increasingly being explored for clinical decision-making and evidence-based information retrieval in dental implantology. However, concerns remain regarding the accuracy, reliability, and clinical applicability of different AI approaches, including general-purpose, fine-tuned, and retrieval-augmented generation (RAG) models. This study aimed to compare their ability to provide accurate, evidence-based, and clinically relevant responses to implantology-related questions.Methods
Sixteen master's-level clinical questions were presented to all four LLMs under controlled conditions. Their anonymized responses were independently evaluated by three dental implantology specialists based on three core criteria: specificity, accuracy, and clinical utility, using a 0-10 scale. Descriptive statistics were reported as medians and interquartile ranges. Differences among models were assessed primarily using Kruskal-Wallis tests, and inter-rater reliability was evaluated using Fleiss' kappa.Results
LLM4 consistently achieved the highest median scores across all criteria (Specificity: median 9.0, IQR 9.0-10.0; Accuracy: median 9.5, IQR 9.0-10.0; Clinical Utility: median 10.0, IQR 9.0-10.0), significantly outperforming the other models (all p < 0.001). Cohen's d revealed large to very large effect sizes in comparisons involving LLM4, particularly against LLM2 and LLM3. LLM2 scored lowest and showed the highest variability. Inter-rater reliability was slight across all metrics (κ = 0.078-0.180), indicating moderate evaluator disagreement.Conclusions
Among the models tested, LLaMA 3.2 + RAG demonstrated superior and more consistent clinical performance in dental implantology. While fine-tuning provided moderate benefits, real-time access to structured guidelines significantly enhanced response quality. Despite low inter-rater agreement, these findings highlight the promise of RAG models in supporting high-complexity, evidence-based clinical domains, while emphasizing the need for standardization in AI evaluation protocols.
Related
- Evaluating 20-session augmented depression therapy for complex depression (ADepT+) versus usual care: a study protocol for a feasibility randomised controlled trial
- Evaluating a shared decision engagement system in the National Cancer Institute Community Oncology Research Program: a multi-level randomized controlled trial
- Evaluating the effectiveness and cost–benefit of a worksite mental health program among manufacturing workers: A study protocol for a quasi-experimental study
- Two-Year Knowledge Retention After a Short Online Palliative Care Intervention for Medical Students: A Controlled Experimental Study
- Evaluating the effects of the educational course on undergraduate students' perceived knowledge, attitude, and motivation: a quasi-experimental study in Iran.
BibTeX
@article{alevizakos2026evaluating,
title = {{Evaluating different large language model performances in evidence-based dental implantology: a blinded controlled experimental evaluation using expert panel assessment.}},
author = {Vasilios Alevizakos and Robert Gaudin and Iman Ghanad and Jonathan von See and Theodor von See and Florin Eggmann and Constantin von See},
journal = {BMC oral health},
year = {2026},
month = jul,
doi = {10.1186/s12903-026-09368-2},
url = {https://europepmc.org/article/MED/42894220},
note = {CC BY-NC-ND},
}
RIS
TY - JOUR
TI - Evaluating different large language model performances in evidence-based dental implantology: a blinded controlled experimental evaluation using expert panel assessment.
AU - Vasilios Alevizakos
AU - Robert Gaudin
AU - Iman Ghanad
AU - Jonathan von See
AU - Theodor von See
AU - Florin Eggmann
AU - Constantin von See
PY - 2026
DA - 2026/07/01
JO - BMC oral health
DO - 10.1186/s12903-026-09368-2
UR - https://europepmc.org/article/MED/42894220
AB - Background Artificial intelligence (AI) and large language models (LLMs) are increasingly being explored for clinical decision-making and evidence-based information retrieval in dental implantology. However, concerns remain regarding the accuracy, reliability, and clinical applicability of different AI approaches, including general-purpose, fine-tuned, and retrieval-augmented generation (RAG) models. This study aimed to compare their ability to provide accurate, evidence-based, and clinically relevant responses to implantology-related questions.Methods Sixteen master's-level clinical questions were presented to all four LLMs under controlled conditions. Their anonymized responses were independently evaluated by three dental implantology specialists based on three core criteria: specificity, accuracy, and clinical utility, using a 0-10 scale. Descriptive statistics were reported as medians and interquartile ranges. Differences among models were assessed primarily using Kruskal-Wallis tests, and inter-rater reliability was evaluated using Fleiss' kappa.Results LLM4 consistently achieved the highest median scores across all criteria (Specificity: median 9.0, IQR 9.0-10.0; Accuracy: median 9.5, IQR 9.0-10.0; Clinical Utility: median 10.0, IQR 9.0-10.0), significantly outperforming the other models (all p < 0.001). Cohen's d revealed large to very large effect sizes in comparisons involving LLM4, particularly against LLM2 and LLM3. LLM2 scored lowest and showed the highest variability. Inter-rater reliability was slight across all metrics (κ = 0.078-0.180), indicating moderate evaluator disagreement.Conclusions Among the models tested, LLaMA 3.2 + RAG demonstrated superior and more consistent clinical performance in dental implantology. While fine-tuning provided moderate benefits, real-time access to structured guidelines significantly enhanced response quality. Despite low inter-rater agreement, these findings highlight the promise of RAG models in supporting high-complexity, evidence-based clinical domains, while emphasizing the need for standardization in AI evaluation protocols.
SN - 1472-6831
ER -