Guideline Concordance of Antibiotic Recommendations From Large Language Models in Dentistry: A Vignette-Based Evaluation in Georgian and English
4 affiliations
- 1Faculty of Medicine, Ivane Javakhishvili Tbilisi State University, Tbilisi, GEO
- 2Faculty of Dentistry, European University, Tbilisi, GEO
- 3Department of Public Health, Georgian Association for Public Health and Well-Being, Tbilisi, GEO
- 4Department of Dentistry, International Medical School, Alte University, Tbilisi, GEO
Abstract
Introduction: Inappropriate antibiotic prescribing in dentistry is a recognised stewardship concern. Large language models (LLMs) are readily accessible sources of prescribing information, but dental evaluations have been conducted almost entirely in English. We compared the guideline concordance of matched Georgian and English LLM recommendations for common dental presentations in a setting in which we could identify no national dental antibiotic prescribing guideline.
Materials and methods: In this preregistered vignette study, 20 standardised scenarios (10 antibiotic-indicated, 10 not) were submitted in Georgian and English to the free-tier interfaces of Claude (Anthropic, San Francisco, USA), ChatGPT (OpenAI, San Francisco, USA) and Gemini (Google LLC, Mountain View, USA) on three occasions (360 responses). Responses were normalised, blinded to system and run, and coded by two dentists against a key based on Scottish Dental Clinical Effectiveness Programme (SDCEP), American Dental Association (ADA) 2019 and American Heart Association (AHA) 2021 guidance on five domains: decision to prescribe, agent, dose and frequency, duration, and referral advice. Disagreements were adjudicated and the regimen domains were re-scored under clarified rules. The primary outcome was full concordance on all applicable domains in the first run, compared across 60 matched scenario-system pairs with the exact McNemar test.
Results: Full concordance was 57/60 (95.0%, 95% CI 86.3 to 98.3) for English and 46/60 (76.7%, 64.6 to 85.6) for Georgian prompts, a paired difference of 18.3 percentage points (95% CI 7.5 to 29.2; p = 0.003); 12 pairs were concordant in English only and one in Georgian only. The decision to prescribe was concordant in all 360 responses and referral advice in 59/60 per language. In antibiotic-indicated scenarios, an accepted first-choice agent was given in 30/30 English and 21/30 Georgian responses (p = 0.004); dose and duration concordance was also lower in Georgian (16/27 and 15/27 versus 26/27 and 25/27) but was scored discordant whenever the agent was discordant. All 30 no-antibiotic pairs were concordant in both languages. The between-system comparison was not significant (Cochran's Q p = 0.14). The agent code was reproducible across three runs in 29/30 English and 26/30 Georgian antibiotic-indicated cells (p = 0.25). No first-run response met a prespecified safety-critical error criterion (0/60 per language, 95% CI 0.0 to 6.0).
Conclusions: Decision concordance was preserved across languages, but full concordance with a UK stewardship standard was lower for Georgian prompts, chiefly because the first-choice agent was broader than the reference accepts. LLM-generated regimen details should be checked against an authoritative source. The findings underscore the value of accessible, authoritative Georgian-language dental prescribing guidance.
Related
- Antibiotic Prophylaxis of Infective Endocarditis in Dentistry: An Update on the ESC, AHA, and NICE Guidelines
- Safety and guideline concordance of public large language model responses to patient-facing questions on active charcot neuro-osteoarthropathy
- Consideration of person-important outcomes by guideline recommendations for the use of tests and testing strategies: protocol for a meta-epidemiological study
- Recommendations for managing multiple long-term conditions: a systemaftic guideline review and modified Delphi study
- Association of geriatric assessment with multidisciplinary treatment recommendations and treatment concordance among older patients with breast cancer
BibTeX
@article{alkhanishvili2026guideline,
title = {{Guideline Concordance of Antibiotic Recommendations From Large Language Models in Dentistry: A Vignette-Based Evaluation in Georgian and English}},
author = {Zurab Alkhanishvili and Ketevan Gogilashvili and Nino Gvasalia and Sopio Samkharadze and Landa Lursmanashvili and Luka Nachkebia and Aleksandre Chkhikvishvili and Ali Qasim Tahir and Nana Tsitskishvili},
journal = {Cureus},
year = {2026},
month = sep,
doi = {10.7759/cureus.117183},
url = {https://europepmc.org/article/PMC/PMC13719903},
note = {CC BY},
}
RIS
TY - JOUR
TI - Guideline Concordance of Antibiotic Recommendations From Large Language Models in Dentistry: A Vignette-Based Evaluation in Georgian and English
AU - Zurab Alkhanishvili
AU - Ketevan Gogilashvili
AU - Nino Gvasalia
AU - Sopio Samkharadze
AU - Landa Lursmanashvili
AU - Luka Nachkebia
AU - Aleksandre Chkhikvishvili
AU - Ali Qasim Tahir
AU - Nana Tsitskishvili
PY - 2026
DA - 2026/09/29
JO - Cureus
DO - 10.7759/cureus.117183
UR - https://europepmc.org/article/PMC/PMC13719903
AB - Introduction: Inappropriate antibiotic prescribing in dentistry is a recognised stewardship concern. Large language models (LLMs) are readily accessible sources of prescribing information, but dental evaluations have been conducted almost entirely in English. We compared the guideline concordance of matched Georgian and English LLM recommendations for common dental presentations in a setting in which we could identify no national dental antibiotic prescribing guideline. Materials and methods: In this preregistered vignette study, 20 standardised scenarios (10 antibiotic-indicated, 10 not) were submitted in Georgian and English to the free-tier interfaces of Claude (Anthropic, San Francisco, USA), ChatGPT (OpenAI, San Francisco, USA) and Gemini (Google LLC, Mountain View, USA) on three occasions (360 responses). Responses were normalised, blinded to system and run, and coded by two dentists against a key based on Scottish Dental Clinical Effectiveness Programme (SDCEP), American Dental Association (ADA) 2019 and American Heart Association (AHA) 2021 guidance on five domains: decision to prescribe, agent, dose and frequency, duration, and referral advice. Disagreements were adjudicated and the regimen domains were re-scored under clarified rules. The primary outcome was full concordance on all applicable domains in the first run, compared across 60 matched scenario-system pairs with the exact McNemar test. Results: Full concordance was 57/60 (95.0%, 95% CI 86.3 to 98.3) for English and 46/60 (76.7%, 64.6 to 85.6) for Georgian prompts, a paired difference of 18.3 percentage points (95% CI 7.5 to 29.2; p = 0.003); 12 pairs were concordant in English only and one in Georgian only. The decision to prescribe was concordant in all 360 responses and referral advice in 59/60 per language. In antibiotic-indicated scenarios, an accepted first-choice agent was given in 30/30 English and 21/30 Georgian responses (p = 0.004); dose and duration concordance was also lower in Georgian (16/27 and 15/27 versus 26/27 and 25/27) but was scored discordant whenever the agent was discordant. All 30 no-antibiotic pairs were concordant in both languages. The between-system comparison was not significant (Cochran's Q p = 0.14). The agent code was reproducible across three runs in 29/30 English and 26/30 Georgian antibiotic-indicated cells (p = 0.25). No first-run response met a prespecified safety-critical error criterion (0/60 per language, 95% CI 0.0 to 6.0). Conclusions: Decision concordance was preserved across languages, but full concordance with a UK stewardship standard was lower for Georgian prompts, chiefly because the first-choice agent was broader than the reference accepts. LLM-generated regimen details should be checked against an authoritative source. The findings underscore the value of accessible, authoritative Georgian-language dental prescribing guidance.
SN - 2168-8184
ER -