Concordance between large language models and orthodontists in photo-based orthodontic assessment.
- 1Department of Orthodontics, Faculty of Dentistry, Giresun University, Giresun, Turkey. [email protected].
Abstract
Background
The widespread use of digital platforms for preliminary health information has increased interest in artificial intelligence (AI)-based tools in dentistry. Large language model (LLM)-based systems are increasingly used for patient education and preliminary clinical guidance. However, their clinical consistency in orthodontic assessments based solely on intraoral photographs remains unclear. This study aimed to evaluate the clinical appropriateness of LLM-generated orthodontic assessments compared with blinded expert orthodontist evaluations.Methods
This blinded comparative agreement study included 104 patients aged ≥ 12 years undergoing orthodontic evaluation. For each case, three standardized intraoral photographs and basic demographic data were submitted to two LLM-based systems (ChatGPT-4o and Grok 4.1) using a standardized prompt. Responses to five clinical questions were independently evaluated by three expert orthodontists using a 5-point Likert scale. Consensus scores were obtained by averaging the ratings of the three evaluators. Inter-system comparisons were performed using the Wilcoxon signed-rank test. Bonferroni and Benjamini-Hochberg false discovery rate (FDR) corrections were applied for multiple comparisons.Results
Mean ratings were generally ≥ 3, indicating moderate clinical acceptability. After correction for multiple comparisons, the most robust finding was the significantly higher rating assigned to ChatGPT-4o for the recommended orthodontic treatment approach in Dental Class I patients. In the 12-15-year subgroup, ChatGPT-4o also received higher ratings for treatment recommendation; however, statistical significance was retained only after FDR correction and not after Bonferroni adjustment. No significant differences were observed across most other evaluated parameters, particularly malocclusion classification. In patients older than 15 years, paired differences between the two systems were highly similar, resulting in an insufficient number of non-zero differences for reliable Wilcoxon signed-rank testing.Conclusions
LLM-based systems may provide clinically acceptable outputs in selected orthodontic assessment domains when evaluations are based solely on intraoral photographs. However, variability persists in complex decision-making tasks, particularly treatment planning. Current evidence supports the use of such systems as supportive tools operating under professional supervision rather than as independent clinical decision-makers.
Related
- Comparative Analysis of Aesthetic Judgment between Orthodontists, General Dentists, and Different Versions of Large Language Models Using the IOTN-AC Index
- Safety and guideline concordance of public large language model responses to patient-facing questions on active charcot neuro-osteoarthropathy
- Evaluation of Large Language Model Concordance With AAOS Clinical Practice Guidelines for Carpal Tunnel Syndrome
- Performance of large language models in answering orthodontic questions: A scoping review.
- Severity Stratification Changes What Hallucination Rates Mean: An Item-Matched Audit of Five Large Language Models in Orthodontic Decision Support.
BibTeX
@article{abay2026concordance,
title = {{Concordance between large language models and orthodontists in photo-based orthodontic assessment.}},
author = {Feridun Abay},
journal = {BMC oral health},
year = {2026},
month = jul,
doi = {10.1186/s12903-026-09380-6},
url = {https://europepmc.org/article/MED/42894149},
note = {CC BY-NC-ND},
}
RIS
TY - JOUR
TI - Concordance between large language models and orthodontists in photo-based orthodontic assessment.
AU - Feridun Abay
PY - 2026
DA - 2026/07/01
JO - BMC oral health
DO - 10.1186/s12903-026-09380-6
UR - https://europepmc.org/article/MED/42894149
AB - Background The widespread use of digital platforms for preliminary health information has increased interest in artificial intelligence (AI)-based tools in dentistry. Large language model (LLM)-based systems are increasingly used for patient education and preliminary clinical guidance. However, their clinical consistency in orthodontic assessments based solely on intraoral photographs remains unclear. This study aimed to evaluate the clinical appropriateness of LLM-generated orthodontic assessments compared with blinded expert orthodontist evaluations.Methods This blinded comparative agreement study included 104 patients aged ≥ 12 years undergoing orthodontic evaluation. For each case, three standardized intraoral photographs and basic demographic data were submitted to two LLM-based systems (ChatGPT-4o and Grok 4.1) using a standardized prompt. Responses to five clinical questions were independently evaluated by three expert orthodontists using a 5-point Likert scale. Consensus scores were obtained by averaging the ratings of the three evaluators. Inter-system comparisons were performed using the Wilcoxon signed-rank test. Bonferroni and Benjamini-Hochberg false discovery rate (FDR) corrections were applied for multiple comparisons.Results Mean ratings were generally ≥ 3, indicating moderate clinical acceptability. After correction for multiple comparisons, the most robust finding was the significantly higher rating assigned to ChatGPT-4o for the recommended orthodontic treatment approach in Dental Class I patients. In the 12-15-year subgroup, ChatGPT-4o also received higher ratings for treatment recommendation; however, statistical significance was retained only after FDR correction and not after Bonferroni adjustment. No significant differences were observed across most other evaluated parameters, particularly malocclusion classification. In patients older than 15 years, paired differences between the two systems were highly similar, resulting in an insufficient number of non-zero differences for reliable Wilcoxon signed-rank testing.Conclusions LLM-based systems may provide clinically acceptable outputs in selected orthodontic assessment domains when evaluations are based solely on intraoral photographs. However, variability persists in complex decision-making tasks, particularly treatment planning. Current evidence supports the use of such systems as supportive tools operating under professional supervision rather than as independent clinical decision-makers.
SN - 1472-6831
ER -