Guideline-Concordant Answer Quality of Five Large Language Models on Indication Criteria for Total Hip Arthroplasty in Hip Osteoarthritis: A Single-Run Benchmark Against the 2021 German S3 Guideline (EKIT-Hip)
- 1Department of Orthopedic and Trauma Surgery, Helios University Hospital, University Witten/Herdecke, 42283 Wuppertal, Germany
- 2Department of Orthopedics and Trauma Surgery, Medical Faculty, University Duesseldorf, 40225 Duesseldorf, Germany
Abstract
Background/Objectives: Hip osteoarthritis (OA) is a highly prevalent musculoskeletal disorder associated with substantial functional impairment and healthcare burden. Although evidence-based clinical practice guidelines exist, including the 2021 guideline of the German Society for Orthopaedics and Trauma Surgery (DGOU), their implementation in routine practice remains incomplete. Large language models (LLMs) are increasingly consulted for guideline-related information, but the concordance of their answers with guideline recommendations is uncertain. This benchmark study evaluated the guideline concordance, comprehensiveness and length of answers generated by five widely used LLMs to questions derived from the 2021 German S3 guideline on evidence- and consensus-based indication criteria for total hip arthroplasty in hip OA (EKIT-Hip; AWMF register no. 187-001; lead society DGOU). It did not evaluate the effectiveness or safety of clinical decision support. Methods: In a cross-sectional, single-run benchmark study, 26 structured, context-free questions covering five of the six thematic complexes of the guideline (diagnosis and imaging; assessment of the level of suffering; evaluation of alternative therapeutic options; contraindications; optimisation of modifiable risk factors) were derived from the guideline recommendations and submitted between 9 November 2024 and 25 February 2025 through the consumer web interfaces of five LLM-based systems (GPT-4, GPT-4o, GPT-4o mini, Gemini Advanced and Consensus Premium), yielding 130 responses (one per system–question pair, each in a newly initiated chat session). Responses were evaluated by three board-certified orthopaedic surgeons blinded to model identity, using a four-point accuracy scale and a five-point comprehensiveness scale. Only a single consensus score per response was recorded, so inter-rater reliability could not be assessed. Response length was quantified in words and characters and benchmarked against the corresponding guideline text. Because all systems answered the same questions, ratings were analysed as paired, question-clustered data with cumulative-link mixed models (random intercept for question), Friedman and Cochran’s Q tests and question-level cluster-bootstrap confidence intervals; unpaired analyses were retained as sensitivity analyses. Results: Accuracy differed across systems (cumulative-link mixed model, likelihood-ratio χ2(4) = 49.0, p < 0.0001; paired Friedman χ2(4) = 41.7, p < 0.0001; unpaired Kruskal–Wallis sensitivity analysis H = 35.6, p < 0.0001). Mean accuracy was highest for GPT-4o (3.69 ± 0.55) and GPT-4 (3.62 ± 0.57), followed by GPT-4o mini (3.27 ± 0.78), Gemini Advanced (3.08 ± 0.84) and Consensus Premium (2.54 ± 0.71). Responses rated good or excellent accounted for 25/26 (96.2%) of GPT-4o and GPT-4 responses versus 15/26 (57.7%) for Consensus Premium. Comprehensiveness followed the same ordering (H = 33.0, p < 0.0001, ε2 = 0.23), with GPT-4o achieving the highest mean score (4.31 ± 0.79). In the cumulative-link mixed model, the odds of a higher accuracy rating relative to Consensus Premium were 51.8 (95% CI 12.4–215.9) for GPT-4o and 35.4 (95% CI 9.2–136.4) for GPT-4; GPT-4o and GPT-4 did not differ (OR 1.47, multiplicity-adjusted 95% CI 0.23–9.21), and their paired mean difference in accuracy (0.08 points, 90% CI −0.11 to 0.26) lay within an equivalence margin of ±0.5 points. Responses rated poor were observed for Consensus Premium (2/26) and Gemini Advanced (1/26) and not for the ChatGPT variants (0/26 each; exact 95% CI 0–13.2%), so a clinically relevant error rate cannot be excluded for any system. Response length differed markedly (Friedman χ2 = 47.3, p < 0.0001), ranging from 32.4 ± 11.5 words for Consensus Premium to 77.8 ± 24.9 words for GPT-4o, corresponding to 1.29-fold and 3.19-fold the length of the source guideline text, respectively. Conclusions: GPT-4o and GPT-4 received the highest ratings for accuracy and comprehensiveness in this single-run benchmark of static, context-free questions. These data describe guideline-concordant answer quality only; they do not establish clinical decision-support effectiveness or safety and do not support unsupervised clinical use of any evaluated system. Evaluation with standardised patient cases, repeated version-controlled runs, clinician-in-the-loop designs and prospective safety endpoints is required.
Related
- Receipt of guideline-concordant prenatal care among Medicaid-insured women with disabilities in Texas
- Association of Demographic, Economic, and Geographic Factors With Guideline-Concordant Infection Prophylaxis in Patients With Antineutrophil Cytoplasmic Antibody-Associated Vasculitis.
- EM-BENCH: A Reproducible Framework for Benchmarking Guideline-Concordant Clinical Decision Making in Large Language Models
- Comparative evaluation of large language models for guideline-concordant decision support in colorectal cancer multidisciplinary team care: A source-masked retrospective study
- Predictors of guideline-concordant treatment intensification in metastatic castration-sensitive prostate cancer: an explainable machine learning study
BibTeX
@article{alalman2026guidelineconcordant,
title = {{Guideline-Concordant Answer Quality of Five Large Language Models on Indication Criteria for Total Hip Arthroplasty in Hip Osteoarthritis: A Single-Run Benchmark Against the 2021 German S3 Guideline (EKIT-Hip)}},
author = {Obada Alalman and Max J. Scheyerer and Wajjahat Syed and Maher Ghandour and Matthias Hermasch and Christoph Beyersdorf and Mohamad Agha Mahmoud and Carl Neuerburg and Koroush Kabir and Ümit Mert},
journal = {Journal of Clinical Medicine},
year = {2026},
month = oct,
doi = {10.3390/jcm15207944},
url = {https://www.mdpi.com/2077-0383/15/20/7944},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Guideline-Concordant Answer Quality of Five Large Language Models on Indication Criteria for Total Hip Arthroplasty in Hip Osteoarthritis: A Single-Run Benchmark Against the 2021 German S3 Guideline (EKIT-Hip)
AU - Obada Alalman
AU - Max J. Scheyerer
AU - Wajjahat Syed
AU - Maher Ghandour
AU - Matthias Hermasch
AU - Christoph Beyersdorf
AU - Mohamad Agha Mahmoud
AU - Carl Neuerburg
AU - Koroush Kabir
AU - Ümit Mert
PY - 2026
DA - 2026/10/01
JO - Journal of Clinical Medicine
DO - 10.3390/jcm15207944
UR - https://www.mdpi.com/2077-0383/15/20/7944
AB - Background/Objectives: Hip osteoarthritis (OA) is a highly prevalent musculoskeletal disorder associated with substantial functional impairment and healthcare burden. Although evidence-based clinical practice guidelines exist, including the 2021 guideline of the German Society for Orthopaedics and Trauma Surgery (DGOU), their implementation in routine practice remains incomplete. Large language models (LLMs) are increasingly consulted for guideline-related information, but the concordance of their answers with guideline recommendations is uncertain. This benchmark study evaluated the guideline concordance, comprehensiveness and length of answers generated by five widely used LLMs to questions derived from the 2021 German S3 guideline on evidence- and consensus-based indication criteria for total hip arthroplasty in hip OA (EKIT-Hip; AWMF register no. 187-001; lead society DGOU). It did not evaluate the effectiveness or safety of clinical decision support. Methods: In a cross-sectional, single-run benchmark study, 26 structured, context-free questions covering five of the six thematic complexes of the guideline (diagnosis and imaging; assessment of the level of suffering; evaluation of alternative therapeutic options; contraindications; optimisation of modifiable risk factors) were derived from the guideline recommendations and submitted between 9 November 2024 and 25 February 2025 through the consumer web interfaces of five LLM-based systems (GPT-4, GPT-4o, GPT-4o mini, Gemini Advanced and Consensus Premium), yielding 130 responses (one per system–question pair, each in a newly initiated chat session). Responses were evaluated by three board-certified orthopaedic surgeons blinded to model identity, using a four-point accuracy scale and a five-point comprehensiveness scale. Only a single consensus score per response was recorded, so inter-rater reliability could not be assessed. Response length was quantified in words and characters and benchmarked against the corresponding guideline text. Because all systems answered the same questions, ratings were analysed as paired, question-clustered data with cumulative-link mixed models (random intercept for question), Friedman and Cochran’s Q tests and question-level cluster-bootstrap confidence intervals; unpaired analyses were retained as sensitivity analyses. Results: Accuracy differed across systems (cumulative-link mixed model, likelihood-ratio χ2(4) = 49.0, p < 0.0001; paired Friedman χ2(4) = 41.7, p < 0.0001; unpaired Kruskal–Wallis sensitivity analysis H = 35.6, p < 0.0001). Mean accuracy was highest for GPT-4o (3.69 ± 0.55) and GPT-4 (3.62 ± 0.57), followed by GPT-4o mini (3.27 ± 0.78), Gemini Advanced (3.08 ± 0.84) and Consensus Premium (2.54 ± 0.71). Responses rated good or excellent accounted for 25/26 (96.2%) of GPT-4o and GPT-4 responses versus 15/26 (57.7%) for Consensus Premium. Comprehensiveness followed the same ordering (H = 33.0, p < 0.0001, ε2 = 0.23), with GPT-4o achieving the highest mean score (4.31 ± 0.79). In the cumulative-link mixed model, the odds of a higher accuracy rating relative to Consensus Premium were 51.8 (95% CI 12.4–215.9) for GPT-4o and 35.4 (95% CI 9.2–136.4) for GPT-4; GPT-4o and GPT-4 did not differ (OR 1.47, multiplicity-adjusted 95% CI 0.23–9.21), and their paired mean difference in accuracy (0.08 points, 90% CI −0.11 to 0.26) lay within an equivalence margin of ±0.5 points. Responses rated poor were observed for Consensus Premium (2/26) and Gemini Advanced (1/26) and not for the ChatGPT variants (0/26 each; exact 95% CI 0–13.2%), so a clinically relevant error rate cannot be excluded for any system. Response length differed markedly (Friedman χ2 = 47.3, p < 0.0001), ranging from 32.4 ± 11.5 words for Consensus Premium to 77.8 ± 24.9 words for GPT-4o, corresponding to 1.29-fold and 3.19-fold the length of the source guideline text, respectively. Conclusions: GPT-4o and GPT-4 received the highest ratings for accuracy and comprehensiveness in this single-run benchmark of static, context-free questions. These data describe guideline-concordant answer quality only; they do not establish clinical decision-support effectiveness or safety and do not support unsupervised clinical use of any evaluated system. Evaluation with standardised patient cases, repeated version-controlled runs, clinician-in-the-loop designs and prospective safety endpoints is required.
SN - 2077-0383
ER -