Diagnostic and treatment recommendation performance of multimodal large language models in failed or painful total hip arthroplasty: an expert-rated benchmark study.
- 1Ortopedia-Traumatologia e Chirurgia protesica e dei reimpianti d'anca e di ginocchio, IRCCS Istituto Ortopedico Rizzoli, Bologna, Italy.
- 2Department of Industrial Engineering (DIN), Alma Mater Studiorum, Università di Bologna, Bologna, Italy.
- 3Dipartimento di Scienze Biomediche e Neuromotorie (DIBINEM), Alma Mater Studiorum, Università di Bologna, Bologna, Italy.
Abstract
The role of multimodal large language models in orthopaedic decision-making remains insufficiently explored, particularly in revision arthroplasty. This exploratory pilot benchmark study compared the diagnostic and treatment recommendation performance of generative pre-trained transformer (GPT)-4o and GPT-5 with that of human evaluators in cases of failed or painful total hip arthroplasty (THA). Twenty cases from a tertiary referral center were retrospectively selected and converted into standardized clinical vignettes, each accompanied by one representative radiological image. The cases were purposively selected, illustrative, with clear dominant failure mechanisms. Cases were independently assessed by GPT-4o, GPT-5, an arthroplasty fellow, an arthroplasty-trained fourth-year orthopaedic resident, and a third-year orthopaedic resident. Responses were rated by two senior reviewers on a 3-point ordinal scale for correctness and completeness. Paired comparisons across raters were performed on the ordinal scores. GPT-4o achieved fully correct and fully complete diagnoses in 60% and 55% of cases, respectively, compared with 80% and 80% for GPT-5. For treatment recommendations, GPT-4o achieved 60% fully correct and 45% fully complete responses, whereas GPT-5 achieved 85% and 70%, respectively. Overall paired comparisons on the ordinal scale showed significant differences across raters for diagnostic correctness (P<0.001), diagnostic completeness (P<0.001), treatment correctness (P=0.004), and treatment completeness (P=0.001). However, no pairwise difference between GPT-5 and GPT-4o remained significant after correction for multiple testing. In this small, purposively selected set of illustrative cases with relatively clear dominant failure mechanisms, GPT-5 showed consistent numerical improvement over GPT-4o across all evaluated domains, but neither model matched the highest-performing individual human raters. Multimodal large language models may support structured clinical reasoning in revision THA, but specialist oversight remains necessary, and these findings should not be extrapolated to more complex real-world revision scenarios.
Related
- Expert-Guided Visual Correction for Characterizing Diagnostic Performance and Error Patterns of Multimodal Large Language Models Using Periodontal In-Service Examination Images
- Conservative Treatment with Adjunctive Use of Teriparatide for Periprosthetic Humeral Fractures after Reverse Total Shoulder Arthroplasty: A Report of 6 Cases
- Multimodal neuroimaging and electrophysiological signatures in amnestic mild cognitive impairment with different amyloid-β deposition status: correlations with cognitive function and diagnostic value
- Navigation-Assisted and Robotic Total Hip, Knee, and Unicompartmental Arthroplasty: Reconciling Positioning Accuracy with Biomaterial Performance and Long-Term Implant Survival.
- Technical Feasibility of a Multimodal Electro-Optical and Infrared Imaging Pipeline for Postoperative Wound Assessment After Total Knee Arthroplasty: Formative Evaluation Study.
BibTeX
@article{castagnini2026diagnostic,
title = {{Diagnostic and treatment recommendation performance of multimodal large language models in failed or painful total hip arthroplasty: an expert-rated benchmark study.}},
author = {Francesco Castagnini and Marco Rotini and Mara Gorgone and Stefano Vitacca and Linda Castagnini and Serena Santoro and Rosalinda Ventimiglia and Francesco Traina},
journal = {Annals of joint},
year = {2026},
month = sep,
doi = {10.21037/aoj-2026-0025},
url = {https://europepmc.org/article/MED/42999487},
note = {CC BY-NC-ND},
}
RIS
TY - JOUR
TI - Diagnostic and treatment recommendation performance of multimodal large language models in failed or painful total hip arthroplasty: an expert-rated benchmark study.
AU - Francesco Castagnini
AU - Marco Rotini
AU - Mara Gorgone
AU - Stefano Vitacca
AU - Linda Castagnini
AU - Serena Santoro
AU - Rosalinda Ventimiglia
AU - Francesco Traina
PY - 2026
DA - 2026/09/29
JO - Annals of joint
DO - 10.21037/aoj-2026-0025
UR - https://europepmc.org/article/MED/42999487
AB - The role of multimodal large language models in orthopaedic decision-making remains insufficiently explored, particularly in revision arthroplasty. This exploratory pilot benchmark study compared the diagnostic and treatment recommendation performance of generative pre-trained transformer (GPT)-4o and GPT-5 with that of human evaluators in cases of failed or painful total hip arthroplasty (THA). Twenty cases from a tertiary referral center were retrospectively selected and converted into standardized clinical vignettes, each accompanied by one representative radiological image. The cases were purposively selected, illustrative, with clear dominant failure mechanisms. Cases were independently assessed by GPT-4o, GPT-5, an arthroplasty fellow, an arthroplasty-trained fourth-year orthopaedic resident, and a third-year orthopaedic resident. Responses were rated by two senior reviewers on a 3-point ordinal scale for correctness and completeness. Paired comparisons across raters were performed on the ordinal scores. GPT-4o achieved fully correct and fully complete diagnoses in 60% and 55% of cases, respectively, compared with 80% and 80% for GPT-5. For treatment recommendations, GPT-4o achieved 60% fully correct and 45% fully complete responses, whereas GPT-5 achieved 85% and 70%, respectively. Overall paired comparisons on the ordinal scale showed significant differences across raters for diagnostic correctness (P<0.001), diagnostic completeness (P<0.001), treatment correctness (P=0.004), and treatment completeness (P=0.001). However, no pairwise difference between GPT-5 and GPT-4o remained significant after correction for multiple testing. In this small, purposively selected set of illustrative cases with relatively clear dominant failure mechanisms, GPT-5 showed consistent numerical improvement over GPT-4o across all evaluated domains, but neither model matched the highest-performing individual human raters. Multimodal large language models may support structured clinical reasoning in revision THA, but specialist oversight remains necessary, and these findings should not be extrapolated to more complex real-world revision scenarios.
SN - 2415-6809
ER -