Large language models in rheumatology: applications, methodological quality and clinical translation – a systematic review
- 1Division of Rheumatology and Systemic Inflammatory Diseases, III. Department of Medicine, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
- 2Department of Rheumatology and Clinical Immunology, Charité – University Medical Center Berlin, Berlin, Germany
Abstract
Objectives: To map applications of large language models (LLMs) in rheumatology and evaluate the methodological quality and clinical translation of the evidence.
Methods: MEDLINE, Embase and Web of Science were searched from inception to 13 July 2026 for primary studies of LLMs in rheumatology. Two reviewers independently screened records and analysed data. Each study was assigned to one application domain and to the highest level of a review-derived translational maturity framework. Risk of bias was assessed with the Alberta Quality Assessment Tool: Risk of Bias (AQAT:RoB; worst-item rule), where human evaluation of free-text answers was reported; findings were synthesised narratively.
Results: 93 studies were included, 79 (84.9%) from 2025 or 2026. Patient education (39, 41.9%) and clinical decision support (38, 40.9%) dominated. Accuracy in clinical decision support was strongly task-dependent, with medians of 90.9% for structured diagnostic tasks and 40.0% for open-ended differential diagnosis (best-performing model per study). Patient-facing output had a median Flesch-Kincaid grade of 12.2; none of the 13 readability studies met the recommended sixth-grade target. 22 studies (23.7%) used real-world clinical data, mostly retrospectively; only four (4.3%) evaluated an LLM prospectively in a clinical workflow. Prompts were fully reported in 59.1% of studies, and answer stability was tested in 19.4%. Among the 68 studies amenable to AQAT:RoB assessment, five (7.4%) were at low risk of bias.
Conclusions: LLM research in rheumatology is expanding rapidly but remains dominated by prompt-response benchmarks with surrogate outcomes, incomplete reporting and, among AQAT:RoB-assessable studies, predominantly unclear or high risk of bias. Whether LLMs improve rheumatology care remains unresolved; prospective evaluations with patient-relevant endpoints are required.
Related
- Clinical translation of artificial intelligence in musculoskeletal care: a systematic review of current applications, evidence gaps, and implementation readiness
- Automating the triage of rheumatology outpatient referrals: a comparative evaluation of 23 large language models under simple and advanced prompting.
- Synthetic data in rheumatology: a systematic literature review.
- Certified large language model-based diagnostic decision support in rheumatology: the ALLIANCE multicentre randomised controlled trial
- Translation as cultural dialogue: An explorative methodological approach to translation reconciliation of DLI-Est
BibTeX
@article{kremer2026large,
title = {{Large language models in rheumatology: applications, methodological quality and clinical translation – a systematic review}},
author = {Phillip Kremer and Isabell Haase and Zhivana Boyadzhieva and Martin Krusche},
journal = {RMD Open},
year = {2026},
month = oct,
doi = {10.1136/rmdopen-2026-007538},
url = {https://doi.org/10.1136/rmdopen-2026-007538},
note = {CC BY-NC 4.0},
}
RIS
TY - JOUR
TI - Large language models in rheumatology: applications, methodological quality and clinical translation – a systematic review
AU - Phillip Kremer
AU - Isabell Haase
AU - Zhivana Boyadzhieva
AU - Martin Krusche
PY - 2026
DA - 2026/10/28
JO - RMD Open
DO - 10.1136/rmdopen-2026-007538
UR - https://doi.org/10.1136/rmdopen-2026-007538
AB - Objectives: To map applications of large language models (LLMs) in rheumatology and evaluate the methodological quality and clinical translation of the evidence. Methods: MEDLINE, Embase and Web of Science were searched from inception to 13 July 2026 for primary studies of LLMs in rheumatology. Two reviewers independently screened records and analysed data. Each study was assigned to one application domain and to the highest level of a review-derived translational maturity framework. Risk of bias was assessed with the Alberta Quality Assessment Tool: Risk of Bias (AQAT:RoB; worst-item rule), where human evaluation of free-text answers was reported; findings were synthesised narratively. Results: 93 studies were included, 79 (84.9%) from 2025 or 2026. Patient education (39, 41.9%) and clinical decision support (38, 40.9%) dominated. Accuracy in clinical decision support was strongly task-dependent, with medians of 90.9% for structured diagnostic tasks and 40.0% for open-ended differential diagnosis (best-performing model per study). Patient-facing output had a median Flesch-Kincaid grade of 12.2; none of the 13 readability studies met the recommended sixth-grade target. 22 studies (23.7%) used real-world clinical data, mostly retrospectively; only four (4.3%) evaluated an LLM prospectively in a clinical workflow. Prompts were fully reported in 59.1% of studies, and answer stability was tested in 19.4%. Among the 68 studies amenable to AQAT:RoB assessment, five (7.4%) were at low risk of bias. Conclusions: LLM research in rheumatology is expanding rapidly but remains dominated by prompt-response benchmarks with surrogate outcomes, incomplete reporting and, among AQAT:RoB-assessable studies, predominantly unclear or high risk of bias. Whether LLMs improve rheumatology care remains unresolved; prospective evaluations with patient-relevant endpoints are required.
SN - 2056-5933
ER -