Automating the triage of rheumatology outpatient referrals: a comparative evaluation of 23 large language models under simple and advanced prompting.
- 1Faculty of Medicine, Nursing, and Health Sciences, Monash University, Clayton, Australia. [email protected].
- 2Digital & Information Division, Monash Health, 246 Clayton Rd, Melbourne, VIC, 3168, Australia. [email protected].
- 3Department of Rheumatology, Monash Health, Melbourne, VIC, Australia. [email protected].
Abstract
Triage of rheumatology outpatient referrals is a high-volume administrative task that consumes senior specialist time without advancing patient care. The human triage system is only moderately accurate and reproducible. We assessed whether contemporary large language models (LLMs) are able to perform well enough to support automating this task in practice. In addition, the effects of different prompting techniques on triage accuracy and cost was assessed to help identify the optimal approach. Twenty referral scenarios spanning the urgency spectrum were created by a certified Australian rheumatologist. Four rheumatologists triaged all cases independently and blinded to produce a consensus reference standard; inter-rater reliability was quantified. Twenty-three LLMs each triaged every referral into one of five urgency categories, three times (1380 outputs per referral). The experiment was run with a simple prompt and repeated with an advanced prompt supplying explicit triage expectations and worked examples. In line with published literature, agreement among the four rheumatologists was moderate (Fleiss kappa 0.60; mean pairwise linear-weighted kappa 0.79), with all four assigning categories within one level of each other in 90% of cases. All 2760 model calls returned an interpretable result. Under the simple prompt, performance separated into distinct tiers, larger models were more accurate (Spearman rho = 0.42; P=.047) and accuracy tracked cost. Advanced prompting reduced between-model variance in accuracy 5.3-fold (0.014 to 0.003; Levene P=.01), abolished the size-accuracy association (rho=-0.05; P=.83) and removed the accuracy-cost relationship. The best models matched expert consensus on 18 of 20 cases, comparable to the reproducibility observed among the rheumatologists themselves. Under-triage errors persisted with some LLMs. Contemporary LLMs categorised rheumatology referral urgency with an accuracy and reproducibility comparable to that reported for human triage systems, although human triage was not tested in this study. Advanced prompting appeared to substitute for the reasoning capability of larger models, suggesting that adequate performance on this task may not require the most expensive models. These findings indicate that automation of this administrative task is technically feasible. Candidate models that warrant prospective evaluation are identified.
Related
- Diagnostic Yield of Internal Medicine Referrals to Rheumatology: A Retrospective Single-Center Study
- Referrals for suicidality to a UK child and adolescent mental health service: an evaluation of the policy and practice context of response to suicidal young people.
- Comparative cost analysis of virtual consultation for outpatient service in Malaysian public health clinics
- Prospective evaluation of a real-time AI assistant for outpatient documentation and quality control
- Evaluation of Fibromyalgia Frequency in Chronic Spontaneous Urticaria Patients Referred to a Tertiary Dermatology Outpatient Clinic.
BibTeX
@article{roberts2026automating,
title = {{Automating the triage of rheumatology outpatient referrals: a comparative evaluation of 23 large language models under simple and advanced prompting.}},
author = {Lynden Roberts},
journal = {Rheumatology international},
year = {2026},
month = oct,
doi = {10.1007/s00296-026-06317-8},
url = {https://europepmc.org/article/MED/42853295},
note = {CC BY},
}
RIS
TY - JOUR
TI - Automating the triage of rheumatology outpatient referrals: a comparative evaluation of 23 large language models under simple and advanced prompting.
AU - Lynden Roberts
PY - 2026
DA - 2026/10/09
JO - Rheumatology international
DO - 10.1007/s00296-026-06317-8
UR - https://europepmc.org/article/MED/42853295
AB - Triage of rheumatology outpatient referrals is a high-volume administrative task that consumes senior specialist time without advancing patient care. The human triage system is only moderately accurate and reproducible. We assessed whether contemporary large language models (LLMs) are able to perform well enough to support automating this task in practice. In addition, the effects of different prompting techniques on triage accuracy and cost was assessed to help identify the optimal approach. Twenty referral scenarios spanning the urgency spectrum were created by a certified Australian rheumatologist. Four rheumatologists triaged all cases independently and blinded to produce a consensus reference standard; inter-rater reliability was quantified. Twenty-three LLMs each triaged every referral into one of five urgency categories, three times (1380 outputs per referral). The experiment was run with a simple prompt and repeated with an advanced prompt supplying explicit triage expectations and worked examples. In line with published literature, agreement among the four rheumatologists was moderate (Fleiss kappa 0.60; mean pairwise linear-weighted kappa 0.79), with all four assigning categories within one level of each other in 90% of cases. All 2760 model calls returned an interpretable result. Under the simple prompt, performance separated into distinct tiers, larger models were more accurate (Spearman rho = 0.42; P=.047) and accuracy tracked cost. Advanced prompting reduced between-model variance in accuracy 5.3-fold (0.014 to 0.003; Levene P=.01), abolished the size-accuracy association (rho=-0.05; P=.83) and removed the accuracy-cost relationship. The best models matched expert consensus on 18 of 20 cases, comparable to the reproducibility observed among the rheumatologists themselves. Under-triage errors persisted with some LLMs. Contemporary LLMs categorised rheumatology referral urgency with an accuracy and reproducibility comparable to that reported for human triage systems, although human triage was not tested in this study. Advanced prompting appeared to substitute for the reasoning capability of larger models, suggesting that adequate performance on this task may not require the most expensive models. These findings indicate that automation of this administrative task is technically feasible. Candidate models that warrant prospective evaluation are identified.
SN - 0172-8172
SN - 1437-160X
ER -