Certified large language model-based diagnostic decision support in rheumatology: the ALLIANCE multicentre randomised controlled trial
- 1Philipps-Universität Marburg, School of Medicine, Institute for Digital Medicine, Marburg, Germany
Abstract
ObjectivesTo evaluate whether access to a certified large language model (LLM)-based clinical decision support system improves physician diagnostic performance in rheumatology compared with conventional diagnostic resources alone.
MethodsIn this multicentre, open-label, randomised controlled trial, 82 physicians from seven hospitals in two countries were randomised 1:1 to conventional diagnostic resources plus Prof. Valmed or conventional resources alone. Participants assessed three rheumatology vignettes before and after assistance. The primary outcome was top-1 diagnostic accuracy. Secondary outcomes included top-3 accuracy, diagnostic reasoning, confidence, case-processing time and perceived support quality.
ResultsTop-1 accuracy increased from 22.2% to 33.3% in the intervention group and from 23.3% to 35.0% in the control group, with no between-group difference in improvement (adjusted OR 0.99, 95% CI 0.45 to 2.19; p=0.979). Differences in top-3 accuracy, diagnostic reasoning and confidence were also not significant. Assisted case-processing time was substantially shorter with LLM support (94 vs 206 s; adjusted mean difference -112 s, 95% CI -141 to -83; p<0.001). Information timeliness and perceived diagnostic support quality were rated significantly higher in the intervention group. Exploratory analyses showed persistent overconfidence and substantial AI over-reliance.
ConclusionsCertified LLM-based diagnostic support did not improve diagnostic accuracy compared with conventional resources, but substantially reduced case-processing time and improved perceived support quality. These findings suggest potential workflow benefits while highlighting overconfidence and over-reliance as important safety considerations.
Trial registration numberNCT07166692
WHAT IS ALREADY KNOWN ON THIS TOPICO_LIAccurate and timely diagnosis in rheumatology is challenging because workforce shortages coincide with non-specific, complex and often rare presentations. C_LIO_LILarge language models are increasingly used for diagnostic decision support and have shown strong performance in benchmarking studies. C_LIO_LIRandomised evidence for physician-facing diagnostic decision support, particularly in rheumatology and for certified LLM-based systems, is lacking. C_LI
WHAT THIS STUDY ADDSO_LIThe ALLIANCE trial is the first randomised controlled trial to evaluate physician-facing diagnostic decision support in rheumatology and the first to assess a medically certified LLM-based clinical decision support system in any clinical specialty. C_LIO_LIA certified LLM-based clinical decision support system did not improve top-1 diagnostic accuracy compared with conventional resources alone, as assistance led to similar gains in both groups. C_LIO_LIUse of a certified LLM-based clinical decision support system was associated with approximately half the assisted case-processing time of conventional resources alone, as well as higher perceived support quality. C_LI
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICYO_LILLM-based diagnostic decision support may be most useful for improving efficiency, support quality, and broadening differential diagnoses rather than increasing top-1 diagnostic accuracy. C_LIO_LISafe clinical implementation will require careful attention to overconfidence, over-reliance, transparency, and trust. C_LIO_LIFurther robust trials, ideally under real-world clinical conditions, are needed to define the role of clinical decision support in routine care. C_LI
Related
- Comparison of ‘quilting sutures’ with ‘conventional sutures and drain placement’ after mastectomy: protocol for the BE-Quilt – a multicentre, pragmatic randomised controlled trial
- Immobilisation versus early active motion following primary flexor pollicis longus tendon zones 1-2 repair: protocol for a registry-based multicentre randomised controlled trial in Sweden.
- Multicentre randomised controlled trial to evaluate the efficacy of DIALIVE 2.0, a liver dialysis device, for the treatment of acute-on-chronic liver failure: study protocol
- Effectiveness and safety of electroacupuncture as an adjunct to guideline-based usual care for acute lumbar disc herniation: a multicentre, randomised, sham-controlled trial protocol.
- Primary repair during the neonatal period for tetralogy of Fallot (RIFAN): a multicentre randomised controlled trial protocol in China
BibTeX
@misc{kremer2026certified,
title = {{Certified large language model-based diagnostic decision support in rheumatology: the ALLIANCE multicentre randomised controlled trial}},
author = {P. Kremer and N. Schlicker and R. Hasnaj and J. Bamberger and T. Witte and I. Haase and A. Mayr and C. Schmidt and N. Osteras and X. Baraliakos and S. Kuhn and M. Krusche and J. Knitza},
howpublished = {medRxiv},
year = {2026},
month = aug,
doi = {10.64898/2026.08.29.26361715},
url = {https://www.medrxiv.org/content/10.64898/2026.08.29.26361715v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Certified large language model-based diagnostic decision support in rheumatology: the ALLIANCE multicentre randomised controlled trial
AU - P. Kremer
AU - N. Schlicker
AU - R. Hasnaj
AU - J. Bamberger
AU - T. Witte
AU - I. Haase
AU - A. Mayr
AU - C. Schmidt
AU - N. Osteras
AU - X. Baraliakos
AU - S. Kuhn
AU - M. Krusche
AU - J. Knitza
PY - 2026
DA - 2026/08/29
PB - medRxiv
DO - 10.64898/2026.08.29.26361715
UR - https://www.medrxiv.org/content/10.64898/2026.08.29.26361715v1
AB - ObjectivesTo evaluate whether access to a certified large language model (LLM)-based clinical decision support system improves physician diagnostic performance in rheumatology compared with conventional diagnostic resources alone. MethodsIn this multicentre, open-label, randomised controlled trial, 82 physicians from seven hospitals in two countries were randomised 1:1 to conventional diagnostic resources plus Prof. Valmed or conventional resources alone. Participants assessed three rheumatology vignettes before and after assistance. The primary outcome was top-1 diagnostic accuracy. Secondary outcomes included top-3 accuracy, diagnostic reasoning, confidence, case-processing time and perceived support quality. ResultsTop-1 accuracy increased from 22.2% to 33.3% in the intervention group and from 23.3% to 35.0% in the control group, with no between-group difference in improvement (adjusted OR 0.99, 95% CI 0.45 to 2.19; p=0.979). Differences in top-3 accuracy, diagnostic reasoning and confidence were also not significant. Assisted case-processing time was substantially shorter with LLM support (94 vs 206 s; adjusted mean difference -112 s, 95% CI -141 to -83; p<0.001). Information timeliness and perceived diagnostic support quality were rated significantly higher in the intervention group. Exploratory analyses showed persistent overconfidence and substantial AI over-reliance. ConclusionsCertified LLM-based diagnostic support did not improve diagnostic accuracy compared with conventional resources, but substantially reduced case-processing time and improved perceived support quality. These findings suggest potential workflow benefits while highlighting overconfidence and over-reliance as important safety considerations. Trial registration numberNCT07166692 WHAT IS ALREADY KNOWN ON THIS TOPICO_LIAccurate and timely diagnosis in rheumatology is challenging because workforce shortages coincide with non-specific, complex and often rare presentations. C_LIO_LILarge language models are increasingly used for diagnostic decision support and have shown strong performance in benchmarking studies. C_LIO_LIRandomised evidence for physician-facing diagnostic decision support, particularly in rheumatology and for certified LLM-based systems, is lacking. C_LI WHAT THIS STUDY ADDSO_LIThe ALLIANCE trial is the first randomised controlled trial to evaluate physician-facing diagnostic decision support in rheumatology and the first to assess a medically certified LLM-based clinical decision support system in any clinical specialty. C_LIO_LIA certified LLM-based clinical decision support system did not improve top-1 diagnostic accuracy compared with conventional resources alone, as assistance led to similar gains in both groups. C_LIO_LIUse of a certified LLM-based clinical decision support system was associated with approximately half the assisted case-processing time of conventional resources alone, as well as higher perceived support quality. C_LI HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICYO_LILLM-based diagnostic decision support may be most useful for improving efficiency, support quality, and broadening differential diagnoses rather than increasing top-1 diagnostic accuracy. C_LIO_LISafe clinical implementation will require careful attention to overconfidence, over-reliance, transparency, and trust. C_LIO_LIFurther robust trials, ideally under real-world clinical conditions, are needed to define the role of clinical decision support in routine care. C_LI
ER -