An LLM-based decision aid for collaborative decision-making in juvenile idiopathic arthritis: A multimethod evaluation
12 affiliations
- 1Institute of Health Policy, Management, and Evaluation, University of Toronto, Toronto, Ontario, Canada.
- 2Centre for Digital Therapeutics, University Health Network, Toronto, Ontario, Canada.
- 3Department of Pediatrics, Cumming School of Medicine, University of Calgary, Calgary, Albereta, Canada.
- 4Children's Health Ireland, and University College Dublin, Trinity College Dublin, Dublin, Ireland.
- 5Child Health Evaluative Sciences, Research Institute, The Hospital for Sick Children, Toronto, Ontario, Canada.
- 6Department of Computer Science, University of Toronto, Toronto, Ontario, Canada.
- 7Telfer School of Management, University of Ottawa, Ottawa, Ontario, Canada.
- 8Department of Pediatric Rheumatology and Immunology, Wilhelmina, Children's Hospital, University Medical Center Utrecht, Utrecht, The Netherlands.
- 9Faculty of Medicine, Utrecht University, Utrecht, The Netherlands.
- 10Center for Translational Immunology, University Medical Center Utrecht, Utrecht, The Netherlands.
- 11Department of Immunology and Medical Science, University of Toronto, Toronto, Ontario, Canada.
- 12Division of Rheumatology, The Hospital for Sick Children, Toronto, Ontario, Canada.
Abstract
Juvenile idiopathic arthritis (JIA) is the most common pediatric chronic inflammatory rheumatic condition, with substantial quality-of-life impacts. Treatment selection is complex due to limited guideline evidence, and difficult trade-offs. Decisions are often solely physician driven. We evaluated the feasibility of Large Language Model-based Decision Aid (LLM-DA) designed to support collaborative rather than solely physician decision-making in JIA care. A multi-method feasibility study was conducted in accordance with the TRIPOD-LLM checklist. The aims were twofold: first, to assess the quality of the LLM-DA (OpenAI’s o3 model) based on evaluations from three pediatric rheumatologists and an exploratory assessment by an LLM-as-a-Judge (OpenAI’s GPT-5 model); and second, to characterize perceived value of the tool from the patient perspective through a mixed methods evaluation (n = 34). For 10 de-identified use cases, rheumatologists and LLM-Judge generated treatment recommendations and scored the LLM-DA outputs on safety, factual consistency, treatment appropriateness, and preference acknowledgment. Rheumatologists rated 97% of outputs as acceptable for safety and preference acknowledgment, 60% for factual consistency, 70% for appropriateness. Treatment recommendation agreement among all evaluators and real-world data was low at <30%. Inter-rater agreement for scores among rheumatologists was almost perfect for safety and preference acknowledgement, moderate for treatment appropriateness, and poor for factual consistency, and remained the same or improved with LLM-as-a-Judge. Variability in evaluator scoring and treatment recommendations may reflect the inherent heterogeneity in acceptable JIA treatment pathways and potential limitations in the LLM-DA’s decision-making logic. Five patient themes were identified: the tool complements rheumatologist expertise, prepares patients through reflection and actionable information, builds confidence for self-advocacy, has variable value depending on the patient-physician relationship, and has potential for broader JIA self-management. These exploratory findings provide a proof-of-concept for using AI to support collaborative decision-making, while highlighting the need for future studies to determine whether LLMs can reliably support clinical practice. Author summary: Our work investigates the feasibility of using large language models (LLMs), to support treatment planning for juvenile idiopathic arthritis (JIA). JIA is hard to treat because evidence in clinical guidelines is low, so physicians often rely on their experiential knowledge to make decisions, and medications can be hard to manage due to harsh side effects. Given these challenges, there is a unique opportunity to investigate whether decision aids that use LLMs can help patients get more involved in their care. Our research provides preliminary evidence that the LLM-DA may support physician-mediated discussions, particularly by helping patients reflect on preferences and prepare for appointments. We also interviewed 34 individuals diagnosed with JIA and found that they thought the decision aid’s outputs were useful and felt more confident to be involved in the decision-making process. By helping young patients prepare, reflect on their preferences, and speak up during appointments, this tool shows potential to support more collaborative conversations.
Related
- Combining A Massively Parallel Reporter Assay and Human Data to Elucidate Genetic Mechanisms Driving Risk for Juvenile Idiopathic Arthritis
- Juvenile idiopathic arthritis and rheumatoid arthritis: a narrative review of age-dependent, subtype-specific immune mechanisms, biomarkers, and treatment strategies
- Shared pathogenic genes and therapeutic targets in periodontitis and systemic juvenile idiopathic arthritis: An integrative bioinformatics and machine learning study
- Realist Collaborative Evaluation of a Work Disability Prevention Program for Breast Cancer Survivors: Protocol for a RECOVA-FASTRACS Realist Evaluation
- Evaluation of the collaborative process of public-private partnerships in the veterinary domain: The case of the sanitary mandate in Tunisia
BibTeX
@article{barbaric2026llmbased,
title = {{An LLM-based decision aid for collaborative decision-making in juvenile idiopathic arthritis: A multimethod evaluation}},
author = {Antonia Barbaric and Susanne M Benseler and Chitra Lalloo and Xinyi Celine Liu and Alex Mariakakis and Quynh Pham and Sebastiaan J Vastert and Rae S M Yeung and Joseph A Cafazzo and Joost F Swart},
journal = {PLOS Digital Health},
year = {2026},
month = oct,
doi = {10.1371/journal.pdig.0001655},
url = {https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001655},
note = {CC BY},
}
RIS
TY - JOUR
TI - An LLM-based decision aid for collaborative decision-making in juvenile idiopathic arthritis: A multimethod evaluation
AU - Antonia Barbaric
AU - Susanne M Benseler
AU - Chitra Lalloo
AU - Xinyi Celine Liu
AU - Alex Mariakakis
AU - Quynh Pham
AU - Sebastiaan J Vastert
AU - Rae S M Yeung
AU - Joseph A Cafazzo
AU - Joost F Swart
PY - 2026
DA - 2026/10/22
JO - PLOS Digital Health
DO - 10.1371/journal.pdig.0001655
UR - https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001655
AB - Juvenile idiopathic arthritis (JIA) is the most common pediatric chronic inflammatory rheumatic condition, with substantial quality-of-life impacts. Treatment selection is complex due to limited guideline evidence, and difficult trade-offs. Decisions are often solely physician driven. We evaluated the feasibility of Large Language Model-based Decision Aid (LLM-DA) designed to support collaborative rather than solely physician decision-making in JIA care. A multi-method feasibility study was conducted in accordance with the TRIPOD-LLM checklist. The aims were twofold: first, to assess the quality of the LLM-DA (OpenAI’s o3 model) based on evaluations from three pediatric rheumatologists and an exploratory assessment by an LLM-as-a-Judge (OpenAI’s GPT-5 model); and second, to characterize perceived value of the tool from the patient perspective through a mixed methods evaluation (n = 34). For 10 de-identified use cases, rheumatologists and LLM-Judge generated treatment recommendations and scored the LLM-DA outputs on safety, factual consistency, treatment appropriateness, and preference acknowledgment. Rheumatologists rated 97% of outputs as acceptable for safety and preference acknowledgment, 60% for factual consistency, 70% for appropriateness. Treatment recommendation agreement among all evaluators and real-world data was low at <30%. Inter-rater agreement for scores among rheumatologists was almost perfect for safety and preference acknowledgement, moderate for treatment appropriateness, and poor for factual consistency, and remained the same or improved with LLM-as-a-Judge. Variability in evaluator scoring and treatment recommendations may reflect the inherent heterogeneity in acceptable JIA treatment pathways and potential limitations in the LLM-DA’s decision-making logic. Five patient themes were identified: the tool complements rheumatologist expertise, prepares patients through reflection and actionable information, builds confidence for self-advocacy, has variable value depending on the patient-physician relationship, and has potential for broader JIA self-management. These exploratory findings provide a proof-of-concept for using AI to support collaborative decision-making, while highlighting the need for future studies to determine whether LLMs can reliably support clinical practice. Author summary: Our work investigates the feasibility of using large language models (LLMs), to support treatment planning for juvenile idiopathic arthritis (JIA). JIA is hard to treat because evidence in clinical guidelines is low, so physicians often rely on their experiential knowledge to make decisions, and medications can be hard to manage due to harsh side effects. Given these challenges, there is a unique opportunity to investigate whether decision aids that use LLMs can help patients get more involved in their care. Our research provides preliminary evidence that the LLM-DA may support physician-mediated discussions, particularly by helping patients reflect on preferences and prepare for appointments. We also interviewed 34 individuals diagnosed with JIA and found that they thought the decision aid’s outputs were useful and felt more confident to be involved in the decision-making process. By helping young patients prepare, reflect on their preferences, and speak up during appointments, this tool shows potential to support more collaborative conversations.
SN - 2767-3170
ER -