Comparative evaluation of large language models for guideline-concordant decision support in colorectal cancer multidisciplinary team care: A source-masked retrospective study
- 1Cancer Center, Yantai Affiliated Hospital of Shandong Medical and Pharmaceutical University, The Second Clinical Medical School of Shandong Medical and Pharmaceutical University, Yantai, Shandong 264100, China
- 2Cancer Center, Yantai Affiliated Hospital of Shandong Medical and Pharmaceutical University, The Second Clinical Medical School of Shandong Medical and Pharmaceutical University, Yantai, Shandong 264100, China; Corresponding authors.
- 3The Second Department of Gastrointestinal Surgery, Affiliated Hospital of North Sichuan Medical College, Nanchong, Sichuan 637000, China; Corresponding authors.
Abstract
Introduction: Multidisciplinary teams (MDTs) are central to colorectal cancer (CRC) care, but real-world recommendations may vary because of information overload, heterogeneous documentation, and imperfect guideline implementation. Large language models (LLMs) have emerged as potential adjunctive clinical decision-support tools, yet their performance within multidisciplinary oncology workflows remains uncertain. Methods: We conducted a source-masked, retrospective comparative study of treatment-naive patients discussed at an MDT between January 1, 2022 and December 31, 2024. De-identified case narratives and key test results were reformatted into a standardized prompt and submitted to five LLMs—DeepSeek-V3.2, Doubao-Seed-1.6, ChatGPT-5, Claude Sonnet 4.5, and Grok 4—using identical prompt templates. Each model was queried once per case in a fresh, independent chat session. Before evaluation, recommendations from the original MDT and the five LLMs were converted into a common source-neutral presentation template. Five experienced oncologists, who were not provided with recommendation-source labels, independently rated each plan using a 5-point Likert scale for clinical appropriateness and guideline concordance, referencing Chinese Society of Clinical Oncology and National Comprehensive Cancer Network CRC guideline versions contemporaneous with the original MDT decision period. Results: Ninety-one cases were included (41 colon and 50 rectal cancers). In the adjusted cumulative link mixed-effects model, all five LLMs had higher cumulative odds of receiving a higher guideline-concordance score than MDT recommendations (odds ratios, 9.7–25.7; all Holm-adjusted P < 0.001). Descriptively, the largest score differences were observed in rectal cancer. LLM outputs more frequently included de-escalation strategies in Stage I disease and neoadjuvant-treatment components in Stage III rectal cancer. Inter-rater agreement varied across sources and was highest for MDT recommendations by intraclass correlation coefficient (2, k ). Conclusions: In this retrospective source-masked evaluation, LLM-generated CRC recommendations received higher reviewer-assigned guideline-concordance ratings than real-world MDT recommendations. These findings support prospective evaluation of LLMs as adjunctive tools for guideline-concordance auditing and documentation support rather than as substitutes for MDT judgment. Because clinical safety, patient outcomes, repeated-generation reproducibility, and successful source masking were not independently established, higher guideline-concordance ratings should not be interpreted as evidence of superior individualized care or clinical safety.
Related
- Application of a Quantitative POCT-FIT Based on Near-Infrared Fluorescence Immunochromatography in Population-Based Colorectal Cancer Screening: A Comparative Evaluation with a Centralized Testing System
- Comparative evaluation of two automated magnetic-bead cfDNA extraction workflows in early colorectal cancer
- Comparative outcomes of natural orifice specimen extraction surgery versus conventional minilaparotomy-assisted specimen extraction in colorectal cancer: A single-center retrospective study.
- Multidisciplinary approach and rehabilitation practice for defecatory disorders in Italy: a snapshot survey on behalf of the Italian Society of Colorectal Surgery (SICCR).
- Comparative evaluation of metagenomic and targeted next-generation sequencing for pathogen detection in bronchoalveolar lavage fluid from patients with lower respiratory tract infections: a retrospective cohort study.
BibTeX
@article{zhang2026comparative,
title = {{Comparative evaluation of large language models for guideline-concordant decision support in colorectal cancer multidisciplinary team care: A source-masked retrospective study}},
author = {Zheng Zhang and Qi Zhao and Shuang Han and Liang Hao and Yuhui Shang and Kaizhen Xu and Wenjian Hu and Wenyu Luan and Zhenxi Xu and Mayin Shan and Lina Wang and Wei Zhang and Xiaolong Tang and Yandong Miao},
journal = {Intelligent Oncology},
year = {2026},
month = oct,
doi = {10.1016/j.intonc.2026.100090},
url = {https://doi.org/10.1016/j.intonc.2026.100090},
note = {CC BY},
}
RIS
TY - JOUR
TI - Comparative evaluation of large language models for guideline-concordant decision support in colorectal cancer multidisciplinary team care: A source-masked retrospective study
AU - Zheng Zhang
AU - Qi Zhao
AU - Shuang Han
AU - Liang Hao
AU - Yuhui Shang
AU - Kaizhen Xu
AU - Wenjian Hu
AU - Wenyu Luan
AU - Zhenxi Xu
AU - Mayin Shan
AU - Lina Wang
AU - Wei Zhang
AU - Xiaolong Tang
AU - Yandong Miao
PY - 2026
DA - 2026/10/25
JO - Intelligent Oncology
DO - 10.1016/j.intonc.2026.100090
UR - https://doi.org/10.1016/j.intonc.2026.100090
AB - Introduction: Multidisciplinary teams (MDTs) are central to colorectal cancer (CRC) care, but real-world recommendations may vary because of information overload, heterogeneous documentation, and imperfect guideline implementation. Large language models (LLMs) have emerged as potential adjunctive clinical decision-support tools, yet their performance within multidisciplinary oncology workflows remains uncertain. Methods: We conducted a source-masked, retrospective comparative study of treatment-naive patients discussed at an MDT between January 1, 2022 and December 31, 2024. De-identified case narratives and key test results were reformatted into a standardized prompt and submitted to five LLMs—DeepSeek-V3.2, Doubao-Seed-1.6, ChatGPT-5, Claude Sonnet 4.5, and Grok 4—using identical prompt templates. Each model was queried once per case in a fresh, independent chat session. Before evaluation, recommendations from the original MDT and the five LLMs were converted into a common source-neutral presentation template. Five experienced oncologists, who were not provided with recommendation-source labels, independently rated each plan using a 5-point Likert scale for clinical appropriateness and guideline concordance, referencing Chinese Society of Clinical Oncology and National Comprehensive Cancer Network CRC guideline versions contemporaneous with the original MDT decision period. Results: Ninety-one cases were included (41 colon and 50 rectal cancers). In the adjusted cumulative link mixed-effects model, all five LLMs had higher cumulative odds of receiving a higher guideline-concordance score than MDT recommendations (odds ratios, 9.7–25.7; all Holm-adjusted P < 0.001). Descriptively, the largest score differences were observed in rectal cancer. LLM outputs more frequently included de-escalation strategies in Stage I disease and neoadjuvant-treatment components in Stage III rectal cancer. Inter-rater agreement varied across sources and was highest for MDT recommendations by intraclass correlation coefficient (2, k ). Conclusions: In this retrospective source-masked evaluation, LLM-generated CRC recommendations received higher reviewer-assigned guideline-concordance ratings than real-world MDT recommendations. These findings support prospective evaluation of LLMs as adjunctive tools for guideline-concordance auditing and documentation support rather than as substitutes for MDT judgment. Because clinical safety, patient outcomes, repeated-generation reproducibility, and successful source masking were not independently established, higher guideline-concordance ratings should not be interpreted as evidence of superior individualized care or clinical safety.
SN - 2950-2616
SN - 2097-7131
ER -