Large language models reveal a systematic preference in clinical immunotherapy guidelines: a multi-model comparative analysis
Abstract
Clinical guidelines may sometimes contain divergent treatment recommendations, creating uncertainty in therapeutic decision-making. To evaluate the performance of four large language models (LLMs) (GPT-4o, Claude 3.7 Sonnet, Gemini 1.5 Pro, Deepseek-V3) in analyzing discrepancies between the American Society of Clinical Oncology (ASCO) and the National Comprehensive Cancer Network (NCCN) immune checkpoint inhibitors (ICIs) guidelines. ICI-related guidelines published by ASCO and NCCN were collected, divergent recommendations were extracted and input into four LLMs for statistical analysis, and each model’s responses were independently evaluated by three oncology experts using Likert scales (focusing on accuracy, comprehensiveness, readability, and potential harm). All models demonstrated a significant preference for NCCN. Gemini 1.5 Pro exhibited extreme rigidity (100% selection of NCCN). The remaining models showed moderate degrees of a systematic preference: Claude 3.7 Sonnet(73.7%) , Deepseek-V3(72.0%), and GPT-4o(71.3%) . Further stratification revealed a relative preference for ASCO guidelines under more conservative treatment protocols. Generative artificial intelligence (AI) demonstrates persistent and systematic preference in conflicting guideline interpretation, with an overall inclination toward NCCN guidelines, potentially related to training corpus distribution and differences in guideline articulation. These findings suggest the need to establish transparent and regulatory-compliant AI decision-making frameworks.
Related
- A Multidimensional Evaluation of Large Language Model Responses to the 2024 ESC Hypertension Guidelines: A Comparative Study
- Comparative barriers to molecular therapy and immunotherapy in glioblastoma: A systematic review.
- Artificial Intelligence Scribes and Same-Day Note Closure by Patient Language Preference and Interpreter Modality
- Evaluation of Large Language Model Concordance With AAOS Clinical Practice Guidelines for Carpal Tunnel Syndrome
- Large Language Model Dietary Recommendations for Multimorbidity with Conflicting Guidelines in Turkish and English
BibTeX
@article{chen2026large,
title = {{Large language models reveal a systematic preference in clinical immunotherapy guidelines: a multi-model comparative analysis}},
author = {Zhenyu Chen and Anqi Lin and Junyi Shen and Wentao Xu and Haoxuan Ying and Aimin Jiang and Chang Qi and Wenyi Gan and Lingxuan Zhu and Weiming Mou and Dongqiang Zeng and Bufu Tang and Mingjia Xiao and Guangdi Chu and Shengkun Peng and Hank Z. H. Wong and Lin Zhang and Hengguo Zhang and Xinpei Deng and Xuanye Cao and Wenjin Chen and Xiao Liu and Qing Zeng and Xiang Wang and Quan Cheng and Jian Zhang and Peng Luo},
journal = {Scientific Reports},
year = {2026},
month = oct,
doi = {10.1038/s41598-026-73694-2},
url = {https://www.nature.com/articles/s41598-026-73694-2},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Large language models reveal a systematic preference in clinical immunotherapy guidelines: a multi-model comparative analysis
AU - Zhenyu Chen
AU - Anqi Lin
AU - Junyi Shen
AU - Wentao Xu
AU - Haoxuan Ying
AU - Aimin Jiang
AU - Chang Qi
AU - Wenyi Gan
AU - Lingxuan Zhu
AU - Weiming Mou
AU - Dongqiang Zeng
AU - Bufu Tang
AU - Mingjia Xiao
AU - Guangdi Chu
AU - Shengkun Peng
AU - Hank Z. H. Wong
AU - Lin Zhang
AU - Hengguo Zhang
AU - Xinpei Deng
AU - Xuanye Cao
AU - Wenjin Chen
AU - Xiao Liu
AU - Qing Zeng
AU - Xiang Wang
AU - Quan Cheng
AU - Jian Zhang
AU - Peng Luo
PY - 2026
DA - 2026/10/08
JO - Scientific Reports
DO - 10.1038/s41598-026-73694-2
UR - https://www.nature.com/articles/s41598-026-73694-2
AB - Clinical guidelines may sometimes contain divergent treatment recommendations, creating uncertainty in therapeutic decision-making. To evaluate the performance of four large language models (LLMs) (GPT-4o, Claude 3.7 Sonnet, Gemini 1.5 Pro, Deepseek-V3) in analyzing discrepancies between the American Society of Clinical Oncology (ASCO) and the National Comprehensive Cancer Network (NCCN) immune checkpoint inhibitors (ICIs) guidelines. ICI-related guidelines published by ASCO and NCCN were collected, divergent recommendations were extracted and input into four LLMs for statistical analysis, and each model’s responses were independently evaluated by three oncology experts using Likert scales (focusing on accuracy, comprehensiveness, readability, and potential harm). All models demonstrated a significant preference for NCCN. Gemini 1.5 Pro exhibited extreme rigidity (100% selection of NCCN). The remaining models showed moderate degrees of a systematic preference: Claude 3.7 Sonnet(73.7%) , Deepseek-V3(72.0%), and GPT-4o(71.3%) . Further stratification revealed a relative preference for ASCO guidelines under more conservative treatment protocols. Generative artificial intelligence (AI) demonstrates persistent and systematic preference in conflicting guideline interpretation, with an overall inclination toward NCCN guidelines, potentially related to training corpus distribution and differences in guideline articulation. These findings suggest the need to establish transparent and regulatory-compliant AI decision-making frameworks.
SN - 2045-2322
ER -