An Explainable Multimodal Deep‐Learning Model Using Histopathological Image and Clinical Information for Prognosis Prediction in Colorectal Cancer: A Multi‐Center Cohort Study
11 affiliations
- 1Department of General Surgery, Peking University Third Hospital Peking University Beijing People's Republic of China
- 2Peking University Third Hospital Cancer Center Peking University Beijing People's Republic of China
- 3Beijing Key Laboratory for Interdisciplinary Research in Gastrointestinal Oncology (BLGO) Beijing People's Republic of China
- 4Department of General Surgery, The First Medical Center Chinese PLA General Hospital Beijing People's Republic of China
- 5School of Biomedical Sciences, Faculty of Medicine and Health UNSW Sydney Kensington Australia
- 6Office of DVCR Monash University Melbourne Australia
- 7Image Processing Center Beihang University Beijing People's Republic of China
- 8State Key Laboratory of Virtual Reality Technology and Systems Beihang University Beijing People's Republic of China
- 9Key Laboratory of Spacecraft Design Optimization and Dynamic Simulation Technology, Ministry of Education Beihang University Beijing People's Republic of China
- 10School of Clinical Medicine, Faculty of Medicine and Health UNSW Sydney Kensington Australia
- 11SREDH Consortium Sydney Australia
Abstract
Accurate prognostication is crucial for the individualized management of patients with colorectal cancer (CRC), while the accuracy and interpretability of current models still need to be improved. This study aims to develop and validate an explainable multimodal model for precise survival prediction. 3381 patients with resected CRC are enrolled from three medical centers, and a multimodal deep‐learning framework integrating clinical variables and histopathological images is developed. The multimodal model exhibits significantly more robust performance than unimodal models in prognosis prediction across all of the cohorts. By stratifying patients into low‐risk and high‐risk groups based on the model‐derived risk scores, high‐risk patients exhibit worse outcomes, and the stratification remains an independent prognostic factor in multivariable Cox regression. The model also identifies patients with markedly poorer survival among the stage II/III subgroup. Adjuvant chemotherapy demonstrates benefits in high‐risk patients with stage III CRC but not in the low‐risk group. Multi‐omic analyses disclose that low‐risk tumors are characterized by enhanced adaptive immune responses, whereas high‐risk tumors exhibit stronger angiogenesis and higher exhaustion scores of T cells, reflecting a hypoxic and immunosuppressive microenvironment. These results support the potential of the multimodal model to effectively predict survival outcomes in resected CRC with great explainability.
Related
- Explainable AI framework for Alzheimer’s disease prediction using multimodal data
- Multimodal Machine Learning for Explainable Disease Risk Prediction and Clinical Decision
- Vehicle lane change prediction with explainable soft mask attention mechanism and heterogeneous information encoder
- Three-dimensional multimodal fractal dynamics integrating spatial metabolism, cytoarchitecture, and perfusion for preoperative prediction of microvascular invasion in colorectal liver metastases
- An Explainable Disease-Aware Intelligent Histopathological Representation Learning Framework for Generalizable Breast Cancer Classification
BibTeX
@article{jiang2026explainable,
title = {{An Explainable Multimodal Deep‐Learning Model Using Histopathological Image and Clinical Information for Prognosis Prediction in Colorectal Cancer: A Multi‐Center Cohort Study}},
author = {Zhihan Jiang and Xiangchao Shi and Chenghan Su and Yao Ma and Rui Li and Menglei Chen and Jianxin Cui and Nicholas Hawkins and Robyn Ward and Xiangzhi Bai and Jitendra Jonnagaddala and Wei Fu and Xin Zhou},
journal = {Advanced Science},
year = {2026},
month = oct,
doi = {10.1002/advs.78417},
url = {https://doi.org/10.1002/advs.78417},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - An Explainable Multimodal Deep‐Learning Model Using Histopathological Image and Clinical Information for Prognosis Prediction in Colorectal Cancer: A Multi‐Center Cohort Study
AU - Zhihan Jiang
AU - Xiangchao Shi
AU - Chenghan Su
AU - Yao Ma
AU - Rui Li
AU - Menglei Chen
AU - Jianxin Cui
AU - Nicholas Hawkins
AU - Robyn Ward
AU - Xiangzhi Bai
AU - Jitendra Jonnagaddala
AU - Wei Fu
AU - Xin Zhou
PY - 2026
DA - 2026/10/26
JO - Advanced Science
DO - 10.1002/advs.78417
UR - https://doi.org/10.1002/advs.78417
AB - Accurate prognostication is crucial for the individualized management of patients with colorectal cancer (CRC), while the accuracy and interpretability of current models still need to be improved. This study aims to develop and validate an explainable multimodal model for precise survival prediction. 3381 patients with resected CRC are enrolled from three medical centers, and a multimodal deep‐learning framework integrating clinical variables and histopathological images is developed. The multimodal model exhibits significantly more robust performance than unimodal models in prognosis prediction across all of the cohorts. By stratifying patients into low‐risk and high‐risk groups based on the model‐derived risk scores, high‐risk patients exhibit worse outcomes, and the stratification remains an independent prognostic factor in multivariable Cox regression. The model also identifies patients with markedly poorer survival among the stage II/III subgroup. Adjuvant chemotherapy demonstrates benefits in high‐risk patients with stage III CRC but not in the low‐risk group. Multi‐omic analyses disclose that low‐risk tumors are characterized by enhanced adaptive immune responses, whereas high‐risk tumors exhibit stronger angiogenesis and higher exhaustion scores of T cells, reflecting a hypoxic and immunosuppressive microenvironment. These results support the potential of the multimodal model to effectively predict survival outcomes in resected CRC with great explainability.
SN - 2198-3844
ER -