Evaluation effects can rival reported concordance gains in cross-cancer survival prediction
- 1University of New South Wales
Abstract
Studies of cross-cancer learning for survival prediction from tumour omics have reported concordance gains of 0.020 and 0.045, but how such gains compare with evaluation effects remains unclear. We examined these effects across 13 TCGA cancers (5,761 patients, 1,526 disease-specific deaths), using five repeats of five-fold cross-validation and paired bootstrap intervals. When model selection did not use test data, our borrowing model gained 0.009 (95% interval 0.001 to 0.016) over its target-fitted reference when cancers received equal weight (post hoc). Random-effects pooling gave a gain of 0.002 (-0.002 to 0.006), with most weight assigned to cancers where the model made little use of the pretrained score. Choosing the fusion weight on test splits raised the apparent equal-weight gain to 0.017. We reimplemented a published method, VAECox, which showed no pretraining gain over random initialisation at the last or validation-selected training step. Choosing each network's best test step, as the published code does, raised concordance by 0.03 to 0.05 across the 13-cancer and ten-cancer overall-survival settings, but barely changed the difference between networks. In a constructed post hoc experiment, choosing the best test step only for the pretrained network changed the equal-weight mean difference from -0.004 to +0.045 (+0.035 to +0.054). Across seven multi-omics comparisons, estimates from single five-fold runs differed from their cancer's five-run mean by at least 0.02 in 11% to 42% of runs. Concordance was about 0.02 lower at held-out tissue source sites. Interpreting a reported gain therefore requires each model's selection rule, variation across repeated partitions, the weighting of cancers and the evaluation population.
Related
- Layer-Specific Source-Anchored Transfer Learning for Multi-Omics Survival Prediction in Younger Breast Cancer Patients
- Alert-List Concordance for Evaluating Imputation Strategies in ICU Sepsis Prediction
- Clinical-Anchored Incremental Survival Learning: A Stability-Constrained Framework for Molecularly Enhanced Survival Prediction in Lung Adenocarcinoma
- CROSS-PLATFORM CONCORDANCE ANALYSIS REVEALS ROBUST AND CLINICALLY RELEVANT GENE SIGNATURES IN BREAST CANCER
- Isolation and Evaluation of the Gut-Associated Bacterium Staphylococcus gallinarum KI4: Effects on Survival and Cocoon Traits of Bombyx mori Under Bacillus cereus Challenge
BibTeX
@misc{nguyen2026evaluation,
title = {{Evaluation effects can rival reported concordance gains in cross-cancer survival prediction}},
author = {R. Nguyen and F. Vafaee},
howpublished = {bioRxiv},
year = {2026},
month = nov,
doi = {10.64898/2026.11.05.764739},
url = {https://www.biorxiv.org/content/10.64898/2026.11.05.764739v1},
note = {CC BY-NC},
}
RIS
TY - UNPB
TI - Evaluation effects can rival reported concordance gains in cross-cancer survival prediction
AU - R. Nguyen
AU - F. Vafaee
PY - 2026
DA - 2026/11/05
PB - bioRxiv
DO - 10.64898/2026.11.05.764739
UR - https://www.biorxiv.org/content/10.64898/2026.11.05.764739v1
AB - Studies of cross-cancer learning for survival prediction from tumour omics have reported concordance gains of 0.020 and 0.045, but how such gains compare with evaluation effects remains unclear. We examined these effects across 13 TCGA cancers (5,761 patients, 1,526 disease-specific deaths), using five repeats of five-fold cross-validation and paired bootstrap intervals. When model selection did not use test data, our borrowing model gained 0.009 (95% interval 0.001 to 0.016) over its target-fitted reference when cancers received equal weight (post hoc). Random-effects pooling gave a gain of 0.002 (-0.002 to 0.006), with most weight assigned to cancers where the model made little use of the pretrained score. Choosing the fusion weight on test splits raised the apparent equal-weight gain to 0.017. We reimplemented a published method, VAECox, which showed no pretraining gain over random initialisation at the last or validation-selected training step. Choosing each network's best test step, as the published code does, raised concordance by 0.03 to 0.05 across the 13-cancer and ten-cancer overall-survival settings, but barely changed the difference between networks. In a constructed post hoc experiment, choosing the best test step only for the pretrained network changed the equal-weight mean difference from -0.004 to +0.045 (+0.035 to +0.054). Across seven multi-omics comparisons, estimates from single five-fold runs differed from their cancer's five-run mean by at least 0.02 in 11% to 42% of runs. Concordance was about 0.02 lower at held-out tissue source sites. Interpreting a reported gain therefore requires each model's selection rule, variation across repeated partitions, the weighting of cancers and the evaluation population.
ER -