Pretrained transformers applied to population cancer registries improve survival prediction in label-scarce and previously unseen cancers
- 1Department of Clinical Laboratory, Sir Run Run Shaw Hospital of Zhejiang University School of Medicine, Hangzhou, Zhejiang, China; Key Laboratory of Precision M
Abstract
Prognostic models in oncology are developed one cancer at a time, from that cancers own labelled outcomes, and fail where prognostic information is scarcest. Rare cancers account for roughly a fifth of diagnoses and most paediatric malignancies, yet seldom supply enough events for a reliable time-to-event model. We therefore asked whether a representation learned without outcome labels can supply what those cohorts cannot. A Transformer encoder was pretrained by masked field-value modelling on 9,425,135 tumour records from the SEER 17 registries, diagnosed in 2000-2023. Only diagnosis-time fields passing a fail-closed coding-verification gate were admitted, and each record was emitted as an era-specific and a harmonised view, keeping two decades of recoding auditable. The encoder was then frozen and read by a linear Cox head for overall survival. Nine rare cancers were removed from the pretraining corpus entirely, each requiring an independent pretraining run. On a sealed test partition, all nine exceeded an architecture-identical random frozen encoder in Harrell concordance by +0.0034 to +0.0368, every lower confidence limit above zero. At 256 labelled patients, all 67 cancers favoured the pretrained representation over budget-matched Cox regression, median difference +0.0283. The advantage was bounded: given the entire training set, Cox regression was favoured in seven of nine rare cancers. The encoder did not outperform a field-frequency baseline on its own objective, so upstream reconstruction did not predict downstream transfer. Outcome-agnostic registry pretraining carries prognostic signal into cancers it has never seen, and is most useful where labels are fewest, without establishing clinical utility.
Related
- Cancer incidence in first-generation migrants and host populations in seven European registries, including Ukrainian-born migrants in Poland: a population-based study
- Standardisation of stage information collected by European population-based cancer registries: an application to breast cancer survival
- A mixed-methods study of methodological approaches in population-based cancer registries in the North and South
- Healthcare Inequities and Structural Racism as Determinants of Treatment and Survival among Individuals Recently Diagnosed with Cervical Cancer from Two Diverse Population-Based Cancer Registries: The ACHIEVE Study
- Development of the OncoScore: A Diagnostic Prediction Score for Previously Unrecognized Malignancy in Patients with Acute Pulmonary Embolism
BibTeX
@misc{gao2026pretrained,
title = {{Pretrained transformers applied to population cancer registries improve survival prediction in label-scarce and previously unseen cancers}},
author = {Y. Gao and S. Yu and Y. Xia and S. Chen and S. Xia and R. An and J. Zeng and F. Zhao and Y. Ma and Y. Wang and X. Xie and J. Zhang},
howpublished = {medRxiv},
year = {2026},
month = aug,
doi = {10.64898/2026.08.30.26361693},
url = {https://www.medrxiv.org/content/10.64898/2026.08.30.26361693v1},
note = {CC BY-NC},
}
RIS
TY - UNPB
TI - Pretrained transformers applied to population cancer registries improve survival prediction in label-scarce and previously unseen cancers
AU - Y. Gao
AU - S. Yu
AU - Y. Xia
AU - S. Chen
AU - S. Xia
AU - R. An
AU - J. Zeng
AU - F. Zhao
AU - Y. Ma
AU - Y. Wang
AU - X. Xie
AU - J. Zhang
PY - 2026
DA - 2026/08/30
PB - medRxiv
DO - 10.64898/2026.08.30.26361693
UR - https://www.medrxiv.org/content/10.64898/2026.08.30.26361693v1
AB - Prognostic models in oncology are developed one cancer at a time, from that cancers own labelled outcomes, and fail where prognostic information is scarcest. Rare cancers account for roughly a fifth of diagnoses and most paediatric malignancies, yet seldom supply enough events for a reliable time-to-event model. We therefore asked whether a representation learned without outcome labels can supply what those cohorts cannot. A Transformer encoder was pretrained by masked field-value modelling on 9,425,135 tumour records from the SEER 17 registries, diagnosed in 2000-2023. Only diagnosis-time fields passing a fail-closed coding-verification gate were admitted, and each record was emitted as an era-specific and a harmonised view, keeping two decades of recoding auditable. The encoder was then frozen and read by a linear Cox head for overall survival. Nine rare cancers were removed from the pretraining corpus entirely, each requiring an independent pretraining run. On a sealed test partition, all nine exceeded an architecture-identical random frozen encoder in Harrell concordance by +0.0034 to +0.0368, every lower confidence limit above zero. At 256 labelled patients, all 67 cancers favoured the pretrained representation over budget-matched Cox regression, median difference +0.0283. The advantage was bounded: given the entire training set, Cox regression was favoured in seven of nine rare cancers. The encoder did not outperform a field-frequency baseline on its own objective, so upstream reconstruction did not predict downstream transfer. Outcome-agnostic registry pretraining carries prognostic signal into cancers it has never seen, and is most useful where labels are fewest, without establishing clinical utility.
ER -