How Deep is the Signal? Time-Context up to 30 Years for Colorectal Cancer Prediction in a Primary-Care EHR Model
Abstract
EHR foundation models read medical events within a fixed context window, and long context has so far been measured only in tokens. How far back in time a model must look to capture the earliest predictive signal is unknown. We introduce time-context, the calendar lookback window before the prediction point, and provide, to our knowledge, the first systematic evaluation of decade-scale time-context for an EHR foundation model. We pretrained a decoder-only transformer (34.6M parameters) on 1.4B coded events from 3.6M primary-care patients in North West London and fine-tuned it to predict incident colorectal cancer under a 12-month pre-diagnostic buffer. A frozen cohort of 1,008 cases and 9,582 matched controls, all registered for at least 30 years, was evaluated at 15 nested lookback windows from 2 to 30 years, so that every truncation is compared within the same patients. Discrimination rose from an AUROC of 0.551 at 2 years to 0.590 at 18 years, then plateaued, with about 80% of the gain reached within 10 years; an XGBoost baseline converged to the same level (peak AUROC 0.589). A simulation further suggests that rotary position embeddings (RoPE) over elapsed time alias beyond about 20 years at the default base frequency, motivating a retuned θ. Time-context is a measurable design axis for clinical prediction models, distinct from token-context, and should be reported.
Related
- Intercellular communication and multi-organ axis interactions in the pathogenesis of MASLD: a signal-class synthesis graded by disease context and causal status
- Anterior cingulate feedback facilitates rapid global context modulation in primary visual cortex
- Continuous Rod Noise and Nonlinear Synaptic Filtering Shape Signal Detection in the Primary Rod Pathway
- All of Us diversity and scale yield context-dependent improvements in polygenic prediction
- Receptor-Side Neuroimmune Remodeling in IBD: Participant-Level Associations with Metabolite-Sensing Programs and Colorectal Neoplasia Context
BibTeX
@misc{raviez2026deep,
title = {{How Deep is the Signal? Time-Context up to 30 Years for Colorectal Cancer Prediction in a Primary-Care EHR Model}},
author = {Achille Raviez and Lan Wang and Brendan Delaney},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.07265},
eprint = {2611.07265},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.07265},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - How Deep is the Signal? Time-Context up to 30 Years for Colorectal Cancer Prediction in a Primary-Care EHR Model
AU - Achille Raviez
AU - Lan Wang
AU - Brendan Delaney
PY - 2026
DA - 2026/11/05
PB - arXiv
DO - 10.48550/arxiv.2611.07265
UR - https://arxiv.org/abs/2611.07265
AB - EHR foundation models read medical events within a fixed context window, and long context has so far been measured only in tokens. How far back in time a model must look to capture the earliest predictive signal is unknown. We introduce time-context, the calendar lookback window before the prediction point, and provide, to our knowledge, the first systematic evaluation of decade-scale time-context for an EHR foundation model. We pretrained a decoder-only transformer (34.6M parameters) on 1.4B coded events from 3.6M primary-care patients in North West London and fine-tuned it to predict incident colorectal cancer under a 12-month pre-diagnostic buffer. A frozen cohort of 1,008 cases and 9,582 matched controls, all registered for at least 30 years, was evaluated at 15 nested lookback windows from 2 to 30 years, so that every truncation is compared within the same patients. Discrimination rose from an AUROC of 0.551 at 2 years to 0.590 at 18 years, then plateaued, with about 80% of the gain reached within 10 years; an XGBoost baseline converged to the same level (peak AUROC 0.589). A simulation further suggests that rotary position embeddings (RoPE) over elapsed time alias beyond about 20 years at the default base frequency, motivating a retuned θ. Time-context is a measurable design axis for clinical prediction models, distinct from token-context, and should be reported.
ER -