The Impossible Trinity of Time-Series Validation: A Conservation Law among Training Sufficiency, Test Coverage, and Temporal Causality
Abstract
Validating a model on a time series asks for three things at once: each training run should use most of the sample (sufficiency), the test sets should together cover most of the sample (coverage), and training data should come before test data (causality). We prove that the three cannot be had together and price each one. Let α be the smallest training fraction over folds, β the fraction of the sample covered by tests, Λ the fraction of the sample used as training data from the future of a test point, and δ the distance from a test point to the nearest training point in its future. Every scheme on a sample of length T satisfies α+β≤ 1+Λ and α+min{β,δ/T} ≤ 1, and under β-mixing the leakage bias at a test point is at most 2Mβmix(δ). In words: going beyond the causal frontier α+β=1 requires training on the future; that future data must sit within (1−α)T of a test point; and its harm depends on its distance, not its amount. Hence expanding walk-forward is exactly the Pareto frontier of causal validation, k-fold cross-validation buys the most future data, and purged k-fold with an embargo pays in distance instead, which is cheap when the process forgets quickly but cannot repair the part of causality demanded by non-stationarity. On pure noise, shuffled 5-fold reports an information coefficient of +0.32, while contiguous 5-fold, using the same amount of future data, reports +0.004.
Related
- Beyond Causal Sufficiency: When Is Optimal Invariant Prediction Possible?
- Sufficiency of Hall's Condition for Graphic List Coloring
- High potential contribution of intercropping to soybean and maize self-sufficiency in Europe
- Classical Sufficiency in Quantum Statistical Experiments
- Regarding the inference of causality in observational non-controlled studies. Comment on “Assessment of sexual function and its association with quality of life and disease severity in patients with atopic dermatitis”
BibTeX
@misc{li2026impossible,
title = {{The Impossible Trinity of Time-Series Validation: A Conservation Law among Training Sufficiency, Test Coverage, and Temporal Causality}},
author = {Jiayu Li},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.29530},
eprint = {2609.29530},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.29530},
}
RIS
TY - UNPB
TI - The Impossible Trinity of Time-Series Validation: A Conservation Law among Training Sufficiency, Test Coverage, and Temporal Causality
AU - Jiayu Li
PY - 2026
DA - 2026/09/02
PB - arXiv
DO - 10.48550/arxiv.2609.29530
UR - https://arxiv.org/abs/2609.29530
AB - Validating a model on a time series asks for three things at once: each training run should use most of the sample (sufficiency), the test sets should together cover most of the sample (coverage), and training data should come before test data (causality). We prove that the three cannot be had together and price each one. Let α be the smallest training fraction over folds, β the fraction of the sample covered by tests, Λ the fraction of the sample used as training data from the future of a test point, and δ the distance from a test point to the nearest training point in its future. Every scheme on a sample of length T satisfies α+β≤ 1+Λ and α+min{β,δ/T} ≤ 1, and under β-mixing the leakage bias at a test point is at most 2Mβ_mix(δ). In words: going beyond the causal frontier α+β=1 requires training on the future; that future data must sit within (1−α)T of a test point; and its harm depends on its distance, not its amount. Hence expanding walk-forward is exactly the Pareto frontier of causal validation, k-fold cross-validation buys the most future data, and purged k-fold with an embargo pays in distance instead, which is cheap when the process forgets quickly but cannot repair the part of causality demanded by non-stationarity. On pure noise, shuffled 5-fold reports an information coefficient of +0.32, while contiguous 5-fold, using the same amount of future data, reports +0.004.
ER -