Correcting test set contamination by spiking the training data
Abstract
The literature on test set contamination largely focuses on detection, but the correction of contaminated test scores is underexplored. Our core proposal is to spike the training data by intentionally contaminating some test examples at known rates. The spiked examples can then be used to calibrate predictors of model memorization which enable principled statistical correction of inflated test scores. To evaluate different correction estimators, we first present a simulation framework based on the Hubble models. Hubble models come in minimal pairs, where the perturbed model was deliberately contaminated with several test sets, while the standard model was not, serving as the counterfactual and correction target. We consider estimators that use information from a memorization predictor, correctness predictor, or both. In simulation, we establish basic statistical intuitions and show that estimators leveraging memorization and correctness information are better than naive estimation which makes no correction at all. We then instantiate several memorization and correctness predictors, and find that simple predictors such as Platt-scaled membership inference metrics provide good signal for correction. Finally, we examine the practical considerations of spiking. Simple memorization predictors need no more than 10 examples for calibration and often transfer from one dataset to another. Taken together, spiking is a promising solution for test set contamination.
Related
- A Decision-Theoretic View of Test-Time Training: When, How Far, and Which Directions to Adapt
- The Impossible Trinity of Time-Series Validation: A Conservation Law among Training Sufficiency, Test Coverage, and Temporal Causality
- Should I Use This Synthetic Dataset for Training? How to Test with Minimal Real Data
- Spiking neuromodulation: a linear control perspective
- Plotkin-Like Bound and Explicit Function-Correcting Code Constructions for Lee Metric Channels
BibTeX
@misc{wei2026correcting,
title = {{Correcting test set contamination by spiking the training data}},
author = {Johnny Tian-Zheng Wei and Jerry Li and Ameya Godbole and Robin Jia},
howpublished = {arXiv},
year = {2026},
month = aug,
doi = {10.48550/arxiv.2605.24818},
eprint = {2605.24818},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.24818},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Correcting test set contamination by spiking the training data
AU - Johnny Tian-Zheng Wei
AU - Jerry Li
AU - Ameya Godbole
AU - Robin Jia
PY - 2026
DA - 2026/08/29
PB - arXiv
DO - 10.48550/arxiv.2605.24818
UR - https://arxiv.org/abs/2605.24818
AB - The literature on test set contamination largely focuses on detection, but the correction of contaminated test scores is underexplored. Our core proposal is to spike the training data by intentionally contaminating some test examples at known rates. The spiked examples can then be used to calibrate predictors of model memorization which enable principled statistical correction of inflated test scores. To evaluate different correction estimators, we first present a simulation framework based on the Hubble models. Hubble models come in minimal pairs, where the perturbed model was deliberately contaminated with several test sets, while the standard model was not, serving as the counterfactual and correction target. We consider estimators that use information from a memorization predictor, correctness predictor, or both. In simulation, we establish basic statistical intuitions and show that estimators leveraging memorization and correctness information are better than naive estimation which makes no correction at all. We then instantiate several memorization and correctness predictors, and find that simple predictors such as Platt-scaled membership inference metrics provide good signal for correction. Finally, we examine the practical considerations of spiking. Simple memorization predictors need no more than 10 examples for calibration and often transfer from one dataset to another. Taken together, spiking is a promising solution for test set contamination.
ER -