Selecting Repetition Counts Across Model Scales in Data-Constrained Pretraining
Abstract
The repetition count that works best for a small language model may not remain best at a larger scale. We study this effect in pretraining with a finite target corpus mixed with generic data at a fixed target fraction. On Wikipedia-derived data and Proof-Pile-2, the ranking of measured repetition counts changes with model size, and a 520M Proof-Pile-2 experiment confirms that reducing repetition from sixteen to eight improves loss while using fewer training tokens. We use loss curves from several smaller models to retain a short list of promising repetition counts for evaluation at a larger scale. On PubMed and Caselaw, candidate sets fixed before target-model training retain the lowest-loss measured count on the original evaluation grids at both 200M and 520M. This supports candidate retention as a practical alternative to exact point prediction. We also relate the pruning regression to an empirical scaling model with two opposing repetition-dependent loss terms. A first-order expansion in log model size yields the linear form used by the selection rule, providing a scaling-based interpretation of the candidate-selection procedure.
Related
- PhysBridge: Response-supervised grid pretraining with physics-constrained local correction for transferable power-system intelligence
- Selecting the number of components in PCA via random signflips
- Score-Based Methods for Selecting Direct Causes in Reweighted Distributions
- Selecting Informative Conformal Prediction Sets with an Optimized FCR-Controlled Approach
- Selecting the 7th Smallest of 16 Elements Takes Exactly 28 Comparisons
BibTeX
@misc{wang2026selecting,
title = {{Selecting Repetition Counts Across Model Scales in Data-Constrained Pretraining}},
author = {Ziyue WANG and T. Kanamori},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.05126},
eprint = {2610.05126},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.05126},
}
RIS
TY - UNPB
TI - Selecting Repetition Counts Across Model Scales in Data-Constrained Pretraining
AU - Ziyue WANG
AU - T. Kanamori
PY - 2026
DA - 2026/10/04
PB - arXiv
DO - 10.48550/arxiv.2610.05126
UR - https://arxiv.org/abs/2610.05126
AB - The repetition count that works best for a small language model may not remain best at a larger scale. We study this effect in pretraining with a finite target corpus mixed with generic data at a fixed target fraction. On Wikipedia-derived data and Proof-Pile-2, the ranking of measured repetition counts changes with model size, and a 520M Proof-Pile-2 experiment confirms that reducing repetition from sixteen to eight improves loss while using fewer training tokens. We use loss curves from several smaller models to retain a short list of promising repetition counts for evaluation at a larger scale. On PubMed and Caselaw, candidate sets fixed before target-model training retain the lowest-loss measured count on the original evaluation grids at both 200M and 520M. This supports candidate retention as a practical alternative to exact point prediction. We also relate the pruning regression to an empirical scaling model with two opposing repetition-dependent loss terms. A first-order expansion in log model size yields the linear form used by the selection rule, providing a scaling-based interpretation of the candidate-selection procedure.
ER -