When do data mixtures improve scaling laws? Insights from high-dimensional regression
Abstract
Modern machine learning systems are trained on mixtures of data from different domains, and choosing the right mixture can substantially improve downstream performance. Despite an extensive literature on data mixing and reweighting, existing work is largely empirical and it remains unclear when auxiliary data genuinely improves scaling laws rather than merely providing more samples. To gain insight into this question, we study a high-dimensional mixed-data regression model with a shared regression function, heterogeneous covariances and noise levels, and dataset sizes that may grow at different rates. We establish the minimax risk under an ellipsoidal parameter constraint for the general covariance structure and derive deterministic equivalents for the test error of ridge regression under commutative covariances. We then specialize to a target domain and an auxiliary domain with aligned power-law covariance spectra, where the theory yields explicit scaling laws in terms of spectral decay, target regularity, and the relative growth of the two datasets. These laws identify regimes in which combining data mixtures provably yields a faster scaling rate than using either dataset alone. In particular, improving the scaling law requires a specific interplay between spectra and relative sample sizes of the domains. Our numerical experiments on language models exhibit the same qualitative phenomenon: appropriate data mixtures yield a faster decrease in target-domain test loss than training on either domain alone.
Related
- A flexible approach to high-dimensional clustering and classification using mixtures of finite mutually repulsive mixtures
- Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning
- SGD in Multiclass Logistic Regression: Sequential Learning and Scaling Laws
- Local scaling limit of the four-dimensional uniform spanning tree
- Scaling limit of the collision measure for two-dimensional random walks
BibTeX
@misc{wu2026when,
title = {{When do data mixtures improve scaling laws? Insights from high-dimensional regression}},
author = {Diyuan Wu and Lehan Chen and Theodor Misiakiewicz and Marco Mondelli},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.38011},
eprint = {2609.38011},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.38011},
note = {CC BY-SA 4.0},
}
RIS
TY - UNPB
TI - When do data mixtures improve scaling laws? Insights from high-dimensional regression
AU - Diyuan Wu
AU - Lehan Chen
AU - Theodor Misiakiewicz
AU - Marco Mondelli
PY - 2026
DA - 2026/09/29
PB - arXiv
DO - 10.48550/arxiv.2609.38011
UR - https://arxiv.org/abs/2609.38011
AB - Modern machine learning systems are trained on mixtures of data from different domains, and choosing the right mixture can substantially improve downstream performance. Despite an extensive literature on data mixing and reweighting, existing work is largely empirical and it remains unclear when auxiliary data genuinely improves scaling laws rather than merely providing more samples. To gain insight into this question, we study a high-dimensional mixed-data regression model with a shared regression function, heterogeneous covariances and noise levels, and dataset sizes that may grow at different rates. We establish the minimax risk under an ellipsoidal parameter constraint for the general covariance structure and derive deterministic equivalents for the test error of ridge regression under commutative covariances. We then specialize to a target domain and an auxiliary domain with aligned power-law covariance spectra, where the theory yields explicit scaling laws in terms of spectral decay, target regularity, and the relative growth of the two datasets. These laws identify regimes in which combining data mixtures provably yields a faster scaling rate than using either dataset alone. In particular, improving the scaling law requires a specific interplay between spectra and relative sample sizes of the domains. Our numerical experiments on language models exhibit the same qualitative phenomenon: appropriate data mixtures yield a faster decrease in target-domain test loss than training on either domain alone.
ER -