New Multiple Descent Phenomena Induced by Heterogeneous Data
Abstract
Modern machine learning models are often trained on large datasets assembled from many different sources, whose statistical properties can differ substantially. Understanding how this heterogeneity affects generalization performance is therefore important for explaining how model performance changes as model size increases. In this work, we study this question in linear random-feature models and show that mixtures of datasets with different covariance structures can give rise to a range of multiple-descent phenomena. We derive exact asymptotic expressions for the test error of each data source in the high-dimensional regime, and show how heterogeneity in covariance scale and spectral structure can introduce additional, group-dependent interpolation thresholds as model size increases. These phenomena arise even with noiseless labels and linear activations, showing that neither label noise nor activation nonlinearity is necessary for multiple-descent. Our results therefore provide a different perspective from classical explanations of multiple descent. More broadly, our results have potential implications for algorithmic fairness by showing that model scaling can affect different data groups unevenly, amplifying disparities in their generalization errors. We validate our theoretical results on synthetic Gaussian mixtures and real-world datasets, including MNIST and CIFAR-10.
Related
- The double descent and Runge phenomena in overparametrized polynomial interpolation
- The Zero Pattern of a Design Matrix Drives Multiple Descent in Over-parameterized Regression
- The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flow
- Factorial Multivariate Bayesian Causal Forests: heterogeneous main and interaction effects of multiple treatments on correlated outcomes
- Localization Landscapes as Operator-Induced Geometry for Heterogeneous Systems
BibTeX
@misc{moisescupareja2026multiple,
title = {{New Multiple Descent Phenomena Induced by Heterogeneous Data}},
author = {Mark-Anthony Moisescu-Pareja and Elvis Dohmatob},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.38002},
eprint = {2610.38002},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.38002},
}
RIS
TY - UNPB
TI - New Multiple Descent Phenomena Induced by Heterogeneous Data
AU - Mark-Anthony Moisescu-Pareja
AU - Elvis Dohmatob
PY - 2026
DA - 2026/10/27
PB - arXiv
DO - 10.48550/arxiv.2610.38002
UR - https://arxiv.org/abs/2610.38002
AB - Modern machine learning models are often trained on large datasets assembled from many different sources, whose statistical properties can differ substantially. Understanding how this heterogeneity affects generalization performance is therefore important for explaining how model performance changes as model size increases. In this work, we study this question in linear random-feature models and show that mixtures of datasets with different covariance structures can give rise to a range of multiple-descent phenomena. We derive exact asymptotic expressions for the test error of each data source in the high-dimensional regime, and show how heterogeneity in covariance scale and spectral structure can introduce additional, group-dependent interpolation thresholds as model size increases. These phenomena arise even with noiseless labels and linear activations, showing that neither label noise nor activation nonlinearity is necessary for multiple-descent. Our results therefore provide a different perspective from classical explanations of multiple descent. More broadly, our results have potential implications for algorithmic fairness by showing that model scaling can affect different data groups unevenly, amplifying disparities in their generalization errors. We validate our theoretical results on synthetic Gaussian mixtures and real-world datasets, including MNIST and CIFAR-10.
ER -