Model Collapse Is Avoidable Even as Real Data Vanishes: A Sharp Criterion for Iterative Gaussian Estimation
Abstract
Large generative models are commonly trained on data collected from the open web, which may contain data produced by previous generations of the same model. Previous work has shown that recursively training on model-generated data can progressively degrade model performance, a phenomenon known as model collapse. Yet, generative models continue to improve dramatically over time. Existing theory suggests that this is enabled by the availability of new real data at each iteration, kept separate from generated data. However, exact separation is not realistic, and uncurated data is still commonly used. We bridge this gap by considering a more realistic setup in which real and generated data are not handled differently during training. Specifically, we present a comprehensive theoretical study of iterative estimation of the mean and covariance of a multivariate Gaussian distribution. At each iteration, samples are drawn from a mixture of the real data distribution and the estimated distribution from the previous iteration, but are used indistinguishably. We provide a single equivalent condition for consistency of the mean and covariance estimators in Euclidean norms, and convergence to zero of the expected squared 2-Wasserstein distance between the estimated and true distributions. Using this result, we show that consistency holds, i.e., model collapse is avoided, for any fixed real data fraction above zero, and even for a fraction that vanishes across iterations, provided it does not vanish too quickly. However, more real data (or data curation) remains important at practical finite times: the convergence rate of the mean estimator deteriorates as the real data fraction approaches zero. We support our results with an empirical investigation of the theoretical setup, and with experiments on iterative training of a diffusion model on CIFAR-10 that correlate with our theory.
Related
- No Model Required: Text Entropy Rate Filtering Mitigates Iterative Fine-Tuning Collapse
- Maximum Likelihood Estimation for Entity Ranking under Iterative Synthetic Data Augmentation
- On the singular layer for the 1-D steady Euler system with heat conductivity in a finite nozzle as the heat conduction coefficient vanishes
- The price of a weak shadow variable: inference under nonignorable missingness as relevance vanishes
- The Willmore Energy Landscape of Spheres and Avoidable Singularities of the Willmore Flow
BibTeX
@misc{turgeman2026model,
title = {{Model Collapse Is Avoidable Even as Real Data Vanishes: A Sharp Criterion for Iterative Gaussian Estimation}},
author = {Roy Turgeman and Tom Tirer},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.00259},
eprint = {2611.00259},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.00259},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Model Collapse Is Avoidable Even as Real Data Vanishes: A Sharp Criterion for Iterative Gaussian Estimation
AU - Roy Turgeman
AU - Tom Tirer
PY - 2026
DA - 2026/11/01
PB - arXiv
DO - 10.48550/arxiv.2611.00259
UR - https://arxiv.org/abs/2611.00259
AB - Large generative models are commonly trained on data collected from the open web, which may contain data produced by previous generations of the same model. Previous work has shown that recursively training on model-generated data can progressively degrade model performance, a phenomenon known as model collapse. Yet, generative models continue to improve dramatically over time. Existing theory suggests that this is enabled by the availability of new real data at each iteration, kept separate from generated data. However, exact separation is not realistic, and uncurated data is still commonly used. We bridge this gap by considering a more realistic setup in which real and generated data are not handled differently during training. Specifically, we present a comprehensive theoretical study of iterative estimation of the mean and covariance of a multivariate Gaussian distribution. At each iteration, samples are drawn from a mixture of the real data distribution and the estimated distribution from the previous iteration, but are used indistinguishably. We provide a single equivalent condition for consistency of the mean and covariance estimators in Euclidean norms, and convergence to zero of the expected squared 2-Wasserstein distance between the estimated and true distributions. Using this result, we show that consistency holds, i.e., model collapse is avoided, for any fixed real data fraction above zero, and even for a fraction that vanishes across iterations, provided it does not vanish too quickly. However, more real data (or data curation) remains important at practical finite times: the convergence rate of the mean estimator deteriorates as the real data fraction approaches zero. We support our results with an empirical investigation of the theoretical setup, and with experiments on iterative training of a diffusion model on CIFAR-10 that correlate with our theory.
ER -