Feature Selective Model Collapse in Diffusion Models: Total Replacement versus Fixed-Budget Training
Abstract
Model collapse arises when generative models are trained on synthetic data produced by earlier models. The phenomenon has attracted considerable attention because of its societal and technical implications. However, previous studies have reached seemingly contradictory conclusions: replacing real data with synthetic data causes collapse (Shumailov et al.), yet accumulating real data alongside synthetic data can prevent it. For diffusion models, we study an intermediate regime typical of finite-budget pipelines: all past datasets and the real data are kept, but each new model is trained on a fixed-size sample from this growing pool, so the real fraction vanishes without any data being removed. Experiments on a 2D spiral dataset as well as the image benchmarks (MNIST, Fashion-MNIST, and CIFAR-10) show that replacement protocol degrades dataset rapidly as in the literature, whereas the fixed budget degrades only partially, sparing some features. A linear-response model of the multi-generational parameter dynamics, analyzed by stochastic recursion, confirms that the two protocols differ: some features will be fragile and lost within a few generations for both protocols, while some will be robust and preserved over practically unbounded horizons under the fixed budget protocol.
Related
- Recursively Trained Diffusion Models: Limiting Collapse Distribution and Spectral Characterization
- Feature Information Dynamics in Diffusion
- Koopman Observers for Diffusion Acceleration: Correcting Feature Forecasts with Shallow Measurements
- Feature Normalization under Finite Training Budgets
- Scalable, Likelihood-Free Calibration of Ice-Sheet Models with Deep Diffusion Emulators and Feature Matching
BibTeX
@misc{malet2026feature,
title = {{Feature Selective Model Collapse in Diffusion Models: Total Replacement versus Fixed-Budget Training}},
author = {Hanna Malet and Gabriel Turinici},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.01318},
eprint = {2610.01318},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.01318},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - Feature Selective Model Collapse in Diffusion Models: Total Replacement versus Fixed-Budget Training
AU - Hanna Malet
AU - Gabriel Turinici
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2610.01318
UR - https://arxiv.org/abs/2610.01318
AB - Model collapse arises when generative models are trained on synthetic data produced by earlier models. The phenomenon has attracted considerable attention because of its societal and technical implications. However, previous studies have reached seemingly contradictory conclusions: replacing real data with synthetic data causes collapse (Shumailov et al.), yet accumulating real data alongside synthetic data can prevent it. For diffusion models, we study an intermediate regime typical of finite-budget pipelines: all past datasets and the real data are kept, but each new model is trained on a fixed-size sample from this growing pool, so the real fraction vanishes without any data being removed. Experiments on a 2D spiral dataset as well as the image benchmarks (MNIST, Fashion-MNIST, and CIFAR-10) show that replacement protocol degrades dataset rapidly as in the literature, whereas the fixed budget degrades only partially, sparing some features. A linear-response model of the multi-generational parameter dynamics, analyzed by stochastic recursion, confirms that the two protocols differ: some features will be fragile and lost within a few generations for both protocols, while some will be robust and preserved over practically unbounded horizons under the fixed budget protocol.
ER -