First Learn, Then Memorize: The Spectral Bias of Diffusion Models
Abstract
Diffusion models trained on a finite dataset first learn to generate novel, high-quality samples and only much later collapse onto their training set. We identify the mechanism behind this separation of timescales and the object that probes it. The training dynamics of the score function are governed---exactly, and at any width---by the Gram matrix of the Neural Tangent Kernel (NTK) evaluated on the noisy training data, so the timescales of generalization and of memorization must be encoded in its spectrum. We show that they are, and that the structure responsible has no analogue in standard kernel settings. The use of multiple noise realizations per sample (m noised copies at a fixed noise level) in the score-matching loss is what restructures the Gram matrix spectrum into two distinct parts. The first, of large eigenvalues, carries the global features of the target distribution and is present already for m=1. The second, which the repeated noising creates, consists of the smallest eigenvalues and is supported on eigenvectors aligned with the sample-specific noise directions; it sets a memorization timescale parametrically larger in the training set size n. We establish this picture on two fronts. Analytically, we solve the spectrum in the lazy high-dimensional limit for both linear (n asymp d) and polynomial (n asymp dk) sample complexities, and prove through a bias--variance decomposition that the first bulk minimizes the approximation error while the second drives the error associated with memorization. Empirically, we show the same two-bulk structure in Convolutional NTKs on CelebA and in finite-width U-Nets trained well beyond the lazy regime, and we make the link causal: truncating the Gram matrix at rank r tunes the generalization--memorization transition, and an L2 penalty targeting the second bulk suppresses memorization in feature-learning U-Nets.
Related
- Why diffusion models do not memorize: the role of implicit dynamical regularization in training
- Thermochemically Constrained Multiscale Laplace Neural Operator to Learn Stiff Chemical Kinetics with Conditional Diffusion Model Sampling
- Anisotropic exciton diffusion in O-IDTBR from first-principles molecular dynamics in the first excited state
- First-principles transition-state tensorial cluster expansion of vacancy diffusion in Ta-W beyond the kinetically-resolved activation approximation
- Acceleration of CTQMC via Denoising Diffusion Models with Auxiliary Spectral Functions
BibTeX
@misc{urfin2026first,
title = {{First Learn, Then Memorize: The Spectral Bias of Diffusion Models}},
author = {Raphaël Urfin and Tony Bonnaire and Giulio Biroli and Marc Mézard},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.35377},
eprint = {2609.35377},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.35377},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - First Learn, Then Memorize: The Spectral Bias of Diffusion Models
AU - Raphaël Urfin
AU - Tony Bonnaire
AU - Giulio Biroli
AU - Marc Mézard
PY - 2026
DA - 2026/09/28
PB - arXiv
DO - 10.48550/arxiv.2609.35377
UR - https://arxiv.org/abs/2609.35377
AB - Diffusion models trained on a finite dataset first learn to generate novel, high-quality samples and only much later collapse onto their training set. We identify the mechanism behind this separation of timescales and the object that probes it. The training dynamics of the score function are governed---exactly, and at any width---by the Gram matrix of the Neural Tangent Kernel (NTK) evaluated on the noisy training data, so the timescales of generalization and of memorization must be encoded in its spectrum. We show that they are, and that the structure responsible has no analogue in standard kernel settings. The use of multiple noise realizations per sample (m noised copies at a fixed noise level) in the score-matching loss is what restructures the Gram matrix spectrum into two distinct parts. The first, of large eigenvalues, carries the global features of the target distribution and is present already for m=1. The second, which the repeated noising creates, consists of the smallest eigenvalues and is supported on eigenvectors aligned with the sample-specific noise directions; it sets a memorization timescale parametrically larger in the training set size n. We establish this picture on two fronts. Analytically, we solve the spectrum in the lazy high-dimensional limit for both linear (n asymp d) and polynomial (n asymp d^k) sample complexities, and prove through a bias--variance decomposition that the first bulk minimizes the approximation error while the second drives the error associated with memorization. Empirically, we show the same two-bulk structure in Convolutional NTKs on CelebA and in finite-width U-Nets trained well beyond the lazy regime, and we make the link causal: truncating the Gram matrix at rank r tunes the generalization--memorization transition, and an L_2 penalty targeting the second bulk suppresses memorization in feature-learning U-Nets.
ER -