Memorisation, convergence and generalisation in generative models
Abstract
Generative neural networks learn how to produce highly realistic images from a large, but finite number of examples - or do they simply memorise their training set? To settle this question, Kadkhodaie, Guth, Simoncelli and Mallat (ICLR '24) trained diffusion models independently on disjoint subsets of a dataset and showed that they converge to nearly the same density when the number of training images is large enough. This result raises two basic questions: how much data do you need for convergence, and what does convergence capture about learning the data distribution? Here, we address these questions by providing an exact analytical characterisation of the transition from memorisation to generalisation in linear generative models. We find that these models memorise at small load, while convergence emerges continuously when the number of samples is linear in the input dimension. Strikingly, we find that convergence is insensitive to recovery of the principal latent factors of the data, which are recovered in a sharp transition. After extending our approach to data with power-law spectra, we find the same distinction between convergence and latent recovery in our experiments with convolutional denoisers and in the data of Kadkhodaie et al. We thus show that generalisation in generative models decomposes into at least two distinct objectives: matching the bulk of the data distribution and recovering the principal latent factors. These objectives correspond to two different distances between true and learnt data distribution, and only the first one is captured by convergence.
Related
- Convergence rates for generative drifting flows: fixed-scale obstructions and multihead acceleration
- Non-asymptotic Convergence of Stochastic Gradient Descent in Score-based Generative Models
- Enhanced convergence guarantees of score-based generative models in 𝒲2-distance beyond log-concavity
- From Global to Tail Convergence: Dynamics of α-Stable Score-Based Generative Models
- Universality and Convergence of Generative Flows
BibTeX
@misc{maillard2026memorisation,
title = {{Memorisation, convergence and generalisation in generative models}},
author = {Antoine Maillard and Sebastian Goldt},
howpublished = {HAL},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.21402},
url = {https://hal.science/hal-05771886v1},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Memorisation, convergence and generalisation in generative models
AU - Antoine Maillard
AU - Sebastian Goldt
PY - 2026
DA - 2026/10/01
PB - HAL
DO - 10.48550/arxiv.2605.21402
UR - https://hal.science/hal-05771886v1
AB - Generative neural networks learn how to produce highly realistic images from a large, but finite number of examples - or do they simply memorise their training set? To settle this question, Kadkhodaie, Guth, Simoncelli and Mallat (ICLR '24) trained diffusion models independently on disjoint subsets of a dataset and showed that they converge to nearly the same density when the number of training images is large enough. This result raises two basic questions: how much data do you need for convergence, and what does convergence capture about learning the data distribution? Here, we address these questions by providing an exact analytical characterisation of the transition from memorisation to generalisation in linear generative models. We find that these models memorise at small load, while convergence emerges continuously when the number of samples is linear in the input dimension. Strikingly, we find that convergence is insensitive to recovery of the principal latent factors of the data, which are recovered in a sharp transition. After extending our approach to data with power-law spectra, we find the same distinction between convergence and latent recovery in our experiments with convolutional denoisers and in the data of Kadkhodaie et al. We thus show that generalisation in generative models decomposes into at least two distinct objectives: matching the bulk of the data distribution and recovering the principal latent factors. These objectives correspond to two different distances between true and learnt data distribution, and only the first one is captured by convergence.
ER -