Weighting Schedules Govern What and When Score-Based Generative Models Learn from Multimodal Data
Abstract
Score-based generative models generate new samples by integrating a time-dependent drift that carries Gaussian noise onto the target distribution. In practice this drift is modeled by a neural network, trained on a loss integrated over time t with a weighting schedule w(t). Along the backward dynamics, and for multi-modal distributions, trajectories commit to modes of the target within a narrow time window, the \textit{speciation time}. In this work, focusing on high-dimensional data, we decompose the integrated loss into its single-time contributions and analyze each at fixed signal-to-noise ratio Λ(t): we show that Λ(t) sets the rate at which each feature of a multimodal target - the mode directions and their relative weights - is acquired during training. Crucially, at high Λ(t) all mode directions are acquired together, on a single timescale insensitive to their amplitudes, while the relative weights are not learned at all. Only near the speciation time, where Λ(t) becomes of order one, do all features become learnable, each on its own timescale: the weights are acquired jointly with the directions, and the directions at rates set by their relative amplitudes. For models trained on time-integrated objectives, the learning dynamics is then governed by how much of the weighting effectively sits near the speciation time, which provides insights on w(t) design choices. These results follow from an exact high-dimensional analysis of the training dynamics of unbalanced and hierarchical Gaussian mixtures. Numerical experiments on image and human genome haplotype generation recover the predicted hierarchy of learning timescales in more complex settings.
Related
- Generative diffusion in long-range correlated landscapes: Speciation cascade and the foam structure of the score
- Towards optimal selection schedules for online active learning
- Adiabatic tracking and geometric schedules for classical and quantum Gibbs-state preparation
- Theory of Trotter Error with Variable-Weight Schedules
- Quantum amplitude estimation beyond power-of-two schedules
BibTeX
@misc{klinger2026weighting,
title = {{Weighting Schedules Govern What and When Score-Based Generative Models Learn from Multimodal Data}},
author = {Jérémie Klinger and Raphaël Urfin and Giulio Biroli and Marylou Gabrié},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.35322},
eprint = {2609.35322},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.35322},
}
RIS
TY - UNPB
TI - Weighting Schedules Govern What and When Score-Based Generative Models Learn from Multimodal Data
AU - Jérémie Klinger
AU - Raphaël Urfin
AU - Giulio Biroli
AU - Marylou Gabrié
PY - 2026
DA - 2026/09/28
PB - arXiv
DO - 10.48550/arxiv.2609.35322
UR - https://arxiv.org/abs/2609.35322
AB - Score-based generative models generate new samples by integrating a time-dependent drift that carries Gaussian noise onto the target distribution. In practice this drift is modeled by a neural network, trained on a loss integrated over time t with a weighting schedule w(t). Along the backward dynamics, and for multi-modal distributions, trajectories commit to modes of the target within a narrow time window, the \textit{speciation time}. In this work, focusing on high-dimensional data, we decompose the integrated loss into its single-time contributions and analyze each at fixed signal-to-noise ratio Λ(t): we show that Λ(t) sets the rate at which each feature of a multimodal target - the mode directions and their relative weights - is acquired during training. Crucially, at high Λ(t) all mode directions are acquired together, on a single timescale insensitive to their amplitudes, while the relative weights are not learned at all. Only near the speciation time, where Λ(t) becomes of order one, do all features become learnable, each on its own timescale: the weights are acquired jointly with the directions, and the directions at rates set by their relative amplitudes. For models trained on time-integrated objectives, the learning dynamics is then governed by how much of the weighting effectively sits near the speciation time, which provides insights on w(t) design choices. These results follow from an exact high-dimensional analysis of the training dynamics of unbalanced and hierarchical Gaussian mixtures. Numerical experiments on image and human genome haplotype generation recover the predicted hierarchy of learning timescales in more complex settings.
ER -