Non-asymptotic Convergence of Stochastic Gradient Descent in Score-based Generative Models
Abstract
Score-based Generative Models (SGMs) have achieved impressive performance in data generation across a wide range of applications. While the statistical properties of their sampling procedures are increasingly well understood, the optimization dynamics underlying their training remain less explored. SGMs are typically trained by minimizing a weighted denoising score-matching objective, yet optimization guarantees with stochastic gradients remain limited. In this work, we study Stochastic Gradient Descent (SGD) for SGMs, contributing results in two complementary regimes. For general score parameterizations, we derive a non-convex analysis of SGD for the weighted denoising score-matching objective, making explicit how the resulting optimization bound depends on the loss weighting and time-sampling distribution. We then consider overparameterized two-layer ReLU networks and develop a Neural Tangent Kernel analysis tailored to diffusion training with stochastic gradients, yielding score-approximation error bounds along the SGD trajectory. Our analysis quantifies the role of the reweighting factor in these bounds, providing a theoretical characterization of weighting choices used in practice.
Related
- A Rate Transfer Method: Convergence Rates in Stochastic Gradient Descent via Stochastic Calculus
- Convergence rate of stochastic gradient descent under Łojasiewicz type condition and polynomially controlled noise
- Large-Step Last-Iterate Convergence of Random-Reshuffling Stochastic Gradient Descent
- Convergence of stochastic gradient descent for learning linear neural networks
- On the convergence of doubly stochastic block gradient descent method for linear inverse problems
BibTeX
@misc{strasman2026nonasymptotic,
title = {{Non-asymptotic Convergence of Stochastic Gradient Descent in Score-based Generative Models}},
author = {Stanislas Strasman and Sobihan Surendran and Sylvain Le Corff},
howpublished = {HAL},
year = {2026},
month = jul,
doi = {10.48550/arxiv.2607.04775},
eprint = {2607.04775},
archivePrefix = {arXiv},
url = {https://hal.science/hal-05678268v2},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Non-asymptotic Convergence of Stochastic Gradient Descent in Score-based Generative Models
AU - Stanislas Strasman
AU - Sobihan Surendran
AU - Sylvain Le Corff
PY - 2026
DA - 2026/07/02
PB - HAL
DO - 10.48550/arxiv.2607.04775
UR - https://hal.science/hal-05678268v2
AB - Score-based Generative Models (SGMs) have achieved impressive performance in data generation across a wide range of applications. While the statistical properties of their sampling procedures are increasingly well understood, the optimization dynamics underlying their training remain less explored. SGMs are typically trained by minimizing a weighted denoising score-matching objective, yet optimization guarantees with stochastic gradients remain limited. In this work, we study Stochastic Gradient Descent (SGD) for SGMs, contributing results in two complementary regimes. For general score parameterizations, we derive a non-convex analysis of SGD for the weighted denoising score-matching objective, making explicit how the resulting optimization bound depends on the loss weighting and time-sampling distribution. We then consider overparameterized two-layer ReLU networks and develop a Neural Tangent Kernel analysis tailored to diffusion training with stochastic gradients, yielding score-approximation error bounds along the SGD trajectory. Our analysis quantifies the role of the reweighting factor in these bounds, providing a theoretical characterization of weighting choices used in practice.
ER -