Gauss-Newton drifting extends natural gradient descent
Abstract
Drifting methods have recently been introduced as a promising new paradigm for training generative models, and have attracted a lot of attention so far. In this work, we focus on how the optimization procedure shapes their convergence properties, shedding light on the underlying machinery that may explain their empirical success. More specifically, when the drift field is a Wasserstein gradient, optimizing the drifting loss via plain Euclidean gradient descent seems to forfeit any hope of exploiting convexity and therefore of obtaining any convergence guarantees. Changing either the drift field or the optimization procedure, however, may lead to different dynamics: we show that keeping the Wasserstein gradient as drift field but replacing gradient descent with a Gauss-Newton scheme amounts to performing a natural gradient descent in parameter space. This connection allows us to provide global convergence guarantees for the optimization procedure under suitable convexity assumptions.
Related
- How Inefficient Is Natural Gradient Descent? From Exact Optimality to Θ( \sqrt{ \log d } ) Divergence
- Kernel-Gradient Drifting Models
- Sparse descent subspace-Newton thresholding algorithm for sparse optimization problems
- Global 𝒪(k−2) Convergence of Multilevel Gradient-Regularized Newton Methods
- Broximal Gradient Descent: A Projection-Free Sister of Projected Gradient Descent
BibTeX
@misc{dumont2026gaussnewton,
title = {{Gauss-Newton drifting extends natural gradient descent}},
author = {Théo Dumont and Théo Lacombe and François-Xavier Vialard},
howpublished = {HAL},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.17167},
eprint = {2609.17167},
archivePrefix = {arXiv},
url = {https://hal.science/hal-05742820v1},
note = {CC BY-NC-SA 4.0},
}
RIS
TY - UNPB
TI - Gauss-Newton drifting extends natural gradient descent
AU - Théo Dumont
AU - Théo Lacombe
AU - François-Xavier Vialard
PY - 2026
DA - 2026/09/08
PB - HAL
DO - 10.48550/arxiv.2609.17167
UR - https://hal.science/hal-05742820v1
AB - Drifting methods have recently been introduced as a promising new paradigm for training generative models, and have attracted a lot of attention so far. In this work, we focus on how the optimization procedure shapes their convergence properties, shedding light on the underlying machinery that may explain their empirical success. More specifically, when the drift field is a Wasserstein gradient, optimizing the drifting loss via plain Euclidean gradient descent seems to forfeit any hope of exploiting convexity and therefore of obtaining any convergence guarantees. Changing either the drift field or the optimization procedure, however, may lead to different dynamics: we show that keeping the Wasserstein gradient as drift field but replacing gradient descent with a Gauss-Newton scheme amounts to performing a natural gradient descent in parameter space. This connection allows us to provide global convergence guarantees for the optimization procedure under suitable convexity assumptions.
ER -