Intrinsic Gradient Barriers in Pushforward Deep Generative Models
Abstract
Broad classes of deep generative models (DGMs) operate by learning to map a simple source distribution, such as a standard Gaussian, to a complex high-dimensional target distribution. Despite their success, DGMs often empirically struggle to accurately represent distributions with heavy tails or well-separated modes. In this work, we show that these difficulties reflect intrinsic properties of the underlying transport problem rather than particular neural network architectures or training procedures. We establish quantitative lower bounds on the gradients and Jacobians of pushforward mappings that transport log-concave source distributions to such targets. For heavy-tailed target distributions, the required gradient magnitude grows as the target develops increasingly large moments. For multimodal target distributions, the low-probability regions between well-separated modes similarly force large gradients in the transport map. We discuss how such large derivatives relate to trade-offs between expressivity and numerical stability. Our results hold under substantially weaker regularity conditions than previous analyses and remain stable under perturbations of the target distribution. Finally, we show that these obstructions can be mitigated when the goal is approximation of a given density rather than generative modeling. Specifically, we propose simple classes of bijective reparametrizations that can alleviate the difficult geometric structure and reduce the gradient burden placed on the learned generative map. Numerical experiments support our theoretical analysis.
Related
- Non-asymptotic Convergence of Stochastic Gradient Descent in Score-based Generative Models
- Gradient-Free Sampling from Generative Models via Stochastic Bounded Extremum Seeking
- Fine-Tuning Generative Models for Extreme Events via CVaR-Penalized Wasserstein Gradient Flows
- Classical BV Pushforward
- A Quadratically Enriched Pushforward for Deligne--Mumford Stacks
BibTeX
@misc{tam2026intrinsic,
title = {{Intrinsic Gradient Barriers in Pushforward Deep Generative Models}},
author = {Edric Tam and Qingyang Liu and Barbara Engelhardt and Ray Bai},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.34484},
eprint = {2610.34484},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.34484},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Intrinsic Gradient Barriers in Pushforward Deep Generative Models
AU - Edric Tam
AU - Qingyang Liu
AU - Barbara Engelhardt
AU - Ray Bai
PY - 2026
DA - 2026/10/25
PB - arXiv
DO - 10.48550/arxiv.2610.34484
UR - https://arxiv.org/abs/2610.34484
AB - Broad classes of deep generative models (DGMs) operate by learning to map a simple source distribution, such as a standard Gaussian, to a complex high-dimensional target distribution. Despite their success, DGMs often empirically struggle to accurately represent distributions with heavy tails or well-separated modes. In this work, we show that these difficulties reflect intrinsic properties of the underlying transport problem rather than particular neural network architectures or training procedures. We establish quantitative lower bounds on the gradients and Jacobians of pushforward mappings that transport log-concave source distributions to such targets. For heavy-tailed target distributions, the required gradient magnitude grows as the target develops increasingly large moments. For multimodal target distributions, the low-probability regions between well-separated modes similarly force large gradients in the transport map. We discuss how such large derivatives relate to trade-offs between expressivity and numerical stability. Our results hold under substantially weaker regularity conditions than previous analyses and remain stable under perturbations of the target distribution. Finally, we show that these obstructions can be mitigated when the goal is approximation of a given density rather than generative modeling. Specifically, we propose simple classes of bijective reparametrizations that can alleviate the difficult geometric structure and reduce the gradient burden placed on the learned generative map. Numerical experiments support our theoretical analysis.
ER -