The Tamed Subgradient Unadjusted Langevin Algorithm beyond Convexity
Abstract
We study the problem of sampling from target distributions whose potentials are simultaneously non-smooth, subject to superlinear gradient growth, and non-convex. We introduce the Subgradient Tamed Unadjusted Langevin Algorithm (SG-TULA), a discretisation of the Langevin diffusion that operates directly on subgradients, without relying on computationally demanding smoothing procedures. To handle the superlinear regime, taming techniques are employed to produce a stable, explicit scheme. We derive non-asymptotic convergence bounds in Wasserstein-2 distance, with all constants tracked explicitly in terms of dimension and inverse temperature, improving upon the currently known rates for subgradient-based Langevin algorithms. We further provide excess risk estimates for the associated optimisation problem. We verify the assumptions, with explicit constants, for the regularized pretraining potential of a LLM in the GPT-2 lineage and the boosted coordinate-wise variant of SG-TULA pretrains the former competitively against finetuned AdamW and Muon, for which no comparable non-asymptotic guarantees are presently available.
Related
- Dimension-free regimes for unadjusted Langevin in ridge-separable models: A subspace robust Wasserstein analysis
- An Optimal Second-Order Algorithm for Finite-Sum Optimization under Average Smoothness and Convexity
- A spectral gap for Metropolis-adjusted Langevin algorithm with a uniformly randomized step size
- Growth beyond exponent 3/2 for convexity and iterated sum sets
- Hyperbolicity Beyond Convexity in the Alexandrov-Fenchel Inequality
BibTeX
@misc{lytras2026tamed,
title = {{The Tamed Subgradient Unadjusted Langevin Algorithm beyond Convexity}},
author = {Iosif Lytras and Nikolaos Makras and Sotirios Sabanis},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2608.06283},
eprint = {2608.06283},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.06283},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - The Tamed Subgradient Unadjusted Langevin Algorithm beyond Convexity
AU - Iosif Lytras
AU - Nikolaos Makras
AU - Sotirios Sabanis
PY - 2026
DA - 2026/10/25
PB - arXiv
DO - 10.48550/arxiv.2608.06283
UR - https://arxiv.org/abs/2608.06283
AB - We study the problem of sampling from target distributions whose potentials are simultaneously non-smooth, subject to superlinear gradient growth, and non-convex. We introduce the Subgradient Tamed Unadjusted Langevin Algorithm (SG-TULA), a discretisation of the Langevin diffusion that operates directly on subgradients, without relying on computationally demanding smoothing procedures. To handle the superlinear regime, taming techniques are employed to produce a stable, explicit scheme. We derive non-asymptotic convergence bounds in Wasserstein-2 distance, with all constants tracked explicitly in terms of dimension and inverse temperature, improving upon the currently known rates for subgradient-based Langevin algorithms. We further provide excess risk estimates for the associated optimisation problem. We verify the assumptions, with explicit constants, for the regularized pretraining potential of a LLM in the GPT-2 lineage and the boosted coordinate-wise variant of SG-TULA pretrains the former competitively against finetuned AdamW and Muon, for which no comparable non-asymptotic guarantees are presently available.
ER -