Statistical and Privacy Guarantees for Deep M-Regression via Stochastic Gradient Langevin Dynamics
Abstract
In this article, we study deep nonparametric regression trained via stochastic gradient Langevin dynamics (SGLD), a variant of stochastic gradient descent that injects carefully calibrated Gaussian noise during optimization. Using a PAC-Bayesian analysis tailored to deep ReLU networks, we establish an excess-risk bound that simultaneously accounts for network approximation error, statistical complexity, and the algorithmic cost accumulated along the SGLD trajectory. Under a Bernstein condition and within a stable SGLD regime, the excess-risk bound contains a stochastic term whose coefficient is inversely proportional to the variance of injected noise, while the estimator retains the optimal rate for nonparametric regression under the Hölder smoothness condition up to logarithmic factors. This result provides an approximation-error-aware, rate-optimal statistical perspective on the widely observed generalization benefits of moderate SGLD noise. For per-sample clipped SGLD, we further establish an exact f-DP guarantee and the corresponding asymptotic μ-GDP guarantee for the proposed estimator. We establish statistical convergence rates and differential privacy guarantees for DNN estimators trained using SGLD, while explicitly accounting for both network approximation error and the effect of injected noise in the excess-risk bound. Extensive simulation studies and analyses of two real datasets examine the finite-sample predictive performance and privacy-utility behavior of the proposed procedure.
Related
- Theoretical guarantees for stochastic gradient Langevin dynamics
- On the computational cost of Stochastic Gradient Langevin Dynamics
- Scalable Statistical Inference in Stochastic Gradient Descent
- Improved Guarantees for Langevin Monte Carlo with Average Smoothness
- The Geometry of Statistical Feature Learning in Mean-Field Langevin Dynamics
BibTeX
@misc{ge2026statistical,
title = {{Statistical and Privacy Guarantees for Deep M-Regression via Stochastic Gradient Langevin Dynamics}},
author = {Yeheng Ge and Jian Huang and Shuyuan Wu},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.07490},
eprint = {2611.07490},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.07490},
}
RIS
TY - UNPB
TI - Statistical and Privacy Guarantees for Deep M-Regression via Stochastic Gradient Langevin Dynamics
AU - Yeheng Ge
AU - Jian Huang
AU - Shuyuan Wu
PY - 2026
DA - 2026/11/05
PB - arXiv
DO - 10.48550/arxiv.2611.07490
UR - https://arxiv.org/abs/2611.07490
AB - In this article, we study deep nonparametric regression trained via stochastic gradient Langevin dynamics (SGLD), a variant of stochastic gradient descent that injects carefully calibrated Gaussian noise during optimization. Using a PAC-Bayesian analysis tailored to deep ReLU networks, we establish an excess-risk bound that simultaneously accounts for network approximation error, statistical complexity, and the algorithmic cost accumulated along the SGLD trajectory. Under a Bernstein condition and within a stable SGLD regime, the excess-risk bound contains a stochastic term whose coefficient is inversely proportional to the variance of injected noise, while the estimator retains the optimal rate for nonparametric regression under the Hölder smoothness condition up to logarithmic factors. This result provides an approximation-error-aware, rate-optimal statistical perspective on the widely observed generalization benefits of moderate SGLD noise. For per-sample clipped SGLD, we further establish an exact f-DP guarantee and the corresponding asymptotic μ-GDP guarantee for the proposed estimator. We establish statistical convergence rates and differential privacy guarantees for DNN estimators trained using SGLD, while explicitly accounting for both network approximation error and the effect of injected noise in the excess-risk bound. Extensive simulation studies and analyses of two real datasets examine the finite-sample predictive performance and privacy-utility behavior of the proposed procedure.
ER -