Finite-Sample Performance of Gradient Descent in Logistic Regression with Gaussian Design
Abstract
We consider the parameter estimation problem in logistic regression with Gaussian design: the estimation of a fixed unknown parameter θ*∈ ℝd (|θ*|2≥ 1) from n i.i.d. samples {(xi,yi)}i=1n, where xi∼ N(0,Id) and yi|xi ∼ Bernoulli(1/(1+exp(−xi⊤ θ*))). Our main aim is to characterize the finite-sample estimation performance and convergence behavior of gradient descent (GD) on the maximum likelihood objective (i.e., the logistic loss). Under small O(1) stepsize and $0$ initialization, we show that GD linearly converges to a small neighborhood of θ* achieving an ℓ2 error of order O(√(|θ*|25d/n)). This substantially goes beyond existing theoretical results that lack non-asymptotic estimation error rate and exhibit much slower parameter convergence. We also establish a faster local linear convergence to the same statistical error under a large Θ(|θ*|2) stepsize. The main technical component is to show that the gradient of the logistic loss satisfies a certain approximate invertibility condition (AIC). To that end, we uniformly control the deviation of the gradient from its population counterpart by covering and peeling arguments, and then show that the population GD is a contraction by a delicate analysis based on the eigenvalues of population Hessian matrices. Finally, we build upon the recent work Matsumoto and Mazumdar (2025) and devise a novel efficient estimator that attains a sharper rate in high dimensions. This indicates that the existing non-asymptotic guarantees exhibit sub-optimal dependence on |θ*|2, and that in many regimes Θ(√(|θ*|2d/n)) is the tight estimation error rate. Numerical examples are provided to corroborate our theoretical results.
Related
- Gradient descent reaches arbitrary accuracy in a fixed number of large steps on multiclass logistic regression
- Ideal Paths for Approximating Logistic Gradient Descent Trajectories at Large Initialization
- Logistic Gaussian process density regression: a generalized Bayesian approach
- Scalable Logistic Gaussian Process Density Regression with Kinetic Langevin Sampling
- Riemannian Gradient Descent for Gaussian Mixture Models with unknown diagonal covariances
BibTeX
@misc{chen2026finitesample,
title = {{Finite-Sample Performance of Gradient Descent in Logistic Regression with Gaussian Design}},
author = {Junren Chen and Arya Mazumdar},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2606.21683},
eprint = {2606.21683},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.21683},
}
RIS
TY - UNPB
TI - Finite-Sample Performance of Gradient Descent in Logistic Regression with Gaussian Design
AU - Junren Chen
AU - Arya Mazumdar
PY - 2026
DA - 2026/09/26
PB - arXiv
DO - 10.48550/arxiv.2606.21683
UR - https://arxiv.org/abs/2606.21683
AB - We consider the parameter estimation problem in logistic regression with Gaussian design: the estimation of a fixed unknown parameter θ^*∈ ℝ^d (|θ^*|_2≥ 1) from n i.i.d. samples {(x_i,y_i)}_i=1^n, where x_i∼ N(0,I_d) and y_i|x_i ∼ Bernoulli(1/(1+exp(−x_i^⊤ θ^*))). Our main aim is to characterize the finite-sample estimation performance and convergence behavior of gradient descent (GD) on the maximum likelihood objective (i.e., the logistic loss). Under small O(1) stepsize and $0$ initialization, we show that GD linearly converges to a small neighborhood of θ^* achieving an ℓ_2 error of order O(√(|θ^*|_2^5d/n)). This substantially goes beyond existing theoretical results that lack non-asymptotic estimation error rate and exhibit much slower parameter convergence. We also establish a faster local linear convergence to the same statistical error under a large Θ(|θ^*|_2) stepsize. The main technical component is to show that the gradient of the logistic loss satisfies a certain approximate invertibility condition (AIC). To that end, we uniformly control the deviation of the gradient from its population counterpart by covering and peeling arguments, and then show that the population GD is a contraction by a delicate analysis based on the eigenvalues of population Hessian matrices. Finally, we build upon the recent work Matsumoto and Mazumdar (2025) and devise a novel efficient estimator that attains a sharper rate in high dimensions. This indicates that the existing non-asymptotic guarantees exhibit sub-optimal dependence on |θ^*|_2, and that in many regimes Θ(√(|θ^*|_2d/n)) is the tight estimation error rate. Numerical examples are provided to corroborate our theoretical results.
ER -