On the Convergence of Stochastic Low-Rank Adaptation
Abstract
Low-rank adaptation (LoRA) optimizes J(B,A)=ℒ(Wbase+sBA) over two adapters B ∈ ℝm × r and A ∈ ℝr × n that form a low-rank update to a frozen pretrained weight matrix Wbase ∈ ℝm × n. The prior analysis shows LoRA-GD takes exp{𝒪(ε−2)} oracle calls to find an ε-stationary point such that |∇ J(B,A)|≤ ε in the deterministic setting. We sharpen the analysis and show that 𝒪(ε−4) full-gradient evaluations suffice for the same first-order criterion. We further study stochastic LoRA under unbiased gradient estimates and finite variance. We propose LoRA-NSGDM, which finds an ε-stationary point with 𝒪(ε−8) stochastic oracle complexity. Under the additional mean-square smoothness condition, we use variance reduction strategy and propose LoRA-STORM, which improves the stochastic oracle complexity to 𝒪(ε−6).
Related
- Convergence rates of randomly pivoted methods for low-rank approximation
- A Rate Transfer Method: Convergence Rates in Stochastic Gradient Descent via Stochastic Calculus
- The Maximum Diagonal Product of a Stochastic Matrix of Bounded Rank
- Global and Unconditional R-Linear Convergence of Rank-Compressed Weighted LMSD Sweeps for Strictly Convex Quadratics
- What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptation
BibTeX
@misc{wang2026convergence,
title = {{On the Convergence of Stochastic Low-Rank Adaptation}},
author = {Ru Wang and Chengchang Liu and John C. S. Lui},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2607.21975},
eprint = {2607.21975},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2607.21975},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - On the Convergence of Stochastic Low-Rank Adaptation
AU - Ru Wang
AU - Chengchang Liu
AU - John C. S. Lui
PY - 2026
DA - 2026/09/26
PB - arXiv
DO - 10.48550/arxiv.2607.21975
UR - https://arxiv.org/abs/2607.21975
AB - Low-rank adaptation (LoRA) optimizes J(B,A)=ℒ(W_base+sBA) over two adapters B ∈ ℝ^m × r and A ∈ ℝ^r × n that form a low-rank update to a frozen pretrained weight matrix W_base ∈ ℝ^m × n. The prior analysis shows LoRA-GD takes exp{𝒪(ε^−2)} oracle calls to find an ε-stationary point such that |∇ J(B,A)|≤ ε in the deterministic setting. We sharpen the analysis and show that 𝒪(ε^−4) full-gradient evaluations suffice for the same first-order criterion. We further study stochastic LoRA under unbiased gradient estimates and finite variance. We propose LoRA-NSGDM, which finds an ε-stationary point with 𝒪(ε^−8) stochastic oracle complexity. Under the additional mean-square smoothness condition, we use variance reduction strategy and propose LoRA-STORM, which improves the stochastic oracle complexity to 𝒪(ε^−6).
ER -