Implicit Bias of (Stochastic) Gradient Descent on Deep Linear Diagonal Networks
Abstract
Modern machine learning models are often overparameterized, i.e., have more parameters than training data. This typically implies that the loss function admits multiple minimizers so that training methods (and their hyperparameter choices) have significant influence on the computed solution. We study this so-called implicit bias phenomenon in the context of training deep linear diagonal networks. Previous work has shown that the gradient flow for learning such networks converges to (almost) sparse solutions when initialized small. For an arbitrary number of layers, we extend this analysis to the practically more realistic (stochastic) gradient descent iterations. Our approach identifies (stochastic) gradient descent as a perturbed (stochastic) mirror gradient descent. This allows us to establish convergence and characterize the limiting solution, relating it to weighted ℓ1-minimization. Moreover, we provide sufficient conditions on the step sizes and quantify the influence of the step sizes on the implicit bias.
Related
- Convergence of stochastic gradient descent for learning linear neural networks
- Stochastic Gradient Descent for Traffic and Pedestrian Models via Neural Networks and Parameter Identification
- Riemannian Gradient Descent for Gaussian Mixture Models with unknown diagonal covariances
- Overfitting of Spectral Gradient Descent: How Matrix Geometry shapes Generalization and Implicit Bias
- A Rate Transfer Method: Convergence Rates in Stochastic Gradient Descent via Stochastic Calculus
BibTeX
@misc{bartolomaeus2026implicit,
title = {{Implicit Bias of (Stochastic) Gradient Descent on Deep Linear Diagonal Networks}},
author = {Wiebke Bartolomaeus and Holger Rauhut},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.39717},
eprint = {2610.39717},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.39717},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Implicit Bias of (Stochastic) Gradient Descent on Deep Linear Diagonal Networks
AU - Wiebke Bartolomaeus
AU - Holger Rauhut
PY - 2026
DA - 2026/10/29
PB - arXiv
DO - 10.48550/arxiv.2610.39717
UR - https://arxiv.org/abs/2610.39717
AB - Modern machine learning models are often overparameterized, i.e., have more parameters than training data. This typically implies that the loss function admits multiple minimizers so that training methods (and their hyperparameter choices) have significant influence on the computed solution. We study this so-called implicit bias phenomenon in the context of training deep linear diagonal networks. Previous work has shown that the gradient flow for learning such networks converges to (almost) sparse solutions when initialized small. For an arbitrary number of layers, we extend this analysis to the practically more realistic (stochastic) gradient descent iterations. Our approach identifies (stochastic) gradient descent as a perturbed (stochastic) mirror gradient descent. This allows us to establish convergence and characterize the limiting solution, relating it to weighted ℓ_1-minimization. Moreover, we provide sufficient conditions on the step sizes and quantify the influence of the step sizes on the implicit bias.
ER -