Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights
Abstract
Standard weight decay treats each weight matrix as a vector and ignores its spectral structure. We introduce spectral weight decay, a post-step decoupled nuclear-norm update that applies additive rather than multiplicative spectral shrinkage. We connect the update to approximate proximal descent and show that its sensitivity to update order can exceed that of conventional ℓ2 weight decay near rank deficiency. Across LLaMA models with $124$M to $500$M parameters, spectral weight decay lowers effective rank and improves SVD-LLM compression at matched validation loss. At $500$M and a 4% distortion budget, it reaches 1.89× compression and 1.18× GPU inference speedup, compared with 1.14× and 1.01× after standard weight decay. Under fixed-horizon training with 60% label noise, it also improves final mean clean-test accuracy over matched ℓ2 regularization by up to $17.8$ points on MNIST and $4.6$ points across four BERT-base tasks. Code is available at https://github.com/brain-lab-research/SpectralWD.
Related
- From Dataset Spectral Geometry to Network Weights: A Geometry-Aware Initialization for Sigmoidal MLPs in Image Classification
- Generalized Hamming Weights and Subcode Support Weight Distributions of Twisted Reed-Solomon Codes
- Efficient and Adaptive Estimation of Portfolio Weights with Spectral Risk Measures
- Bridging Network Psychometrics and Artificial Intelligence: An Ising-Potts Model with LLM-Derived Weights
- Spectral sequences, décalage, and the Beilinson t -structure
BibTeX
@misc{andriianov2026spectral,
title = {{Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights}},
author = {Dmitrii Andriianov and Andrey Veprikov and Aleksandr Beznosikov},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.11730},
eprint = {2610.11730},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.11730},
}
RIS
TY - UNPB
TI - Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights
AU - Dmitrii Andriianov
AU - Andrey Veprikov
AU - Aleksandr Beznosikov
PY - 2026
DA - 2026/10/08
PB - arXiv
DO - 10.48550/arxiv.2610.11730
UR - https://arxiv.org/abs/2610.11730
AB - Standard weight decay treats each weight matrix as a vector and ignores its spectral structure. We introduce spectral weight decay, a post-step decoupled nuclear-norm update that applies additive rather than multiplicative spectral shrinkage. We connect the update to approximate proximal descent and show that its sensitivity to update order can exceed that of conventional ℓ_2 weight decay near rank deficiency. Across LLaMA models with $124$M to $500$M parameters, spectral weight decay lowers effective rank and improves SVD-LLM compression at matched validation loss. At $500$M and a 4% distortion budget, it reaches 1.89× compression and 1.18× GPU inference speedup, compared with 1.14× and 1.01× after standard weight decay. Under fixed-horizon training with 60% label noise, it also improves final mean clean-test accuracy over matched ℓ_2 regularization by up to $17.8$ points on MNIST and $4.6$ points across four BERT-base tasks. Code is available at https://github.com/brain-lab-research/SpectralWD.
ER -