What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptation
Abstract
The rank r in LoRA is widely treated as a capacity control: a smaller rank is assumed to yield a simpler model that generalizes better. We show that, under hard per-factor norm budgets---the idealization of the weight decay and norm control used in practice---this intuition breaks down. The reason is structural: under such budgets, the updates LoRA can reach are exactly the matrices of rank at most r inside a nuclear-norm ball, and every complexity and displacement functional we analyze is maximized over this set by a rank-one update---so the rank cap never binds. The consequences follow directly. The linear-readout model class we study is identical for every r ≥ 1, its Rademacher complexity carries no dependence on r, and the distance the adaptation can move the source distribution obeys a rank-independent upper bound that we show is sharp. If rank does not control capacity, where does it act? We identify two places. Statistically, replacing the per-factor budgets with a joint budget on the product restores a data-dependent, rank-sensitive complexity bound---though the gain appears only for well-spread feature distributions, and the worst case remains rank-free. Spectrally, rank sets the price of adaptation: canceling the leading singular directions of the pretrained weight requires both sufficient rank and sufficient budget. We bound the smallest rank achieving a desired source--target alignment, with upper and lower bounds that match under two-sided spectral decay. Together, these results recast rank as governing which updates are reachable and what cancellation costs---not how much capacity the model has.
Related
- Spectral Differential Properties of the Distributional FKF Transform with Applications to Resilient Supply-Chain Systems
- From What Is Used to What Is Missing: Rethinking Dynamic Rank in LoRA
- On the Convergence of Stochastic Low-Rank Adaptation
- LoRA-RC: Reservoir Computing with Low-Rank Adaptation
- What the submission does not show: Complexifying cognitive offloading in students' mathematical work with LLMs
BibTeX
@misc{barazandeh2026what,
title = {{What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptation}},
author = {Babak Barazandeh},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2609.32002},
eprint = {2609.32002},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.32002},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptation
AU - Babak Barazandeh
PY - 2026
DA - 2026/10/19
PB - arXiv
DO - 10.48550/arxiv.2609.32002
UR - https://arxiv.org/abs/2609.32002
AB - The rank r in LoRA is widely treated as a capacity control: a smaller rank is assumed to yield a simpler model that generalizes better. We show that, under hard per-factor norm budgets---the idealization of the weight decay and norm control used in practice---this intuition breaks down. The reason is structural: under such budgets, the updates LoRA can reach are exactly the matrices of rank at most r inside a nuclear-norm ball, and every complexity and displacement functional we analyze is maximized over this set by a rank-one update---so the rank cap never binds. The consequences follow directly. The linear-readout model class we study is identical for every r ≥ 1, its Rademacher complexity carries no dependence on r, and the distance the adaptation can move the source distribution obeys a rank-independent upper bound that we show is sharp. If rank does not control capacity, where does it act? We identify two places. Statistically, replacing the per-factor budgets with a joint budget on the product restores a data-dependent, rank-sensitive complexity bound---though the gain appears only for well-spread feature distributions, and the worst case remains rank-free. Spectrally, rank sets the price of adaptation: canceling the leading singular directions of the pretrained weight requires both sufficient rank and sufficient budget. We bound the smallest rank achieving a desired source--target alignment, with upper and lower bounds that match under two-sided spectral decay. Together, these results recast rank as governing which updates are reachable and what cancellation costs---not how much capacity the model has.
ER -