Spectral and Subspace Dynamics of LoRA in Supervised Fine-Tuning and Reinforcement Learning

Abstract

Low-rank adaptation (LoRA) is commonly used in both supervised fine-tuning (SFT) and reinforcement learning (RL), often under similar optimizer settings. Yet, how the different dynamics of the two stages manifest in the low-rank increment that LoRA optimizes remains unclear. Tracking the singular spectrum and subspaces of LoRA updates on mathematical reasoning tasks, we find that RL concentrates the spectrum at every learning rate, while SFT expands it at moderate learning rates; moreover, RL rotates the singular subspaces much faster than SFT at small learning rates. Interpolating between SFT and RL one ingredient at a time, we pinpoint group-mean baseline subtraction as the source of both contrasts. A quadratic multi-index model updated via LoRA unveils the underlying mechanism: group-mean baseline subtraction tilts the balance between two competing forces, coherent fitting and gradient noise, toward spectral concentration and faster subspace rotation, while forcing the risk-favorable learning rate to be low. Guided by these dynamics, we tailor LoRA updates to each stage: in SFT, we flatten the spectrum of the full update to spread adaptation across singular directions, whereas in RL, we flatten the part of the update outside the current LoRA increment span to promote subspace rotation. Experiments on mathematical reasoning tasks across multiple models show that the respective configurations for SFT and RL yield the best performance among the compared methods in most evaluated settings.

Survival Reinforcement Learning: Toward Scalable Self-Supervised RL
Franki Nguimatsia-Tiofack, Fabian Schramm, Théotime Le Hellard +1
HAL18 Sep 2026PreprintCC BY-NC-SA 4.0Full textStatisticsHAL
Is Spectral Self-Supervised Learning a Minimax Optimal Method of Density Ratio Estimation?
Hiroki Waida, Takafumi Kanamori
arXiv22 Oct 2026PreprintarXiv licenceFull textStatisticsarXiv
Dual-domain self-supervised feature alignment via spectral–spatial representation learning for deepfake anomaly detection
Yi He
PLOS One11 Sep 2026Peer-reviewedCC BY 4.0Full textBioinformatics & computational biology ProbabilityPLOS
The Geometry of Multitask Learning: Sharp Rates for Shared Subspace Estimation
Tony Cai, Abhinav Chakraborty, Samory Kpotufe
arXiv29 Oct 2026PreprintCC BY 4.0Full textStatisticsarXiv
Measurement-dependent reduction of open dynamics in the quantum phase estimation algorithm: the visible subspace and the damped Fejér kernel
Marina O. Gordeichuk, Oleg M. Kiselev
arXiv26 Oct 2026PreprintarXiv licenceFull textMathematical physics Numerical analysis & scientific computingarXiv

Cite

BibTeX RIS

BibTeX

@misc{liu2026spectral,
  title = {{Spectral and Subspace Dynamics of LoRA in Supervised Fine-Tuning and Reinforcement Learning}},
  author = {Chenruo Liu and Zihan Wang and Yijun Dong and Qi Lei},
  howpublished = {arXiv},
  year = {2026},
  month = nov,
  doi = {10.48550/arxiv.2611.08414},
  eprint = {2611.08414},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08414},
}

RIS

TY  - UNPB
TI  - Spectral and Subspace Dynamics of LoRA in Supervised Fine-Tuning and Reinforcement Learning
AU  - Chenruo Liu
AU  - Zihan Wang
AU  - Yijun Dong
AU  - Qi Lei
PY  - 2026
DA  - 2026/11/05
PB  - arXiv
DO  - 10.48550/arxiv.2611.08414
UR  - https://arxiv.org/abs/2611.08414
AB  - Low-rank adaptation (LoRA) is commonly used in both supervised fine-tuning (SFT) and reinforcement learning (RL), often under similar optimizer settings. Yet, how the different dynamics of the two stages manifest in the low-rank increment that LoRA optimizes remains unclear. Tracking the singular spectrum and subspaces of LoRA updates on mathematical reasoning tasks, we find that RL concentrates the spectrum at every learning rate, while SFT expands it at moderate learning rates; moreover, RL rotates the singular subspaces much faster than SFT at small learning rates. Interpolating between SFT and RL one ingredient at a time, we pinpoint group-mean baseline subtraction as the source of both contrasts. A quadratic multi-index model updated via LoRA unveils the underlying mechanism: group-mean baseline subtraction tilts the balance between two competing forces, coherent fitting and gradient noise, toward spectral concentration and faster subspace rotation, while forcing the risk-favorable learning rate to be low. Guided by these dynamics, we tailor LoRA updates to each stage: in SFT, we flatten the spectrum of the full update to spread adaptation across singular directions, whereas in RL, we flatten the part of the update outside the current LoRA increment span to promote subspace rotation. Experiments on mathematical reasoning tasks across multiple models show that the respective configurations for SFT and RL yield the best performance among the compared methods in most evaluated settings.
ER  -

Abstract

Related

Cite