The Intruder Threshold: A Spectral Law for LoRA Fine-Tuning
Abstract
LoRA fine-tuning can create intruder dimensions: new leading singular vectors of the updated weight matrix W+BA that are nearly orthogonal to all pretrained singular vectors and that drive catastrophic forgetting. Since their discovery, no theory has predicted, layer by layer on measured spectra, when they appear. We derive a per-layer critical update strength s∗=θ̅/(γσ1(BA)), computed from the measured spectrum of W alone through the rectangular spiked-deformation transform, together with an exact secular-equation characterization of the updated spectrum, with no fitted parameters. In a pre-specified study spanning four dense Transformer families, a state-space model, a mixture-of-experts model, and an encoder-decoder (18 adapters, 9{,}840 layer scans), the law localizes the empirical threshold within a factor of two on 82% of layers, separates intruder-bearing from intruder-free layers at deployment with a mean AUC of $0.89$, holds unchanged on six third-party adapters, and predicts where WikiText-2 perplexity begins to degrade; a combination of the two pre-specified edge evaluations reaches 98% and is confirmed out-of-bag on the external adapters ($0.997$). Full fine-tuning disperses its update far below the threshold of every layer, which resolves the asymmetry between LoRA and full fine-tuning. Norm-matched interventions confirm that threshold-crossing layers, rather than update magnitude, carry the forgetting, and a spike-budget rule derived from the thresholds, requiring one SVD and no validation sweeps, reduces forgetting by 62% on the most fragile model at no task cost.
Related
- Spectral and Subspace Dynamics of LoRA in Supervised Fine-Tuning and Reinforcement Learning
- ProLand: A New Infeasible Manifold Optimization Method for LoRA Fine-Tuning
- How Low Can You Go: Fine-Tuning LLMs with Bayesian Optimization (LlMBO)
- Statistical Benefits of Fine-Tuning from Pretrained Initialization in Diagonal Linear Networks
- Two-Timescale Fine-tuning Provably Learns New Features for Two-Layer ReLU Networks
BibTeX
@misc{xie2026intruder,
title = {{The Intruder Threshold: A Spectral Law for LoRA Fine-Tuning}},
author = {Peng Xie and Amr Alanwar},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2607.23711},
eprint = {2607.23711},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2607.23711},
}
RIS
TY - UNPB
TI - The Intruder Threshold: A Spectral Law for LoRA Fine-Tuning
AU - Peng Xie
AU - Amr Alanwar
PY - 2026
DA - 2026/09/12
PB - arXiv
DO - 10.48550/arxiv.2607.23711
UR - https://arxiv.org/abs/2607.23711
AB - LoRA fine-tuning can create intruder dimensions: new leading singular vectors of the updated weight matrix W+BA that are nearly orthogonal to all pretrained singular vectors and that drive catastrophic forgetting. Since their discovery, no theory has predicted, layer by layer on measured spectra, when they appear. We derive a per-layer critical update strength s^∗=θ̅/(γσ_1(BA)), computed from the measured spectrum of W alone through the rectangular spiked-deformation transform, together with an exact secular-equation characterization of the updated spectrum, with no fitted parameters. In a pre-specified study spanning four dense Transformer families, a state-space model, a mixture-of-experts model, and an encoder-decoder (18 adapters, 9{,}840 layer scans), the law localizes the empirical threshold within a factor of two on 82% of layers, separates intruder-bearing from intruder-free layers at deployment with a mean AUC of $0.89$, holds unchanged on six third-party adapters, and predicts where WikiText-2 perplexity begins to degrade; a combination of the two pre-specified edge evaluations reaches 98% and is confirmed out-of-bag on the external adapters ($0.997$). Full fine-tuning disperses its update far below the threshold of every layer, which resolves the asymmetry between LoRA and full fine-tuning. Norm-matched interventions confirm that threshold-crossing layers, rather than update magnitude, carry the forgetting, and a spike-budget rule derived from the thresholds, requiring one SVD and no validation sweeps, reduces forgetting by 62% on the most fragile model at no task cost.
ER -