Rank-Efficient LoRA via Joint Tangent-Space Optimization under Isotropic Curvature
Abstract
Low-Rank Adaptation (LoRA) is an effective approach for adapting large pretrained models by learning low-rank weight updates. In practice, the LoRA rank is used to control an adapter's parameter budget and representational capacity. We show that this view is incomplete: while the nominal rank determines the representational capacity, the optimizer shapes how much of that capacity is used in the induced weight-space updates. In a case study of GPT-2 adaptation with LoRA, we observe a strong rank-dependent optimizer effect. Despite using the same nominal rank, AdamW often produces per-step updates with concentrated singular spectra and low effective rank, whereas Muon uses a richer set of directions and benefits more consistently from increasing LoRA rank. These observations motivate ISO-LoRA, an optimizer that couples the LoRA factor updates through spectral descent on the induced tangent perturbation in weight space. ISO-LoRA promotes updates that distribute energy more evenly across singular directions, improving rank utilization while preserving compatibility with the LoRA parameterization. We complement this design with theoretical guarantees showing that ISO-LoRA can achieve higher effective rank than standard factor-wise optimizers through a one-step analysis under a stylized spiked-gradient model. We validate this design on language-model adaptation across 0.1B-7B-parameter models, where ISO-LoRA improves effective rank and downstream performance, with the strongest gains at moderate-to-large LoRA ranks. Our results highlight rank utilization as a key factor in LoRA optimization and suggest that optimizer design offers an important path toward stronger parameter-efficient adaptation.
Related
- Efficient Joint Trajectory Optimization and Resource Allocation for Secure UAV Networks: A Deep Reinforcement Learning Approach
- Geodesics with Unified Tangent-constrained Priors and Curvature Regularization
- Curvature pinching for minimal Tangent-Yang-Mills hypersurfaces in spheres
- Scalar curvature bounds and tangent cones for Kähler--Ricci shrinkers
- Minimal Two-Spheres and Manifolds with Positive Isotropic Curvature
BibTeX
@misc{zhu2026rankefficient,
title = {{Rank-Efficient LoRA via Joint Tangent-Space Optimization under Isotropic Curvature}},
author = {Zihan Zhu and Zhehang Du and Xuyang Chen and Tim Tsz-Kit Lau and Jiayuan Wu and X. Y. Han and Qi Long and Weijie Su},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.12123},
eprint = {2609.12123},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.12123},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Rank-Efficient LoRA via Joint Tangent-Space Optimization under Isotropic Curvature
AU - Zihan Zhu
AU - Zhehang Du
AU - Xuyang Chen
AU - Tim Tsz-Kit Lau
AU - Jiayuan Wu
AU - X. Y. Han
AU - Qi Long
AU - Weijie Su
PY - 2026
DA - 2026/09/10
PB - arXiv
DO - 10.48550/arxiv.2609.12123
UR - https://arxiv.org/abs/2609.12123
AB - Low-Rank Adaptation (LoRA) is an effective approach for adapting large pretrained models by learning low-rank weight updates. In practice, the LoRA rank is used to control an adapter's parameter budget and representational capacity. We show that this view is incomplete: while the nominal rank determines the representational capacity, the optimizer shapes how much of that capacity is used in the induced weight-space updates. In a case study of GPT-2 adaptation with LoRA, we observe a strong rank-dependent optimizer effect. Despite using the same nominal rank, AdamW often produces per-step updates with concentrated singular spectra and low effective rank, whereas Muon uses a richer set of directions and benefits more consistently from increasing LoRA rank. These observations motivate ISO-LoRA, an optimizer that couples the LoRA factor updates through spectral descent on the induced tangent perturbation in weight space. ISO-LoRA promotes updates that distribute energy more evenly across singular directions, improving rank utilization while preserving compatibility with the LoRA parameterization. We complement this design with theoretical guarantees showing that ISO-LoRA can achieve higher effective rank than standard factor-wise optimizers through a one-step analysis under a stylized spiked-gradient model. We validate this design on language-model adaptation across 0.1B-7B-parameter models, where ISO-LoRA improves effective rank and downstream performance, with the strongest gains at moderate-to-large LoRA ranks. Our results highlight rank utilization as a key factor in LoRA optimization and suggest that optimizer design offers an important path toward stronger parameter-efficient adaptation.
ER -