ProLand: A New Infeasible Manifold Optimization Method for LoRA Fine-Tuning
Abstract
Low-rank adaptation (LoRA) enables parameter-efficient fine-tuning but can suffer from redundant basis directions. Constraining the basis factor to the Stiefel manifold addresses this redundancy through orthonormality. Landing methods allow temporarily infeasible iterates to explore a neighborhood of the manifold while progressively restoring orthogonality without per-step retractions. To improve optimization efficiency while retaining this flexibility, we introduce ProLand (Product-space Landing), a retraction-free optimizer that combines coordinated factor updates with Muon-style spectral directions. By designing the updates in the space of the represented weight increment, ProLand separates objective optimization from orthogonality restoration, supporting efficient feasibility recovery. This coordinated construction also enables Muon-style directions to guide objective optimization while preserving that separation. Experiments cover four GLUE validation tasks on DeBERTa-v3-base and seven language benchmarks on Qwen2.5-3B, Llama-3.2-3B, and Moonlight-16B-A3B across multiple adapter ranks. Across these 11 tasks, ProLand achieves the highest average score among the compared methods in every tested encoder and decoder model-rank configuration.
Related
- Infeasible Momentum Orthogonalized Methods for Optimization on the Stiefel Manifold
- A Feasible-Direction-Type Method for Nonconvex Optimization with Infeasible Starts
- An Infeasible Method with Feasibility Safeguard for Nonsmooth Composite Optimization Over Manifolds
- Proximal-Linearized NEP Method for Composite Optimization with Set-Inclusion and Manifold Constraints
- Gaussian Linear Functional Manifold Method for Massive Point Cloud Data
BibTeX
@misc{zhou2026proland,
title = {{ProLand: A New Infeasible Manifold Optimization Method for LoRA Fine-Tuning}},
author = {Chenyu Zhou and Yuxuan Zeng and Qi Deng and Dongdong Ge and Yinyu Ye},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.00236},
eprint = {2611.00236},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.00236},
}
RIS
TY - UNPB
TI - ProLand: A New Infeasible Manifold Optimization Method for LoRA Fine-Tuning
AU - Chenyu Zhou
AU - Yuxuan Zeng
AU - Qi Deng
AU - Dongdong Ge
AU - Yinyu Ye
PY - 2026
DA - 2026/11/01
PB - arXiv
DO - 10.48550/arxiv.2611.00236
UR - https://arxiv.org/abs/2611.00236
AB - Low-rank adaptation (LoRA) enables parameter-efficient fine-tuning but can suffer from redundant basis directions. Constraining the basis factor to the Stiefel manifold addresses this redundancy through orthonormality. Landing methods allow temporarily infeasible iterates to explore a neighborhood of the manifold while progressively restoring orthogonality without per-step retractions. To improve optimization efficiency while retaining this flexibility, we introduce ProLand (Product-space Landing), a retraction-free optimizer that combines coordinated factor updates with Muon-style spectral directions. By designing the updates in the space of the represented weight increment, ProLand separates objective optimization from orthogonality restoration, supporting efficient feasibility recovery. This coordinated construction also enables Muon-style directions to guide objective optimization while preserving that separation. Experiments cover four GLUE validation tasks on DeBERTa-v3-base and seven language benchmarks on Qwen2.5-3B, Llama-3.2-3B, and Moonlight-16B-A3B across multiple adapter ranks. Across these 11 tasks, ProLand achieves the highest average score among the compared methods in every tested encoder and decoder model-rank configuration.
ER -