PRISM-LoRA: Principal-Direction-Guided Single-LoRA Merging Framework for Continual Learning of Large Language Models
- 1Department of Computing, Gachon University, Seongnam-si 13120, Republic of Korea
Abstract
Continual learning of large language models requires incorporating new knowledge while limiting catastrophic forgetting; however, many low-rank adaptation (LoRA)-based methods retain task-specific modules or separate learning spaces that grow with the task sequence. We propose PRISM-LoRA, a replay-free framework that repeatedly trains a single LoRA module, merges its update into the backbone, and reinitializes it for the next task. Unlike LoRA-based continual-learning approaches that retain task-specific modules or allocate separate task-specific learning spaces, PRISM-LoRA uses the directional structure of the accumulated backbone weight change to guide both subsequent training and consolidation while reusing a single LoRA module. Before each task, PRISM-LoRA decomposes the accumulated backbone weight change using singular value decomposition and uses the squared singular values as directional spectral energies. These energies are treated as empirical weight-space indicators of directions associated with retained performance, rather than as direct evidence of stored knowledge. Principal-direction regularization penalizes overlap with high-energy principal directions during training, whereas dynamic merge applies direction-wise scaling to the learned weight change before merging it into the backbone. On T5-Large, PRISM-LoRA achieved Avg OP scores of 79.4±0.2% on Standard CL and 73.3±0.3% on the 15-task Long Sequence benchmark across three random seeds. The Standard CL result was comparable to CLoRA, whereas the Long Sequence result was 1.6%p higher than CSF. On LLaMA2-7B, PRISM-LoRA achieved Avg OP scores of 80.4 ± 0.1% and 75.9 ± 0.1% on the Standard CL and Long Sequence benchmarks, respectively. Ablation and performance-recovery analyses further showed that both directional protection stages contributed to retention and that a small number of high-energy directions were associated with a substantial portion of the retained performance.
Related
- Reliable Replay through Spatial Coherence in Online Continual Learning
- Fast Weight Attention for Continual Learning
- Generative Replay for Equilibrium Propagation: Toward Edge-Friendly Continual Learning in Hopfield Networks
- ChainLoRA: Geometry-Preserving Task Vector Merging for Continual Learning in LLMs
- A Dynamical Theory of LoRA in Continual Learning
BibTeX
@article{kwon2026prismlora,
title = {{PRISM-LoRA: Principal-Direction-Guided Single-LoRA Merging Framework for Continual Learning of Large Language Models}},
author = {Taehyeong Kwon and Okran Jeong},
journal = {Mathematics},
year = {2026},
month = oct,
doi = {10.3390/math14193616},
url = {https://www.mdpi.com/2227-7390/14/19/3616},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - PRISM-LoRA: Principal-Direction-Guided Single-LoRA Merging Framework for Continual Learning of Large Language Models
AU - Taehyeong Kwon
AU - Okran Jeong
PY - 2026
DA - 2026/10/06
JO - Mathematics
DO - 10.3390/math14193616
UR - https://www.mdpi.com/2227-7390/14/19/3616
AB - Continual learning of large language models requires incorporating new knowledge while limiting catastrophic forgetting; however, many low-rank adaptation (LoRA)-based methods retain task-specific modules or separate learning spaces that grow with the task sequence. We propose PRISM-LoRA, a replay-free framework that repeatedly trains a single LoRA module, merges its update into the backbone, and reinitializes it for the next task. Unlike LoRA-based continual-learning approaches that retain task-specific modules or allocate separate task-specific learning spaces, PRISM-LoRA uses the directional structure of the accumulated backbone weight change to guide both subsequent training and consolidation while reusing a single LoRA module. Before each task, PRISM-LoRA decomposes the accumulated backbone weight change using singular value decomposition and uses the squared singular values as directional spectral energies. These energies are treated as empirical weight-space indicators of directions associated with retained performance, rather than as direct evidence of stored knowledge. Principal-direction regularization penalizes overlap with high-energy principal directions during training, whereas dynamic merge applies direction-wise scaling to the learned weight change before merging it into the backbone. On T5-Large, PRISM-LoRA achieved Avg OP scores of 79.4±0.2% on Standard CL and 73.3±0.3% on the 15-task Long Sequence benchmark across three random seeds. The Standard CL result was comparable to CLoRA, whereas the Long Sequence result was 1.6%p higher than CSF. On LLaMA2-7B, PRISM-LoRA achieved Avg OP scores of 80.4 ± 0.1% and 75.9 ± 0.1% on the Standard CL and Long Sequence benchmarks, respectively. Ablation and performance-recovery analyses further showed that both directional protection stages contributed to retention and that a small number of high-energy directions were associated with a substantial portion of the retained performance.
SN - 2227-7390
ER -