A Dynamical Theory of LoRA in Continual Learning
Abstract
Despite the widespread use of Low-Rank Adaptation (LoRA), little is known about its dynamics in continual learning and the mechanisms by which low-rank updates affect catastrophic forgetting. We provide an asymptotically exact dynamical characterization of LoRA in a solvable two-task teacher-student model. In the high-dimensional online-learning limit, we derive a closed system of ordinary differential equations for a finite set of macroscopic order parameters, yielding exact expressions for the generalization errors throughout both the initial Task 1 learning phase and the subsequent LoRA fine-tuning on Task 2. The theory quantitatively matches finite-dimensional simulations and exposes two characteristic effects of LoRA: low-rank adaptation reduces interference with features learned on the first task, but its initialization slows adaptation to the second task. Building on this mechanistic picture, we analyze a state-dependent masking strategy that freezes hidden units carrying the strongest first-task representations and restricts adaptation to the complementary subspace. This structural partitioning markedly reduces forgetting, while preserving plasticity on the new task. Our framework further clarifies the role of adapter rank: transfer improves only up to the intrinsic dimensionality of the target task and saturates beyond it, while forgetting continues to grow with rank. These results provide a dynamical and geometric account of how low-rank adaptation organizes information across sequential tasks and are qualitatively reproduced on a sequential MNIST benchmark.
Related
- PRISM-LoRA: Principal-Direction-Guided Single-LoRA Merging Framework for Continual Learning of Large Language Models
- Reliable Replay through Spatial Coherence in Online Continual Learning
- Fast Weight Attention for Continual Learning
- Generative Replay for Equilibrium Propagation: Toward Edge-Friendly Continual Learning in Hopfield Networks
- ChainLoRA: Geometry-Preserving Task Vector Merging for Continual Learning in LLMs
BibTeX
@misc{marchetta2026dynamical,
title = {{A Dynamical Theory of LoRA in Continual Learning}},
author = {Théo Marchetta and Filippo Alessandroni and Alessandro Breccia and Alessandro Ingrosso and Federica Gerace},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.39367},
eprint = {2609.39367},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.39367},
}
RIS
TY - UNPB
TI - A Dynamical Theory of LoRA in Continual Learning
AU - Théo Marchetta
AU - Filippo Alessandroni
AU - Alessandro Breccia
AU - Alessandro Ingrosso
AU - Federica Gerace
PY - 2026
DA - 2026/09/30
PB - arXiv
DO - 10.48550/arxiv.2609.39367
UR - https://arxiv.org/abs/2609.39367
AB - Despite the widespread use of Low-Rank Adaptation (LoRA), little is known about its dynamics in continual learning and the mechanisms by which low-rank updates affect catastrophic forgetting. We provide an asymptotically exact dynamical characterization of LoRA in a solvable two-task teacher-student model. In the high-dimensional online-learning limit, we derive a closed system of ordinary differential equations for a finite set of macroscopic order parameters, yielding exact expressions for the generalization errors throughout both the initial Task 1 learning phase and the subsequent LoRA fine-tuning on Task 2. The theory quantitatively matches finite-dimensional simulations and exposes two characteristic effects of LoRA: low-rank adaptation reduces interference with features learned on the first task, but its initialization slows adaptation to the second task. Building on this mechanistic picture, we analyze a state-dependent masking strategy that freezes hidden units carrying the strongest first-task representations and restricts adaptation to the complementary subspace. This structural partitioning markedly reduces forgetting, while preserving plasticity on the new task. Our framework further clarifies the role of adapter rank: transfer improves only up to the intrinsic dimensionality of the target task and saturates beyond it, while forgetting continues to grow with rank. These results provide a dynamical and geometric account of how low-rank adaptation organizes information across sequential tasks and are qualitatively reproduced on a sequential MNIST benchmark.
ER -