Effect of Spherical Parameter Space on Number of Non-Forgetting Tasks and Capacity of Continual Learning
Abstract
We propose a continuous learning formulation for T non-forgetting tasks with the spherical parameter space 𝕊d−1(√d) in asymptotic regime d→ ∞. Each task i has an optimum Θi, minimizing its corresponding loss function Ri(w) on sphere. We reduce each task i to an acceptable loss region modeled by a geodesic ball BΘi(ri(d)) on 𝕊d−1(√d) centered at Θi with radius ri(d). To model task dependencies (similarities), task optima Θi are distributed by a Neyman-Scott clustered point process Φ. Cluster centers {Yj}j=1Np form a homogeneous Poisson point process on sphere, while intra-cluster task optima follow a von Mises-Fisher distribution with mean Yj and concentration κ. Each task t∈ {1, ..., T} learning algorithm is considered to be a channel Qt(dwt | wt−1, 𝒦t−1(θ1:t−1), θt), where Wt is the learned parameter of task t and 𝒦t−1(Θ1:t−1)=⋂i=1t−1BΘi(ri(d)). Since in continual learning, the learner does not need to reach Θt exactly to successfully learn task t, to avoid forgetting as well as reliable learning, it only needs for the learned parameter vector Wt to enter BΘt(rt(d)) ∩ 𝒦t−1(Θ1:t−1). Using this model, we derive an upper bound on T ensuring ℙ[𝒦T(Θ1:T) ≠ ∅] → 1 as d→ ∞ and κ/d → α, where α>0 is a constant. Finally, for T tasks sampled uniformly from Φ, we derive an upper bound for total information theoretic capacity under the non-forgetting constraint for each task.
Related
- Forgetting matters: when subspace optimization accelerates without the strong convexity parameter
- Learning Style, Forgetting Semantics: A Case Study of SFT and RFT on Classification Tasks
- Reliable Replay through Spatial Coherence in Online Continual Learning
- Fast Weight Attention for Continual Learning
- Generative Replay for Equilibrium Propagation: Toward Edge-Friendly Continual Learning in Hopfield Networks
BibTeX
@misc{mirhosseini2026effect,
title = {{Effect of Spherical Parameter Space on Number of Non-Forgetting Tasks and Capacity of Continual Learning}},
author = {Nazanin Mirhosseini},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.17453},
eprint = {2610.17453},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.17453},
}
RIS
TY - UNPB
TI - Effect of Spherical Parameter Space on Number of Non-Forgetting Tasks and Capacity of Continual Learning
AU - Nazanin Mirhosseini
PY - 2026
DA - 2026/10/12
PB - arXiv
DO - 10.48550/arxiv.2610.17453
UR - https://arxiv.org/abs/2610.17453
AB - We propose a continuous learning formulation for T non-forgetting tasks with the spherical parameter space 𝕊^d−1(√d) in asymptotic regime d→ ∞. Each task i has an optimum Θ_i, minimizing its corresponding loss function R_i(w) on sphere. We reduce each task i to an acceptable loss region modeled by a geodesic ball B_Θ_i(r_i(d)) on 𝕊^d−1(√d) centered at Θ_i with radius r_i(d). To model task dependencies (similarities), task optima Θ_i are distributed by a Neyman-Scott clustered point process Φ. Cluster centers {Y_j}_j=1^N_p form a homogeneous Poisson point process on sphere, while intra-cluster task optima follow a von Mises-Fisher distribution with mean Y_j and concentration κ. Each task t∈ {1, ..., T} learning algorithm is considered to be a channel Q_t(dw_t | w_t−1, 𝒦_t−1(θ_1:t−1), θ_t), where W_t is the learned parameter of task t and 𝒦_t−1(Θ_1:t−1)=⋂_i=1^t−1B_Θ_i(r_i(d)). Since in continual learning, the learner does not need to reach Θ_t exactly to successfully learn task t, to avoid forgetting as well as reliable learning, it only needs for the learned parameter vector W_t to enter B_Θ_t(r_t(d)) ∩ 𝒦_t−1(Θ_1:t−1). Using this model, we derive an upper bound on T ensuring ℙ[𝒦_T(Θ_1:T) ≠ ∅] → 1 as d→ ∞ and κ/d → α, where α>0 is a constant. Finally, for T tasks sampled uniformly from Φ, we derive an upper bound for total information theoretic capacity under the non-forgetting constraint for each task.
ER -