CARE-MoE: Conflict-Aware Routing-Guided Expert Mixture-of-Experts for Continual Learning
- 1Department of Computing, Gachon University, Seongnam 13120, Republic of Korea
- 2National Institute of Advanced Industrial Science and Technology (AIST), Tokyo 1350064, Japan
Abstract
Continual learning (CL) aims to learn new tasks sequentially while retaining prior knowledge. Although recent mixture-of-experts (MoE) architectures mitigate catastrophic forgetting by reducing global parameter sharing, they introduce a new bottleneck: expert-level interference. Because multiple tasks inevitably share a subset of experts, these shared parameters are repeatedly exposed to heterogeneous task updates, which may contribute to the degradation of previously encoded knowledge. To tackle this, we propose CARE-MoE (Conflict-Aware Routing-Guided Expert Mixture-of-Experts), a unified framework designed to limit repeated expert sharing, characterize expert utilization, and regulate expert-level updates. First, CARE-MoE aims to limit excessive expert sharing via dual-granularity routing, which leverages both task-level structural and input-level semantic information to distribute representations effectively. Second, rather than using routing merely for expert selection, it diagnoses highly exposed experts via multi-dimensional routing signal aggregation, quantifying expert importance through selection frequency, cross-task overlap, and routing consistency. Finally, conflict-aware expert stabilization translates this diagnosis into targeted optimization, selectively directing preservation updates to the routing parameters associated with heavily shared experts and modulating their strength according to the directional agreement between adaptation and preservation, while allowing task-specific experts to flexibly adapt. Extensive experiments on standard and long-sequence CL benchmarks show that CARE-MoE achieves higher overall performance than the results reported for existing methods. In-depth analyses further indicate that our framework bridges routing behavior with parameter optimization, achieving a favorable balance between stability and plasticity in long-term continual learning.
Related
- When Does Sparse Routing Help? The Geometry of Expert Separation in Mixture of Experts
- MoRE: Scaling mixture of experts with hardware-aware low-rank routing
- MAEM: Mitigating Recency Bias via a Replay-Free, Capacity-Controlled Dynamic Mixture of Experts for Class-Incremental Graph Learning
- Mixture of Censored Partially Linear Experts Models for Symmetric or Asymmetric Data with or without Outlying Observations: A novel approach with two-piece scale-mixture of normal distributions
- The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Experts
BibTeX
@article{lee2026caremoe,
title = {{CARE-MoE: Conflict-Aware Routing-Guided Expert Mixture-of-Experts for Continual Learning}},
author = {SeonGyoung Lee and DaeHo Kim and SoYeop Yoo and Ok-Ran Jeong},
journal = {Mathematics},
year = {2026},
month = sep,
doi = {10.3390/math14193557},
url = {https://www.mdpi.com/2227-7390/14/19/3557},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - CARE-MoE: Conflict-Aware Routing-Guided Expert Mixture-of-Experts for Continual Learning
AU - SeonGyoung Lee
AU - DaeHo Kim
AU - SoYeop Yoo
AU - Ok-Ran Jeong
PY - 2026
DA - 2026/09/30
JO - Mathematics
DO - 10.3390/math14193557
UR - https://www.mdpi.com/2227-7390/14/19/3557
AB - Continual learning (CL) aims to learn new tasks sequentially while retaining prior knowledge. Although recent mixture-of-experts (MoE) architectures mitigate catastrophic forgetting by reducing global parameter sharing, they introduce a new bottleneck: expert-level interference. Because multiple tasks inevitably share a subset of experts, these shared parameters are repeatedly exposed to heterogeneous task updates, which may contribute to the degradation of previously encoded knowledge. To tackle this, we propose CARE-MoE (Conflict-Aware Routing-Guided Expert Mixture-of-Experts), a unified framework designed to limit repeated expert sharing, characterize expert utilization, and regulate expert-level updates. First, CARE-MoE aims to limit excessive expert sharing via dual-granularity routing, which leverages both task-level structural and input-level semantic information to distribute representations effectively. Second, rather than using routing merely for expert selection, it diagnoses highly exposed experts via multi-dimensional routing signal aggregation, quantifying expert importance through selection frequency, cross-task overlap, and routing consistency. Finally, conflict-aware expert stabilization translates this diagnosis into targeted optimization, selectively directing preservation updates to the routing parameters associated with heavily shared experts and modulating their strength according to the directional agreement between adaptation and preservation, while allowing task-specific experts to flexibly adapt. Extensive experiments on standard and long-sequence CL benchmarks show that CARE-MoE achieves higher overall performance than the results reported for existing methods. In-depth analyses further indicate that our framework bridges routing behavior with parameter optimization, achieving a favorable balance between stability and plasticity in long-term continual learning.
SN - 2227-7390
ER -