Fast Training of Mixture-of-Experts for Time Series Forecasting via Expert Loss Integration
Abstract
We propose a novel Mixture-of-Experts framework, referred to as Expert Loss Integration MoE (EliMoE), for time series forecasting. EliMoE addresses the optimization problem arising from small gating weights by incorporating expert-specific losses, which provide each expert with a direct learning signal independent of the gate-assigned weight. Specifically, the overall objective comprises the base forecasting loss and expert-specific losses, allowing individual expert prediction errors to directly influence parameter updates alongside the aggregate forecasting error. The framework also encourages different experts to learn from different temporal segments of the data. The proposed framework is further combined with a partial online learning strategy that enables efficient incremental updates of model parameters. By integrating expert-level loss information with partial online learning, EliMoE improves forecasting performance while retaining computational efficiency. Empirical results across five benchmark datasets from diverse application domains with different sampling frequencies show that the proposed approach generally outperforms state-of-the-art supervised neural forecasting models, including Transformer-based architectures such as PatchTST, as well as zero-shot time-series foundation models such as TimeMoE. Furthermore, ablation studies confirm the effectiveness of the expert-specific loss integration strategy, highlighting its contribution to enhancing predictive performance.
Related
- ExpertMuon-Compass: Alignment-Guided Step Sizes for Mixture-of-Experts Training
- Mixture of Censored Partially Linear Experts Models for Symmetric or Asymmetric Data with or without Outlying Observations: A novel approach with two-piece scale-mixture of normal distributions
- The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Experts
- Non-asymptotic oracle inequalities for the Lasso in high-dimensional mixture of experts
- A Mixture-of-Experts Framework for Multi-Stream Quickest Change Detection
BibTeX
@misc{mahtout2026fast,
title = {{Fast Training of Mixture-of-Experts for Time Series Forecasting via Expert Loss Integration}},
author = {Btissame El Mahtout and Florian Ziel},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2605.10330},
eprint = {2605.10330},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.10330},
}
RIS
TY - UNPB
TI - Fast Training of Mixture-of-Experts for Time Series Forecasting via Expert Loss Integration
AU - Btissame El Mahtout
AU - Florian Ziel
PY - 2026
DA - 2026/09/26
PB - arXiv
DO - 10.48550/arxiv.2605.10330
UR - https://arxiv.org/abs/2605.10330
AB - We propose a novel Mixture-of-Experts framework, referred to as Expert Loss Integration MoE (EliMoE), for time series forecasting. EliMoE addresses the optimization problem arising from small gating weights by incorporating expert-specific losses, which provide each expert with a direct learning signal independent of the gate-assigned weight. Specifically, the overall objective comprises the base forecasting loss and expert-specific losses, allowing individual expert prediction errors to directly influence parameter updates alongside the aggregate forecasting error. The framework also encourages different experts to learn from different temporal segments of the data. The proposed framework is further combined with a partial online learning strategy that enables efficient incremental updates of model parameters. By integrating expert-level loss information with partial online learning, EliMoE improves forecasting performance while retaining computational efficiency. Empirical results across five benchmark datasets from diverse application domains with different sampling frequencies show that the proposed approach generally outperforms state-of-the-art supervised neural forecasting models, including Transformer-based architectures such as PatchTST, as well as zero-shot time-series foundation models such as TimeMoE. Furthermore, ablation studies confirm the effectiveness of the expert-specific loss integration strategy, highlighting its contribution to enhancing predictive performance.
ER -