REVIEW 11 cited by
Theory on Mixture-of-Experts in Continual Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Theory on Mixture-of-Experts in Continual Learning
read the original abstract
Continual learning (CL) has garnered significant attention because of its ability to adapt to new tasks that arrive over time. Catastrophic forgetting (of old tasks) has been identified as a major issue in CL, as the model adapts to new tasks. The Mixture-of-Experts (MoE) model has recently been shown to effectively mitigate catastrophic forgetting in CL, by employing a gating network to sparsify and distribute diverse tasks among multiple experts. However, there is a lack of theoretical analysis of MoE and its impact on the learning performance in CL. This paper provides the first theoretical results to characterize the impact of MoE in CL via the lens of overparameterized linear regression tasks. We establish the benefit of MoE over a single expert by proving that the MoE model can diversify its experts to specialize in different tasks, while its router learns to select the right expert for each task and balance the loads across all experts. Our study further suggests an intriguing fact that the MoE in CL needs to terminate the update of the gating network after sufficient training rounds to attain system convergence, which is not needed in the existing MoE studies that do not consider the continual task arrival. Furthermore, we provide explicit expressions for the expected forgetting and overall generalization error to characterize the benefit of MoE in the learning performance in CL. Interestingly, adding more experts requires additional rounds before convergence, which may not enhance the learning performance. Finally, we conduct experiments on both synthetic and real datasets to extend these insights from linear models to deep neural networks (DNNs), which also shed light on the practical algorithm design for MoE in CL.
Forward citations
Cited by 11 Pith papers
-
STM3: Mixture of Multiscale Mamba for Long-Term Spatio-Temporal Time-Series Prediction
STM3 is a spatio-temporal mixture-of-multiscale-Mamba model that uses disentangled experts, stable routing, and causal contrastive learning to achieve state-of-the-art long-term forecasting on ten real-world benchmarks.
-
MoRe: Modular Representations for Principled Continual Representation Learning on Sequential Data
MoRe identifies modular structure in representations themselves to enable principled reuse, alignment, and expansion of modules during continual adaptation on sequential data.
-
PLATE: Plasticity-Tunable Efficient Adapters for Geometry-Aware Continual Learning
PLATE constructs frozen weight-derived bases B and Q and trains only a small core A, reducing catastrophic forgetting during data-free continual fine-tuning.
-
Continual Video-MLLM Adaptation over Evolving Domains
A frozen Video-MLLM plus per-domain LoRA expert pools with distribution-aware routing improves domain-incremental video QA from 64.38% to 67.59% average accuracy over the strongest prior method.
-
Multi-scale Mixture of World Models for Embodied Agents in Evolving Environments
MuSix introduces scale-aware world model mixtures with experiential-distance routing and adaptive forgetting to improve multi-scale reasoning and dynamic adaptation in embodied agents.
-
MoRe: Modular Representations for Principled Continual Representation Learning on Sequential Data
MoRe decomposes representations into identifiable hierarchical modules to enable principled continual adaptation on sequential data.
-
MoRe: Modular Representations for Principled Continual Representation Learning on Sequential Data
MoRe identifies modular representations in sequential data for continual learning with identifiability guarantees, enabling principled adaptation without disrupting old modules.
-
FLAME: Adaptive Mixture-of-Experts for Continual Multimodal Multi-Task Learning
FLAME is an MoE architecture using modality-specific routers and low-rank compression of expert knowledge to support efficient continual multimodal multi-task learning while reducing catastrophic forgetting.
-
Model Predictive Control and Moving Horizon Estimation using Statistically Weighted Data-Based Ensemble Models
Ensemble MPC with Mahalanobis-distance weighting that adapts across the horizon and a matching MHE observer, shown on a multi-condition energy benchmark.
-
Seeing Clearly, Forgetting Deeply: Revisiting Fine-Tuned Video Generators for Driving Simulation
Fine-tuning video generators on driving data can improve visual fidelity while degrading how accurately the model predicts the movement of cars and pedestrians.
-
STM3: Mixture of Multiscale Mamba for Long-Term Spatio-Temporal Time-Series Prediction
STM3 is a new multiscale Mamba mixture-of-experts model with graph causal networks and contrastive routing that reports state-of-the-art results on 10 long-term spatio-temporal forecasting benchmarks.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.