REVIEW 4 cited by
PMoE: Progressive Mixture of Experts with Asymmetric Transformer for Continual Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large Language Models (LLMs) encounter significant challenges in continual learning due to catastrophic forgetting, where new information overwrites previously acquired knowledge. This limitation leads to substantial environmental and economic waste. In this study, we introduce the PMoE, Progressive Mixture of Experts with Asymmetric Transformer, which aims to minimize forgetting by utilizing an asymmetric design with shallow layers dedicated to general knowledge and deep layers for new knowledge. PMoE incorporates progressively added experts in deep layers and a router that allocates new knowledge to the appropriate experts efficiently. The router, positioned adjacent to the deep layers, utilizes deep features aggregating consolidated information. This enables the router to perform efficiently, allocating new knowledge to the appropriate experts, which progressively increase in the deep layers. Extensive experiments on TRACE datasets and general language understanding datasets demonstrate that the proposed PMoE outperforms previous state-of-the-art approaches.
Forward citations
Cited by 4 Pith papers
-
RoME: Robust Mixture of Low-Rank Experts against Multiple Adversarial Perturbations
Low-rank additive experts with dual-scale gating and threat-guided diversification improve multi-perturbation adversarial robustness by routing different threat types through distinct model pathways.
-
Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
D-MoLE dynamically allocates LoRA experts to the most gradient-sensitive layers of a vision-language model for each new task, and routes inputs via per-task autoencoders, reporting near-zero forgetting on a nine-task ...
-
Mixture-of-Experts for Personalized and Semantic-Aware Next Location Prediction
NextLocMoE combines a location-semantics expert mixer and a persona-based expert mixer inside a trimmed LLaMA backbone to set new accuracy marks on next-location prediction benchmarks.
-
Memorization and Knowledge Injection in Gated LLMs
MEGa injects episodic memories into separate gated LoRA adapters selected by embedding similarity, mitigating catastrophic forgetting and enabling recall, QA, and compositional questions on two datasets.
Discussion (0). Continue with ORCID to comment.