REVIEW 7 cited by
MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Adapting large language models (LLMs) to new domains/tasks and enabling them to be efficient lifelong learners is a pivotal challenge. In this paper, we propose MoRAL, i.e., Mixture-of-Experts augmented Low-Rank Adaptation for Lifelong Learning. MoRAL combines the multi-tasking abilities of MoE with the fine-tuning abilities of LoRA for effective life-long learning of LLMs. In contrast to the conventional approaches that use factual triplets as inputs MoRAL relies on simple question-answer pairs, which is a more practical and effective strategy for robust and efficient learning. Owing to new data settings, we introduce a new evaluation benchmark namely: Life Long Learning of LLM (5L-bench) encompassing a newly curated dataset of question-answer pairs, and a set of evaluation metrics for rigorous evaluation of MoRAL in open-book and closed-book settings. Experimental evaluation shows (i) LLMs learn fast in open-book settings with up to 30.15% improvement in "RA" for Phi-2-2.7B compared to closed-book (for models fine-tuned with MoRAL); (ii) MoRAL shows higher performance improvement for models with a greater number of parameters; (iii) MoRAL is robust to catastrophic forgetting offering better knowledge retention compared to baselines.
Forward citations
Cited by 7 Pith papers
-
Localized LoRA-MoE: Block-wise Low-Rank Experts With Adaptive Routing
Block-wise and cell-wise LoRA-MoE routers break static LoRA gradient conflicts on multi-context matrix tasks, with cell-level gates matching a global router on uniform shifts and beating it on heterogeneous ones.
-
LoRA-Gen: Specializing Large Language Model via Online LoRA Generation
LoRA-Gen generates task-specific LoRA weights with a cloud-side LLM and reparameterizes them into a smaller edge model, enabling training-free specialization with compressed context.
-
TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree
A gradient-similarity tree of LoRA adapters, searched by an LCB bandit algorithm, delivers state-of-the-art continual learning accuracy with up to 3.2x faster training on ViTs and 2.4x on LLMs.
-
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.
-
Mixture of LoRA Experts with Multi-Modal and Multi-Granularity LLM Generative Error Correction for Accented Speech Recognition
A three-stage pipeline of mono-accent LoRA experts, hierarchical routing, and phoneme-plus-word LLM error correction cuts accented-English WER from 6.34% to 2.07% on a combined 9-accent test set.
-
ScatterPrism: convergence for generative simulation and inverse problems in particle and nuclear physics
Standard Conditional Flow Matching loss is a misleading early plateau; physics-informed metrics keep improving, so ScatterPrism and multi-metric diagnostics are needed for kinematic fidelity.
-
Continual Learning for Generative AI: From LLMs to MLLMs and Beyond
A survey that categorizes continual learning methods for generative models into architecture-based, regularization-based, and replay-based paradigms across four model families.
Discussion (0). Sign in to comment.