Pith. sign in

REVIEW 10 cited by

MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.11260 v1 pith:J22BAQJB submitted 2024-02-17 cs.CL cs.AI

Shu Yang , Muhammad Asif Ali , Cheng-Long Wang , Lijie Hu , Di Wang This is my paper · ORCID
classification cs.CLcs.AI
keywords morallearningevaluationllmslifelongmodelssettingsabilities
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Adapting large language models (LLMs) to new domains/tasks and enabling them to be efficient lifelong learners is a pivotal challenge. In this paper, we propose MoRAL, i.e., Mixture-of-Experts augmented Low-Rank Adaptation for Lifelong Learning. MoRAL combines the multi-tasking abilities of MoE with the fine-tuning abilities of LoRA for effective life-long learning of LLMs. In contrast to the conventional approaches that use factual triplets as inputs MoRAL relies on simple question-answer pairs, which is a more practical and effective strategy for robust and efficient learning. Owing to new data settings, we introduce a new evaluation benchmark namely: Life Long Learning of LLM (5L-bench) encompassing a newly curated dataset of question-answer pairs, and a set of evaluation metrics for rigorous evaluation of MoRAL in open-book and closed-book settings. Experimental evaluation shows (i) LLMs learn fast in open-book settings with up to 30.15% improvement in "RA" for Phi-2-2.7B compared to closed-book (for models fine-tuned with MoRAL); (ii) MoRAL shows higher performance improvement for models with a greater number of parameters; (iii) MoRAL is robust to catastrophic forgetting offering better knowledge retention compared to baselines.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Localized LoRA-MoE: Block-wise Low-Rank Experts With Adaptive Routing

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Block-wise and cell-wise LoRA-MoE routers break static LoRA gradient conflicts on multi-context matrix tasks, with cell-level gates matching a global router on uniform shifts and beating it on heterogeneous ones.

  2. LoRA-Gen: Specializing Large Language Model via Online LoRA Generation

    cs.CL 2025-06 conditional novelty 6.0 of 10

    LoRA-Gen generates task-specific LoRA weights with a cloud-side LLM and reparameterizes them into a smaller edge model, enabling training-free specialization with compressed context.

  3. TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A gradient-similarity tree of LoRA adapters, searched by an LCB bandit algorithm, delivers state-of-the-art continual learning accuracy with up to 3.2x faster training on ViTs and 2.4x on LLMs.

  4. Understanding and Mitigating Cross-lingual Privacy Leakage via Language-specific and Universal Privacy Neurons

    cs.CL 2025-06 reject novelty 6.0 of 10

    Cross-lingual privacy leakage in LLMs is driven by a mix of language-universal and language-specific neurons, and deactivating those neurons lowers measured leakage by 23.3% to 31.6%.

  5. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.

  6. Mixture of LoRA Experts with Multi-Modal and Multi-Granularity LLM Generative Error Correction for Accented Speech Recognition

    cs.SD 2025-07 conditional novelty 5.0 of 10

    A three-stage pipeline of mono-accent LoRA experts, hierarchical routing, and phoneme-plus-word LLM error correction cuts accented-English WER from 6.34% to 2.07% on a combined 9-accent test set.

  7. Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuning

    cs.LG 2025-02 conditional novelty 5.0 of 10

    HILO assigns shallow layers fewer and lower-rank LoRA experts and deeper layers more and higher-rank experts, reporting accuracy gains with 37.5 percent fewer trainable or active parameters on Llama 2-7B.

  8. ScatterPrism: convergence for generative simulation and inverse problems in particle and nuclear physics

    cs.LG 2026-04 unverdicted novelty 4.0 of 10

    Standard Conditional Flow Matching loss is a misleading early plateau; physics-informed metrics keep improving, so ScatterPrism and multi-metric diagnostics are needed for kinematic fidelity.

  9. Continual Learning for Generative AI: From LLMs to MLLMs and Beyond

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A survey that categorizes continual learning methods for generative models into architecture-based, regularization-based, and replay-based paradigms across four model families.

  10. Mechanistic Unveiling of Transformer Circuits: Self-Influence as a Key to Model Reasoning

    cs.AI 2025-02 reject novelty 4.0 of 10

    SICAF traces per-token self-influence inside extracted circuits to map GPT-2's reasoning on the IOI task.

Pith tools