REVIEW 21 cited by
LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin
read the original abstract
Supervised fine-tuning (SFT) is a crucial step for large language models (LLMs), enabling them to align with human instructions and enhance their capabilities in downstream tasks. Increasing instruction data substantially is a direct solution to align the model with a broader range of downstream tasks or notably improve its performance on a specific task. However, we find that large-scale increases in instruction data can damage the world knowledge previously stored in LLMs. To address this challenge, we propose LoRAMoE, a novelty framework that introduces several low-rank adapters (LoRA) and integrates them by using a router network, like a plugin version of Mixture of Experts (MoE). It freezes the backbone model and forces a portion of LoRAs to focus on leveraging world knowledge to solve downstream tasks, to alleviate world knowledge-edge forgetting. Experimental results show that, as the instruction data increases, LoRAMoE can significantly improve the ability to process downstream tasks, while maintaining the world knowledge stored in the LLM.
Forward citations
Cited by 21 Pith papers
-
Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning
DRAPE generates query-image conditioned prompts on the fly for multimodal continual instruction tuning and reports SOTA results on MCIT benchmarks.
-
Beyond GSD-as-Token: Continuous Scale Conditioning for Remote Sensing VLMs
ScaleEarth conditions remote sensing VLMs on continuous GSD via CS-HLoRA and a visual GSD predictor, creating a closed training loop with GeoScale-VQA to achieve SOTA on Earth observation benchmarks.
-
Adaptive and Fine-grained Module-wise Expert Pruning for Efficient LoRA-MoE Fine-Tuning
DMEP prunes experts module-by-module in LoRA-MoE and removes load balancing after pruning, cutting trainable parameters 35-43% and raising throughput ~10% while matching or exceeding uniform baselines on reasoning tasks.
-
Orthogonal Subspace Projection for Continual Machine Unlearning via SVD-Based LoRA
SVD-guided orthogonal subspace projection in LoRA enables stable performance across 30 sequential unlearning tasks on CIFAR-100 and MNIST, unlike naive static fusion which collapses retained accuracy.
-
Progressive Multimodal Alignment for Continual Instruction Tuning
Adding distribution-shift-triggered projector experts, mixed by a router and anchored by the frozen pretrained projector, improves multimodal continual instruction tuning accuracy and reduces forgetting.
-
MoE$^2$-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
Routing LoRA adapters with the frozen base router's logits plus a shared cross-layer adapter pool gives the best PEFT accuracy and retention on three MoE backbones.
-
Online Data Selection Is Implicit Alignment
Online SFT data selection acts as an implicit preference model, shifting refusal rates, verbosity, and sycophancy in directions predictable from the selected data's attribute mixture.
-
Localized LoRA-MoE: Block-wise Low-Rank Experts With Adaptive Routing
Block-wise and cell-wise LoRA-MoE routers break static LoRA gradient conflicts on multi-context matrix tasks, with cell-level gates matching a global router on uniform shifts and beating it on heterogeneous ones.
-
Behavioral and Representational Evidence of Binomial Ordering Preferences in Large Language Models
LLMs recover dominant binomial orders from corpora but align less closely with exact preference distributions, with preference strength partially encoded in middle-to-late layers and manipulable via steering.
-
STAR: Rethinking MoE Routing as Structure-Aware Subspace Learning
STAR rethinks MoE routing as structure-aware subspace learning by adding a GHA-tracked principal subspace to standard routers, yielding more stable specialization and better performance on synthetic, language, and vis...
-
SMILE-Next: Teaching Large Language Models to Detect, Classify, and Reason about Laughter
SMILE-Next dataset and MoLE framework allow LLMs to handle laughter detection, type classification, and reasoning tasks, with reported gains over multimodal baselines.
-
MP-ISMoE: Mixed-Precision Interactive Side Mixture-of-Experts for Efficient Transfer Learning
MP-ISMoE uses Gaussian noise perturbed iterative quantization and interactive side mixture-of-experts to deliver higher accuracy than prior memory-efficient transfer learning methods while keeping similar parameter an...
-
CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging
CoMoL represents every LoRA expert as a shared-basis core matrix and merges token-selected experts in that core space, reaching standard LoRA parameter counts while outperforming MoE-LoRA baselines on math and code.
-
Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Experts
MoRAM frames continual learning as incremental addition of rank-1 adapters viewed as self-activating key-value associative memory units in a mixture-of-experts setup.
-
LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing
LoRA-Mixer routes modular LoRA experts into attention projection matrices with an adaptive Routing Specialization Loss to improve multi-task performance while using fewer trainable parameters than prior LoRA-MoE methods.
-
Progressive Multimodal Alignment for Continual Instruction Tuning
Progressive Multimodal Alignment expands projector experts only when multimodal distribution shifts are detected, reducing projector-level forgetting and boosting MCIT baselines with sub-linear growth.
-
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.
-
Unifying Search and Recommendation in LLMs via Gradient Multi-Subspace Tuning
GEMS proposes a multi-subspace gradient-tuning framework for unified search and recommendation in LLMs, but its null-space projection equation is inverted and its adaptive gate has no training rule.
-
Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompression
Introduces Tree Generation (TG-SFT) to generate synthetic instruction-tuning data from LLMs, reducing catastrophic forgetting when fine-tuning MLLMs on domain-specific or multimodal data.
-
ScatterPrism: convergence for generative simulation and inverse problems in particle and nuclear physics
CFM training loss plateaus while physics-informed metrics continue improving; ScatterPrism and a multi-metric protocol are proposed to restore kinematic fidelity without memorization.
-
ScatterPrism: convergence for generative simulation and inverse problems in particle and nuclear physics
Standard Conditional Flow Matching loss is a misleading early plateau; physics-informed metrics keep improving, so ScatterPrism and multi-metric diagnostics are needed for kinematic fidelity.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.