REVIEW 20 cited by
Higher Layers Need More LoRA Experts
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Parameter-efficient tuning (PEFT) techniques like low-rank adaptation (LoRA) offer training efficiency on Large Language Models, but their impact on model performance remains limited. Recent efforts integrate LoRA and Mixture-of-Experts (MoE) to improve the performance of PEFT methods. Despite promising results, research on improving the efficiency of LoRA with MoE is still in its early stages. Recent studies have shown that experts in the MoE architecture have different strengths and also exhibit some redundancy. Does this statement also apply to parameter-efficient MoE? In this paper, we introduce a novel parameter-efficient MoE method, \textit{\textbf{M}oE-L\textbf{o}RA with \textbf{L}ayer-wise Expert \textbf{A}llocation (MoLA)} for Transformer-based models, where each model layer has the flexibility to employ a varying number of LoRA experts. We investigate several architectures with varying layer-wise expert configurations. Experiments on six well-known NLP and commonsense QA benchmarks demonstrate that MoLA achieves equal or superior performance compared to all baselines. We find that allocating more LoRA experts to higher layers further enhances the effectiveness of models with a certain number of experts in total. With much fewer parameters, this allocation strategy outperforms the setting with the same number of experts in every layer. This work can be widely used as a plug-and-play parameter-efficient tuning approach for various applications. The code is available at https://github.com/GCYZSL/MoLA.
Forward citations
Cited by 20 Pith papers
-
CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging
CoMoL represents every LoRA expert as a shared-basis core matrix and merges token-selected experts in that core space, reaching standard LoRA parameter counts while outperforming MoE-LoRA baselines on math and code.
-
FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning
A federated LoRA fine-tuning method that builds a client-similarity tree and adapts aggregation depth layer-by-layer outperforms flat or global aggregation baselines on NLU and NLG tasks.
-
Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
D-MoLE dynamically allocates LoRA experts to the most gradient-sensitive layers of a vision-language model for each new task, and routes inputs via per-task autoencoders, reporting near-zero forgetting on a nine-task ...
-
LoRA-Gen: Specializing Large Language Model via Online LoRA Generation
LoRA-Gen generates task-specific LoRA weights with a cloud-side LLM and reparameterizes them into a smaller edge model, enabling training-free specialization with compressed context.
-
Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation
Gradually increasing the probability that LoRA adapters stay active during fine-tuning improves generalization, merging, and pruning robustness.
-
Multi-objective Large Language Model Alignment with Hierarchical Experts
HoE claims to align a single LLM to any preference vector over multiple objectives using training-free LoRA experts, lightweight trained routers, and nearest-neighbor preference routing.
-
Learning without Isolation: Pathway Protection for Continual Learning
LwI fuses old and new models with graph matching, matching similar channels in shallow layers and dissimilar channels in deep layers, to reduce catastrophic forgetting without storing old data.
-
Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocation
Fed-HeLLo allocates different LoRA layers to clients of different resource levels using importance scores and geometric patterns, improving federated fine-tuning accuracy over random allocation baselines.
-
Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts
A layer-wise expert allocation algorithm based on hidden-state similarity, plus a routing classifier, improves parameter efficiency and reduces forgetting when expanding LLMs to new languages.
-
Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuning
HILO assigns shallow layers fewer and lower-rank LoRA experts and deeper layers more and higher-rank experts, reporting accuracy gains with 37.5 percent fewer trainable or active parameters on Llama 2-7B.
-
Ensembles of Low-Rank Expert Adapters
ELREA clusters instruction-tuning data by gradient direction, trains one LoRA expert per cluster, and routes new instructions to experts via gradient similarity, giving modest benchmark gains over full-data LoRA.
-
MoSLD: An Extremely Parameter-Efficient Mixture-of-Shared LoRAs for Multi-Task Learning
A new LoRA-MoE variant that shares the up-projection matrix across experts and adds dropout on it improves multi-task fine-tuning accuracy on six commonsense reasoning datasets.
-
Planning vs Reasoning: Ablations to Test Capabilities of LoRA layers
On GPT-2 hash tasks, LoRA layers improved a synthetic reasoning task more than a planning task, and the authors interpret this as evidence that reasoning is inherently low rank.
-
PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Model
PERFT inserts routed, parameter-efficient LoRA experts into MoE layers and reports consistent gains over attention-only LoRA on reasoning benchmarks.
-
Curvature-Weighted Capacity Allocation: A Minimum Description Length Framework for Layer-Adaptive Large Language Model Optimization
A curvature-based layer gain is converted into optimal allocation and pruning via water-filling and convex penalties, but the experiments substitute the baseline's influence scores and show only modest, mixed gains.
-
CoLA: Collaborative Low-Rank Adaptation
CoLA generalizes LoRA to multiple A and B matrices with a principal-component initialization and reports gains of roughly 2-4 accuracy points over PiSSA on low-sample fine-tuning benchmarks.
-
Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devices
Assigning federated fine-tuning devices different numbers of LoRA layers near the output, with ranks increasing toward the output, reaches target accuracy 1.5-2.8x faster and with up to 42.3% less communication than e...
-
Advancing Single and Multi-task Text Classification through Large Language Model Fine-tuning
Fully fine-tuned Llama3-70B beats RoBERTa-large and other LLMs on 20 Newsgroups and MASSIVE, and a single consolidated multi-task LLM matches separated dual models.
-
DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Models
DP-FedLoRA clips and adds Gaussian noise to per-client LoRA matrices in federated LLM fine-tuning, claiming unbiased updates and bounded variance, but the privacy calibration and experiments have significant gaps.
-
A Survey on Inference Optimization Techniques for Mixture of Experts Models
A structured survey of MoE inference optimization that categorizes existing techniques into model, system, and hardware levels and summarizes reported speedups and memory savings.
Discussion (0). Continue with ORCID to comment.