Pith. sign in

REVIEW 18 cited by

MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.12851 v1 pith:PF25UWJ5 submitted 2024-02-20 cs.CL

Tongxu Luo , Jiahe Lei , Fangyu Lei , Weihao Liu , Shizhu He , Jun Zhao , Kang Liu This is my paper · ORCID
classification cs.CL
keywords loraexpertsfine-tuningmodelsmoeloraparameterspeftreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Fine-tuning is often necessary to enhance the adaptability of Large Language Models (LLM) to downstream tasks. Nonetheless, the process of updating billions of parameters demands significant computational resources and training time, which poses a substantial obstacle to the widespread application of large-scale models in various scenarios. To address this issue, Parameter-Efficient Fine-Tuning (PEFT) has emerged as a prominent paradigm in recent research. However, current PEFT approaches that employ a limited set of global parameters (such as LoRA, which adds low-rank approximation matrices to all weights) face challenges in flexibly combining different computational modules in downstream tasks. In this work, we introduce a novel PEFT method: MoELoRA. We consider LoRA as Mixture of Experts (MoE), and to mitigate the random routing phenomenon observed in MoE, we propose the utilization of contrastive learning to encourage experts to learn distinct features. We conducted experiments on 11 tasks in math reasoning and common-sense reasoning benchmarks. With the same number of parameters, our approach outperforms LoRA significantly. In math reasoning, MoELoRA achieved an average performance that was 4.2% higher than LoRA, and demonstrated competitive performance compared to the 175B GPT-3.5 on several benchmarks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Six-Dimensional Taxonomy of Post-Training Adaptation Techniques with Applications in AI Governance

    cs.LG 2026-08 conditional novelty 6.0 of 10

    A new taxonomy characterizes 48 post-training AI adaptation techniques on six axes and maps them to regulatory documentation requirements.

  2. MoE$^2$-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Routing LoRA adapters with the frozen base router's logits plus a shared cross-layer adapter pool gives the best PEFT accuracy and retention on three MoE backbones.

  3. RoCo-ACE: Rollout-Conditioned Online Distillation for Retention-Aware Knowledge Injection

    cs.AI 2026-06 conditional novelty 6.0 of 10

    A rollout-conditioned contrastive distillation loss plus sparse anchored cross-entropy improves injected-knowledge accuracy in MLLMs while keeping retention close to the base model.

  4. CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging

    cs.CL 2026-02 conditional novelty 6.0 of 10

    CoMoL represents every LoRA expert as a shared-basis core matrix and merges token-selected experts in that core space, reaching standard LoRA parameter counts while outperforming MoE-LoRA baselines on math and code.

  5. Reasoning and Tool-use Compete in Agentic RL:From Quantifying Interference to Disentangled Tuning

    cs.AI 2026-02 conditional novelty 6.0 of 10

    Jointly optimizing reasoning and tool use in agentic RL interferes with both; separating the two into disjoint LoRA adapters (DART) improves retrieval-augmented QA.

  6. Memorization and Knowledge Injection in Gated LLMs

    cs.CL 2025-04 conditional novelty 6.0 of 10

    MEGa injects episodic memories into separate gated LoRA adapters selected by embedding similarity, mitigating catastrophic forgetting and enabling recall, QA, and compositional questions on two datasets.

  7. Onsager Principle-Based Domain Embedding for Thermodynamically Consistent Cahn-Hilliard Model in Arbitrary Domain

    math.NA 2025-08 unverdicted novelty 5.0 of 10

    As submitted, the paper is unverifiable: the abstract claims a thermodynamically consistent Cahn-Hilliard domain-embedding derivation, while the full text is a different paper on person re-identification.

  8. Sci-LoRA: Mixture of Scientific LoRAs for Cross-Domain Lay Paraphrasing

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Sci-LoRA dynamically mixes domain-specific LoRA adapters and achieves state-of-the-art lay paraphrasing across twelve domains without needing domain labels at inference.

  9. HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation

    cs.CV 2025-02 reject novelty 5.0 of 10

    HealthGPT unifies medical image comprehension and generation in a single autoregressive model using heterogeneous low-rank adaptation, reporting strong benchmark results.

  10. Ensembles of Low-Rank Expert Adapters

    cs.CL 2025-01 conditional novelty 5.0 of 10

    ELREA clusters instruction-tuning data by gradient direction, trains one LoRA expert per cluster, and routes new instructions to experts via gradient similarity, giving modest benchmark gains over full-data LoRA.

  11. MoSLD: An Extremely Parameter-Efficient Mixture-of-Shared LoRAs for Multi-Task Learning

    cs.LG 2024-12 conditional novelty 5.0 of 10

    A new LoRA-MoE variant that shares the up-projection matrix across experts and adds dropout on it improves multi-task fine-tuning accuracy on six commonsense reasoning datasets.

  12. PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Model

    cs.LG 2024-11 conditional novelty 5.0 of 10

    PERFT inserts routed, parameter-efficient LoRA experts into MoE layers and reports consistent gains over attention-only LoRA on reasoning benchmarks.

  13. ScatterPrism: convergence for generative simulation and inverse problems in particle and nuclear physics

    cs.LG 2026-04 unverdicted novelty 4.0 of 10

    Standard Conditional Flow Matching loss is a misleading early plateau; physics-informed metrics keep improving, so ScatterPrism and multi-metric diagnostics are needed for kinematic fidelity.

  14. MoE-MLoRA for Multi-Domain CTR Prediction: Efficient Adaptation with Expert Specialization

    cs.IR 2025-06 conditional novelty 4.0 of 10

    A three-stage MoE variant of MLoRA improves multi-domain CTR prediction on sparse Taobao data but not on denser Movielens splits.

  15. Why Do More Experts Fail? A Theoretical Analysis of Model Merging

    cs.LG 2025-05 reject novelty 4.0 of 10

    The paper claims to prove an upper bound and diminishing returns in model merging, but the proofs are not sound and the heavy-tailed claim is contradicted by its own equations.

  16. VELoRA: A Low-Rank Adaptation Approach for Efficient RGB-Event based Recognition

    cs.CV 2024-12 conditional novelty 4.0 of 10

    VELoRA applies modality-specific and modality-shared LoRA adapters to a frozen CLIP ViT for RGB-event recognition, reaching 57.99% on PokerEvent and 50.89% on HARDVS.

  17. Advancing Single and Multi-task Text Classification through Large Language Model Fine-tuning

    cs.CL 2024-12 conditional novelty 4.0 of 10

    Fully fine-tuned Llama3-70B beats RoBERTa-large and other LLMs on 20 Newsgroups and MASSIVE, and a single consolidated multi-task LLM matches separated dual models.

  18. A Survey on Inference Optimization Techniques for Mixture of Experts Models

    cs.LG 2024-12 conditional novelty 3.0 of 10

    A structured survey of MoE inference optimization that categorizes existing techniques into model, system, and hardware levels and summarizes reported speedups and memory savings.

Pith tools