Pith. sign in

REVIEW 21 cited by

MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.15159 v3 pith:QOJJAV5V submitted 2024-04-22 cs.CL cs.AI

MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts

Dengchun Li , Yingzi Ma , Naizheng Wang , Zhengmao Ye , Zhiyuan Cheng , Yinghao Tang , Yan Zhang , Lei Duan
show 3 more authors
Jie Zuo Cal Yang Mingjie Tang
This is my paper
classification cs.CL cs.AI
keywords mixloramodelsmemoryduringfine-tuningloralora-basedmethods
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Fine-tuning Large Language Models (LLMs) is a common practice to adapt pre-trained models for specific applications. While methods like LoRA have effectively addressed GPU memory constraints during fine-tuning, their performance often falls short, especially in multi-task scenarios. In contrast, Mixture-of-Expert (MoE) models, such as Mixtral 8x7B, demonstrate remarkable performance in multi-task learning scenarios while maintaining a reduced parameter count. However, the resource requirements of these MoEs remain challenging, particularly for consumer-grade GPUs with less than 24GB memory. To tackle these challenges, we propose MixLoRA, an approach to construct a resource-efficient sparse MoE model based on LoRA. MixLoRA inserts multiple LoRA-based experts within the feed-forward network block of a frozen pre-trained dense model and employs a commonly used top-k router. Unlike other LoRA-based MoE methods, MixLoRA enhances model performance by utilizing independent attention-layer LoRA adapters. Additionally, an auxiliary load balance loss is employed to address the imbalance problem of the router. Our evaluations show that MixLoRA improves about 9% accuracy compared to state-of-the-art PEFT methods in multi-task learning scenarios. We also propose a new high-throughput framework to alleviate the computation and memory bottlenecks during the training and inference of MOE models. This framework reduces GPU memory consumption by 40% and token computation latency by 30% during both training and inference.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 21 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Predicting Mergeability of Parameter-Efficient Fine-Tuning Updates

    cs.LG 2026-06 unverdicted novelty 7.0

    MergeProbe forecasts LoRA adapter mergeability from first-few-percent training signals and outperforms interference-aware baselines on retention while adding low overhead on a five-domain benchmark.

  2. Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation

    cs.CL 2026-05 unverdicted novelty 7.0

    MoLF routes updates between full fine-tuning and LoRA at the optimizer level to match or exceed the better of either static method, with an efficient LoRA-only variant outperforming prior adaptive approaches.

  3. Adaptive and Fine-grained Module-wise Expert Pruning for Efficient LoRA-MoE Fine-Tuning

    cs.LG 2026-04 unverdicted novelty 7.0

    DMEP prunes experts module-by-module in LoRA-MoE and removes load balancing after pruning, cutting trainable parameters 35-43% and raising throughput ~10% while matching or exceeding uniform baselines on reasoning tasks.

  4. Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMs

    cs.LG 2024-10 unverdicted novelty 7.0

    UQ4CT integrates functional-level uncertainty calibration into mixture-of-experts LoRA fine-tuning via a dedicated loss, cutting expected calibration error by over 25% on multiple-choice and generative QA tasks.

  5. Online Data Selection Is Implicit Alignment

    cs.LG 2026-07 conditional novelty 6.0

    Online SFT data selection acts as an implicit preference model, shifting refusal rates, verbosity, and sycophancy in directions predictable from the selected data's attribute mixture.

  6. Clinically Structured Rank-Gated LoRA for Cross-Benchmark Medical Question Answering

    cs.CL 2026-06 unverdicted novelty 6.0

    BiRG-LoRA reaches 69.31% macro-average accuracy across CMB, CMExam, MedQA and MedMCQA, outperforming MoELoRA by 0.89 points with 28.1% fewer parameters under a matched single-seed protocol.

  7. Clinically Structured Rank-Gated LoRA for Cross-Benchmark Medical Question Answering

    cs.CL 2026-06 unverdicted novelty 6.0

    BiRG-LoRA achieves 69.31% macro-average accuracy across CMB, CMExam, MedQA and MedMCQA using a rank-gated LoRA with biaxial clinical gating, outperforming MoELoRA by 0.89 points with 28.1% fewer parameters.

  8. Mixture of Debaters: Learn to Debate at Architectural Level in Multi-Agent Reasoning

    cs.AI 2026-06 unverdicted novelty 6.0

    Mixture of Debaters uses MoE to enable dynamic self-debate inside one model, claiming better accuracy than multi-agent systems at 3.7x lower latency and 87% fewer tokens on multimodal benchmarks.

  9. Behavioral and Representational Evidence of Binomial Ordering Preferences in Large Language Models

    cs.CL 2026-06 unverdicted novelty 6.0

    LLMs recover dominant binomial orders from corpora but align less closely with exact preference distributions, with preference strength partially encoded in middle-to-late layers and manipulable via steering.

  10. Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories

    cs.LG 2026-06 unverdicted novelty 6.0

    Language models can use a two-stage sleep process of upward distillation for memory consolidation and RL-based dreaming for unsupervised self-improvement to enable continual learning.

  11. Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation

    cs.CL 2026-05 unverdicted novelty 6.0

    MoLF routes updates between full fine-tuning and LoRA at the optimizer level to match or exceed the better of the two static methods on SQL, medical QA, and counterfactual tasks while an efficient variant outperforms ...

  12. ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning

    cs.CL 2026-04 unverdicted novelty 6.0

    ShadowPEFT replaces distributed low-rank weight perturbations with a centralized, depth-shared shadow module that evolves parallel hidden states layer by layer, matching or beating LoRA and DoRA on generation and unde...

  13. TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Models

    cs.LG 2026-04 unverdicted novelty 6.0

    TalkLoRA equips MoE-LoRA experts with a communication module that smooths routing dynamics and improves performance on language tasks under similar parameter budgets.

  14. Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Experts

    cs.LG 2025-06 unverdicted novelty 6.0

    MoRAM frames continual learning as incremental addition of rank-1 adapters viewed as self-activating key-value associative memory units in a mixture-of-experts setup.

  15. LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing

    cs.LG 2025-06 unverdicted novelty 6.0

    LoRA-Mixer routes modular LoRA experts into attention projection matrices with an adaptive Routing Specialization Loss to improve multi-task performance while using fewer trainable parameters than prior LoRA-MoE methods.

  16. MLorc: Momentum Low-rank Compression for Memory Efficient Large Language Model Adaptation

    cs.LG 2025-06 conditional novelty 6.0

    MLorc compresses optimizer momentum with low-rank methods to enable memory-efficient full fine-tuning of LLMs, outperforming LoRA and GaLore while matching full-parameter performance at small ranks.

  17. Clinically Structured Rank-Gated LoRA for Cross-Benchmark Medical Question Answering

    cs.CL 2026-06 unverdicted novelty 5.0

    BiRG-LoRA achieves 69.31% macro-average accuracy across CMB, CMExam, MedQA, and MedMCQA, outperforming MoELoRA by 0.89 points with 28.1% fewer trainable parameters under a matched Qwen3-8B protocol.

  18. TriageRA-CCF: Source-Side Clinical Confidence and Coverage Signals for Adaptive Rank Budgeting in Medical LLMs

    cs.CL 2026-06 unverdicted novelty 5.0

    TriageRA-CCF combines source-side confidence, coverage, and counterfactual signals to supervise an adaptive LoRA rank router, reporting modest average accuracy gains over LoRA/DoRA/MoELoRA baselines on two 8B models u...

  19. Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning

    cs.LG 2026-06 unverdicted novelty 5.0

    SETA decomposes parameters into task-specific and shared sparse experts with adaptive anchoring and routing regularization to improve retention and backward transfer in LLM continual learning.

  20. BioFact-MoE: Biologically Factorized Mixture of Experts for Vision-Language Prognostic Modeling in Hepatocellular Carcinoma

    cs.CV 2026-05 unverdicted novelty 5.0

    BioFact-MoE applies a biologically factorized MoE architecture to multimodal MRI-report data and reports improved 12-24 month survival AUCs plus selective embedding associations in an N=588 HCC cohort.

  21. Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

    cs.LG 2024-03 accept novelty 4.0

    A comprehensive survey of PEFT algorithms for large models, covering their performance, overhead, applications, and real-world system implementations.