Pith. sign in

REVIEW 30 cited by

Mixture of LoRA Experts

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.13628 v1 pith:GGVX5D33 submitted 2024-04-21 cs.CL cs.LGcs.MM

Xun Wu , Shaohan Huang , Furu Wei This is my paper · ORCID
classification cs.CLcs.LGcs.MM
keywords loralorasfusionmoleapproacharithmeticchallengesdirect
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

LoRA has gained widespread acceptance in the fine-tuning of large pre-trained models to cater to a diverse array of downstream tasks, showcasing notable effectiveness and efficiency, thereby solidifying its position as one of the most prevalent fine-tuning techniques. Due to the modular nature of LoRA's plug-and-play plugins, researchers have delved into the amalgamation of multiple LoRAs to empower models to excel across various downstream tasks. Nonetheless, extant approaches for LoRA fusion grapple with inherent challenges. Direct arithmetic merging may result in the loss of the original pre-trained model's generative capabilities or the distinct identity of LoRAs, thereby yielding suboptimal outcomes. On the other hand, Reference tuning-based fusion exhibits limitations concerning the requisite flexibility for the effective combination of multiple LoRAs. In response to these challenges, this paper introduces the Mixture of LoRA Experts (MoLE) approach, which harnesses hierarchical control and unfettered branch selection. The MoLE approach not only achieves superior LoRA fusion performance in comparison to direct arithmetic merging but also retains the crucial flexibility for combining LoRAs effectively. Extensive experimental evaluations conducted in both the Natural Language Processing (NLP) and Vision & Language (V&L) domains substantiate the efficacy of MoLE.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 30 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LoRA of Change: Learning to Generate LoRA for the Editing Instruction from A Single Before-After Image Pair

    cs.CV 2024-11 conditional novelty 7.0 of 10

    A hypernetwork generates a per-instruction LoRA from a before-after image pair, and a reverse training loss allows learning from paired data alone.

  2. SAM+D: Parameter-Efficient Dimensional Lifting of SAM-Family Models via Depth-Routed LoRA and Depth Shifting

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Depth-routed LoRA and a depth-shift module lift frozen SAM and SAM2 to 3D and 3D+T segmentation using less than ~3.7% trainable parameters.

  3. DA-MergeLoRA: Hypernetwork-Based LoRA Merging for Few-Shot Test-Time Domain Adaptation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A hypernetwork generates per-column merging weights to combine source LoRA modules on CLIP, achieving state-of-the-art few-shot test-time domain adaptation.

  4. Transferable Low-Rank Convolutional Bases for Onboarding Unseen Medical Imaging Modalities

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A convolutional low-rank basis learned on CT and MRI transfers to chest X-ray, enabling onboarding of an unseen modality at 0.78% of full-fine-tuning parameters with exactly zero forgetting.

  5. Parametric Memory Decoding for Zero-Shot Routing in LoRA-Based External Parametric Memory

    cs.LG 2026-07 conditional novelty 6.0 of 10

    PMDRouter selects LoRAs zero-shot by decoding scale-normalized linear response energy from one adapter-free backbone prefill, and leads most internal-signal baselines on a new multi-granularity EPM bench.

  6. Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    Hard top-1 token routing over frozen, independently RLVF-trained LoRA experts matches or beats soft-routing mixtures with about 8–10x fewer trainable parameters.

  7. Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Sleep-time Knowledge Seeding plus Dreaming lets LLMs expand capacity, distill fragile in-context memories into stable parameters, and self-improve without human labels.

  8. CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging

    cs.CL 2026-02 conditional novelty 6.0 of 10

    CoMoL represents every LoRA expert as a shared-basis core matrix and merges token-selected experts in that core space, reaching standard LoRA parameter counts while outperforming MoE-LoRA baselines on math and code.

  9. R^2MoE: Redundancy-Removal Mixture of Experts for Lifelong Concept Learning

    cs.CV 2025-07 conditional novelty 6.0 of 10

    R2MoE adds per-concept LoRA experts with routing distillation and expert pruning, reporting 0.19% forgetting and 15.2M added parameters on CustomConcept101.

  10. Cartridges: Lightweight and general-purpose long context representations via self-study

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A per-corpus trained KV cache, called a Cartridge, matches full-context in-context learning quality on long-document benchmarks while using up to 38.6x less serving memory.

  11. MultLFG: Training-free Multi-LoRA composition using Frequency-domain Guidance

    cs.CV 2025-05 conditional novelty 6.0 of 10

    MultLFG merges multiple LoRA adapters by adaptively weighting them in wavelet frequency subbands per denoising timestep, improving multi-concept composition on the ComposLoRA benchmark compared to prior training-free methods.

  12. Guiding the Experts: Semantic Priors for Efficient and Focused MoE Routing

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Guiding Soft MoE dispatch weights with foreground segmentation masks plus a zero-initialized LayerScale improves ImageNet-1K top-1 by 0.6% and ImageNet-100 by 1.4% over a reproduced baseline.

  13. TransMLA: Multi-Head Latent Attention Is All You Need

    cs.LG 2025-02 conditional novelty 6.0 of 10

    A framework that converts pretrained GQA models into MLA models using RoRoPE, FreqFold, and balanced KV low-rank compression, regaining baseline performance with only a few billion fine-tuning tokens.

  14. When One LLM Drools, Multi-LLM Collaboration Rules

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A position paper that introduces a four-level taxonomy of multi-LLM collaboration (API, text, logit, weight) and argues it is essential for reliability, pluralism, and democratization.

  15. Multi-Dimensional Quality Assessment for AI-Generated Human-Centric Videos: Dataset and Model

    cs.CV 2026-07 conditional novelty 5.0 of 10

    A new large benchmark for AI-generated human-centric video quality with pairwise preferences, plus a Mixture-of-Experts MLLM that outperforms prior methods on rating, comparison, and Q&A.

  16. EmoStyle: Affective Conditioning of Style-Specialist Experts for Emotional Image Generation

    cs.CV 2026-07 conditional novelty 5.0 of 10

    EmoStyle injects LLM-inferred valence-arousal and emotion labels into Z-Image via AdaLN-style residual modulation over style-bucket LoRA experts, plus VLM candidate ranking, and ranked first on AffectiveArt Track 1.

  17. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.

  18. Joint Information Extraction Across Classical and Modern Chinese with Tea-MOELoRA

    cs.CL 2025-09 conditional novelty 5.0 of 10

    Tea-MOELORA uses separate task and era gates over LoRA experts to jointly train relation and event extraction across classical and modern Chinese, improving F1 over joint LoRA and existing LoRA-MoE baselines on most datasets.

  19. ICM-Fusion: In-Context Meta-Optimized LoRA Fusion for Multi-Task Adaptation

    cs.CV 2025-08 reject novelty 5.0 of 10

    ICM-Fusion uses a conditional VAE plus task-vector guidance to fuse multiple LoRA adapters into one model, reporting marginal average gains on vision and language benchmarks and larger gains in a few-shot long-tail setup.

  20. PFMBench: Protein Foundation Model Benchmark

    q-bio.BM 2025-06 conditional novelty 5.0 of 10

    A comprehensive benchmark of 17 protein foundation models across 38 tasks yields task correlations, a streamlined protocol, and identifies ProTrek as the strongest general performer.

  21. CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing

    cs.CL 2025-02 conditional novelty 5.0 of 10

    CITER trains a token-level router with preference optimization to route non-critical tokens to a small model and critical tokens to a large model, reducing inference cost on QA and math benchmarks.

  22. Ensembles of Low-Rank Expert Adapters

    cs.CL 2025-01 conditional novelty 5.0 of 10

    ELREA clusters instruction-tuning data by gradient direction, trains one LoRA expert per cluster, and routes new instructions to experts via gradient similarity, giving modest benchmark gains over full-data LoRA.

  23. ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing

    cs.LG 2024-12 conditional novelty 5.0 of 10

    ReLU-based routing with adaptive L1 regularization outperforms TopK routing in MoE language models across model sizes, expert counts, and granularity.

  24. Task Arithmetic Through The Lens Of One-Shot Federated Learning

    cs.LG 2024-11 conditional novelty 5.0 of 10

    Task arithmetic is exactly one-shot FedAvg with outer step size beta = lambda T, and FedNova, FedGMA, Median, and CCLIP can often improve merged model performance.

  25. Multi LoRA Meets Vision: Merging multiple adapters to create a multi task model

    cs.CV 2024-11 conditional novelty 5.0 of 10

    Merging LoRA adapters across six vision tasks produces usable multitask models, with performance depending strongly on dataset similarity.

  26. From Holistic to Localized: Local Enhanced Adapters for Efficient Visual Instruction Fine-Tuning

    cs.CV 2024-11 conditional novelty 5.0 of 10

    Dual-LoRA plus Visual Cue Enhancement improves efficient visual instruction tuning over LoRA and LoRA-MoE baselines with near-vanilla-LoRA inference time.

  27. PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Model

    cs.LG 2024-11 conditional novelty 5.0 of 10

    PERFT inserts routed, parameter-efficient LoRA experts into MoE layers and reports consistent gains over attention-only LoRA on reasoning benchmarks.

  28. $\mu$-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts

    cs.LG 2025-05 conditional novelty 4.0 of 10

    Test-time Wanda pruning, reframed as a mixture of micro-experts, adapts the sparse weight mask to each prompt and improves perplexity and VQA accuracy over static pruning baselines.

  29. MMRL++: Parameter-Efficient and Interaction-Aware Representation Learning for Vision-Language Models

    cs.CV 2025-05 conditional novelty 4.0 of 10

    MMRL++ inserts shared, learnable representation tokens into the upper layers of CLIP's image and text encoders and uses low-rank shared aligners, achieving state-of-the-art base-to-novel harmonic mean accuracy on 11 d...

  30. Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges

    cs.LG 2024-12 conditional

    A broad but error-prone survey of LLM and MLLM architectures, training methods, benchmarks, and challenges.

Pith tools