REVIEW 6 cited by
Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning
read the original abstract
Low-Rank Adaptation (LoRA) is widely used for adapting large language models (LLMs) to specific domains due to its efficiency and modularity. Meanwhile, vanilla LoRA struggles with task conflicts in multi-task scenarios. Recent works adopt Mixture of Experts (MoE) by treating each LoRA module as an expert, thereby mitigating task interference through multiple specialized LoRA modules. While effective, these methods often isolate knowledge within individual tasks, failing to fully exploit the shared knowledge across related tasks. In this paper, we establish a connection between single LoRA and multi-LoRA MoE, integrating them into a unified framework. We demonstrate that the dynamic routing of multiple LoRAs is functionally equivalent to rank partitioning and block-level activation within a single LoRA. We further empirically demonstrate that finer-grained LoRA partitioning, within the same total and activated parameter constraints, leads to better performance gains across heterogeneous tasks. Building on these findings, we propose Single-ranked Mixture of Experts LoRA (\textbf{SMoRA}), which embeds MoE into LoRA by \textit{treating each rank as an independent expert}. With a \textit{dynamic rank-wise activation} mechanism, SMoRA promotes finer-grained knowledge sharing while mitigating task conflicts. Experiments demonstrate that SMoRA activates fewer parameters yet achieves better performance in multi-task scenarios.
Forward citations
Cited by 6 Pith papers
-
Clinically Structured Rank-Gated LoRA for Cross-Benchmark Medical Question Answering
BiRG-LoRA reaches 69.31% macro-average accuracy across CMB, CMExam, MedQA and MedMCQA, outperforming MoELoRA by 0.89 points with 28.1% fewer parameters under a matched single-seed protocol.
-
Clinically Structured Rank-Gated LoRA for Cross-Benchmark Medical Question Answering
BiRG-LoRA achieves 69.31% macro-average accuracy across CMB, CMExam, MedQA and MedMCQA using a rank-gated LoRA with biaxial clinical gating, outperforming MoELoRA by 0.89 points with 28.1% fewer parameters.
-
Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation
PrunedLoRA tunes large-rank LoRA adapters and progressively structurally prunes A and B to a target low rank, empirically closing much of the LoRA-to-full-fine-tuning gap.
-
Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMs
DERN prunes SMoE LLMs by decomposing removed experts into neuron segments, reassigning the best-matching ones to kept experts, and clustering them into compact replacements, beating prior pruning baselines without retraining.
-
Clinically Structured Rank-Gated LoRA for Cross-Benchmark Medical Question Answering
BiRG-LoRA achieves 69.31% macro-average accuracy across CMB, CMExam, MedQA, and MedMCQA, outperforming MoELoRA by 0.89 points with 28.1% fewer trainable parameters under a matched Qwen3-8B protocol.
-
DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
DAG-MoE uses a lightweight module to learn DAG-based structural aggregation of selected experts, expanding combination space and enabling intra-layer multi-step reasoning compared to standard weighted-sum MoE.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.