Pith. sign in

REVIEW 10 cited by

Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2409.16167 v3 pith:4LMG65MX submitted 2024-09-24 cs.LG cs.AIcs.CL

Ziyu Zhao , Tao Shen , Didi Zhu , Zexi Li , Jing Su , Xuwu Wang , Kun Kuang , Fei Wu This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords loralorasmergingmsusclusteringdemonstrateenablingexisting
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Low-Rank Adaptation (LoRA) has emerged as a popular technique for fine-tuning large language models (LLMs) to various domains due to its modular design and widespread availability on platforms like Huggingface. This modularity has sparked interest in combining multiple LoRAs to enhance LLM capabilities. However, existing methods for LoRA composition primarily focus on task-specific adaptations that require additional training, and current model merging techniques often fail to fully leverage LoRA's modular nature, leading to parameter interference and performance degradation. In this paper, we investigate the feasibility of disassembling and reassembling multiple LoRAs at a finer granularity, analogous to assembling LEGO blocks. We introduce the concept of Minimal Semantic Units (MSUs), where the parameters corresponding to each rank in LoRA function as independent units. These MSUs demonstrate permutation invariance and concatenation-summation equivalence properties, enabling flexible combinations to create new LoRAs. Building on these insights, we propose the LoRA-LEGO framework. This framework conducts rank-wise parameter clustering by grouping MSUs from different LoRAs into $k$ clusters. The centroid of each cluster serves as a representative MSU, enabling the assembly of a merged LoRA with an adjusted rank of $k$. Additionally, we apply a dual reweighting strategy to optimize the scale of the merged LoRA. Experiments across various benchmarks demonstrate that our method outperforms existing approaches in LoRA merging.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SkillSmith: Learning to Compose Parametric Skills and Textual Knowledge

    cs.CL 2026-07 conditional novelty 7.0 of 10

    SkillSmith, an LLM augmented to ingest prefix-weights and text, generates target-task prefix-weights that beat text-only and weight-only baselines, especially as fine-tuning initialization.

  2. Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Sleep-time Knowledge Seeding plus Dreaming lets LLMs expand capacity, distill fragile in-context memories into stable parameters, and self-improve without human labels.

  3. Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMs

    cs.CL 2025-09 conditional novelty 6.0 of 10

    DERN prunes SMoE LLMs by decomposing removed experts into neuron segments, reassigning the best-matching ones to kept experts, and clustering them into compact replacements, beating prior pruning baselines without retraining.

  4. Cartridges: Lightweight and general-purpose long context representations via self-study

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A per-corpus trained KV cache, called a Cartridge, matches full-context in-context learning quality on long-document benchmarks while using up to 38.6x less serving memory.

  5. FedRPCA: Enhancing Federated LoRA Aggregation Using Robust PCA

    cs.LG 2025-06 conditional novelty 6.0 of 10

    FedRPCA decomposes federated LoRA client updates with Robust PCA into common and client-specific components, averaging the common part and scaled-averaging the sparse part, which improves accuracy and convergence over...

  6. Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Initializing LoRA's A matrix with the smallest-eigenvalue eigenvectors of other tasks' feature covariance reduces interference when merging task-specific LoRA models, improving average merged accuracy.

  7. One Rank at a Time: Cascading Error Dynamics in Sequential Learning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Errors from each rank-1 step in sequential low-rank learning compound through factors that grow when singular values are close, so early steps deserve more compute.

  8. MultLFG: Training-free Multi-LoRA composition using Frequency-domain Guidance

    cs.CV 2025-05 conditional novelty 6.0 of 10

    MultLFG merges multiple LoRA adapters by adaptively weighting them in wavelet frequency subbands per denoising timestep, improving multi-concept composition on the ComposLoRA benchmark compared to prior training-free methods.

  9. Decouple and Orthogonalize: A Data-Free Framework for LoRA Merging

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A data-free LoRA merging framework that decouples weight magnitude from direction and orthogonalizes directions to reduce task interference, outperforming existing merging methods across vision, language and multimoda...

  10. Sci-LoRA: Mixture of Scientific LoRAs for Cross-Domain Lay Paraphrasing

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Sci-LoRA dynamically mixes domain-specific LoRA adapters and achieves state-of-the-art lay paraphrasing across twelve domains without needing domain labels at inference.

Pith tools