Pith. sign in

REVIEW 14 cited by

MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.12130 v1 pith:X733VSZB submitted 2024-05-20 cs.CL cs.LG

Ting Jiang , Shaohan Huang , Shengyue Luo , Zihan Zhang , Haizhen Huang , Furu Wei , Weiwei Deng , Feng Sun
show 3 more authors
Qi Zhang Deqing Wang Fuzhen Zhuang
This is my paper · ORCID
classification cs.CLcs.LG
keywords methodupdatingloralow-ranktasksachievedimensionfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Low-rank adaptation is a popular parameter-efficient fine-tuning method for large language models. In this paper, we analyze the impact of low-rank updating, as implemented in LoRA. Our findings suggest that the low-rank updating mechanism may limit the ability of LLMs to effectively learn and memorize new knowledge. Inspired by this observation, we propose a new method called MoRA, which employs a square matrix to achieve high-rank updating while maintaining the same number of trainable parameters. To achieve it, we introduce the corresponding non-parameter operators to reduce the input dimension and increase the output dimension for the square matrix. Furthermore, these operators ensure that the weight can be merged back into LLMs, which makes our method can be deployed like LoRA. We perform a comprehensive evaluation of our method across five tasks: instruction tuning, mathematical reasoning, continual pretraining, memory and pretraining. Our method outperforms LoRA on memory-intensive tasks and achieves comparable performance on other tasks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 14 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Low-Rank Tuning: Model Prior-Guided Rank Allocation for Effective Transfer in Low-Data and Large-Gap Regimes

    cs.CV 2025-06 conditional novelty 6.0 of 10

    SR-LoRA sets each LoRA layer's rank to the stable rank of that layer's pretrained weight matrix, improving few-shot transfer on large domain gaps without rank search.

  2. PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    PLoP selects LoRA adapter placement by ranking normalized feature norms and placing adapters on the lowest-scoring module types, using only forward passes.

  3. Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helps

    cs.LG 2025-06 conditional novelty 6.0 of 10

    SeLoRA reparameterizes LoRA updates as inverse Fourier or wavelet transforms of sparsely masked spectral coefficients, improving fine-tuning accuracy on LLaMA models with fewer trainable parameters.

  4. Adapt Once, Thrive with Updates: Transferable Parameter-Efficient Fine-Tuning on Evolving Base Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Trans-PEFT uses random FFN masking and layer dropping during fine-tuning so PEFT modules trained on an old base model transfer to a continually updated base model without retuning.

  5. On the Convergence Analysis of Muon

    stat.ML 2025-05 unverdicted novelty 6.0 of 10

    Muon's convergence rate depends on an average Hessian curvature along its update directions, which can be much smaller than the worst-case Lipschitz constant when Hessians are low-rank.

  6. Fragments to Facts: Partial-Information Fragment Inference from LLMs

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Fine-tuned LLMs leak private fragment-level information to adversaries holding only a few unordered public fragments, as shown by two probe attacks (LR-Attack and PRISM) on medical and legal summarization tasks.

  7. CURing Large Models: Compression via CUR Decomposition

    cs.LG 2025-01 conditional novelty 6.0 of 10

    CUR decomposition with WANDA-and-DEIM row/column selection compresses LLM weights quickly, and the linking matrix U can be fine-tuned as a PEFT-style healing step.

  8. S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity

    cs.LG 2024-12 conditional novelty 6.0 of 10

    S2FT selects a few attention heads and FFN channels, permutes the neighboring weight matrices so the selected parts form dense blocks, and fine-tunes only those blocks, reporting better generalization and efficiency t...

  9. CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuning

    cs.LG 2024-11 reject novelty 6.0 of 10

    Attention pairs (Q-K and V-O) are SVD-decomposed so pruning or fine-tuning touches only a small singular-factor matrix, yielding better pruning tolerance and small PEFT gains.

  10. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.

  11. Beyond End-to-End Video Models: An LLM-Based Multi-Agent System for Educational Video Generation

    cs.AI 2026-02 conditional novelty 5.0 of 10

    LASEV, a multi-agent LLM system that compiles structured 'executable video scripts' into educational videos, reports 92-96% expert-rated publishable quality and more than one million videos per day at 95% lower cost.

  12. FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts

    cs.LG 2025-05 conditional novelty 5.0 of 10

    FLoE uses Fisher information to pick the transformer layers that matter and a Bayesian optimizer to set LoRA rank, cutting trainable parameters while keeping or improving accuracy.

  13. Fine Tuning without Catastrophic Forgetting via Selective Low Rank Adaptation

    cs.CV 2025-01 conditional novelty 4.0 of 10

    Fine-tuning a pretrained vision model with only 2-6% of its LoRA adapters active preserves zero-shot and out-of-distribution performance while matching full-rank LoRA accuracy.

  14. BoRA: Bi-dimensional Weight-Decomposed Low-Rank Adaptation

    cs.CL 2024-12 conditional novelty 4.0 of 10

    BoRA extends DoRA by adding a row-magnitude vector alongside the column-magnitude vector, reporting improved MT-Bench and commonsense reasoning scores over LoRA and DoRA on three LLMs.

Pith tools