Pith. sign in

REVIEW 5 cited by

MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.12130 v1 pith:X733VSZB submitted 2024-05-20 cs.CL cs.LG

Ting Jiang , Shaohan Huang , Shengyue Luo , Zihan Zhang , Haizhen Huang , Furu Wei , Weiwei Deng , Feng Sun
show 3 more authors
Qi Zhang Deqing Wang Fuzhen Zhuang
This is my paper
classification cs.CLcs.LG
keywords methodupdatingloralow-ranktasksachievedimensionfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Low-rank adaptation is a popular parameter-efficient fine-tuning method for large language models. In this paper, we analyze the impact of low-rank updating, as implemented in LoRA. Our findings suggest that the low-rank updating mechanism may limit the ability of LLMs to effectively learn and memorize new knowledge. Inspired by this observation, we propose a new method called MoRA, which employs a square matrix to achieve high-rank updating while maintaining the same number of trainable parameters. To achieve it, we introduce the corresponding non-parameter operators to reduce the input dimension and increase the output dimension for the square matrix. Furthermore, these operators ensure that the weight can be merged back into LLMs, which makes our method can be deployed like LoRA. We perform a comprehensive evaluation of our method across five tasks: instruction tuning, mathematical reasoning, continual pretraining, memory and pretraining. Our method outperforms LoRA on memory-intensive tasks and achieves comparable performance on other tasks.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 6 citations worldwide. Full citation record

  1. Beyond Low-Rank Tuning: Model Prior-Guided Rank Allocation for Effective Transfer in Low-Data and Large-Gap Regimes

    cs.CV 2025-06 conditional novelty 6.0 of 10

    SR-LoRA sets each LoRA layer's rank to the stable rank of that layer's pretrained weight matrix, improving few-shot transfer on large domain gaps without rank search.

  2. PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    PLoP selects LoRA adapter placement by ranking normalized feature norms and placing adapters on the lowest-scoring module types, using only forward passes.

  3. Adapt Once, Thrive with Updates: Transferable Parameter-Efficient Fine-Tuning on Evolving Base Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Trans-PEFT uses random FFN masking and layer dropping during fine-tuning so PEFT modules trained on an old base model transfer to a continually updated base model without retuning.

  4. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.

  5. Beyond End-to-End Video Models: An LLM-Based Multi-Agent System for Educational Video Generation

    cs.AI 2026-02 conditional novelty 5.0 of 10

    LASEV, a multi-agent LLM system that compiles structured 'executable video scripts' into educational videos, reports 92-96% expert-rated publishable quality and more than one million videos per day at 95% lower cost.

Pith tools