Pith. sign in

REVIEW 5 cited by

ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.16187 v2 pith:33XDFVVV submitted 2024-03-24 cs.CL

Zequan Liu , Jiawen Lyn , Wei Zhu , Xing Tian , Yvette Graham This is my paper
classification cs.CL
keywords adaptationloraaloralow-rankmethodrankab-loraallocating
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Parameter-efficient fine-tuning (PEFT) is widely studied for its effectiveness and efficiency in the era of large language models. Low-rank adaptation (LoRA) has demonstrated commendable performance as a popular and representative method. However, it is implemented with a fixed intrinsic rank that might not be the ideal setting for the downstream tasks. Recognizing the need for more flexible downstream task adaptation, we extend the methodology of LoRA to an innovative approach we call allocating low-rank adaptation (ALoRA) that enables dynamic adjustments to the intrinsic rank during the adaptation process. First, we propose a novel method, AB-LoRA, that can effectively estimate the importance score of each LoRA rank. Second, guided by AB-LoRA, we gradually prune abundant and negatively impacting LoRA ranks and allocate the pruned LoRA budgets to important Transformer modules needing higher ranks. We have conducted experiments on various tasks, and the experimental results demonstrate that our ALoRA method can outperform the recent baselines with comparable tunable parameters.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints

    cs.CV 2025-07 conditional novelty 6.0 of 10

    LoRA adapters can be initialized with a closed-form estimate derived from constraint sets linking source and target activations, improving fine-tuning speed and accuracy.

  2. Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Models

    cs.LG 2025-09 conditional novelty 5.0 of 10

    Sensitivity-LoRA allocates LoRA ranks across layers using Hessian-based sensitivity metrics, improving average GLUE score by 0.74 over AdaLoRA on RoBERTa-base.

  3. QR-LoRA: QR-Based Low-Rank Adaptation for Efficient Fine-Tuning of Large Language Models

    cs.LG 2025-08 conditional novelty 5.0 of 10

    QR-LoRA trains only scalar coefficients on a pivoted QR basis of pretrained weights, reportedly matching full fine-tuning on GLUE with 601 parameters.

  4. FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts

    cs.LG 2025-05 conditional novelty 5.0 of 10

    FLoE uses Fisher information to pick the transformer layers that matter and a Bayesian optimizer to set LoRA rank, cutting trainable parameters while keeping or improving accuracy.

  5. GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptation

    cs.LG 2025-08 conditional novelty 4.0 of 10

    GEM selects fine-tuning parameters by gradient-to-weight ratio and distributes the budget by layer entropy, reaching 0.1% parameter updates with small accuracy gains on several NLP tasks.

Pith tools