REVIEW 10 cited by
IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuning
read the original abstract
With the increasing size of pre-trained language models (PLMs), fine-tuning all the parameters in the model is not efficient, especially when there are a large number of downstream tasks, which incur significant training and storage costs. Many parameter-efficient fine-tuning (PEFT) approaches have been proposed, among which, Low-Rank Adaptation (LoRA) is a representative approach that injects trainable rank decomposition matrices into every target module. Yet LoRA ignores the importance of parameters in different modules. To address this problem, many works have been proposed to prune the parameters of LoRA. However, under limited training conditions, the upper bound of the rank of the pruned parameter matrix is still affected by the preset values. We, therefore, propose IncreLoRA, an incremental parameter allocation method that adaptively adds trainable parameters during training based on the importance scores of each module. This approach is different from the pruning method as it is not limited by the initial number of training parameters, and each parameter matrix has a higher rank upper bound for the same training overhead. We conduct extensive experiments on GLUE to demonstrate the effectiveness of IncreLoRA. The results show that our method owns higher parameter efficiency, especially when under the low-resource settings where our method significantly outperforms the baselines. Our code is publicly available.
Forward citations
Cited by 10 Pith papers
-
MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning
MatryoshkaLoRA inserts a crafted diagonal matrix P into LoRA to learn accurate nested low-rank adapters that support dynamic rank selection with minimal performance drop.
-
Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation
MoLF routes updates between full fine-tuning and LoRA at the optimizer level to match or exceed the better of either static method, with an efficient LoRA-only variant outperforming prior adaptive approaches.
-
Clinically Structured Rank-Gated LoRA for Cross-Benchmark Medical Question Answering
BiRG-LoRA reaches 69.31% macro-average accuracy across CMB, CMExam, MedQA and MedMCQA, outperforming MoELoRA by 0.89 points with 28.1% fewer parameters under a matched single-seed protocol.
-
Clinically Structured Rank-Gated LoRA for Cross-Benchmark Medical Question Answering
BiRG-LoRA achieves 69.31% macro-average accuracy across CMB, CMExam, MedQA and MedMCQA using a rank-gated LoRA with biaxial clinical gating, outperforming MoELoRA by 0.89 points with 28.1% fewer parameters.
-
AdaPaD: Adaptive Parallel Deflation for PEFT with Self-Correcting Rank Discovery
AdaPaD performs parallel low-rank adaptation with self-correcting deflation targets and dynamic per-module rank growth, yielding competitive GLUE and SQuAD results at 30% smaller average adapter size.
-
Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation
MoLF routes updates between full fine-tuning and LoRA at the optimizer level to match or exceed the better of the two static methods on SQL, medical QA, and counterfactual tasks while an efficient variant outperforms ...
-
Clinically Structured Rank-Gated LoRA for Cross-Benchmark Medical Question Answering
BiRG-LoRA achieves 69.31% macro-average accuracy across CMB, CMExam, MedQA, and MedMCQA, outperforming MoELoRA by 0.89 points with 28.1% fewer trainable parameters under a matched Qwen3-8B protocol.
-
TriageRA-CCF: Source-Side Clinical Confidence and Coverage Signals for Adaptive Rank Budgeting in Medical LLMs
TriageRA-CCF combines source-side confidence, coverage, and counterfactual signals to supervise an adaptive LoRA rank router, reporting modest average accuracy gains over LoRA/DoRA/MoELoRA baselines on two 8B models u...
-
Low-Rank Adaptation Redux for Large Models
An overview revisits LoRA variants by categorizing advances in architectural design, efficient optimization, and applications while linking them to classical signal processing tools for principled fine-tuning.
-
Accelerating Attention with Basis Decomposition
A low-rank factorization of attention projection matrices (basis plus coefficients) gives modest FLOP savings in exact arithmetic, but the claimed losslessness and novelty are not supported.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.