Pith. sign in

REVIEW 10 cited by

Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matrices

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2309.02411 v1 pith:GGPSIDEL submitted 2023-09-05 cs.LG

Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matrices

Bojia Zi , Xianbiao Qi , Lingzhi Wang , Jianan Wang , Kam-Fai Wong , Lei Zhang This is my paper
classification cs.LG
keywords delta-loralow-rankmatricesadaptationdeltalearningloramethods
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

In this paper, we present Delta-LoRA, which is a novel parameter-efficient approach to fine-tune large language models (LLMs). In contrast to LoRA and other low-rank adaptation methods such as AdaLoRA, Delta-LoRA not only updates the low-rank matrices $\bA$ and $\bB$, but also propagate the learning to the pre-trained weights $\bW$ via updates utilizing the delta of the product of two low-rank matrices ($\bA^{(t+1)}\bB^{(t+1)} - \bA^{(t)}\bB^{(t)}$). Such a strategy effectively addresses the limitation that the incremental update of low-rank matrices is inadequate for learning representations capable for downstream tasks. Moreover, as the update of $\bW$ does not need to compute the gradients of $\bW$ and store their momentums, Delta-LoRA shares comparable memory requirements and computational costs with LoRA. Extensive experiments show that Delta-LoRA significantly outperforms existing low-rank adaptation methods. We further support these results with comprehensive analyses that underscore the effectiveness of Delta-LoRA.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs

    cs.LG 2025-09 conditional novelty 7.0

    By sharing the B matrix across adapters instead of the A matrix, ALoRA and Fed-ALoRA deliver more balanced performance in multi-task and federated LLM fine-tuning.

  2. \k{appa}-LoRA: Condition Numbers Reveal Which LoRA Matrices Worth Updating

    cs.LG 2026-07 conditional novelty 6.0

    Selecting the top 50% of LoRA weight matrices by condition number halves trainable parameters and cuts fine-tuning time by about 16% while roughly matching full-LoRA accuracy.

  3. EPTS: Elastic Post-Training Sparsity for Efficient Large Language Model Compression

    cs.LG 2026-06 unverdicted novelty 6.0

    EPTS is a multi-sparsity framework using MS-HiLoRA and MSFM to create an elastic LLM from one-shot post-training optimization that matches SparseGPT and Wanda performance.

  4. NP-LoRA: Null Space Projection for Subject-Style LoRA Fusion

    cs.CV 2025-11 unverdicted novelty 6.0

    NP-LoRA fuses subject and style LoRAs via null-space projection of the content update onto the orthogonal complement of the style subspace, with a soft variant controlled by one parameter.

  5. LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing

    cs.LG 2025-06 unverdicted novelty 6.0

    LoRA-Mixer routes modular LoRA experts into attention projection matrices with an adaptive Routing Specialization Loss to improve multi-task performance while using fewer trainable parameters than prior LoRA-MoE methods.

  6. MLorc: Momentum Low-rank Compression for Memory Efficient Large Language Model Adaptation

    cs.LG 2025-06 conditional novelty 6.0

    MLorc compresses optimizer momentum with low-rank methods to enable memory-efficient full fine-tuning of LLMs, outperforming LoRA and GaLore while matching full-parameter performance at small ranks.

  7. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    cs.LG 2026-06 conditional novelty 5.0

    A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.

  8. The Hidden Power of Scaling Factor in LoRA Optimization

    cs.AI 2026-06 unverdicted novelty 5.0

    Alpha in LoRA outperforms learning-rate scaling, follows a square-root law with rank, and enables a minimalist LoRA-alpha method that improves performance across tasks.

  9. EinSort: Sorting is All We Need for Tensorizing LLM

    cs.LG 2026-06 unverdicted novelty 5.0

    Sorting tensor indices enables an adaptive tensorization method that discovers low-rank structure in LLM weights and KV caches, yielding better reconstruction quality than baselines.

  10. TLoRA+: A Low-Rank Parameter-Efficient Fine-Tuning Method for Large Language Models

    cs.CL 2026-04 unverdicted novelty 3.0

    TLoRA+ augments LoRA with a dedicated optimizer to improve fine-tuning performance on GLUE tasks without meaningful added compute.