Pith. sign in

REVIEW 1 cited by

SVFit: Parameter-Efficient Fine-Tuning of Large Pre-Trained Models Using Singular Values

Not yet reviewed by Pith; the record is open.

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2409.05926 v1 pith:PKQYGAXX submitted 2024-09-09 cs.LG cs.CL

SVFit: Parameter-Efficient Fine-Tuning of Large Pre-Trained Models Using Singular Values

classification cs.LG cs.CL
keywords singularmatrixmodelsparameterspre-trainedsvfitvaluesfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Large pre-trained models (LPMs) have demonstrated exceptional performance in diverse natural language processing and computer vision tasks. However, fully fine-tuning these models poses substantial memory challenges, particularly in resource-constrained environments. Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, mitigate this issue by adjusting only a small subset of parameters. Nevertheless, these methods typically employ random initialization for low-rank matrices, which can lead to inefficiencies in gradient descent and diminished generalizability due to suboptimal starting points. To address these limitations, we propose SVFit, a novel PEFT approach that leverages singular value decomposition (SVD) to initialize low-rank matrices using critical singular values as trainable parameters. Specifically, SVFit performs SVD on the pre-trained weight matrix to obtain the best rank-r approximation matrix, emphasizing the most critical singular values that capture over 99% of the matrix's information. These top-r singular values are then used as trainable parameters to scale the fundamental subspaces of the matrix, facilitating rapid domain adaptation. Extensive experiments across various pre-trained models in natural language understanding, text-to-image generation, and image classification tasks reveal that SVFit outperforms LoRA while requiring 16 times fewer trainable parameters.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Localized LoRA-MoE: Block-wise Low-Rank Experts With Adaptive Routing

    cs.LG 2026-07 conditional novelty 6.0

    Block-wise and cell-wise LoRA-MoE routers break static LoRA gradient conflicts on multi-context matrix tasks, with cell-level gates matching a global router on uniform shifts and beating it on heterogeneous ones.