Pith. sign in

REVIEW 5 cited by

SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.19597 v1 pith:27PUZSXT submitted 2024-05-30 cs.LG cs.AIcs.CL

Vijay Lingam , Atula Tejaswi , Aditya Vavre , Aneesh Shetty , Gautham Krishna Gudur , Joydeep Ghosh , Alex Dimakis , Eunsol Choi
show 2 more authors
Aleksandar Bojchevski Sujay Sanghavi
This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords methodsfine-tuningsvftdeltaonlyperformancevectorsapproach
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Popular parameter-efficient fine-tuning (PEFT) methods, such as LoRA and its variants, freeze pre-trained model weights \(W\) and inject learnable matrices \(\Delta W\). These \(\Delta W\) matrices are structured for efficient parameterization, often using techniques like low-rank approximations or scaling vectors. However, these methods typically show a performance gap compared to full fine-tuning. Although recent PEFT methods have narrowed this gap, they do so at the cost of additional learnable parameters. We propose SVFT, a simple approach that fundamentally differs from existing methods: the structure imposed on \(\Delta W\) depends on the specific weight matrix \(W\). Specifically, SVFT updates \(W\) as a sparse combination of outer products of its singular vectors, training only the coefficients (scales) of these sparse combinations. This approach allows fine-grained control over expressivity through the number of coefficients. Extensive experiments on language and vision benchmarks show that SVFT recovers up to 96% of full fine-tuning performance while training only 0.006 to 0.25% of parameters, outperforming existing methods that only recover up to 85% performance using 0.03 to 0.8% of the trainable parameter budget.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models

    cs.LG 2025-05 conditional novelty 6.0 of 10

    RefLoRA picks a per-step optimal low-rank factorization (a matrix geometric mean) that balances LoRA's factors, improving fine-tuning convergence and accuracy.

  2. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.

  3. QR-LoRA: Efficient and Disentangled Fine-tuning via QR Decomposition for Customized Generation

    cs.CV 2025-07 conditional novelty 4.0 of 10

    QR-LoRA freezes the QR-decomposed basis of pretrained weights, trains only a residual matrix, and reports halved trainable parameters with improved content-style disentanglement in diffusion models.

  4. CoLA: Collaborative Low-Rank Adaptation

    cs.CL 2025-05 conditional novelty 4.0 of 10

    CoLA generalizes LoRA to multiple A and B matrices with a principal-component initialization and reports gains of roughly 2-4 accuracy points over PiSSA on low-sample fine-tuning benchmarks.

  5. Dual Decomposition of Weights and Singular Value Low Rank Adaptation

    cs.CL 2025-05 conditional novelty 4.0 of 10

    DuDe combines DoRA's magnitude-direction decomposition with PiSSA's SVD-based initialization, reporting consistent but modest accuracy gains over LoRA, DoRA, and PiSSA on commonsense reasoning, GPQA, MMLU, and GSM8K.

Pith tools