Pith. sign in

REVIEW 15 cited by

PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.02948 v4 pith:NYYNY7G3 submitted 2024-04-03 cs.LG cs.AI

PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models

Fanxu Meng , Zhaohui Wang , Muhan Zhang This is my paper
classification cs.LG cs.AI
keywords pissaloramathbbprincipalsingulartimesadaptationcomponents
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

To parameter-efficiently fine-tune (PEFT) large language models (LLMs), the low-rank adaptation (LoRA) method approximates the model changes $\Delta W \in \mathbb{R}^{m \times n}$ through the product of two matrices $A \in \mathbb{R}^{m \times r}$ and $B \in \mathbb{R}^{r \times n}$, where $r \ll \min(m, n)$, $A$ is initialized with Gaussian noise, and $B$ with zeros. LoRA freezes the original model $W$ and updates the "Noise & Zero" adapter, which may lead to slow convergence. To overcome this limitation, we introduce Principal Singular values and Singular vectors Adaptation (PiSSA). PiSSA shares the same architecture as LoRA, but initializes the adaptor matrices $A$ and $B$ with the principal components of the original matrix $W$, and put the remaining components into a residual matrix $W^{res} \in \mathbb{R}^{m \times n}$ which is frozen during fine-tuning. Compared to LoRA, PiSSA updates the principal components while freezing the "residual" parts, allowing faster convergence and enhanced performance. Comparative experiments of PiSSA and LoRA across 12 different models, ranging from 184M to 70B, encompassing 5 NLG and 8 NLU tasks, reveal that PiSSA consistently outperforms LoRA under identical experimental setups. On the GSM8K benchmark, Mistral-7B fine-tuned with PiSSA achieves an accuracy of 72.86%, surpassing LoRA's 67.7% by 5.16%. Due to the same architecture, PiSSA is also compatible with quantization to further reduce the memory requirement of fine-tuning. Compared to QLoRA, QPiSSA exhibits smaller quantization errors in the initial stages. Fine-tuning LLaMA-3-70B on GSM8K, QPiSSA attains an accuracy of 86.05%, exceeding the performances of QLoRA at 81.73%. Leveraging a fast SVD technique, PiSSA can be initialized in only a few seconds, presenting a negligible cost for transitioning from LoRA to PiSSA. Code is available at https://github.com/GraphPKU/PiSSA.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. BoostLoRA: Growing Effective Rank by Boosting Adapters

    cs.LG 2026-04 unverdicted novelty 7.0

    BoostLoRA grows effective adapter rank linearly via iterative boosting on hard examples with orthogonal low-rank updates, outperforming both single-shot ultra-low-rank adapters and full fine-tuning on math and code ta...

  2. Spectral Rewiring for Exploration, Purification, and Model Merging

    cs.LG 2026-07 conditional novelty 6.5

    Subspace-Aligned Rewiring projects RL weight updates onto the base model’s SVD basis, retaining a compact rewiring matrix that preserves reasoning and improves exploration and multi-domain merging.

  3. \k{appa}-LoRA: Condition Numbers Reveal Which LoRA Matrices Worth Updating

    cs.LG 2026-07 conditional novelty 6.0

    Selecting the top 50% of LoRA weight matrices by condition number halves trainable parameters and cuts fine-tuning time by about 16% while roughly matching full-LoRA accuracy.

  4. ReCoLoRA: Spectrum-Aware Recursive Consolidation for Continual LLM Fine-Tuning

    cs.LG 2026-07 conditional novelty 6.0

    Spectrum-initialized LoRA with elbow ranks and recursive SVD consolidation of the effective weight beats rank-swept PEFT baselines on three of four 7–8B models in continual GLUE fine-tuning.

  5. Nonlinearity-Aware LoRA: Structured Gate Adaptation under Low-Rank Constraints

    cs.LG 2026-06 unverdicted novelty 6.0

    NA-LoRA introduces derivative-based temporal-importance masks and activation-specific step scaling to LoRA to reduce selection misalignment in self-gated FFNs, with reported gains on language and vision-language fine-...

  6. FedSmoothLoRA: Toward Smoother and Faster Convergence in Federated Low-Rank Adaptation

    cs.CV 2026-05 unverdicted novelty 6.0

    FedSmoothLoRA improves federated LoRA fine-tuning by constructing local initializations from a round-matching matrix for cross-round continuity and a gradient-aligned matrix for client-specific guidance, yielding fast...

  7. Rotation-Preserving Supervised Fine-Tuning

    cs.LG 2026-05 unverdicted novelty 6.0

    RPSFT improves the in-domain versus out-of-domain performance trade-off during LLM supervised fine-tuning by penalizing rotations in pretrained singular subspaces as a proxy for loss-sensitive directions.

  8. Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models

    cs.LG 2026-01 reject novelty 6.0

    SALR combines static pruning of frozen weights with a trainable truncated-SVD low-rank residual adapter to match LoRA accuracy at 50% sparsity, cutting model size ~2x and giving ~1.7x inference speedup.

  9. Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Experts

    cs.LG 2025-06 unverdicted novelty 6.0

    MoRAM frames continual learning as incremental addition of rank-1 adapters viewed as self-activating key-value associative memory units in a mixture-of-experts setup.

  10. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    cs.LG 2026-06 conditional novelty 5.0

    A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.

  11. On the Scaling of PEFT: Towards Million Personal Models of Trillion Parameters

    cs.LG 2026-06 unverdicted novelty 5.0

    PEFT adapters are positioned as persistent personal state on foundation models, organized via Scale Up, Scale Down, and Scale Out axes, with MinT as an infrastructure example for managing them.

  12. Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgetting

    cs.CL 2026-05 unverdicted novelty 5.0

    A plug-and-play KL regularizer that masks the target token and renormalizes probabilities to improve the learning-forgetting trade-off in LoRA adaptation of LLMs.

  13. When Top-1 Fails: Calibrating LoRA Monitors for Masked Diffusion LMs

    cs.LG 2026-06 unverdicted novelty 4.0

    Empirical test shows top-1 argmax concentration has zero precision as collapse warning in DLM LoRA training due to pre-equilibrium saturation while max gradient norm provides usable but family-specific detection on sh...

  14. LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models

    cs.CL 2024-03 unverdicted novelty 4.0

    LlamaFactory provides a unified no-code framework for efficient fine-tuning of 100+ LLMs via an integrated web UI and has been released on GitHub.

  15. TLoRA+: A Low-Rank Parameter-Efficient Fine-Tuning Method for Large Language Models

    cs.CL 2026-04 unverdicted novelty 3.0

    TLoRA+ augments LoRA with a dedicated optimizer to improve fine-tuning performance on GLUE tasks without meaningful added compute.