Pith. sign in

REVIEW 3 cited by

The Impact of Initialization on LoRA Finetuning Dynamics

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.08447 v1 pith:2LPXYQG7 submitted 2024-06-12 cs.LG cs.AIcs.CLstat.ML

Soufiane Hayou , Nikhil Ghosh , Bin Yu This is my paper · ORCID
classification cs.LGcs.AIcs.CLstat.ML
keywords initializationfinetuningfirstlearningschemezerocomparedlora
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In this paper, we study the role of initialization in Low Rank Adaptation (LoRA) as originally introduced in Hu et al. (2021). Essentially, to start from the pretrained model as initialization for finetuning, one can either initialize B to zero and A to random (default initialization in PEFT package), or vice-versa. In both cases, the product BA is equal to zero at initialization, which makes finetuning starts from the pretrained model. These two initialization schemes are seemingly similar. They should in-principle yield the same performance and share the same optimal learning rate. We demonstrate that this is an incorrect intuition and that the first scheme (initializing B to zero and A to random) on average yields better performance compared to the other scheme. Our theoretical analysis shows that the reason behind this might be that the first initialization allows the use of larger learning rates (without causing output instability) compared to the second initialization, resulting in more efficient learning of the first scheme. We validate our results with extensive experiments on LLMs.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HRP: High-Rank Preheating for Superior LoRA Initialization

    cs.LG 2025-02 conditional novelty 6.0 of 10

    HRP initializes LoRA with the top singular vectors of a briefly preheated high-rank adapter, improving fine-tuning results over random initialization in experiments.

  2. TuneComp: Joint Fine-tuning and Compression for Large Foundation Models

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Jointly fine-tuning and compressing a ViT into pruned low-rank factors with progressive distillation achieves better accuracy for smaller parameter counts than sequential fine-tune-then-compress pipelines on CIFAR-100.

  3. CoLA: Collaborative Low-Rank Adaptation

    cs.CL 2025-05 conditional novelty 4.0 of 10

    CoLA generalizes LoRA to multiple A and B matrices with a principal-component initialization and reports gains of roughly 2-4 accuracy points over PiSSA on low-sample fine-tuning benchmarks.

Pith tools