REVIEW 2 cited by
On the Crucial Role of Initialization for Matrix Factorization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
This work revisits the classical low-rank matrix factorization problem and unveils the critical role of initialization in shaping convergence rates for such nonconvex and nonsmooth optimization. We introduce Nystrom initialization, which significantly improves the global convergence of Scaled Gradient Descent (ScaledGD) in both symmetric and asymmetric matrix factorization tasks. Specifically, we prove that ScaledGD with Nystrom initialization achieves quadratic convergence in cases where only linear rates were previously known. Furthermore, we extend this initialization to low-rank adapters (LoRA) commonly used for finetuning foundation models. Our approach, NoRA, i.e., LoRA with Nystrom initialization, demonstrates superior performance across various downstream tasks and model scales, from 1B to 7B parameters, in large language and diffusion models.
Forward citations
Cited by 2 Pith papers
-
HRP: High-Rank Preheating for Superior LoRA Initialization
HRP initializes LoRA with the top singular vectors of a briefly preheated high-rank adapter, improving fine-tuning results over random initialization in experiments.
-
TuneComp: Joint Fine-tuning and Compression for Large Foundation Models
Jointly fine-tuning and compressing a ViT into pruned low-rank factors with progressive distillation achieves better accuracy for smaller parameter counts than sequential fine-tune-then-compress pipelines on CIFAR-100.
Discussion (0). Continue with ORCID to comment.