REVIEW 2 cited by
LoRA Training in the NTK Regime has No Spurious Local Minima
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Low-rank adaptation (LoRA) has become the standard approach for parameter-efficient fine-tuning of large language models (LLM), but our theoretical understanding of LoRA has been limited. In this work, we theoretically analyze LoRA fine-tuning in the neural tangent kernel (NTK) regime with $N$ data points, showing: (i) full fine-tuning (without LoRA) admits a low-rank solution of rank $r\lesssim \sqrt{N}$; (ii) using LoRA with rank $r\gtrsim \sqrt{N}$ eliminates spurious local minima, allowing gradient descent to find the low-rank solutions; (iii) the low-rank solution found using LoRA generalizes well.
Forward citations
Cited by 2 Pith papers
-
When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index models
In a Gaussian single-index model with one-pass SGD, the LoRA escape time scales as τ(μ) log d / 2, where τ(μ) increases with pre-training strength μ and diverges for odd Hermite activations at a critical μ.
-
FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregation
FedHL aggregates heterogeneous LoRA updates against a full-rank global baseline and claims O(1/sqrt T) convergence, with small gains on three LLM fine-tuning datasets.
Discussion (0). Continue with ORCID to comment.