REVIEW 5 cited by
The Expressive Power of Low-Rank Adaptation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method that leverages low-rank adaptation of weight matrices, has emerged as a prevalent technique for fine-tuning pre-trained models such as large language models and diffusion models. Despite its huge success in practice, the theoretical underpinnings of LoRA have largely remained unexplored. This paper takes the first step to bridge this gap by theoretically analyzing the expressive power of LoRA. We prove that, for fully connected neural networks, LoRA can adapt any model $f$ to accurately represent any smaller target model $\overline{f}$ if LoRA-rank $\geq(\text{width of }f) \times \frac{\text{depth of }\overline{f}}{\text{depth of }f}$. We also quantify the approximation error when LoRA-rank is lower than the threshold. For Transformer networks, we show any model can be adapted to a target model of the same size with rank-$(\frac{\text{embedding size}}{2})$ LoRA adapters.
Forward citations
Cited by 5 Pith papers
-
The impact of allocation strategies in subset learning on the expressive power of neural networks
In a teacher-student setup, maximal expressive power for a fixed learnable-weight budget is characterized by even row or column distribution in linear RNNs and feedforward networks.
-
HRP: High-Rank Preheating for Superior LoRA Initialization
HRP initializes LoRA with the top singular vectors of a briefly preheated high-rank adapter, improving fine-tuning results over random initialization in experiments.
-
Tight Sample Complexity for Low-Rank Adaptation: Matching Bounds and Rank Selection
LoRA fine-tuning has Θ~(rd/n) sample complexity, and over-ranking strictly hurts unregularized empirical risk minimization while being harmless for nuclear-norm-style adaptive estimators.
-
MoLEx: Mixture of LoRA Experts in Speech Self-Supervised Models for Audio Deepfake Detection
MoLEx combines LoRA adapters with a top-K expert router inside a frozen WavLM model, achieving 5.56% EER on ASVSpoof 5 without augmentation.
-
Probabilistic Forecasting for Building Energy Systems using Time-Series Foundation Models
Fine-tuned time-series foundation models, especially Chronos with LoRA, outperform trained-from-scratch deep forecasters on multi-signal building energy forecasting with limited data.
Discussion (0). Continue with ORCID to comment.