REVIEW 13 cited by
Asymmetry in Low-Rank Adapters of Foundation Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Parameter-efficient fine-tuning optimizes large, pre-trained foundation models by updating a subset of parameters; in this class, Low-Rank Adaptation (LoRA) is particularly effective. Inspired by an effort to investigate the different roles of LoRA matrices during fine-tuning, this paper characterizes and leverages unexpected asymmetry in the importance of low-rank adapter matrices. Specifically, when updating the parameter matrices of a neural network by adding a product $BA$, we observe that the $B$ and $A$ matrices have distinct functions: $A$ extracts features from the input, while $B$ uses these features to create the desired output. Based on this observation, we demonstrate that fine-tuning $B$ is inherently more effective than fine-tuning $A$, and that a random untrained $A$ should perform nearly as well as a fine-tuned one. Using an information-theoretic lens, we also bound the generalization of low-rank adapters, showing that the parameter savings of exclusively training $B$ improves the bound. We support our conclusions with experiments on RoBERTa, BART-Large, LLaMA-2, and ViTs.
Forward citations
Cited by 13 Pith papers
-
Noise Consistency Regularization for Improved Subject-Driven Image Synthesis
Adding consistency-to-pretrained and multiplicative-noise consistency losses to fine-tuning improves subject identity and background diversity over DreamBooth on a 30-subject benchmark.
-
CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning
CL-LoRA adds a fixed random-orthogonal shared LoRA branch for cross-task knowledge and task-specific LoRA branches with block-wise weights, improving rehearsal-free class-incremental learning accuracy at low parameter cost.
-
DeCAF: Decentralized Consensus-And-Factorization for Low-Rank Adaptation of Foundation Models
A truncated-SVD consensus step for decentralized LoRA is claimed to reach O(1/sqrt T) convergence, matching decentralized SGD, with supporting CLIP and LLAMA2-7B experiments.
-
HRP: High-Rank Preheating for Superior LoRA Initialization
HRP initializes LoRA with the top singular vectors of a briefly preheated high-rank adapter, improving fine-tuning results over random initialization in experiments.
-
Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection
Finetuning forgetting follows a multiplicative scaling law in model size, finetuning tokens, and injected pretraining fraction, with 1% injection nearly eliminating forgetting.
-
MakeAnything: Harnessing Diffusion Transformers for Multi-Domain Procedural Sequence Generation
Fine-tuning a diffusion transformer with asymmetric LoRA plus a new 24,000-sequence dataset enables multi-domain, step-by-step procedural generation and image-to-process reconstruction.
-
S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity
S2FT selects a few attention heads and FFN channels, permutes the neighboring weight matrices so the selected parts form dense blocks, and fine-tunes only those blocks, reporting better generalization and efficiency t...
-
Continual Knowledge Consolidation LORA for Domain Incremental Learning
CONEC-LoRA reports state-of-the-art accuracy on four domain-incremental benchmarks by combining task-shared and task-specific LoRAs with a stochastic classifier and a learned domain-ID selector.
-
Theoretical Foundations of Communication-Efficient, Robust, and Practical Distributed and Federated Optimization
A thesis proving communication-acceleration guarantees for local-step, compressed, Byzantine-robust, and low-rank federated optimization methods, assembled from the author's own published papers.
-
CoLA: Collaborative Low-Rank Adaptation
CoLA generalizes LoRA to multiple A and B matrices with a principal-component initialization and reports gains of roughly 2-4 accuracy points over PiSSA on low-sample fine-tuning benchmarks.
-
Rate-My-LoRA: Efficient and Adaptive Federated Model Tuning for Cardiac MRI Segmentation
An adaptive LoRA-aggregation method for federated cardiac MRI segmentation improves Dice scores and cuts bandwidth compared to full-weight FedAvg and FedPETuning.
-
BoRA: Bi-dimensional Weight-Decomposed Low-Rank Adaptation
BoRA extends DoRA by adding a row-magnitude vector alongside the column-magnitude vector, reporting improved MT-Bench and commonsense reasoning scores over LoRA and DoRA on three LLMs.
-
LoRA-Mini : Adaptation Matrices Decomposition and Selective Training
Decomposing each LoRA matrix into frozen and trainable halves and training only the inner matrices cuts trainable parameters by up to roughly 20x while keeping GLUE and WMT16 scores near LoRA.
Discussion (0). Continue with ORCID to comment.