REVIEW 8 cited by
A Survey on LoRA of Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Low-Rank Adaptation~(LoRA), which updates the dense neural network layers with pluggable low-rank matrices, is one of the best performed parameter efficient fine-tuning paradigms. Furthermore, it has significant advantages in cross-task generalization and privacy-preserving. Hence, LoRA has gained much attention recently, and the number of related literature demonstrates exponential growth. It is necessary to conduct a comprehensive overview of the current progress on LoRA. This survey categorizes and reviews the progress from the perspectives of (1) downstream adaptation improving variants that improve LoRA's performance on downstream tasks; (2) cross-task generalization methods that mix multiple LoRA plugins to achieve cross-task generalization; (3) efficiency-improving methods that boost the computation-efficiency of LoRA; (4) data privacy-preserving methods that use LoRA in federated learning; (5) application. Besides, this survey also discusses the future directions in this field. At last, we provide a Github page~\footnote{\href{https://github.com/ZJU-LLMs/Awesome-LoRAs.git}{https://github.com/ZJU-LLMs/Awesome-LoRAs.git}} for readers to check the updates and initiate discussions on this survey paper.
Forward citations
Cited by 8 Pith papers
-
LobRA: Multi-tenant Fine-tuning over Heterogeneous Data
LobRA reduces GPU seconds for multi-tenant LoRA fine-tuning by 45.03%-60.67% through heterogeneous FT replicas and per-step workload-balanced dispatching.
-
FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregation
FedHL aggregates heterogeneous LoRA updates against a full-rank global baseline and claims O(1/sqrt T) convergence, with small gains on three LLM fine-tuning datasets.
-
Sparse Gradient Compression for Fine-Tuning Large Language Models
SGC compresses LLM optimizer states into a low-dimensional subspace via top-k gradient sparsification and OMP recovery, claiming comparable fine-tuning accuracy with fewer optimizer states.
-
Reassessing Layer Pruning in LLMs: New Insights and Methods
Trimming the final 25% of layers and fine-tuning the head and last three layers outperforms sophisticated pruning metrics and LoRA-based recovery for LLM compression.
-
Does Low Rank Adaptation Lead to Lower Robustness against Training-Time Attacks?
LoRA fine-tuning is more robust than full fine-tuning against backdoor attacks but more vulnerable to untargeted label-flipping poisoning, based on BERT/GLUE experiments and NTK-information-geometry analysis.
-
Low-rank Adaptation-based All-Weather Removal for Autonomous Navigation
LoRA and a new SVD-aligned variant, LoRA-A, adapt pre-trained all-weather restoration models to unseen degradations while retaining performance on original tasks.
-
The Scaling Law for LoRA Base on Mutual Information Upper Bound
The claimed mutual information upper bound for LoRA scaling laws is invalid because the key inequality in the proof is false.
-
Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directions
LLM robustness research is organized into adversarial robustness, out-of-distribution robustness, and evaluation, with an accompanying GitHub collection of papers.
Discussion (0). Continue with ORCID to comment.