REVIEW 7 cited by
AutoLoRA: Automatically Tuning Matrix Ranks in Low-Rank Adaptation Based on Meta Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large-scale pretraining followed by task-specific finetuning has achieved great success in various NLP tasks. Since finetuning all parameters of large pretrained models poses substantial computational and memory challenges, several efficient finetuning methods have been developed. Among them, low-rank adaptation (LoRA), which finetunes low-rank incremental update matrices on top of frozen pretrained weights, has proven particularly effective. Nonetheless, LoRA's uniform rank assignment across all layers, along with its reliance on an exhaustive search to find the best rank, leads to high computation costs and suboptimal finetuning performance. To address these limitations, we introduce AutoLoRA, a meta learning based framework for automatically identifying the optimal rank of each LoRA layer. AutoLoRA associates each rank-1 matrix in a low-rank update matrix with a selection variable, which determines whether the rank-1 matrix should be discarded. A meta learning based method is developed to learn these selection variables. The optimal rank is determined by thresholding the values of these variables. Our comprehensive experiments on natural language understanding, generation, and sequence labeling demonstrate the effectiveness of AutoLoRA.
Forward citations
Cited by 7 Pith papers
-
DynImmune-BERT: Dynamic Immune Repertoire Modeling with Neural ODE Driven Continuous Transformers
DynImmune-BERT shows that event-aware continuous-time modeling of longitudinal TCR repertoires improves cancer-status AUC over static and simpler temporal baselines, but external validation is limited by small cohorts.
-
Improved Representation Steering for Language Models
RePS, a reference-free bidirectional preference optimization objective, improves representation steering and suppression for Gemma models, outperforming language-modeling objectives and approaching prompting performance.
-
LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning
LAARA allocates LoRA ranks per layer from diagonal Fisher (gradient-based) estimates, reporting improved accuracy with fewer trainable parameters on GLUE and MathInstruct.
-
A Language-Guided Bayesian Optimization for Efficient LoRA Hyperparameter Search
LLM embeddings plus Bayesian optimization find better LoRA hyperparameters in ~30 proxy trials than standard published settings.
-
Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Models
Sensitivity-LoRA allocates LoRA ranks across layers using Hessian-based sensitivity metrics, improving average GLUE score by 0.74 over AdaLoRA on RoBERTa-base.
-
Regularizing Subspace Redundancy of Low-Rank Adaptation
ReSoRA adds a penalty that reduces redundancy among rank-1 subspaces of LoRA-style adapters, producing modest accuracy improvements on vision-language retrieval and visual classification.
-
The Future of Continual Learning in the Era of Foundation Models: Three Key Directions
Continual learning should pivot from weight-update-based methods to continual compositionality and orchestration of foundation models and agents.
Discussion (0). Continue with ORCID to comment.