REVIEW 18 cited by
VeRA: Vector-based Random Matrix Adaptation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
VeRA: Vector-based Random Matrix Adaptation
read the original abstract
Low-rank adapation (LoRA) is a popular method that reduces the number of trainable parameters when finetuning large language models, but still faces acute storage challenges when scaling to even larger models or deploying numerous per-user or per-task adapted models. In this work, we present Vector-based Random Matrix Adaptation (VeRA), which significantly reduces the number of trainable parameters compared to LoRA, yet maintains the same performance. It achieves this by using a single pair of low-rank matrices shared across all layers and learning small scaling vectors instead. We demonstrate its effectiveness on the GLUE and E2E benchmarks, image classification tasks, and show its application in instruction-tuning of 7B and 13B language models.
Forward citations
Cited by 18 Pith papers
-
BoostLoRA: Growing Effective Rank by Boosting Adapters
BoostLoRA grows effective adapter rank linearly via iterative boosting on hard examples with orthogonal low-rank updates, outperforming both single-shot ultra-low-rank adapters and full fine-tuning on math and code ta...
-
TerraNova: A Foundation Model for the Anthropocene
A multi-geometry model, trained on 512 gridded Earth-system fields and 512 national indicators, couples physical and societal data in one latent space and supports uncertainty-aware reconstruction and downscaling.
-
RSRA: Training-Free Probing of Representation Sensitivity for Efficient LoRA Rank Allocation
Training-free LoRA rank allocation via virtual low-rank noise probing that scores modules by Fréchet distance and effective rank of activation shifts outperforms prior heuristic and training-based allocators on standa...
-
RSRA: Training-Free Probing of Representation Sensitivity for Efficient LoRA Rank Allocation
RSRA, a forward-only rank allocator combining spectral effective rank with Fréchet-distance sensitivity probing, reports higher average LoRA accuracy on three commonsense/GLUE settings with lower pre-allocation cost.
-
ARIADNE: Agnostic Routing for Inference-time Adapter DyNamic sElection
ARIADNE routes queries to the best adapter via embedding-space centroid proximity, recovering 97.44% of upper-bound performance on 23 NLP tasks and 89.7% selection accuracy on 44 tasks without training or internal access.
-
TLoRA: Task-aware Low Rank Adaptation of Large Language Models
TLoRA jointly optimizes LoRA initialization via task-data SVD and sensitivity-driven rank allocation, delivering stronger results than standard LoRA across NLU, reasoning, math, code, and chat tasks while using fewer ...
-
NP-LoRA: Null Space Projection for Subject-Style LoRA Fusion
NP-LoRA fuses subject and style LoRAs via null-space projection of the content update onto the orthogonal complement of the style subspace, with a soft variant controlled by one parameter.
-
HyperAdapt: Simple High-Rank Adaptation
HyperAdapt performs parameter-efficient fine-tuning by row- and column-wise diagonal scaling to induce high-rank updates with only n+m trainable parameters.
-
Preference Goal Tuning: Post-Training as Latent Control for Frozen Policies
PGT optimizes latent goal embeddings for frozen policies via trajectory-level preference objectives, reporting 72-81.6% relative gains on 17 Minecraft tasks and 13.4% better OOD performance than fine-tuning.
-
Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning
A retraction-free Stiefel manifold optimization algorithm with a fixed penalty parameter is proposed and applied to LoRA fine-tuning, claiming faster convergence and better downstream performance.
-
Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning
Applying mHC as a PEFT method shows that learned residual mixing is unnecessary — even harmful — in finetuning, and mHC+LoRA combinations give small task-dependent gains.
-
Mixture-of-Control: State-Aware Fine-Tuning for Transformer-based Models
Mixture-of-Control adaptively combines local and global control states in transformer fine-tuning by treating per-block states as experts in a sparse MoE setup to improve cross-block communication while keeping memory...
-
BaRA: Bayesian Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning
BaRA adds Bayesian adaptive rank allocation to LoRA fine-tuning by activating sparse instance-specific latent factors, with a generalization bound depending on learned joint effective rank rather than fixed maximum rank.
-
Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgetting
A plug-and-play KL regularizer that masks the target token and renormalizes probabilities to improve the learning-forgetting trade-off in LoRA adaptation of LLMs.
-
SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning
SMoA is a new PEFT adapter that uses block-wise Hadamard-modulated low-rank branches on spectral partitions to cover more pretrained spectral directions than standard LoRA under a smaller parameter budget.
-
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
A comprehensive survey of PEFT algorithms for large models, covering their performance, overhead, applications, and real-world system implementations.
-
The Hitchhiker's Guide to Agentic AI: From Foundations to Systems
A comprehensive reference book organizing existing techniques for agentic AI systems across LLM substrate, reasoning, agent design patterns, inter-agent coordination, and production deployment.
-
The Hitchhiker's Guide to Agentic AI: From Foundations to Systems
A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.