Pith. sign in

REVIEW 9 cited by

DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2210.07558 v2 pith:AC6QXDGZ submitted 2022-10-14 cs.CL cs.LG

Mojtaba Valipour , Mehdi Rezagholizadeh , Ivan Kobyzev , Ali Ghodsi This is my paper
classification cs.CLcs.LG
keywords lorablocksmodelsdifferentdyloradynamiclow-rankpretrained
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

With the ever-growing size of pretrained models (PMs), fine-tuning them has become more expensive and resource-hungry. As a remedy, low-rank adapters (LoRA) keep the main pretrained weights of the model frozen and just introduce some learnable truncated SVD modules (so-called LoRA blocks) to the model. While LoRA blocks are parameter-efficient, they suffer from two major problems: first, the size of these blocks is fixed and cannot be modified after training (for example, if we need to change the rank of LoRA blocks, then we need to re-train them from scratch); second, optimizing their rank requires an exhaustive search and effort. In this work, we introduce a dynamic low-rank adaptation (DyLoRA) technique to address these two problems together. Our DyLoRA method trains LoRA blocks for a range of ranks instead of a single rank by sorting the representation learned by the adapter module at different ranks during training. We evaluate our solution on different natural language understanding (GLUE benchmark) and language generation tasks (E2E, DART and WebNLG) using different pretrained models such as RoBERTa and GPT with different sizes. Our results show that we can train dynamic search-free models with DyLoRA at least 4 to 7 times (depending to the task) faster than LoRA without significantly compromising performance. Moreover, our models can perform consistently well on a much larger range of ranks compared to LoRA.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MoKA: Mixture of Kronecker Adapters

    cs.LG 2025-08 conditional novelty 6.0 of 10

    MoKA models weight updates as a gated mixture of Kronecker products and reports higher accuracy with up to 27x fewer parameters than QLoRA on 4-bit LLaMA models.

  2. ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints

    cs.CV 2025-07 conditional novelty 6.0 of 10

    LoRA adapters can be initialized with a closed-form estimate derived from constraint sets linking source and target activations, improving fine-tuning speed and accuracy.

  3. Beyond Low-Rank Tuning: Model Prior-Guided Rank Allocation for Effective Transfer in Low-Data and Large-Gap Regimes

    cs.CV 2025-06 conditional novelty 6.0 of 10

    SR-LoRA sets each LoRA layer's rank to the stable rank of that layer's pretrained weight matrix, improving few-shot transfer on large domain gaps without rank search.

  4. LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning

    cs.LG 2026-07 conditional novelty 5.0 of 10

    LAARA allocates LoRA ranks per layer from diagonal Fisher (gradient-based) estimates, reporting improved accuracy with fewer trainable parameters on GLUE and MathInstruct.

  5. A Language-Guided Bayesian Optimization for Efficient LoRA Hyperparameter Search

    cs.CL 2026-01 conditional novelty 5.0 of 10

    LLM embeddings plus Bayesian optimization find better LoRA hyperparameters in ~30 proxy trials than standard published settings.

  6. Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Models

    cs.LG 2025-09 conditional novelty 5.0 of 10

    Sensitivity-LoRA allocates LoRA ranks across layers using Hessian-based sensitivity metrics, improving average GLUE score by 0.74 over AdaLoRA on RoBERTa-base.

  7. Language-Aware Information Maximization for Transductive Few-Shot CLIP

    cs.CV 2025-08 conditional novelty 5.0 of 10

    LIMO, a transductive loss combining mutual information, zero-shot KL regularization, and LoRA, sets new state-of-the-art few-shot accuracy for CLIP on 11 datasets.

  8. Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergence

    cs.LG 2025-06 conditional novelty 5.0 of 10

    CorDA++ uses data-driven SVD to initialize LoRA adapters, adding per-layer covariance selection and rank allocation that reduce forgetting and speed convergence compared to LoRA, PiSSA, QLoRA, and other baselines.

  9. Find Before You Fine-Tune: A Diagnostic Study of Small LLMs for Cybersecurity QA

    cs.CL 2026-07 conditional novelty 4.0 of 10

    Pre-fine-tuning scores on a three-task diagnostic can predict the direction of post-fine-tuning change in small LLMs for cybersecurity QA, but not the magnitude or rank-preservation, which is regime-dependent.

Pith tools