Pith. sign in

REVIEW 59 cited by

Towards a Unified View of Parameter-Efficient Transfer Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2110.04366 v3 pith:EYCV5NY6 submitted 2021-10-08 cs.CL cs.LG

Junxian He , Chunting Zhou , Xuezhe Ma , Taylor Berg-Kirkpatrick , Graham Neubig This is my paper · ORCID
classification cs.CLcs.LG
keywords methodsdesignlearningparameter-efficientparameterstransferunifiedfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Fine-tuning large pre-trained language models on downstream tasks has become the de-facto learning paradigm in NLP. However, conventional approaches fine-tune all the parameters of the pre-trained model, which becomes prohibitive as the model size and the number of tasks grow. Recent work has proposed a variety of parameter-efficient transfer learning methods that only fine-tune a small number of (extra) parameters to attain strong performance. While effective, the critical ingredients for success and the connections among the various methods are poorly understood. In this paper, we break down the design of state-of-the-art parameter-efficient transfer learning methods and present a unified framework that establishes connections between them. Specifically, we re-frame them as modifications to specific hidden states in pre-trained models, and define a set of design dimensions along which different methods vary, such as the function to compute the modification and the position to apply the modification. Through comprehensive empirical studies across machine translation, text summarization, language understanding, and text classification benchmarks, we utilize the unified view to identify important design choices in previous methods. Furthermore, our unified framework enables the transfer of design elements across different approaches, and as a result we are able to instantiate new parameter-efficient fine-tuning methods that tune less parameters than previous methods while being more effective, achieving comparable results to fine-tuning all parameters on all four tasks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 59 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LoRA-DA: Data-Aware Initialization for Low-Rank Adaptation via Asymptotic Analysis

    cs.LG 2025-10 reject novelty 7.0 of 10

    LoRA-DA initializes LoRA's low-rank subspace with the smallest eigenvectors of a guidance matrix built from Fisher information and a Fisher-gradient estimate of the target displacement.

  2. Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting

    cs.CV 2025-08 unverdicted novelty 7.0 of 10

    The paper offers a comprehensive survey and proposes a new taxonomy for continual learning strategies in VLMs and MLLMs to combat catastrophic forgetting beyond traditional methods.

  3. GAPrompt: Geometry-Aware Point Cloud Prompt for 3D Vision Model

    cs.CV 2025-05 conditional novelty 7.0 of 10

    A geometry-aware prompting method reaches competitive 3D point cloud classification accuracy with roughly 2% of full fine-tuning parameters.

  4. Test-Time Scaling via Error Localization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    TTEL uses feedback-induced token probability drops to localize the first error in a failed reasoning trace and branch a new generation from that prefix, improving pass@k per token on coding and math benchmarks.

  5. RSRA: Training-Free Probing of Representation Sensitivity for Efficient LoRA Rank Allocation

    cs.CV 2026-07 reject novelty 6.0 of 10

    Training-free LoRA rank allocation via virtual low-rank noise probing that scores modules by Fréchet distance and effective rank of activation shifts outperforms prior heuristic and training-based allocators on standa...

  6. CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging

    cs.CL 2026-02 conditional novelty 6.0 of 10

    CoMoL represents every LoRA expert as a shared-basis core matrix and merges token-selected experts in that core space, reaching standard LoRA parameter counts while outperforming MoE-LoRA baselines on math and code.

  7. CeRA: Breaking the Linear Ceiling of Low-Rank Adaptation with Non-linearity Retained at Inference

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    CeRA overcomes LoRA's linear ceiling by injecting non-linear SiLU gating and dropout, outperforming high-rank LoRA on complex math reasoning with 1/8 the parameters.

  8. PLATE: Plasticity-Tunable Efficient Adapters for Geometry-Aware Continual Learning

    cs.LG 2026-02 conditional novelty 6.0 of 10

    PLATE constructs frozen weight-derived bases B and Q and trains only a small core A, reducing catastrophic forgetting during data-free continual fine-tuning.

  9. MHA-RAG: Improving Efficiency, Accuracy, and Consistency by Encoding Exemplars as Soft Prompts

    cs.AI 2025-10 conditional novelty 6.0 of 10

    MHA-RAG encodes retrieved exemplars into order-invariant soft prompts via multi-head attention, claiming ~20-point effective-accuracy gains over RAG at ~10x lower inference FLOPs.

  10. PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    PLoP selects LoRA adapter placement by ranking normalized feature norms and placing adapters on the lowest-scoring module types, using only forward passes.

  11. Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helps

    cs.LG 2025-06 conditional novelty 6.0 of 10

    SeLoRA reparameterizes LoRA updates as inverse Fourier or wavelet transforms of sparsely masked spectral coefficients, improving fine-tuning accuracy on LLaMA models with fewer trainable parameters.

  12. Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    ALTA adapts a frozen masked-pretrained X-ray encoder to language with 8% trainable parameters and temporal-multiview inputs, improving medical retrieval and zero-shot classification.

  13. GaRA-SAM: Robustifying Segment Anything Model with Gated-Rank Adaptation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    GaRA-SAM improves SAM's robustness to image corruption by using input-dependent gating to adjust the effective rank of low-rank adapters, beating prior methods on robust segmentation benchmarks.

  14. iDPA: Instance Decoupled Prompt Attention for Incremental Medical Object Detection

    cs.CV 2025-05 conditional novelty 6.0 of 10

    iDPA improves incremental medical object detection by generating instance-level prompts from bounding-box regions and decoupling prompt attention in a frozen GLIP model.

  15. Improved Representation Steering for Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    RePS, a reference-free bidirectional preference optimization objective, improves representation steering and suppression for Gemma models, outperforming language-modeling objectives and approaching prompting performance.

  16. SPECI: Skill Prompts based Hierarchical Continual Imitation Learning for Robot Manipulation

    cs.RO 2025-04 conditional novelty 6.0 of 10

    A hierarchical continual imitation learning policy with an expandable skill codebook and CP-decomposed task-specific attention parameters outperforms prior CIL methods on the LIBERO robot manipulation benchmark.

  17. Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Finetuning forgetting follows a multiplicative scaling law in model size, finetuning tokens, and injected pretraining fraction, with 1% injection nearly eliminating forgetting.

  18. Sparse Gradient Compression for Fine-Tuning Large Language Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    SGC compresses LLM optimizer states into a low-dimensional subspace via top-k gradient sparsification and OMP recovery, claiming comparable fine-tuning accuracy with fewer optimizer states.

  19. Algorithm for Automatic Legislative Text Consolidation

    cs.CL 2025-01 conditional novelty 6.0 of 10

    A LoRA-fine-tuned 13B language model can automatically consolidate French legislative texts, outperforming a span-extraction baseline and approaching GPT-4 on a subset of a real finance bill.

  20. Decentralized Low-Rank Fine-Tuning of Large Language Models

    cs.LG 2025-01 conditional novelty 6.0 of 10

    Decentralized LoRA fine-tuning with gossip averaging converges to a stationary point at O(1/sqrt(T)) and matches centralized LoRA accuracy on several NLP tasks.

  21. FlexiGPT: Pruning and Extending Large Language Models with Low-Rank Weight Sharing

    cs.CL 2025-01 conditional novelty 6.0 of 10

    A block-pruning and model-extension method for LLMs that replaces pruned blocks with weight-shared blocks plus low-rank adapters, reporting state-of-the-art recovery on several benchmarks.

  22. TADFormer : Task-Adaptive Dynamic Transformer for Efficient Multi-Task Learning

    cs.CV 2025-01 conditional novelty 6.0 of 10

    A prompt-and-dynamic-filter PEFT design for multi-task dense prediction beats MTLoRA on PASCAL-Context with fewer trainable parameters.

  23. CM3T: Framework for Efficient Multimodal Learning for Inhomogeneous Interaction Datasets

    cs.CV 2025-01 conditional novelty 6.0 of 10

    CM3T shows that multi-head vision adapters plus cross-attention adapters can adapt frozen supervised-pretrained video transformers with a fraction of the trainable parameters of full fine-tuning.

  24. All-in-One Tuning and Structural Pruning for Domain-Specific LLMs

    cs.CL 2024-12 conditional novelty 6.0 of 10

    ATP jointly searches for pruning decisions and fine-tunes LLaMA models with LoRA in one stage, outperforming two-stage pruning on domain-specific tasks.

  25. S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity

    cs.LG 2024-12 conditional novelty 6.0 of 10

    S2FT selects a few attention heads and FFN channels, permutes the neighboring weight matrices so the selected parts form dense blocks, and fine-tunes only those blocks, reporting better generalization and efficiency t...

  26. Unlocking Tuning-Free Few-Shot Adaptability in Visual Foundation Models by Recycling Pre-Tuned LoRAs

    cs.CV 2024-12 conditional novelty 6.0 of 10

    LoRA Recycle distills a meta-LoRA from diverse pre-tuned LoRAs using data generated by inverting those LoRAs, enabling tuning-free few-shot classification at test time.

  27. Safety Alignment Backfires: Preventing the Re-emergence of Suppressed Concepts in Fine-tuned Text-to-Image Diffusion Models

    cs.AI 2024-11 conditional novelty 6.0 of 10

    Fine-tuning text-to-image diffusion models on benign data can reactivate suppressed unsafe concepts, and training the task adapter separately from a frozen safety LoRA prevents this.

  28. SimCMF: A Simple Cross-modal Fine-tuning Strategy from Vision Foundation Models to Any Imaging Modality

    cs.CV 2024-11 conditional novelty 6.0 of 10

    A small convolutional adapter plus a frozen patch embedding lets SAM segment depth, thermal, polarization, HHA, and NIR images far better than training from scratch, with parameter-efficient fine-tuning matching full ...

  29. AI-driven Remote Facial Skin Hydration and TEWL Assessment from Selfie Images: A Systematic Solution

    cs.CV 2025-09 conditional novelty 5.0 of 10

    Selfie images can be used to predict skin hydration and water loss with R2 up to about 0.35, using a new dataset of 336 panelists and an adapter-based vision transformer.

  30. PrunePEFT: Iterative Hybrid Pruning for Parameter-Efficient Fine-tuning of LLMs

    cs.LG 2025-06 conditional novelty 5.0 of 10

    An iterative hybrid pruning method selects which PEFT modules to keep at each transformer layer, matching or improving fixed PEFT baselines on GLUE at 1% trainable parameters.

  31. A Systematic Review of Poisoning Attacks Against Large Language Models

    cs.CR 2025-06 conditional novelty 5.0 of 10

    A systematic review that organizes 65 LLM poisoning papers into a threat model with four attack specifications and generalized metrics.

  32. SenWiCh: Sense-Annotation of Low-Resource Languages for WiC using Hybrid Methods

    cs.CL 2025-05 conditional novelty 5.0 of 10

    The authors release sense-annotated WSD/WiC datasets for ten low-resource languages and report that English-based zero-shot transfer often beats small in-language fine-tuning, while mixed training usually helps.

  33. HOFT: Householder Orthogonal Fine-tuning

    cs.LG 2025-05 conditional novelty 5.0 of 10

    HOFT and SHOFT fine-tune foundation models with two Householder-built orthogonal matrices, matching or beating LoRA, DoRA, OFT, BOFT and HRA on reasoning, translation, image generation and math.

  34. Task-Adapter++: Task-specific Adaptation with Order-aware Alignment for Few-shot Action Recognition

    cs.CV 2025-05 conditional novelty 5.0 of 10

    Task-Adapter++ adapts frozen CLIP encoders with task-specific visual adapters and order-aware semantic adapters, and reports state-of-the-art results on five few-shot action recognition benchmarks.

  35. DiTASK: Multi-Task Fine-Tuning with Diffeomorphic Transformations

    cs.CV 2025-02 conditional novelty 5.0 of 10

    DiTASK fine-tunes vision transformers for multiple tasks by applying learnable diffeomorphic maps to the singular values of frozen weights, and reports state-of-the-art multi-task performance with fewer parameters.

  36. MPLinker: Multi-template Prompt-tuning with Adversarial Training for Issue-commit Link Recovery

    cs.SE 2025-01 conditional novelty 5.0 of 10

    MPLinker reframes issue-commit link recovery as a masked-language-model cloze task with multi-template averaging and adversarial training, reporting an average F1 of 96.10% on six projects.

  37. SAM-DA: Decoder Adapter for Efficient Medical Domain Adaptation

    cs.CV 2025-01 conditional novelty 5.0 of 10

    A zero-initialized attention adapter placed in SAM's mask decoder matches full fine-tuning on medical segmentation and improves cross-domain generalization with under 1% trainable parameters.

  38. DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models

    cs.CL 2024-12 conditional novelty 5.0 of 10

    Initializing tensor adapters from the MPO decomposition of pretrained weights boosts fine-tuning accuracy and parameter efficiency over random and SVD-based initialization in LLaMA2-7B and LLaMA3-8B.

  39. Theoretical Foundations of Communication-Efficient, Robust, and Practical Distributed and Federated Optimization

    cs.LG 2026-08 conditional novelty 4.0 of 10

    A thesis proving communication-acceleration guarantees for local-step, compressed, Byzantine-robust, and low-rank federated optimization methods, assembled from the author's own published papers.

  40. Capacity and Redundancy Trade-offs in Multi-Task Learning

    cs.LG 2026-07 conditional novelty 4.0 of 10

    A shared representation's total per-task information is bounded by capacity plus label redundancy; clustered sharing wins exactly when interference reduction exceeds the redundancy it loses.

  41. Weight Spectra Induced Efficient Model Adaptation

    cs.LG 2025-05 reject novelty 4.0 of 10

    Fine-tuning mostly amplifies and reorients the top singular directions of weight matrices, and SpecLoRA learns to rescale a top-left block plus LoRA to improve PEFT performance.

  42. MAP: Revisiting Weight Decomposition for Low-Rank Adaptation

    cs.LG 2025-05 conditional novelty 4.0 of 10

    MAP decouples a weight matrix's direction and magnitude by normalizing the whole matrix and the low-rank update by their Frobenius norms and scaling each with a learnable scalar.

  43. Cuff-KT: Tackling Learners' Real-time Learning Pattern Adjustment via Tuning-Free Knowledge State Guided Model Updating

    cs.LG 2025-05 conditional novelty 4.0 of 10

    Cuff-KT generates personalized output-layer parameters for knowledge tracing models without fine-tuning, reporting AUC improvements of about 10% and 4% under intra- and inter-learner shifts.

  44. Mixture of Low Rank Adaptation with Partial Parameter Sharing for Time Series Forecasting

    cs.LG 2025-05 conditional novelty 4.0 of 10

    MoLA adapts a pre-trained short-horizon forecaster to multiple forecast steps via segment-specific mixtures of shared low-rank adapters, reporting modest mean-squared-error gains over the base models on most of eight ...

  45. A Systematic Literature Review of Parameter-Efficient Fine-Tuning for Large Code Models

    cs.SE 2025-04 conditional novelty 4.0 of 10

    This review synthesizes 28 studies to classify PEFT use in software engineering, finding LoRA and adapters dominant and PEFT often competitive with full fine-tuning.

  46. Transfer Learning in Physics-Informed Neural Networks: Full Fine-Tuning, Lightweight Fine-Tuning, and Low-Rank Adaptation

    cs.LG 2025-02 conditional novelty 4.0 of 10

    A benchmark comparing full fine-tuning, lightweight fine-tuning, and LoRA for transferring PINNs across boundary conditions, materials, and geometries, finding full fine-tuning and LoRA help while last-layer fine-tuni...

  47. Fine Tuning without Catastrophic Forgetting via Selective Low Rank Adaptation

    cs.CV 2025-01 conditional novelty 4.0 of 10

    Fine-tuning a pretrained vision model with only 2-6% of its LoRA adapters active preserves zero-shot and out-of-distribution performance while matching full-rank LoRA accuracy.

  48. OMoE: Diversifying Mixture of Low-Rank Adaptation by Orthogonal Finetuning

    cs.LG 2025-01 conditional novelty 4.0 of 10

    OMoE applies Gram-Schmidt orthogonalization to expert representations in LoRA-based mixture-of-experts, improving fine-tuning performance with only two experts.

  49. ChronoLLM: A Framework for Customizing Large Language Model for Digital Twins generalization based on PyChrono

    cs.SE 2025-01 conditional novelty 4.0 of 10

    Fine-tuning LLMs on PyChrono-specific data improves their success rate at generating runnable simulation code from about 40% to about 85%, compared to prompting general models.

  50. When SAM2 Meets Video Shadow and Mirror Detection

    cs.CV 2024-12 conditional novelty 4.0 of 10

    Evaluating SAM2 on video shadow and mirror detection shows strong results with first-frame mask prompts and poor results with point prompts.

  51. Adapter-based Approaches to Knowledge-enhanced Language Models -- A Survey

    cs.CL 2024-11 conditional novelty 4.0 of 10

    A systematic review of adapter-based knowledge-enhanced language models, covering 26 papers, popular adapter types, and biomedical performance comparisons.

  52. Learning Text Styles: A Study on Transfer, Attribution, and Verification

    cs.CL 2025-07 conditional novelty 3.0 of 10

    A thesis compiles published work claiming that lightweight adapters, contrastive disentanglement, and instruction tuning improve text style transfer, authorship attribution, and authorship verification.

  53. PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Models

    cs.CL 2025-04 conditional novelty 3.0 of 10

    A survey that organizes PEFT methods into additive, selective, reparameterized, hybrid, and unified families, but with no new method or verified experiments.

  54. Parameter-Efficient Continual Fine-Tuning: A Survey

    cs.LG 2025-04 conditional novelty 3.0 of 10

    A survey and comparison of Parameter-Efficient Continual Fine-Tuning methods for adapting large pre-trained models to sequential tasks without catastrophic forgetting.

  55. SSH: Sparse Spectrum Adaptation via Discrete Hartley Transformation

    cs.CV 2025-02 conditional novelty 3.0 of 10

    SSH fine-tunes large models by learning sparse Hartley-spectrum coefficients selected by energy of the pretrained weights, matching or beating LoRA and FourierFT with fewer parameters.

  56. A Survey on Large Language Models with some Insights on their Capabilities and Limitations

    cs.CL 2025-01 unverdicted novelty 3.0 of 10

    A broad survey of LLM methods and applications, plus an empirical section on how code-rich pretraining may influence chain-of-thought reasoning, the details of which are not visible in the supplied text.

  57. Slimming Down LLMs Without Losing Their Minds

    cs.CL 2025-06 reject novelty 2.0 of 10

    QLoRA fine-tuning on Alpaca data drops a 1B model's math accuracy from 33.5% to 3.7% and reduces MMLU-CS accuracy by 13 points, while HellaSwag stays flat.

  58. Parameter-Efficient Fine-Tuning for Foundation Models

    cs.CL 2025-01 conditional novelty 2.0 of 10

    A survey that categorizes and summarizes parameter-efficient fine-tuning methods across large language, vision, and multimodal models.

  59. Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges

    cs.LG 2024-12 conditional

    A broad but error-prone survey of LLM and MLLM architectures, training methods, benchmarks, and challenges.

Pith tools