Pith. sign in

REVIEW 21 cited by

Parameter-Efficient Transfer Learning with Diff Pruning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2012.07463 v2 pith:KSIZGZAD submitted 2020-12-14 cs.CL cs.LG

Demi Guo , Alexander M. Rush , Yoon Kim This is my paper · ORCID
classification cs.CLcs.LG
keywords difftaskspretrainedpruningvectorfinetuninglearningparameter-efficient
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

While task-specific finetuning of pretrained networks has led to significant empirical advances in NLP, the large size of networks makes finetuning difficult to deploy in multi-task, memory-constrained settings. We propose diff pruning as a simple approach to enable parameter-efficient transfer learning within the pretrain-finetune framework. This approach views finetuning as learning a task-specific diff vector that is applied on top of the pretrained parameter vector, which remains fixed and is shared across different tasks. The diff vector is adaptively pruned during training with a differentiable approximation to the L0-norm penalty to encourage sparsity. Diff pruning becomes parameter-efficient as the number of tasks increases, as it requires storing only the nonzero positions and weights of the diff vector for each task, while the cost of storing the shared pretrained model remains constant. It further does not require access to all tasks during training, which makes it attractive in settings where tasks arrive in stream or the set of tasks is unknown. We find that models finetuned with diff pruning can match the performance of fully finetuned baselines on the GLUE benchmark while only modifying 0.5% of the pretrained model's parameters per task.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 21 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MOON: Multi-Objective OrthoNormalized Updates for Multitask Learning

    cs.LG 2026-08 conditional novelty 7.0 of 10

    MOON applies spectral-nuclear-norm geometry to multi-objective gradient manipulation and uses polar-factor updates, with O(T^-1/2) deterministic and O(T^-1/4) stochastic convergence to Pareto stationarity.

  2. The impact of allocation strategies in subset learning on the expressive power of neural networks

    cs.LG 2025-02 conditional novelty 7.0 of 10

    In a teacher-student setup, maximal expressive power for a fixed learnable-weight budget is characterized by even row or column distribution in linear RNNs and feedforward networks.

  3. RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models

    cs.LG 2025-05 conditional novelty 6.0 of 10

    RefLoRA picks a per-step optimal low-rank factorization (a matrix geometric mean) that balances LoRA's factors, improving fine-tuning convergence and accuracy.

  4. Not All Parameters Matter: Masking Diffusion Models for Enhancing Generation Ability

    cs.CV 2025-05 conditional novelty 6.0 of 10

    MaskUNet masks U-Net weights with a timestep- and sample-dependent binary mask, improving zero-shot FID on COCO by about 1.1 to 1.5 points while leaving pre-trained weights frozen.

  5. Densely Connected Parameter-Efficient Tuning for Referring Image Segmentation

    cs.CV 2025-01 conditional novelty 6.0 of 10

    DETRIS uses dense mixtures of convolutions and cross-attention adapters to tune a frozen DINOv2/CLIP pair, achieving top reported IoU on three referring image segmentation benchmarks while updating only a small fracti...

  6. S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity

    cs.LG 2024-12 conditional novelty 6.0 of 10

    S2FT selects a few attention heads and FFN channels, permutes the neighboring weight matrices so the selected parts form dense blocks, and fine-tunes only those blocks, reporting better generalization and efficiency t...

  7. CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuning

    cs.LG 2024-11 reject novelty 6.0 of 10

    Attention pairs (Q-K and V-O) are SVD-decomposed so pruning or fine-tuning touches only a small singular-factor matrix, yielding better pruning tolerance and small PEFT gains.

  8. LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning

    cs.LG 2026-07 conditional novelty 5.0 of 10

    LAARA allocates LoRA ranks per layer from diagonal Fisher (gradient-based) estimates, reporting improved accuracy with fewer trainable parameters on GLUE and MathInstruct.

  9. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.

  10. FeRA: Frequency-Energy Constrained Routing for Effective Diffusion Adaptation Fine-Tuning

    cs.CV 2025-11 conditional novelty 5.0 of 10

    A frequency-energy router that blends LoRA experts according to the latent's bandwise energy improves diffusion fine-tuning quality and style consistency across multiple backbones.

  11. Tensorized Clustered LoRA Merging for Multi-Task Interference

    cs.LG 2025-08 unverdicted novelty 5.0 of 10

    Clustering training data by embedding similarity and jointly CP-decomposing LoRA adapters cuts multi-task merging interference: +1.4% on Phi-3 and +2.3% on Mistral-7B over SVD baselines.

  12. Accelerating Diffusion Transformer via Error-Optimized Cache

    cs.CV 2025-01 conditional novelty 5.0 of 10

    A precomputed trend correction injected into cached diffusion transformer blocks reduces FID by up to 28.8% at heavy caching with negligible added compute.

  13. Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Models

    cs.CL 2024-12 conditional novelty 5.0 of 10

    A static sparsity mask chosen by plain gradients matches or beats LoRA and second-order salience metrics across NLP fine-tuning benchmarks.

  14. GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptation

    cs.LG 2025-08 conditional novelty 4.0 of 10

    GEM selects fine-tuning parameters by gradient-to-weight ratio and distributes the budget by layer entropy, reaching 0.1% parameter updates with small accuracy gains on several NLP tasks.

  15. Backbone Augmented Training for Adaptations

    cs.LG 2025-06 reject novelty 4.0 of 10

    Selecting backbone pre-training data with an influence-based score and adding it to the adaptation set can improve LoRA and DreamBooth training when adaptation data are scarce.

  16. A Systematic Literature Review of Parameter-Efficient Fine-Tuning for Large Code Models

    cs.SE 2025-04 conditional novelty 4.0 of 10

    This review synthesizes 28 studies to classify PEFT use in software engineering, finding LoRA and adapters dominant and PEFT often competitive with full fine-tuning.

  17. CrackESS: A Self-Prompting Crack Segmentation System for Edge Devices

    cs.CV 2024-12 conditional novelty 4.0 of 10

    CrackESS combines YOLOv8, a lightly fine-tuned EdgeSAM, and a refinement module to segment concrete cracks on edge devices, achieving about four times the speed of CrackSAM with comparable accuracy.

  18. A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learning

    cs.LG 2025-04 conditional novelty 3.0 of 10

    A review that sorts recent federated-learning PEFT approaches into additive, selective, and reparameterized (LoRA-style) families and maps them onto NLP and vision applications.

  19. PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Models

    cs.CL 2025-04 conditional novelty 3.0 of 10

    A survey that organizes PEFT methods into additive, selective, reparameterized, hybrid, and unified families, but with no new method or verified experiments.

  20. Progtuning: Progressive Fine-tuning Framework for Transformer-based Language Models

    cs.CL 2025-06 reject novelty 2.0 of 10

    A progressive scheduling trick that updates only the last remaining blocks in later epochs reduces parameter-update counts by about 25% with roughly unchanged GLUE and SQuAD scores.

  21. Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges

    cs.LG 2024-12 conditional

    A broad but error-prone survey of LLM and MLLM architectures, training methods, benchmarks, and challenges.

Pith tools