REVIEW 19 cited by
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
read the original abstract
Sequential fine-tuning and multi-task learning are methods aiming to incorporate knowledge from multiple tasks; however, they suffer from catastrophic forgetting and difficulties in dataset balancing. To address these shortcomings, we propose AdapterFusion, a new two stage learning algorithm that leverages knowledge from multiple tasks. First, in the knowledge extraction stage we learn task specific parameters called adapters, that encapsulate the task-specific information. We then combine the adapters in a separate knowledge composition step. We show that by separating the two stages, i.e., knowledge extraction and knowledge composition, the classifier can effectively exploit the representations learned from multiple tasks in a non-destructive manner. We empirically evaluate AdapterFusion on 16 diverse NLU tasks, and find that it effectively combines various types of knowledge at different layers of the model. We show that our approach outperforms traditional strategies such as full fine-tuning as well as multi-task learning. Our code and adapters are available at AdapterHub.ml.
Forward citations
Cited by 19 Pith papers
-
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention
LLaMA-Adapter turns frozen LLaMA 7B into a capable instruction follower using only 1.2M new parameters and zero-init attention, matching Alpaca while extending to image-conditioned reasoning on ScienceQA and COCO.
-
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Prefix-tuning matches or exceeds fine-tuning on NLG tasks by optimizing a continuous prefix using 0.1% of parameters while keeping the LM frozen.
-
Test-Time Scaling via Error Localization
TTEL uses feedback-induced token probability drops to localize the first error in a failed reasoning trace and branch a new generation from that prefix, improving pass@k per token on coding and math benchmarks.
-
Substrate Asymmetry in User-Side Memory: A Diagnostic Framework
User memory in LLMs factors into three orthogonal axes where parametric adapters and retrieval show opposite strengths, with causal evidence from attention interventions and an alignment tax on RLHF models.
-
PEML: Parameter-efficient Multi-Task Learning with Optimized Continuous Prompts
PEML co-optimizes continuous prompts and low-rank adaptations to deliver up to 6.67% average accuracy gains over existing multi-task PEFT methods on GLUE, SuperGLUE, and other benchmarks.
-
SECOS: Semantic Capture for Rigorous Classification in Open-World Semi-Supervised Learning
SECOS enables direct semantic label prediction in open-world semi-supervised learning by aligning representations with external knowledge for novel classes, outperforming prior methods by up to 5.4% even without post-...
-
TLoRA: Task-aware Low Rank Adaptation of Large Language Models
TLoRA jointly optimizes LoRA initialization via task-data SVD and sensitivity-driven rank allocation, delivering stronger results than standard LoRA across NLU, reasoning, math, code, and chat tasks while using fewer ...
-
CrossEarth-Gate: Fisher-Guided Adaptive Tuning Engine for Efficient Adaptation of Cross-Domain Remote Sensing Semantic Segmentation
A Fisher-information-guided dynamic selection over a toolbox of LoRA, adapter, and frequency-adapter modules improves cross-domain remote sensing segmentation over static PEFT methods.
-
DivMerge: A divergence-based model merging method for multi-tasking
DivMerge learns task-arithmetic merging weights by minimizing Jensen-Shannon divergence between each specialist model and the merged model, improving multi-task performance and scalability.
-
AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
AdaLoRA uses SVD-based pruning to allocate the parameter budget for low-rank fine-tuning updates according to per-matrix importance scores, yielding better performance than uniform allocation especially under tight budgets.
-
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.
-
Extending Tabular Denoising Diffusion Probabilistic Models for Time-Series Data Generation
A temporal extension of TabDDPM generates coherent synthetic time-series sequences on the WISDM dataset that match real distributions and support downstream classification with macro F1 of 0.64.
-
Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Models
Sensitivity-LoRA allocates LoRA ranks across layers using Hessian-based sensitivity metrics, improving average GLUE score by 0.74 over AdaLoRA on RoBERTa-base.
-
LARA: Lightweight Adapters in the Residual Stream for Composable Adaptation and Alignment
Low-rank residual-stream adapters match LoRA at near-equal parameter counts on code fine-tuning and DPO, and support inference-time scaling plus per-token routing of seven behaviors on one frozen 1.5B model.
-
GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptation
GEM selects fine-tuning parameters by gradient-to-weight ratio and distributes the budget by layer entropy, reaching 0.1% parameter updates with small accuracy gains on several NLP tasks.
-
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
A comprehensive survey of PEFT algorithms for large models, covering their performance, overhead, applications, and real-world system implementations.
-
Neural Network Optimization Reimagined: Decoupled Techniques for Scratch and Fine-Tuning
DualOpt decouples optimization by using real-time layer-wise weight decay for scratch training and weight rollback for fine-tuning to improve convergence, generalization, and reduce knowledge forgetting.
-
Structure-Learnable Adapter Fine-Tuning for Parameter-Efficient Large Language Models
A gated-adapter method with a sparsity regularizer is reported to match or slightly beat full fine-tuning on MNLI and BoolQ using 1.4% of parameters, but lacks the experimental detail needed to verify the claim.
-
Small Language Models (SLMs) Can Still Pack a Punch: A survey (updated 2026)
A literature survey of Small Language Models (1-8B parameters) that can perform comparably or better than larger models, covering general-purpose and task-specific approaches plus creation techniques.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.