Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks
read the original abstract
State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. However, such modules are trained separately for each task and thus do not enable sharing information across tasks. In this paper, we show that we can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks, which condition on task, adapter position, and layer id in a transformer model. This parameter-efficient multi-task learning framework allows us to achieve the best of both worlds by sharing knowledge across tasks via hypernetworks while enabling the model to adapt to each individual task through task-specific adapters. Experiments on the well-known GLUE benchmark show improved performance in multi-task learning while adding only 0.29% parameters per task. We additionally demonstrate substantial performance improvements in few-shot domain generalization across a variety of tasks. Our code is publicly available in https://github.com/rabeehk/hyperformer.
This paper has not been read by Pith yet.
Forward citations
Cited by 5 Pith papers
-
Hypernetwork-Conditioned WENO5 Conservative-Form CNNs for One-Dimensional Conservation Laws
A hypernetwork conditions a conservative-form CNN to predict WENO5 weights from mesh and initial-condition metadata, preserving conservation and generalizing across resolutions for 1D hyperbolic conservation laws.
-
Retrieval-Augmented Personalization with Foundation Models for Wearable Stress Detection
Retrieval from out-of-domain foundation models enables personalization of a lightweight transformer for stress detection, yielding +3.92% accuracy and +4.76% F1 gains on WESAD without user labels.
-
Deep Reprogramming Distillation for Medical Foundation Models
DRD introduces a reprogramming module and CKA-based distillation to enable efficient, robust adaptation of medical foundation models to downstream 2D/3D classification and segmentation tasks, outperforming prior PEFT ...
-
Towards Unified Multi-task EEG Analysis with Low-Rank Adaptation
MTEEG uses task-specific LoRA modules to jointly adapt a pre-trained EEG model across multiple tasks, outperforming single-task baselines on most metrics in evaluations on six downstream tasks.
-
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
A comprehensive survey of PEFT algorithms for large models, covering their performance, overhead, applications, and real-world system implementations.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.