REVIEW 9 cited by
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. However, such modules are trained separately for each task and thus do not enable sharing information across tasks. In this paper, we show that we can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks, which condition on task, adapter position, and layer id in a transformer model. This parameter-efficient multi-task learning framework allows us to achieve the best of both worlds by sharing knowledge across tasks via hypernetworks while enabling the model to adapt to each individual task through task-specific adapters. Experiments on the well-known GLUE benchmark show improved performance in multi-task learning while adding only 0.29% parameters per task. We additionally demonstrate substantial performance improvements in few-shot domain generalization across a variety of tasks. Our code is publicly available in https://github.com/rabeehk/hyperformer.
Forward citations
Cited by 9 Pith papers
-
Hypernetwork-Conditioned WENO5 Conservative-Form CNNs for One-Dimensional Conservation Laws
A hypernetwork conditions a conservative-form CNN to predict WENO5 weights from mesh and initial-condition metadata, preserving conservation and generalizing across resolutions for 1D hyperbolic conservation laws.
-
Retrieval-Augmented Personalization with Foundation Models for Wearable Stress Detection
Retrieval from out-of-domain foundation models enables personalization of a lightweight transformer for stress detection, yielding +3.92% accuracy and +4.76% F1 gains on WESAD without user labels.
-
UnHype: CLIP-Guided Hypernetworks for Dynamic LoRA Unlearning
UnHype generates concept-specific LoRA unlearning weights on the fly from CLIP text embeddings by training a hypernetwork to follow the gradient of an unlearning loss, enabling single- and multi-concept erasure in dif...
-
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.
-
Deep Reprogramming Distillation for Medical Foundation Models
DRD introduces a reprogramming module and CKA-based distillation to enable efficient, robust adaptation of medical foundation models to downstream 2D/3D classification and segmentation tasks, outperforming prior PEFT ...
-
(Almost) Free Modality Stitching of Foundation Models
A hypernetwork that generates connector weights for all image-text model pairs can rank pairs like grid search at about 10x lower training cost, but the best connector lags grid search by a few points.
-
Capacity and Redundancy Trade-offs in Multi-Task Learning
A shared representation's total per-task information is bounded by capacity plus label redundancy; clustered sharing wins exactly when interference reduction exceeds the redundancy it loses.
-
Towards Unified Multi-task EEG Analysis with Low-Rank Adaptation
MTEEG uses task-specific LoRA modules to jointly adapt a pre-trained EEG model across multiple tasks, outperforming single-task baselines on most metrics in evaluations on six downstream tasks.
-
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
A comprehensive survey of PEFT algorithms for large models, covering their performance, overhead, applications, and real-world system implementations.
Discussion (0). Sign in to comment.