Pith. sign in

REVIEW 21 cited by

Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2106.04489 v1 pith:BE23THTY submitted 2021-06-08 cs.CL

Rabeeh Karimi Mahabadi , Sebastian Ruder , Mostafa Dehghani , James Henderson This is my paper · ORCID
classification cs.CL
keywords tasktasksacrossadapterhypernetworksmodelmulti-taskparameter-efficient
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. However, such modules are trained separately for each task and thus do not enable sharing information across tasks. In this paper, we show that we can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks, which condition on task, adapter position, and layer id in a transformer model. This parameter-efficient multi-task learning framework allows us to achieve the best of both worlds by sharing knowledge across tasks via hypernetworks while enabling the model to adapt to each individual task through task-specific adapters. Experiments on the well-known GLUE benchmark show improved performance in multi-task learning while adding only 0.29% parameters per task. We additionally demonstrate substantial performance improvements in few-shot domain generalization across a variety of tasks. Our code is publicly available in https://github.com/rabeehk/hyperformer.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 21 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MOON: Multi-Objective OrthoNormalized Updates for Multitask Learning

    cs.LG 2026-08 conditional novelty 7.0 of 10

    MOON applies spectral-nuclear-norm geometry to multi-objective gradient manipulation and uses polar-factor updates, with O(T^-1/2) deterministic and O(T^-1/4) stochastic convergence to Pareto stationarity.

  2. UnHype: CLIP-Guided Hypernetworks for Dynamic LoRA Unlearning

    cs.CV 2026-02 conditional novelty 6.0 of 10

    UnHype generates concept-specific LoRA unlearning weights on the fly from CLIP text embeddings by training a hypernetwork to follow the gradient of an unlearning loss, enabling single- and multi-concept erasure in dif...

  3. Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A residual adapter called DFN plus singular-value sharpness regularization improves cross-domain few-shot segmentation by 2.69% and 4.68% MIoU over prior state-of-the-art in 1-shot and 5-shot settings.

  4. Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A task-conditioned projection operator converts a large transformer's weights into a smaller task-specialized transformer that outperforms same-size universal conditional models.

  5. Sparse Gradient Compression for Fine-Tuning Large Language Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    SGC compresses LLM optimizer states into a low-dimensional subspace via top-k gradient sparsification and OMP recovery, claiming comparable fine-tuning accuracy with fewer optimizer states.

  6. TADFormer : Task-Adaptive Dynamic Transformer for Efficient Multi-Task Learning

    cs.CV 2025-01 conditional novelty 6.0 of 10

    A prompt-and-dynamic-filter PEFT design for multi-task dense prediction beats MTLoRA on PASCAL-Context with fewer trainable parameters.

  7. CM3T: Framework for Efficient Multimodal Learning for Inhomogeneous Interaction Datasets

    cs.CV 2025-01 conditional novelty 6.0 of 10

    CM3T shows that multi-head vision adapters plus cross-attention adapters can adapt frozen supervised-pretrained video transformers with a fraction of the trainable parameters of full fine-tuning.

  8. Learning to Forget using Hypernetworks

    cs.LG 2024-12 conditional novelty 6.0 of 10

    A diffusion-based hypernetwork can generate classifier weights with near-zero accuracy on a requested forget class and near-retrained accuracy on retained classes.

  9. Safety Alignment Backfires: Preventing the Re-emergence of Suppressed Concepts in Fine-tuned Text-to-Image Diffusion Models

    cs.AI 2024-11 conditional novelty 6.0 of 10

    Fine-tuning text-to-image diffusion models on benign data can reactivate suppressed unsafe concepts, and training the task adapter separately from a frozen safety LoRA prevents this.

  10. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.

  11. (Almost) Free Modality Stitching of Foundation Models

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A hypernetwork that generates connector weights for all image-text model pairs can rank pairs like grid search at about 10x lower training cost, but the best connector lags grid search by a few points.

  12. Text-to-LoRA: Instant Transformer Adaption

    cs.LG 2025-06 conditional novelty 5.0 of 10

    A hypernetwork can generate task-specific LoRA adapters from a text description, and when trained with supervised fine-tuning it zero-shot outperforms a multi-task LoRA baseline on ten benchmarks.

  13. Direct Preference Optimization-Enhanced Multi-Guided Diffusion Model for Traffic Scenario Generation

    cs.LG 2025-02 reject novelty 5.0 of 10

    MuDi-Pro fine-tunes a multi-guided diffusion transformer with DPO using guidance-score preferences to improve controllability of traffic scenario generation on nuScenes.

  14. DiTASK: Multi-Task Fine-Tuning with Diffeomorphic Transformations

    cs.CV 2025-02 conditional novelty 5.0 of 10

    DiTASK fine-tunes vision transformers for multiple tasks by applying learnable diffeomorphic maps to the singular values of frozen weights, and reports state-of-the-art multi-task performance with fewer parameters.

  15. SAM-DA: Decoder Adapter for Efficient Medical Domain Adaptation

    cs.CV 2025-01 conditional novelty 5.0 of 10

    A zero-initialized attention adapter placed in SAM's mask decoder matches full fine-tuning on medical segmentation and improves cross-domain generalization with under 1% trainable parameters.

  16. Capacity and Redundancy Trade-offs in Multi-Task Learning

    cs.LG 2026-07 conditional novelty 4.0 of 10

    A shared representation's total per-task information is bounded by capacity plus label redundancy; clustered sharing wins exactly when interference reduction exceeds the redundancy it loses.

  17. Meta-aware Learning in text-to-SQL Large Language Model

    cs.AI 2025-05 conditional novelty 4.0 of 10

    Combining schema, chain-of-thought, metadata knowledge, and tokenized prompt structures during fine-tuning improves text-to-SQL execution accuracy on private business databases compared to schema-only fine-tuning.

  18. RepCali: High Efficient Fine-tuning Via Representation Calibration in Latent Space for Pre-trained Language Models

    cs.CL 2025-05 conditional novelty 4.0 of 10

    Adding a single learned, input-independent offset to encoder outputs before decoding produces small but consistent downstream gains on many encoder-decoder PLMs.

  19. ChameleonLLM: Batch-Aware Dynamic Low-Rank Adaptation via Inference-Time Clusters

    cs.CL 2025-02 reject novelty 4.0 of 10

    ChameleonLLM generates low-rank LoRA updates from clustered batch statistics via a hypernetwork, claiming better perplexity than static LoRA, but the evidence is undercut by implausible baselines and confounded comparisons.

  20. Multimodal Large Language Models for Medicine: A Comprehensive Survey

    cs.LG 2025-04 conditional novelty 2.0 of 10

    A comprehensive review cataloging medical MLLMs, their uses in report generation, diagnosis, and treatment, and the challenges of accuracy, hallucination, fairness, privacy, and deployment.

  21. Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges

    cs.LG 2024-12 conditional

    A broad but error-prone survey of LLM and MLLM architectures, training methods, benchmarks, and challenges.

Pith tools