Pith. sign in

REVIEW 11 cited by

AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2205.12410 v2 pith:CQ4VVR5U submitted 2022-05-24 cs.CL cs.AIcs.LG

Yaqing Wang , Sahaj Agarwal , Subhabrata Mukherjee , Xiaodong Liu , Jing Gao , Ahmed Hassan Awadallah , Jianfeng Gao This is my paper
classification cs.CLcs.AIcs.LG
keywords adamixfine-tuningpeftmethodmixtureparameter-efficientparameterstasks
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Standard fine-tuning of large pre-trained language models (PLMs) for downstream tasks requires updating hundreds of millions to billions of parameters, and storing a large copy of the PLM weights for every task resulting in increased cost for storing, sharing and serving the models. To address this, parameter-efficient fine-tuning (PEFT) techniques were introduced where small trainable components are injected in the PLM and updated during fine-tuning. We propose AdaMix as a general PEFT method that tunes a mixture of adaptation modules -- given the underlying PEFT method of choice -- introduced in each Transformer layer while keeping most of the PLM weights frozen. For instance, AdaMix can leverage a mixture of adapters like Houlsby or a mixture of low rank decomposition matrices like LoRA to improve downstream task performance over the corresponding PEFT methods for fully supervised and few-shot NLU and NLG tasks. Further, we design AdaMix such that it matches the same computational cost and the number of tunable parameters as the underlying PEFT method. By only tuning 0.1-0.2% of PLM parameters, we show that AdaMix outperforms SOTA parameter-efficient fine-tuning and full model fine-tuning for both NLU and NLG tasks.

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Substrate Asymmetry in User-Side Memory: A Diagnostic Framework

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    User memory in LLMs factors into three orthogonal axes where parametric adapters and retrieval show opposite strengths, with causal evidence from attention interventions and an alignment tax on RLHF models.

  2. 5% > 100%: Flatness Preference is All You Need for Multimodal Parameter-Efficient Fine-Tuning

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    Flatness Preference Optimization (FlatPO) improves multimodal PEFT generalization by flattening a small set of sharp dimensions that dominate performance.

  3. LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning

    cs.LG 2026-07 conditional novelty 5.0 of 10

    LAARA allocates LoRA ranks per layer from diagonal Fisher (gradient-based) estimates, reporting improved accuracy with fewer trainable parameters on GLUE and MathInstruct.

  4. Unified Semantic Modeling Framework for Large-Scale Job Understanding at LinkedIn

    cs.AI 2026-06 conditional novelty 5.0 of 10

    A GPT-4-distilled small LM plus grouped LoRA adapters improves LinkedIn's job-attribute classification over legacy models.

  5. STK-Adapter: Incorporating Evolving Graph and Event Chain for Temporal Knowledge Graph Extrapolation

    cs.IR 2026-04 unverdicted novelty 5.0 of 10

    STK-Adapter adds Spatial-Temporal MoE, Event-Aware MoE, and Cross-Modality Alignment MoE to integrate evolving TKG graphs and event chains into LLMs, reducing information loss and improving extrapolation performance o...

  6. Parameter-Efficient Multi-Task Fine-Tuning in Code-Related Tasks

    cs.SE 2026-01 conditional novelty 5.0 of 10

    Multi-task QLoRA on Qwen2.5-Coder matches or beats single-task QLoRA and full fine-tuning for code generation and Python summarization, but lags in Java-to-C# translation.

  7. MoLEx: Mixture of LoRA Experts in Speech Self-Supervised Models for Audio Deepfake Detection

    cs.SD 2025-09 conditional novelty 5.0 of 10

    MoLEx combines LoRA adapters with a top-K expert router inside a frozen WavLM model, achieving 5.56% EER on ASVSpoof 5 without augmentation.

  8. Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

    cs.LG 2024-03 accept novelty 4.0 of 10

    A comprehensive survey of PEFT algorithms for large models, covering their performance, overhead, applications, and real-world system implementations.

  9. Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security

    cs.HC 2024-01 unverdicted novelty 3.0 of 10

    This survey discusses key components and challenges for Personal LLM Agents and reviews solutions for their capability, efficiency, and security.

  10. Small Language Models (SLMs) Can Still Pack a Punch: A survey (updated 2026)

    cs.CL 2025-01 unverdicted novelty 2.0 of 10

    A literature survey of Small Language Models (1-8B parameters) that can perform comparably or better than larger models, covering general-purpose and task-specific approaches plus creation techniques.

  11. A Comprehensive Overview of Large Language Models

    cs.CL 2023-07 unverdicted novelty 2.0 of 10

    A survey paper providing an overview of Large Language Models, their background, and recent advances in the field.

Pith tools