Pith. sign in

REVIEW 17 cited by

LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2304.01933 v3 pith:3IK7DXWL submitted 2023-04-04 cs.CL

LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models

Zhiqiang Hu , Lei Wang , Yihuai Lan , Wanyu Xu , Ee-Peng Lim , Lidong Bing , Xing Xu , Soujanya Poria
show 1 more author
Roy Ka-Wei Lee
This is my paper
classification cs.CL
keywords llmsmethodsadapter-basedadaptersfine-tuningpeftreasoningadapter
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

The success of large language models (LLMs), like GPT-4 and ChatGPT, has led to the development of numerous cost-effective and accessible alternatives that are created by finetuning open-access LLMs with task-specific data (e.g., ChatDoctor) or instruction data (e.g., Alpaca). Among the various fine-tuning methods, adapter-based parameter-efficient fine-tuning (PEFT) is undoubtedly one of the most attractive topics, as it only requires fine-tuning a few external parameters instead of the entire LLMs while achieving comparable or even better performance. To enable further research on PEFT methods of LLMs, this paper presents LLM-Adapters, an easy-to-use framework that integrates various adapters into LLMs and can execute these adapter-based PEFT methods of LLMs for different tasks. The framework includes state-of-the-art open-access LLMs such as LLaMA, BLOOM, and GPT-J, as well as widely used adapters such as Series adapters, Parallel adapter, Prompt-based learning and Reparametrization-based methods. Moreover, we conduct extensive empirical studies on the impact of adapter types, placement locations, and hyper-parameters to the best design for each adapter-based methods. We evaluate the effectiveness of the adapters on fourteen datasets from two different reasoning tasks, Arithmetic Reasoning and Commonsense Reasoning. The results demonstrate that using adapter-based PEFT in smaller-scale LLMs (7B) with few extra trainable parameters yields comparable, and in some cases superior, performance to powerful LLMs (175B) in zero-shot inference on both reasoning tasks.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. R-CoT: A Reasoning-Layer Watermark via Redundant Chain-of-Thought in Large Language Models

    cs.CR 2026-04 unverdicted novelty 7.0

    R-CoT embeds watermarks into LLM reasoning paths via redundant CoT and GRPO-based dual optimization, maintaining over 95% true positive rate under fine-tuning and post-training changes.

  2. Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning

    cs.LG 2026-04 unverdicted novelty 7.0

    A new SFT framework for MoE models combines bias-driven sparsification with gated condenser experts to retain long-tailed expert information, outperforming DenseMixer and ESFT by over 2.5% on math reasoning and common...

  3. LoRA-DA: Data-Aware Initialization for Low-Rank Adaptation via Asymptotic Analysis

    cs.LG 2025-10 conditional novelty 7.0

    LoRA-DA derives an optimal data-aware LoRA initialization by solving an optimization problem from asymptotic analysis of parameter discrepancy using Fisher-gradient bias and Fisher-information variance terms.

  4. WizardLM: Empowering large pre-trained language models to follow complex instructions

    cs.CL 2023-04 conditional novelty 7.0

    WizardLM uses LLM-driven iterative rewriting to generate complex instruction data and fine-tunes LLaMA to reach over 90% of ChatGPT capacity on 17 of 29 evaluated skills.

  5. Behavioral and Representational Evidence of Binomial Ordering Preferences in Large Language Models

    cs.CL 2026-06 unverdicted novelty 6.0

    LLMs recover dominant binomial orders from corpora but align less closely with exact preference distributions, with preference strength partially encoded in middle-to-late layers and manipulable via steering.

  6. One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs

    cs.LG 2026-05 conditional novelty 6.0

    Heavy-tail guided layerwise learning rates improve LLM convergence speed and generalization across LLaMA, GPT variants, AdamW and Muon optimizers from 60M to 1B parameters.

  7. FuRA: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioning

    cs.LG 2026-05 unverdicted novelty 6.0

    FuRA uses block tensor-train factorization with fixed pretrained SVD basis to achieve full-rank spectral preconditioning, outperforming Full FT by +1.37 on LLaMA-3-8B commonsense reasoning and surpassing QLoRA in quan...

  8. TLoRA: Task-aware Low Rank Adaptation of Large Language Models

    cs.CL 2026-04 unverdicted novelty 6.0

    TLoRA jointly optimizes LoRA initialization via task-data SVD and sensitivity-driven rank allocation, delivering stronger results than standard LoRA across NLU, reasoning, math, code, and chat tasks while using fewer ...

  9. CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging

    cs.CL 2026-02 conditional novelty 6.0

    CoMoL represents every LoRA expert as a shared-basis core matrix and merges token-selected experts in that core space, reaching standard LoRA parameter counts while outperforming MoE-LoRA baselines on math and code.

  10. Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models

    cs.LG 2026-01 reject novelty 6.0

    SALR combines static pruning of frozen weights with a trainable truncated-SVD low-rank residual adapter to match LoRA accuracy at 50% sparsity, cutting model size ~2x and giving ~1.7x inference speedup.

  11. HyperAdapt: Simple High-Rank Adaptation

    cs.LG 2025-09 unverdicted novelty 6.0

    HyperAdapt performs parameter-efficient fine-tuning by row- and column-wise diagonal scaling to induce high-rank updates with only n+m trainable parameters.

  12. Feature Geometry of LoRA Adapters: A Sparse Autoencoder Analysis of Representational Divergence in Fine-Tuned Language Models

    cs.LG 2026-05 unverdicted novelty 5.0

    LoRA fine-tuning produces feature dictionaries in language models that show weak alignment with pretrained SAE features and are better reconstructed by adapter-specific SAEs.

  13. One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs

    cs.LG 2026-05 unverdicted novelty 5.0

    LLR uses heavy-tailed self-regularization theory to set per-layer learning rates in Transformers, yielding faster convergence and higher zero-shot accuracy than uniform rates across model scales.

  14. Polynomial Expansion Rank Adaptation: Enhancing Low-Rank Fine-Tuning with High-Order Interactions

    cs.LG 2026-04 unverdicted novelty 5.0

    PERA augments low-rank adaptation by expanding factors with polynomial terms to capture higher-order interactions, yielding better benchmark performance than prior linear methods without raising rank or inference cost.

  15. BoHA: Blockwise Hadamard Product Adaptation for Parameter-Efficient Fine-Tuning

    cs.LG 2025-09 unverdicted novelty 5.0

    BoHA partitions frozen weights into a b by b grid and applies independent low-rank Hadamard factors per block, outperforming LoRA on matched-budget single-task averages while retaining 57.66% first-stage accuracy in a...

  16. Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

    cs.LG 2024-03 accept novelty 4.0

    A comprehensive survey of PEFT algorithms for large models, covering their performance, overhead, applications, and real-world system implementations.

  17. A Comprehensive Overview of Large Language Models

    cs.CL 2023-07 unverdicted novelty 2.0

    A survey paper providing an overview of Large Language Models, their background, and recent advances in the field.