Pith. sign in

REVIEW 33 cited by

Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.12148 v1 pith:WU7WW4YZ submitted 2023-12-19 cs.CL

Lingling Xu , Haoran Xie , Si-Zhao Joe Qin , Xiaohui Tao , Fu Lee Wang This is my paper
classification cs.CL
keywords peftfine-tuningmethodslanguagemodelsplmsparametersapplications
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

With the continuous growth in the number of parameters of transformer-based pretrained language models (PLMs), particularly the emergence of large language models (LLMs) with billions of parameters, many natural language processing (NLP) tasks have demonstrated remarkable success. However, the enormous size and computational demands of these models pose significant challenges for adapting them to specific downstream tasks, especially in environments with limited computational resources. Parameter Efficient Fine-Tuning (PEFT) offers an effective solution by reducing the number of fine-tuning parameters and memory usage while achieving comparable performance to full fine-tuning. The demands for fine-tuning PLMs, especially LLMs, have led to a surge in the development of PEFT methods, as depicted in Fig. 1. In this paper, we present a comprehensive and systematic review of PEFT methods for PLMs. We summarize these PEFT methods, discuss their applications, and outline future directions. Furthermore, we conduct experiments using several representative PEFT methods to better understand their effectiveness in parameter efficiency and memory efficiency. By offering insights into the latest advancements and practical applications, this survey serves as an invaluable resource for researchers and practitioners seeking to navigate the challenges and opportunities presented by PEFT in the context of PLMs.

Discussion (0). Sign in to comment.

Forward citations

Cited by 33 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. HeiSD: Hybrid Speculative Decoding for Embodied Vision-Language-Action Models with Kinematic Awareness

    cs.RO 2026-03 unverdicted novelty 7.0 of 10

    HeiSD delivers up to 2.45x faster inference for embodied VLA models by hybridizing speculative decoding with kinematic boundary detection and error-mitigation tricks while preserving task success rates.

  2. When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index models

    cs.LG 2026-02 conditional novelty 7.0 of 10

    In a Gaussian single-index model with one-pass SGD, the LoRA escape time scales as τ(μ) log d / 2, where τ(μ) increases with pre-training strength μ and diverges for odd Hermite activations at a critical μ.

  3. Key and Value Weights Are Probably All You Need: On the Necessity of the Query, Key, Value weight Triplet in Self-Attention Transformers

    cs.LG 2025-10 unverdicted novelty 7.0 of 10

    One of the Q, K or V weights in transformer self-attention is redundant and replaceable by the identity matrix under mild assumptions, reducing parameters by 25 percent with no loss in small-model performance.

  4. Task Prompt Vectors: Effective Initialization through Multi-Task Soft-Prompt Transfer

    cs.CL 2024-08 unverdicted novelty 7.0 of 10

    Task prompt vectors, formed by subtracting random initialization from tuned soft prompts, support low-resource initialization and arithmetic combination across tasks on 12 NLU datasets while remaining independent of i...

  5. Fine-Tuning Integrity for Modern Neural Networks: Structured Drift Proofs via Norm, Rank, and Sparsity Certificates

    cs.CR 2026-04 conditional novelty 6.5 of 10

    Succinct zero-knowledge proofs can certify that a fine-tuned model differs from a base model only by norm-bounded, low-rank, or sparse parameter drift, with cost set by that structure rather than model size.

  6. FMplex: Model Virtualization for Serving Extensible Foundation Models

    cs.DC 2026-06 unverdicted novelty 6.0 of 10

    FMplex is a serving system that virtualizes FM backbones for sharing across tasks, claiming up to 80% lower latency and 6x more tasks hosted versus prior approaches.

  7. World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoning

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    Distilling view-consistent future views and action-outcome supervision from a generative world model into a VLM via two-stage post-training improves dynamic spatial reasoning on SAT-Real, VSI-Bench and similar benchma...

  8. Leveraging LLMs for Multi-File DSL Code Generation: An Industrial Case Study

    cs.SE 2026-04 unverdicted novelty 6.0 of 10

    Fine-tuning 7B code LLMs on a custom multi-file DSL dataset achieves structural fidelity of 1.00, high exact-match accuracy, and practical utility validated by expert survey and execution checks.

  9. SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learning

    cs.CL 2026-04 unverdicted novelty 6.0 of 10

    SAMoRA is a parameter-efficient fine-tuning framework that uses semantic-aware routing and task-adaptive scaling within a Mixture of LoRA Experts to improve multi-task performance and generalization over prior methods.

  10. TLoRA: Task-aware Low Rank Adaptation of Large Language Models

    cs.CL 2026-04 unverdicted novelty 6.0 of 10

    TLoRA jointly optimizes LoRA initialization via task-data SVD and sensitivity-driven rank allocation, delivering stronger results than standard LoRA across NLU, reasoning, math, code, and chat tasks while using fewer ...

  11. Fine-Tuning Integrity for Modern Neural Networks: Structured Drift Proofs via Norm, Rank, and Sparsity Certificates

    cs.CR 2026-04 unverdicted novelty 6.0 of 10

    Succinct Model Difference Proofs certify that a neural-network update stays inside a policy-defined drift class using zero-knowledge proofs whose cost depends only on the drift structure.

  12. Task-Centric Personalized Federated Fine-Tuning of Language Models

    cs.LG 2026-03 unverdicted novelty 6.0 of 10

    FedRouter clusters adapters locally per task samples and globally across clients to create task-centric personalized models, improving generalization and reducing task interference in federated fine-tuning.

  13. Do Fine-Tuned LLMs Understand Vulnerabilities? An Investigation into the Semantic Trap

    cs.CR 2026-01 unverdicted novelty 6.0 of 10

    Fine-tuned decoder-only LLMs fall into a Semantic Trap on vulnerability detection, achieving high scores on unpaired normal code but failing on paired vulnerable-patched code, semantic perturbations, and gap analysis,...

  14. PEFT-Bench: A Parameter-Efficient Fine-Tuning Methods Benchmark

    cs.CL 2025-11 unverdicted novelty 6.0 of 10

    PEFT-Bench is a standardized end-to-end benchmark for 7 PEFT methods across 27 NLP datasets on autoregressive LLMs, accompanied by the PSCP metric that penalizes based on trainable parameters, inference speed, and tra...

  15. Point of Order: Action-Aware LLM Persona Modeling for Data-Grounded Civic Deliberation

    cs.CL 2025-11 conditional novelty 6.0 of 10

    Fine-tuning on speaker-attributed, action-tagged transcripts from public meetings lets LLM agents mimic government meeting participants well enough that human judges often cannot tell them from real people.

  16. Why Do MLLMs Struggle with Spatial Understanding? A Systematic Analysis from Data to Architecture

    cs.CV 2025-09 conditional novelty 6.0 of 10

    Spatial understanding in multimodal LLMs plateaus quickly as training data grows, and position encoding in the visual encoder is the more influential factor.

  17. GRID: Scaling Task-Agnostic Inference in Continual Prompt Tuning

    cs.LG 2025-07 conditional novelty 6.0 of 10

    GRID combines output-space constrained decoding with gradient-guided prompt compression for task-agnostic, bounded-memory continual prompt tuning.

  18. Implementing Adaptations for Vision AutoRegressive Model

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Fine-tuned Vision AutoRegressive models mostly beat a strong diffusion baseline on downstream image generation, but DP fine-tuning yields poor FID scores.

  19. Defending Against Prompt Injection With a Few DefensiveTokens

    cs.CR 2025-07 conditional novelty 6.0 of 10

    Five optimized token embeddings prepended to the prompt reduce prompt-injection attack success to near zero on standard benchmarks while preserving most model utility.

  20. Exploring Post-Training Alignment of Small Language Models for Biomedical Data-to-Text Generation: A Case Study of Medication Leaflet

    cs.CL 2026-07 conditional novelty 5.0 of 10

    In medication-leaflet generation, ORPO and GRPO beat SFT and a zero-shot GPT-5 baseline, but GRPO's cross-domain advantage is partly built into its reward functions.

  21. Soft Specialists: $\alpha$-R\'enyi Ensembles for Uncertainty-Aware LLM Post-Training

    stat.ML 2026-05 unverdicted novelty 5.0 of 10

    An α-Rényi variational ensemble method learns distributions over LoRA adapter parameters for uncertainty-aware LLM post-training, balancing individual model plausibility with complementary specialization.

  22. Reconstructing Item Characteristic Curves using Fine-Tuned Large Language Models

    cs.CL 2026-01 conditional novelty 5.0 of 10

    Fine-tuned LLMs can reconstruct item characteristic curves from multiple-choice item text, giving useful estimates of IRT difficulty and discrimination without live student response data.

  23. PEFT-Factory: Unified Parameter-Efficient Fine-Tuning of Autoregressive Large Language Models

    cs.CL 2025-12 unverdicted novelty 5.0 of 10

    PEFT-Factory supplies a ready-to-use, extensible codebase that unifies 19 PEFT methods and evaluation pipelines for fine-tuning large autoregressive language models.

  24. Echoes of Human Malice in Agents: Benchmarking LLMs for Multi-Turn Online Harassment Attacks

    cs.AI 2025-10 conditional novelty 5.0 of 10

    Fine-tuning an LLM on synthetic toxic dialogues makes it harass in 95–97% of multi-turn conversations in Llama and ~99% in Gemini; memory and planning attacks also raise closed-source vulnerability.

  25. PB-IAD: Utilizing multimodal foundation models for semantic industrial anomaly detection in dynamic manufacturing environments

    cs.CV 2025-08 conditional novelty 5.0 of 10

    With carefully layered prompts and one or three reference samples, GPT-4.1 detects anomalies in cable images and crimp-force features at F1 levels that PatchCore and Isolation Forest reach only after training on dozen...

  26. Sealing The Backdoor: Unlearning Adversarial Text Triggers In Diffusion Models Using Knowledge Distillation

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    SKD-CAG erases adversarial text triggers from diffusion models by distilling the model's own clean outputs through cross-attention guidance, claiming 100% and 93% removal for pixel and style backdoors.

  27. CIgrate: Automating CI Service Migration with Large Language Models

    cs.SE 2025-07 reject novelty 5.0 of 10

    The abstract reports that a fine-tuned Gemma 3 12B outperforms the rule-based CIMig tool on Travis-to-GitHub Actions CI migration, but the manuscript body contains only a planned study with no experimental results.

  28. The Impact of Fine-tuning Large Language Models on Automated Program Repair

    cs.SE 2025-07 conditional novelty 5.0 of 10

    On three Java APR benchmarks, LoRA and IA3 adapters match or beat full-model fine-tuning for most tested code LLMs while training less than one percent of parameters.

  29. Enhancing RLHF with Human Gaze Modeling

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Using predicted human gaze to shape rewards lets RLHF converge 1.3 to 2 times faster with similar final policy quality.

  30. Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

    cs.LG 2024-03 accept novelty 4.0 of 10

    A comprehensive survey of PEFT algorithms for large models, covering their performance, overhead, applications, and real-world system implementations.

  31. DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Models

    cs.CR 2025-09 reject novelty 3.0 of 10

    DP-FedLoRA clips and adds Gaussian noise to per-client LoRA matrices in federated LLM fine-tuning, claiming unbiased updates and bounded variance, but the privacy calibration and experiments have significant gaps.

  32. Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translation

    cs.CL 2025-04 unverdicted novelty 3.0 of 10

    A literature survey that organizes prompting, fine-tuning, preference optimization, and context-aware techniques for LLM-based machine translation with emphasis on low-resource languages.

  33. A Survey on the Memory Mechanism of Large Language Model based Agents

    cs.AI 2024-04 accept novelty 3.0 of 10

    A systematic review of memory designs, evaluation methods, applications, limitations, and future directions for LLM-based agents.

Pith tools