Pith. sign in

REVIEW 9 cited by

Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.08915 v3 pith:EFPN6KAE submitted 2023-10-13 cs.AI

Yuxin Zhang , Lirui Zhao , Mingbao Lin , Yunyun Sun , Yiwu Yao , Xingjia Han , Jared Tanner , Shiwei Liu
show 1 more author
Rongrong Ji
This is my paper · ORCID
classification cs.AI
keywords llmssparsedsnotfine-tuningtrainingdynamicsparsitytraining-free
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

The ever-increasing large language models (LLMs), though opening a potential path for the upcoming artificial general intelligence, sadly drops a daunting obstacle on the way towards their on-device deployment. As one of the most well-established pre-LLMs approaches in reducing model complexity, network pruning appears to lag behind in the era of LLMs, due mostly to its costly fine-tuning (or re-training) necessity under the massive volumes of model parameter and training data. To close this industry-academia gap, we introduce Dynamic Sparse No Training (DSnoT), a training-free fine-tuning approach that slightly updates sparse LLMs without the expensive backpropagation and any weight updates. Inspired by the Dynamic Sparse Training, DSnoT minimizes the reconstruction error between the dense and sparse LLMs, in the fashion of performing iterative weight pruning-and-growing on top of sparse LLMs. To accomplish this purpose, DSnoT particularly takes into account the anticipated reduction in reconstruction error for pruning and growing, as well as the variance w.r.t. different input data for growing each weight. This practice can be executed efficiently in linear time since its obviates the need of backpropagation for fine-tuning LLMs. Extensive experiments on LLaMA-V1/V2, Vicuna, and OPT across various benchmarks demonstrate the effectiveness of DSnoT in enhancing the performance of sparse LLMs, especially at high sparsity levels. For instance, DSnoT is able to outperform the state-of-the-art Wanda by 26.79 perplexity at 70% sparsity with LLaMA-7B. Our paper offers fresh insights into how to fine-tune sparse LLMs in an efficient training-free manner and open new venues to scale the great potential of sparsity to LLMs. Codes are available at https://github.com/zyxxmu/DSnoT.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Accelerating LLM Inference with Flexible N:M Sparsity via A Fully Digital Compute-in-Memory Accelerator

    cs.LG 2025-04 conditional novelty 7.0 of 10

    A pruning method and a digital compute-in-memory accelerator that jointly support layer-wise flexible N:M sparsity, improving LLM perplexity and zero-shot accuracy over fixed N:M baselines while cutting simulated infe...

  2. Delta Activations: A Representation for Finetuned Large Language Models

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Delta Activations embed finetuned LLMs as the average difference in hidden states between the finetuned model and its base model on a small set of generic prompts, yielding domain clusters and approximate additive com...

  3. Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Models

    cs.LG 2025-08 unverdicted novelty 6.0 of 10

    Amber Pruner proposes training-free N:M activation sparsity for LLM prefill; however, the supplied manuscript body is an unrelated paper.

  4. Toward Efficient SpMV in Sparse LLMs via Block Extraction and Compressed Storage

    cs.DC 2025-07 conditional novelty 6.0 of 10

    EC-SpMV combines hierarchical block extraction with delta-indexed compressed storage to speed up sparse matrix-vector multiplication during LLM decoding.

  5. Mitigating Non-IID Drift in Zeroth-Order Federated LLM Fine-Tuning with Transferable Sparsity

    cs.LG 2025-06 reject novelty 6.0 of 10

    Sparse zeroth-order federated fine-tuning with shared seeds and GradIP-based early stopping matches or beats full-parameter ZO while using far less communication.

  6. Perturbation-efficient Zeroth-order Optimization for Hardware-friendly On-device Training

    cs.LG 2025-04 conditional novelty 5.0 of 10

    PeZO makes zeroth-order learning hardware-friendly by reusing a small pool of uniform random perturbations rescaled to match Gaussian length statistics, achieving accuracy close to MeZO with far fewer random number ge...

  7. HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMs

    stat.ML 2025-02 conditional novelty 5.0 of 10

    HASSLE-free gives a fuller-Hessian alternating-minimization recipe for sparse-plus-low-rank LLM compression and reports perplexity improvements over OATS on Llama-3 and Llama-3.2 models.

  8. Symmetric Pruning of Large Language Models

    cs.LG 2025-01 conditional novelty 4.0 of 10

    SymWanda expresses Wanda and RIA pruning scores as special cases of a symmetric input-output reconstruction objective, and R2-DSnoT adds modest training-free post-pruning gains.

  9. Strategies for Improving Communication Efficiency in Distributed and Federated Learning: Compression, Local Training, and Personalization

    cs.LG 2025-09 conditional novelty 3.0 of 10

    A PhD dissertation showing unified compression theory, personalized accelerated local training, and pruning methods that reduce communication costs in federated learning and maintain accuracy in LLM pruning.

Pith tools