Pith. sign in

REVIEW 8 cited by

LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.18403 v5 pith:IWHWVQCF submitted 2023-05-28 cs.LG cs.CV

Mingyang Zhang , Hao Chen , Chunhua Shen , Zhen Yang , Linlin Ou , Xinyi Yu , Bohan Zhuang This is my paper
classification cs.LGcs.CV
keywords pruningllmslorapruneloraweightsgradientsmodelperformance
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs), such as LLaMA and T5, have shown exceptional performance across various tasks through fine-tuning. Although low-rank adaption (LoRA) has emerged to cheaply fine-tune these LLMs on downstream tasks, their deployment is still hindered by the vast model scale and computational costs. Post-training model pruning offers a way to compress LLMs. However, the current pruning methods designed for LLMs are not compatible with LoRA. This is due to their utilization of unstructured pruning on LLMs, impeding the merging of LoRA weights, or their dependence on the gradients of pre-trained weights to guide pruning, which can impose significant memory overhead. To this end, we propose LoRAPrune, a new framework that delivers an accurate structured pruned model in a highly memory-efficient manner. Specifically, we first design a LoRA-guided pruning criterion, which uses the weights and gradients of LoRA, rather than the gradients of pre-trained weights for importance estimation. We subsequently integrate this criterion into an iterative pruning process, effectively removing redundant channels and heads. Extensive experimental results demonstrate the superior performance of our LoRAPrune over existing approaches on the LLaMA series models. At a 50\% compression rate, LoRAPrune demonstrates superior performance over LLM-Pruner, achieving a reduction in perplexity by 4.81 on WikiText2 and 3.46 on PTB, while also decreasing memory usage by 52.6%. Besides, LoRAPrune also matches semi-structural pruning across multiple LLMs, proving its wide applicability. The code is available at https://github.com/aim-uofa/LoRAPrune.

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation

    cs.LG 2026-05 unverdicted novelty 8.0 of 10

    GPTQ-intrinsic LoRA augments GPTQ with intrinsic low-rank compensation via Hessian modification to achieve layer-wise reconstruction bounds that match information-theoretic lower bounds under structural assumptions.

  2. MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on LLMs

    cs.LG 2025-06 unverdicted novelty 6.0 of 10

    MaskPro learns categorical distributions over groups of M weights to generate exact (N:M) sparsity via N-way sampling without replacement and stabilizes training with a moving average tracker of loss residuals.

  3. PALS: Percentile-Aware Layerwise Sparsity for LLM Pruning

    cs.CL 2026-07 conditional novelty 5.0 of 10

    Activation-percentile-based per-layer sparsity allocation improves LLaMA-2-7B pruning by 15% perplexity over uniform Wanda, while gradient-based allocation fails worse than random.

  4. GradMAP: Faster Layer Pruning with Gradient Metric and Projection Compensation

    cs.CL 2026-02 conditional novelty 5.0 of 10

    Layer pruning is accelerated and accuracy improved by using one-backward-pass gradient scores for layer selection and a single projection-compensation matrix on the most-drifted layer.

  5. RAP: Runtime Adaptive Pruning for LLM Inference

    cs.LG 2025-05 unverdicted novelty 5.0 of 10

    RAP is a reinforcement learning framework for runtime-adaptive pruning of LLMs that jointly optimizes model weights and KV-cache usage under varying memory budgets.

  6. Constraint-Driven Model Optimization: An Industry Framework for Selecting Compression and Acceleration Techniques in Modern Machine Learning Systems

    cs.LG 2026-07 conditional novelty 4.0 of 10

    A decision framework mapping five production constraints to established model-optimization techniques, synthesized from 25+ cited papers, with four worked industry pipelines.

  7. Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

    cs.LG 2024-03 accept novelty 4.0 of 10

    A comprehensive survey of PEFT algorithms for large models, covering their performance, overhead, applications, and real-world system implementations.

  8. Will LLMs Scaling Hit the Wall? Breaking Barriers via Distributed Resources on Massive Edge Devices

    cs.DC 2025-03 unverdicted novelty 2.0 of 10

    Position paper claiming that distributed training across massive edge devices can overcome data depletion and centralized compute monopolies in LLM scaling.

Pith tools