Pith. sign in

REVIEW 35 cited by

SliceGPT: Compress Large Language Models by Deleting Rows and Columns

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.15024 v2 pith:TJINR2UM submitted 2024-01-26 cs.LG cs.CL

Saleh Ashkboos , Maximilian L. Croci , Marcelo Gennari do Nascimento , Torsten Hoefler , James Hensman This is my paper · ORCID
classification cs.LGcs.CL
keywords modelsslicegptdensegpuslanguagemodelreducesparsification
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models have become the cornerstone of natural language processing, but their use comes with substantial costs in terms of compute and memory resources. Sparsification provides a solution to alleviate these resource constraints, and recent works have shown that trained models can be sparsified post-hoc. Existing sparsification techniques face challenges as they need additional data structures and offer constrained speedup with current hardware. In this paper we present SliceGPT, a new post-training sparsification scheme which replaces each weight matrix with a smaller (dense) matrix, reducing the embedding dimension of the network. Through extensive experimentation, we show that SliceGPT can remove up to 25% of the model parameters (including embeddings) for LLAMA2-70B, OPT 66B and Phi-2 models while maintaining 99%, 99% and 90% zero-shot task performance of the dense model respectively. Our sliced models run on fewer GPUs and run faster without any additional code optimization: on 24GB consumer GPUs we reduce the total compute for inference on LLAMA2-70B to 64% of that of the dense model; on 40GB A100 GPUs we reduce it to 66%. We offer a new insight, computational invariance in transformer networks, which enables SliceGPT and we hope it will inspire and enable future avenues to reduce memory and computation demands for pre-trained models. Code is available at: https://github.com/microsoft/TransformerCompression

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 35 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FPTQuant: Function-Preserving Transforms for LLM Quantization

    cs.LG 2025-06 conditional novelty 7.0 of 10

    FPTQuant introduces function-preserving transforms that make transformer activations amenable to static 4-bit quantization with minimal inference overhead.

  2. You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning

    cs.CL 2025-01 conditional novelty 7.0 of 10

    PruneNet learns a calibration-free stochastic policy to prune FFN rows in LLMs while preserving the singular-value spectrum, retaining about 80% of LLaMA-2-7B's zero-shot accuracy at 30% compression.

  3. Searching for Task-Specific Vision Paths: Evolutionary Block Pruning Across Vision-Language Models

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Evolutionary search improves fixed-budget vision-block skipping, but capability-specific routes are unstable and fail to transfer across models and sources.

  4. It Takes a MAESTRO To Prune Bad Experts

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Pruning MoE LLMs according to the stationary distribution of a Markov chain over (layer, expert) routing transitions retains more task performance than local importance heuristics, with up to ~3.5% relative gains over...

  5. When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMs

    cs.LG 2025-10 conditional novelty 6.0 of 10

    Removing just one or two layers from a reasoning-tuned LLM destroys its test-time scaling, and neither LoRA nor full fine-tuning restores it.

  6. Prune&Comp: Free Lunch for Layer-Pruned LLMs via Iterative Pruning with Magnitude Compensation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A magnitude-compensation step that rescales the weights before a pruned layer, combined with iterative pruning, consistently improves perplexity and question-answering accuracy of layer-pruned LLMs.

  7. GeLaCo: An Evolutionary Approach to Layer Compression

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Evolutionary search over layer-merging configurations, scored by module-wise activation similarity, yields competitive LLM compression and the first size-quality Pareto fronts.

  8. GPTailor: Large Language Model Pruning Through Layer Cutting and Stitching

    cs.CL 2025-06 conditional novelty 6.0 of 10

    GPTailor searches over layer removal, layer selection, and layer merging across fine-tuned model variants to produce smaller LLMs that retain more benchmark performance than single-model pruning.

  9. ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformations

    cs.CL 2025-06 conditional novelty 6.0 of 10

    ProcrustesGPT searches for per-layer orthogonal rotations that make pretrained LLM weights fit Kronecker or GS structured matrices, cutting 14 to 36 percent of parameters without fine-tuning.

  10. Forget the Data and Fine-Tuning! Just Fold the Network to Compress

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Model folding compresses a network by k-means clustering similar neurons across adjacent layers and repairing activation statistics without data (Fold-AR, Fold-DIR), surpassing prior data-free methods at high sparsity.

  11. RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    RoSTE couples quantization-aware supervised fine-tuning with per-layer Hadamard rotation selection, reducing quantization outliers and improving 4-bit quantized LLM accuracy over SFT-then-PTQ baselines.

  12. EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Pruning-aware pretraining iteratively deletes low-saliency parameter groups from a larger pretrained LLM during continued training, yielding sub-1B EfficientLLM models that outperform direct-pretraining baselines.

  13. Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Dobi-SVD compresses LLMs via differentiable SVD rank selection, IPCA-based weight reconstruction, and quantized storage remapping, reporting competitive perplexity at 40% parameters.

  14. Lightweight and Post-Training Structured Pruning for On-Device Large Lanaguage Models

    cs.LG 2025-01 conditional novelty 6.0 of 10

    COMP prunes LLMs post-training by removing redundant layers and then neurons ranked by a matrix condition-number metric, recovering accuracy via mask tuning.

  15. FlexiGPT: Pruning and Extending Large Language Models with Low-Rank Weight Sharing

    cs.CL 2025-01 conditional novelty 6.0 of 10

    A block-pruning and model-extension method for LLMs that replaces pruned blocks with weight-shared blocks plus low-rank adapters, reporting state-of-the-art recovery on several benchmarks.

  16. Prune, Update and Trim: Robust Structured Pruning for Large Language Models

    cs.LG 2026-05 unverdicted novelty 5.5 of 10

    Putri is a structured pruning technique for LLMs that compensates for pruning errors via weight updates and sequential processing while pruning at the attention-head level to reach state-of-the-art results at extreme ...

  17. PALS: Percentile-Aware Layerwise Sparsity for LLM Pruning

    cs.CL 2026-07 conditional novelty 5.0 of 10

    Activation-percentile-based per-layer sparsity allocation improves LLaMA-2-7B pruning by 15% perplexity over uniform Wanda, while gradient-based allocation fails worse than random.

  18. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.

  19. A Hamiltonian-Inspired Local-Operator Ansatz for Slimming Large Language Models

    cs.CL 2026-05 unverdicted novelty 5.0 of 10

    Tensor Mixture (MixT) compresses transformer linear layers into local tensor sums that execute directly, preserving MMLU accuracy up to a sharp per-model boundary and cutting LLaMA2-7B parameters by 47.5%.

  20. Collaborative Lossless LLM Inference Serving with Offloading-based Pipeline Parallelism on Edge Devices

    cs.DC 2025-12 reject novelty 5.0 of 10

    This preprint claims large latency speedups for collaborative lossless edge LLM inference via interleaved pipeline parallelism with SSD offloading, but the abstract and body describe different systems (LOIP vs. LIME) ...

  21. Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM

    cs.CL 2025-10 conditional novelty 5.0 of 10

    A single per-layer error tolerance, instead of a per-layer rank budget, improves zero-shot low-rank compression of LLMs and VLMs empirically, but the theorem that this uniform tolerance is Pareto-optimal is proven onl...

  22. AQUA: Attention via QUery mAgnitudes for Memory and Compute Efficient Inference in LLMs

    cs.LG 2025-09 conditional novelty 5.0 of 10

    A training-free method that prunes low-magnitude dimensions of projected query/key vectors in attention, cutting dot-product cost by 25% with small benchmark degradation.

  23. Pruning Large Language Models by Identifying and Preserving Functional Networks

    cs.CL 2025-08 reject novelty 5.0 of 10

    An ICA-based structured pruning method that preserves functional neuron networks shows mixed performance against FLAP, with overclaimed state-of-the-art results.

  24. Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Structural pruning with finetuning plus hidden-state distillation recovers most performance in multimodal LLMs, with 5% of training data sufficient at moderate compression levels.

  25. CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge

    cs.AR 2025-06 conditional novelty 5.0 of 10

    CLONE combines generative pruning, LoRA adapters, a parameter-free Mixture-of-Experts router, and learning-based DVFS to make LLM inference on edge devices faster and more energy-efficient, claiming up to 11.92x speed...

  26. Pangu Light: Weight Re-Initialization for Pruning and Accelerating LLMs

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Pangu Light prunes LLMs across width, depth, and attention heads, then re-initializes remaining weights, achieving up to 4.2x throughput with modest benchmark loss.

  27. Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfer

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Neural Parameter Search (NPS) prunes fine-tuned models by evolutionary reweighting of magnitude-based task vector subspaces, improving transfer, fusion, and compression.

  28. TriSP: Tri-Signal Structured Pruning for Large Language Models

    cs.AI 2026-06 conditional novelty 4.0 of 10

    TriSP prunes LLM attention heads and MLP neurons by combining magnitude, activation norm, and gradient sensitivity, achieving the best reported quality at 20–50% pruning on 7B models with LoRA recovery.

  29. Enabling On-Device Medical AI Assistants via Input-Driven Saliency Adaptation

    cs.CL 2025-06 reject novelty 4.0 of 10

    A saliency-based pruning plus 4-bit quantization pipeline runs Gemma 7B and LLaMA 8B on edge hardware, but medical QA accuracy drops by up to 27 points, contradicting the 'minimal accuracy loss' claim.

  30. SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decoupling

    cs.CL 2025-06 conditional novelty 4.0 of 10

    SkipGPT learns per-token, per-module routers that dynamically skip attention and MLP blocks, then uses LoRA to recover most of the original model's quality.

  31. Rethinking the Outlier Distribution in Large Language Models: An In-depth Study

    cs.CL 2025-05 reject novelty 4.0 of 10

    The paper claims that most massive activations in LLMs are residual-propagated 'fake' outliers that can be removed without accuracy loss.

  32. UniCP: A Unified Caching and Pruning Framework for Efficient Video Generation

    cs.CV 2025-02 reject novelty 4.0 of 10

    UniCP combines error-aware caching and PCA-based pruning to speed up diffusion-transformer video generation by about 1.6x.

  33. FASP: Fast and Accurate Structured Pruning of Large Language Models

    cs.LG 2025-01 conditional novelty 4.0 of 10

    FASP prunes LLMs by deleting coupled rows and columns across adjacent layers, using a Wanda-style column score and a least-squares weight restoration, and reports faster, lower-perplexity pruning than SliceGPT, NASLLM...

  34. Survey of Specialized Large Language Model

    cs.CL 2025-08 conditional novelty 2.0 of 10

    A survey of 24 specialized LLMs (2022-2025) claims a shift from domain fine-tuning to native architectures, but the synthesis is undermined by citation errors and selection bias.

  35. On Accelerating Edge AI: Optimizing Resource-Constrained Environments

    cs.LG 2025-01 conditional novelty 2.0 of 10

    The paper argues that model compression, neural architecture search, and compiler optimizations work together to accelerate edge AI, but it provides no new experimental evidence.

Pith tools