Pith. sign in

REVIEW 5 cited by

AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.10912 v1 pith:YWNPY4TG submitted 2024-10-14 cs.LG stat.ML

Haiquan Lu , Yefan Zhou , Shiwei Liu , Zhangyang Wang , Michael W. Mahoney , Yaoqing Yang This is my paper
classification cs.LGstat.ML
keywords pruningalphapruningllmslargelayerwiseratiosempiricalexisting
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Recent work on pruning large language models (LLMs) has shown that one can eliminate a large number of parameters without compromising performance, making pruning a promising strategy to reduce LLM model size. Existing LLM pruning strategies typically assign uniform pruning ratios across layers, limiting overall pruning ability; and recent work on layerwise pruning of LLMs is often based on heuristics that can easily lead to suboptimal performance. In this paper, we leverage Heavy-Tailed Self-Regularization (HT-SR) Theory, in particular the shape of empirical spectral densities (ESDs) of weight matrices, to design improved layerwise pruning ratios for LLMs. Our analysis reveals a wide variability in how well-trained, and thus relatedly how prunable, different layers of an LLM are. Based on this, we propose AlphaPruning, which uses shape metrics to allocate layerwise sparsity ratios in a more theoretically principled manner. AlphaPruning can be used in conjunction with multiple existing LLM pruning methods. Our empirical results show that AlphaPruning prunes LLaMA-7B to 80% sparsity while maintaining reasonable perplexity, marking a first in the literature on LLMs. We have open-sourced our code at https://github.com/haiquanlu/AlphaPruning.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Omega-S: A Functional Resilience Index for LLM Fine-Tuning

    cs.LG 2026-08 conditional novelty 6.0 of 10

    Omega-S, a penalty on node-degree variance in the weight matrix, improves code retention during LoRA fine-tuning of Llama-3-8B, while its advertised clustering/topological channel is inert.

  2. Dynamic Sparse Training of Diagonally Sparse Networks

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A dynamic sparse training method that restricts weights to a learnable set of diagonals, preserving sparsity in both forward and backward passes to obtain GPU speedups at accuracy close to unstructured sparsity.

  3. Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Aspect ratio of weight matrices biases heavy-tail spectral metrics; the new FARMS subsampling method removes this bias and improves downstream layer-wise tuning.

  4. A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs

    cs.LG 2026-02 reject novelty 5.0 of 10

    Replicate-and-Quantize copies the busiest MoE expert as a quantized duplicate and compresses the least important expert, lowering a new Load-Imbalance Score by up to 1.4x while accuracy varies by roughly -1.2 to +3.0 points.

  5. SHUFFLESPARSE: Learned Shuffles for Structured Sparse Networks

    cs.LG 2025-10 conditional novelty 5.0 of 10

    Learned per-layer permutations bring structured sparse training (block, N:M, diagonal) close to unstructured sparse-training accuracy at 90-95% sparsity, with inference speedups up to 2.9x.

Pith tools