Pith. sign in

REVIEW 9 cited by

MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2409.17481 v2 pith:G5YKEWUE submitted 2024-09-26 cs.AI cs.CLcs.LG

Gongfan Fang , Hongxu Yin , Saurav Muralidharan , Greg Heinrich , Jeff Pool , Jan Kautz , Pavlo Molchanov , Xinchao Wang This is my paper · ORCID
classification cs.AIcs.CLcs.LG
keywords maskllmsparsitylearnablemaskslargellmsmodelsdatasets
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Large Language Models (LLMs) are distinguished by their massive parameter counts, which typically result in significant redundancy. This work introduces MaskLLM, a learnable pruning method that establishes Semi-structured (or ``N:M'') Sparsity in LLMs, aimed at reducing computational overhead during inference. Instead of developing a new importance criterion, MaskLLM explicitly models N:M patterns as a learnable distribution through Gumbel Softmax sampling. This approach facilitates end-to-end training on large-scale datasets and offers two notable advantages: 1) High-quality Masks - our method effectively scales to large datasets and learns accurate masks; 2) Transferability - the probabilistic modeling of mask distribution enables the transfer learning of sparsity across domains or tasks. We assessed MaskLLM using 2:4 sparsity on various LLMs, including LLaMA-2, Nemotron-4, and GPT-3, with sizes ranging from 843M to 15B parameters, and our empirical results show substantial improvements over state-of-the-art methods. For instance, leading approaches achieve a perplexity (PPL) of 10 or greater on Wikitext compared to the dense model's 5.12 PPL, but MaskLLM achieves a significantly lower 6.72 PPL solely by learning the masks with frozen weights. Furthermore, MaskLLM's learnable nature allows customized masks for lossless application of 2:4 sparsity to downstream tasks or domains. Code is available at https://github.com/NVlabs/MaskLLM.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TinyFusion: Diffusion Transformers Learned Shallow

    cs.CV 2024-12 conditional novelty 7.0 of 10

    A learnable depth-pruning method that optimizes post-fine-tuning recoverability produces a 14-layer DiT-XL with FID 2.86 and a 2x speedup at 7% of the original training cost.

  2. Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Models

    cs.LG 2025-08 unverdicted novelty 6.0 of 10

    Amber Pruner proposes training-free N:M activation sparsity for LLM prefill; however, the supplied manuscript body is an unrelated paper.

  3. CoNNect: Connectivity-Based Regularization for Structural Pruning

    cs.LG 2025-02 conditional novelty 6.0 of 10

    CoNNect is a differentiable connectivity regularizer that encourages sparse but connected networks, improving structural pruning accuracy when added to DepGraph and LLM-pruner.

  4. ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs

    cs.LG 2025-02 conditional novelty 6.0 of 10

    ProxSparse learns 2:4 semi-structured sparsity masks for pretrained LLMs via regularized proximal-gradient optimization, outperforming heuristic baselines on seven models.

  5. Expand Neurons, Not Parameters

    cs.LG 2025-10 reject novelty 5.0 of 10

    Fixed Parameter Expansion — duplicating neurons and partitioning their incoming weights into disjoint sparse sub-neurons at constant non-zero parameter count — reduces measured feature interference and improves classi...

  6. Pruning Weights but Not Truth: Safeguarding Truthfulness While Pruning LLMs

    cs.LG 2025-08 reject novelty 5.0 of 10

    LLM pruning damages the internal features used to detect false statements; the proposed TPLO method reallocates sparsity to protect them, though the measured improvements are modest and potentially confounded by leaka...

  7. SepPrune: Structured Pruning for Efficient Deep Speech Separation

    cs.SD 2025-05 conditional novelty 5.0 of 10

    SepPrune applies differentiable channel masks to compress speech separation models, reporting stronger accuracy than existing pruning baselines at matched FLOPs after fine-tuning.

  8. SPAP: Structured Pruning via Alternating Optimization and Penalty Methods

    cs.LG 2025-05 conditional novelty 5.0 of 10

    SPAP combines a mixed-integer relaxation with a penalty method and alternating minimization to prune MLP columns in LLMs, reporting lower perplexity than CFSP, FLAP, SliceGPT and FASP at matched sparsity.

  9. FASP: Fast and Accurate Structured Pruning of Large Language Models

    cs.LG 2025-01 conditional novelty 4.0 of 10

    FASP prunes LLMs by deleting coupled rows and columns across adjacent layers, using a Wanda-style column score and a least-squares weight restoration, and reports faster, lower-perplexity pruning than SliceGPT, NASLLM...

Pith tools