Pith. sign in

REVIEW 4 cited by

Structured Pruning of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1910.04732 v2 pith:SK6IGSFM submitted 2019-10-10 cs.CL cs.LGstat.ML

Ziheng Wang , Jeremy Wohlwend , Tao Lei This is my paper
classification cs.CLcs.LGstat.ML
keywords languagemodelspruninglargestructuredapproachcompressionduring
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models have recently achieved state of the art performance across a wide variety of natural language tasks. Meanwhile, the size of these models and their latency have significantly increased, which makes their usage costly, and raises an interesting question: do language models need to be large? We study this question through the lens of model compression. We present a generic, structured pruning approach by parameterizing each weight matrix using its low-rank factorization, and adaptively removing rank-1 components during training. On language modeling tasks, our structured approach outperforms other unstructured and block-structured pruning baselines at various compression levels, while achieving significant speedups during both training and inference. We also demonstrate that our method can be applied to pruning adaptive word embeddings in large language models, and to pruning the BERT model on several downstream fine-tuning classification benchmarks.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Boosting Parameter Efficiency in LLM-Based Recommendation through Sophisticated Pruning

    cs.IR 2025-07 conditional novelty 5.0 of 10

    PruneRec prunes attention heads, embedding dimensions, MLP units, and layers from a recommendation-tuned LLM, retaining 88% of accuracy with under 5% of non-embedding parameters.

  2. BnBERT-iPET: Sparse Few-Shot Language Modeling for Bengali via Lottery Ticket Pruning

    cs.LG 2026-08 reject novelty 4.0 of 10

    A 90%-pruned few-shot Bengali model is reported to rival larger baselines on some tasks, but the reported F1 scores contradict the paper's own precision and recall values.

  3. Delta-SVD: Efficient Compression for Personalized Text-to-Image Models

    cs.CV 2025-08 conditional novelty 4.0 of 10

    DreamBooth fine-tuning updates are low-rank, so post-hoc SVD with energy-based truncation compresses personalized models by up to 270x with mild quality loss.

  4. Towards Inclusive NLP: Assessing Compressed Multilingual Transformers across Diverse Language Benchmarks

    cs.CL 2025-07 reject novelty 3.0 of 10

    Across Arabic, English, and Kannada benchmarks, 4-bit and 8-bit quantization preserves most accuracy while aggressive pruning degrades larger multilingual models more than smaller ones.

Pith tools