REVIEW 7 cited by
The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Transformer-based language models have become a key building block for natural language processing. While these models are extremely accurate, they can be too large and computationally intensive to run on standard deployments. A variety of compression methods, including distillation, quantization, structured and unstructured pruning are known to decrease model size and increase inference speed, with low accuracy loss. In this context, this paper's contributions are two-fold. We perform an in-depth study of the accuracy-compression trade-off for unstructured weight pruning of BERT models. We introduce Optimal BERT Surgeon (oBERT), an efficient and accurate weight pruning method based on approximate second-order information, which we show to yield state-of-the-art results in both stages of language tasks: pre-training and fine-tuning. Specifically, oBERT extends existing work on unstructured second-order pruning by allowing for pruning blocks of weights, and by being applicable at the BERT scale. Second, we investigate the impact of this pruning method when compounding compression approaches to obtain highly compressed but accurate models for deployment on edge devices. These models significantly push boundaries of the current state-of-the-art sparse BERT models with respect to all metrics: model size, inference speed and task accuracy. For example, relative to the dense BERT-base, we obtain 10x model size compression (in MB) with < 1% accuracy drop, 10x CPU-inference speedup with < 2% accuracy drop, and 29x CPU-inference speedup with < 7.5% accuracy drop. Our code, fully integrated with Transformers and SparseML, is available at https://github.com/neuralmagic/sparseml/tree/main/research/optimal_BERT_surgeon_oBERT.
Forward citations
Cited by 7 Pith papers
-
MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference
MXSens allocates 8-bit precision to the 32 most sensitive columns per layer, 6-bit to moderately sensitive columns, and 4-bit elsewhere in MXINT, improving WikiText-2 perplexity over prior 4-bit LLM quantization methods.
-
DarwinLM: Evolutionary Structured Pruning of Large Language Models
DarwinLM uses evolutionary search with training-aware offspring selection to prune LLMs, beating ShearedLlama with 5x less post-training data.
-
Scalable iterative pruning of large language and vision models using block coordinate descent
An iterative block-coordinate-descent version of the Combinatorial Brain Surgeon outperforms one-shot Wanda pruning on DeiT and Mistral-7b, but without code release or error bars.
-
Verifiable Unlearning on Edge
A pruning-plus-OBS unlearning method is wrapped in a proposed zk-SNARK verification protocol, but no proof-generation evaluation is provided and the single experiment is self-referential.
-
HessFormer: Hessians at Foundation Scale
HessFormer delivers distributed Hessian-vector products for 70B-parameter models and reports the first spectral-density plots, including large negative eigenvalues, for DeepSeek-LLM-70B.
-
HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMs
HASSLE-free gives a fuller-Hessian alternating-minimization recipe for sparse-plus-low-rank LLM compression and reports perplexity improvements over OATS on Llama-3 and Llama-3.2 models.
-
Efficient Shapley Value-based Non-Uniform Pruning of Large Language Models
SV-NUP allocates non-uniform per-layer pruning budgets based on Shapley values and reports 18% lower perplexity than SparseGPT on LLaMA-7B at 70% sparsity.
Discussion (0). Continue with ORCID to comment.