REVIEW 6 cited by
Sparse Fine-tuning for Inference Acceleration of Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We consider the problem of accurate sparse fine-tuning of large language models (LLMs), that is, fine-tuning pretrained LLMs on specialized tasks, while inducing sparsity in their weights. On the accuracy side, we observe that standard loss-based fine-tuning may fail to recover accuracy, especially at high sparsities. To address this, we perform a detailed study of distillation-type losses, determining an L2-based distillation approach we term SquareHead which enables accurate recovery even at higher sparsities, across all model types. On the practical efficiency side, we show that sparse LLMs can be executed with speedups by taking advantage of sparsity, for both CPU and GPU runtimes. While the standard approach is to leverage sparsity for computational reduction, we observe that in the case of memory-bound LLMs sparsity can also be leveraged for reducing memory bandwidth. We exhibit end-to-end results showing speedups due to sparsity, while recovering accuracy, on T5 (language translation), Whisper (speech translation), and open GPT-type (MPT for text generation). For MPT text generation, we show for the first time that sparse fine-tuning can reach 75% sparsity without accuracy drops, provide notable end-to-end speedups for both CPU and GPU inference, and highlight that sparsity is also compatible with quantization approaches. Models and software for reproducing our results are provided in Section 6.
Forward citations
Cited by 6 Pith papers
-
Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interface
Attackers can use the loss signal from a remote LLM fine-tuning API to optimize adversarial prefix and suffix tokens, turning existing prompt injections into high-success attacks on closed-weight Gemini models.
-
Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models
SALR combines static pruning of frozen weights with a trainable truncated-SVD low-rank residual adapter to match LoRA accuracy at 50% sparsity, cutting model size ~2x and giving ~1.7x inference speedup.
-
Puzzle: Distillation-Based NAS for Inference-Optimized LLMs
Puzzle uses blockwise local distillation and mixed-integer programming to search per-layer replacements, yielding Llama derivatives that retain 98.4% of the parent's accuracy at 2.17x inference throughput on one H100 GPU.
-
Fast RoPE Attention: Combining the Polynomial Method and Fast Fourier Transform
A new algorithm computes RoPE attention in n^{1+o(1)} time under bounded entries by decomposing the attention matrix into a small sum of rescaled Toeplitz matrices, then using FFT.
-
DLP: Dynamic Layerwise Pruning in Large Language Models
DLP assigns each LLM layer a sparsity rate derived from the median of Wanda-style weight-activation scores, improving perplexity and zero-shot accuracy at high sparsity versus uniform and outlier-based layerwise pruning.
-
Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapse
A residual self-attention network with all weight entries bounded by a small η can be approximated by one layer to error O(η)‖X‖∞, so skip connections do not prevent layer collapse.
Discussion (0). Continue with ORCID to comment.