REVIEW 7 cited by
A deeper look at depth pruning of LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) are not only resource-intensive to train but even more costly to deploy in production. Therefore, recent work has attempted to prune blocks of LLMs based on cheap proxies for estimating block importance, effectively removing 10% of blocks in well-trained LLaMa-2 and Mistral 7b models without any significant degradation of downstream metrics. In this paper, we explore different block importance metrics by considering adaptive metrics such as Shapley value in addition to static ones explored in prior work. We show that adaptive metrics exhibit a trade-off in performance between tasks i.e., improvement on one task may degrade performance on the other due to differences in the computed block influences. Furthermore, we extend this analysis from a complete block to individual self-attention and feed-forward layers, highlighting the propensity of the self-attention layers to be more amendable to pruning, even allowing removal of upto 33% of the self-attention layers without incurring any performance degradation on MMLU for Mistral 7b (significant reduction in costly maintenance of KV-cache). Finally, we look at simple performance recovery techniques to emulate the pruned layers by training lightweight additive bias or low-rank linear adapters. Performance recovery using emulated updates avoids performance degradation for the initial blocks (up to 5% absolute improvement on MMLU), which is either competitive or superior to the learning-based technique.
Forward citations
Cited by 7 Pith papers
-
When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMs
Removing just one or two layers from a reasoning-tuned LLM destroys its test-time scaling, and neither LoRA nor full fine-tuning restores it.
-
OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
OrthoRank selects tokens for computation by their orthogonality to the sink token in normalized hidden states, yielding better perplexity and accuracy than layer pruning at matched sparsity without additional training.
-
Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN
Mix-LN, which uses Post-LN in early layers and Pre-LN in later layers, gives more uniform layer gradients and better LLM pretraining and fine-tuning results than Pre-LN or Post-LN alone.
-
Reassessing Layer Pruning in LLMs: New Insights and Methods
Trimming the final 25% of layers and fine-tuning the head and last three layers outperforms sophisticated pruning metrics and LoRA-based recovery for LLM compression.
-
DeltaLLM: Compress LLMs with Low-Rank Deltas between Shared Weights
Sharing weights between transformer layers with low-rank delta corrections compresses LLMs by 12 to 25 percent while retaining about 90 percent of benchmark performance.
-
Analyzing Memorization in Large Language Models through the Lens of Model Attribution
Bypassing attention in the deepest transformer layers reduces extractable memorization with little loss on standard benchmarks, while early-layer bypass collapses the model.
-
SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decoupling
SkipGPT learns per-token, per-module routers that dynamically skip attention and MLP blocks, then uses LoRA to recover most of the original model's quality.
Discussion (0). Continue with ORCID to comment.