REVIEW 8 cited by
Investigating Layer Importance in Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Investigating Layer Importance in Large Language Models
read the original abstract
Large language models (LLMs) have gained increasing attention due to their prominent ability to understand and process texts. Nevertheless, LLMs largely remain opaque. The lack of understanding of LLMs has obstructed the deployment in safety-critical scenarios and hindered the development of better models. In this study, we advance the understanding of LLM by investigating the significance of individual layers in LLMs. We propose an efficient sampling method to faithfully evaluate the importance of layers using Shapley values, a widely used explanation framework in feature attribution and data valuation. In addition, we conduct layer ablation experiments to assess the performance degradation resulting from the exclusion of specific layers. Our findings reveal the existence of cornerstone layers, wherein certain early layers can exhibit a dominant contribution over others. Removing one cornerstone layer leads to a drastic collapse of the model performance, often reducing it to random guessing. Conversely, removing non-cornerstone layers results in only marginal performance changes. This study identifies cornerstone layers in LLMs and underscores their critical role for future research.
Forward citations
Cited by 8 Pith papers
-
Are LLM Uncertainty and Correctness Encoded by the Same Features? A Functional Dissociation via Sparse Autoencoders
Uncertainty and correctness in LLMs are encoded by distinct feature populations, with suppression of confounded features improving accuracy and reducing entropy.
-
Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training
A single middle transformer layer trained in isolation recovers most RL post-training gains in LLMs, with gains concentrated in middle layers across models, algorithms, and tasks.
-
Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training
Training a single middle transformer layer recovers most gains from full-parameter RL post-training across models, tasks, and algorithms.
-
Statistically-Lossless Quantization of Large Language Models
SLQ achieves task-lossless LLM quantization below 4 bits per parameter and distribution-lossless at 5-6 bits on average, with 1.7-3.6x speedups over FP16.
-
You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations
TAQ estimates per-layer importance from hidden representations and output sensitivity on task calibration data to allocate mixed precision in a training-free PTQ setting, outperforming task-agnostic baselines on accur...
-
EvolKV: Evolutionary KV Cache Compression for LLM Inference
CMA-ES search over per-layer KV cache budgets beats uniform and pyramidal compression heuristics on LongBench, NIAH, RULER, and GSM8K, and edges past the full cache on one code dataset at 1.5% of the budget.
-
You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations
A per-layer, per-task bit-allocation rule based on hidden-activation entropy and variance reportedly matches full-precision TriviaQA accuracy within about one point — but the paper's tables and abstract are internally...
-
RAP: Runtime Adaptive Pruning for LLM Inference
RAP is a reinforcement learning framework for runtime-adaptive pruning of LLMs that jointly optimizes model weights and KV-cache usage under varying memory budgets.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.