arxiv: 2604.24647 · v1 · submitted 2026-04-27 · 💻 cs.CL · cs.AI

Recognition: unknown

DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inference

Zahra Dehghanighobadi , Asja Fischer

Authors on Pith no claims yet

Pith reviewed 2026-05-08 03:43 UTC · model grok-4.3

classification 💻 cs.CL cs.AI

keywords KV cache pruninglong-context inferencelayer-dependent allocationLLM memory optimizationattention-based pruningtransformer efficiencyinference accelerationcontext length scaling

0 comments

The pith

LLMs perform better on long-context tasks when KV cache pruning ratios vary by layer sensitivity instead of staying uniform.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that current KV cache pruning methods waste budget by discarding the same fraction of tokens in every layer. It demonstrates that layers actually differ markedly in how much their outputs suffer when tokens are removed. DepthKV measures these differences and shifts the total pruning allowance so that sensitive layers keep more of their cache while less sensitive layers prune harder. Because the overall memory footprint stays fixed, the method improves accuracy on tasks that need long contexts without requiring extra hardware. This matters for practical deployment of LLMs that must reason over documents, code, or conversations spanning thousands of tokens.

Core claim

We show that the assumption of uniform contribution across layers in KV cache pruning is suboptimal because layers differ significantly in their sensitivity to pruning. We propose DepthKV, a layer-dependent pruning framework that allocates a fixed global KV budget across layers based on their sensitivity, rather than using a uniform allocation. Across multiple models and tasks, DepthKV consistently outperforms uniform pruning at the same global pruning ratio, demonstrating more effective utilization of the KV cache budget through layer-dependent allocation.

What carries the argument

DepthKV, the framework that first estimates each layer's sensitivity to token removal and then assigns different pruning ratios to meet a single total KV cache limit.

If this is right

At any fixed KV cache size, models retain higher accuracy on long-sequence reasoning tasks.
The same memory budget supports longer input sequences before performance degrades.
Inference becomes more memory-efficient without retraining or architectural changes to the base model.
Pruning can be pushed to higher overall ratios while preserving more of the original capability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same sensitivity principle could be applied to other memory structures such as activation caches or attention head pruning.
Dynamic re-estimation of sensitivities on the fly might further adapt to different input domains like code versus narrative text.
If layer sensitivities prove stable across model families, the method could be pre-computed once and reused for many deployments.

Load-bearing premise

That each layer's sensitivity to KV pruning can be measured reliably ahead of time and stays stable enough to guide budget allocation without adding noticeable overhead or error.

What would settle it

Run DepthKV and uniform pruning on the same models at identical global pruning ratios on long-document QA or summarization benchmarks; if the accuracy gap disappears or reverses, the layer-dependent allocation claim is falsified.

Figures

Figures reproduced from arXiv: 2604.24647 by Asja Fischer, Zahra Dehghanighobadi.

**Figure 1.** Figure 1: Uniform vs. layer-dependent KV allocation. Uniform allocation (left) assigns an equal KV budget across transformer layers. DepthKV (right) reallocates this budget based on sensitivity to pruning, retaining more tokens in critical layers (highlighted) and pruning less important ones more aggressively. Token rank denotes relative importance. suggests that intermediate transformer layers may play a more criti… view at source ↗

**Figure 2.** Figure 2: Single-layer KV cache pruning. Layer-wise ROUGE-1 under KV cache pruning of individual layers, standardized within each model–dataset pair (z-score; mean = 0, standard deviation = 1). Markers indicate the layer with the largest performance drop for each dataset. between layers are statistically significant. However, the magnitude of these differences (i.e., the effect size) depends on the dataset and the … view at source ↗

**Figure 4.** Figure 4: Layer Importance vs. InfoNCE. Standardized InfoNCE (post-attention) and ROUGE-1 performance drop across layers under KV cache pruning on the arXiv dataset. role (Skean et al., 2025), and further supported by our preliminary analysis, we preserve a subset of middle layers while pruning the remaining layers uniformly. Specifically, we define the middle layers as those surrounding the network midpoint, name… view at source ↗

**Figure 5.** Figure 5: GSM-∞ accuracy. Performance on the GSM- ∞ benchmark across different KV cache pruning settings. As shown in view at source ↗

read the original abstract

Long-context reasoning is a critical capability of large language models (LLMs), enabling applications such as long-document understanding, summarization, and code generation. However, efficient autoregressive inference relies on the key-value (KV) cache, whose memory footprint grows linearly with sequence length, leading to a major memory bottleneck. To mitigate this overhead, KV cache pruning methods discard cached tokens with low attention scores during inference. Most existing methods apply a uniform pruning ratio across layers, implicitly assuming that all layers contribute equally to overall model performance. We show that this assumption is suboptimal, as layers differ significantly in their sensitivity to pruning. We propose DepthKV, a layer-dependent pruning framework that allocates a fixed global KV budget across layers based on their sensitivity, rather than using a uniform allocation. Across multiple models and tasks, DepthKV consistently outperforms uniform pruning at the same global pruning ratio, demonstrating more effective utilization of the KV cache budget through layer-dependent allocation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

DepthKV shows that uniform KV pruning wastes budget because layers vary in sensitivity, and reallocating by measured sensitivity beats the uniform baseline at fixed global size.

read the letter

The central point is that most KV pruning methods assume every layer can lose the same fraction of tokens without much harm, but this paper argues and tests that some layers are far more sensitive than others. DepthKV measures that difference and gives the more sensitive layers a larger share of the fixed cache budget. The result is better task performance at the same total memory footprint, which is the kind of targeted fix that matters for long-context inference where cache size is the real constraint. The experiments apparently cover multiple models and tasks and show the advantage holds, which is the right way to make the claim credible rather than just theoretical. That part is useful and worth knowing about if you work on inference efficiency. The main soft spot is how the sensitivity scores are actually computed and whether they stay stable across tasks or require per-task tuning. If the measurement adds non-trivial overhead or turns out to be brittle, the practical gain shrinks. The paper also needs to be clear that the reported wins come from the allocation rule itself and not from extra hyperparameter search that the uniform baseline did not receive. Overall this is a clean, incremental idea aimed at people who already run long-context models and are hitting memory walls. It is not revolutionary, but the comparison is fair and the claim is easy to check, so it deserves a serious referee who can look at the exact sensitivity metric and the full experimental controls.

Referee Report

3 major / 1 minor

Summary. The paper claims that uniform KV cache pruning across layers is suboptimal because layers differ significantly in sensitivity to pruning. It proposes DepthKV, a layer-dependent framework that reallocates a fixed global KV budget according to per-layer sensitivity measurements rather than applying a uniform ratio. The central empirical claim is that DepthKV consistently outperforms uniform pruning at matched global pruning ratios across multiple models and tasks.

Significance. If the empirical results hold with the claimed consistency, the work would demonstrate a practical way to improve KV cache utilization in long-context inference without increasing the global budget or adding substantial overhead. This addresses a real memory bottleneck and could be adopted in production systems if the sensitivity allocation proves robust and low-cost.

major comments (3)

[Abstract] Abstract: the claim that DepthKV 'consistently outperforms uniform pruning at the same global pruning ratio' is presented without any quantitative results, specific metrics, error bars, or even the magnitude of improvement. This absence makes it impossible to assess whether the outperformance is statistically meaningful or practically relevant.
[Abstract] Abstract: the procedure for determining layer sensitivity is not described at all. It is unclear what metric is used (e.g., attention score drop, downstream task degradation), on what calibration data it is computed, whether the measurement is performed once or per sequence, and whether it introduces non-negligible extra computation or latency.
[Abstract] Abstract: no information is given on the experimental setup (models, sequence lengths, tasks, pruning ratios tested, or baselines beyond uniform pruning), nor on any error analysis or ablation of the sensitivity-based allocation itself.

minor comments (1)

[Abstract] The abstract would be clearer if it briefly named the models and tasks on which the consistent outperformance was observed.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback. We will revise the abstract to incorporate more specific details on our method, results, and experimental setup as suggested. The full details are already present in the body of the paper, but enhancing the abstract will improve clarity.

read point-by-point responses

Referee: [Abstract] Abstract: the claim that DepthKV 'consistently outperforms uniform pruning at the same global pruning ratio' is presented without any quantitative results, specific metrics, error bars, or even the magnitude of improvement. This absence makes it impossible to assess whether the outperformance is statistically meaningful or practically relevant.

Authors: We agree that the abstract should provide some quantitative indication of the improvements. In the revised version, we will include a summary of the key results, such as the average performance gains observed across the evaluated models and tasks at matched global pruning ratios. The detailed metrics, including any error bars and statistical analysis, are presented in the experimental section of the manuscript. revision: yes
Referee: [Abstract] Abstract: the procedure for determining layer sensitivity is not described at all. It is unclear what metric is used (e.g., attention score drop, downstream task degradation), on what calibration data it is computed, whether the measurement is performed once or per sequence, and whether it introduces non-negligible extra computation or latency.

Authors: The procedure for determining layer sensitivity is described in Section 3 of the manuscript. We will add a concise summary of this procedure to the abstract to address the lack of description there. This will clarify the metric used, the calibration data, and confirm that it is a one-time offline computation with negligible inference overhead. revision: yes
Referee: [Abstract] Abstract: no information is given on the experimental setup (models, sequence lengths, tasks, pruning ratios tested, or baselines beyond uniform pruning), nor on any error analysis or ablation of the sensitivity-based allocation itself.

Authors: We acknowledge that the abstract is high-level and omits these details. The manuscript provides comprehensive information on the experimental setup in Section 4, including the models used, sequence lengths, tasks, pruning ratios, and additional baselines. Ablations and error analysis are also included. We will revise the abstract to briefly mention the scope of the experiments. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical comparison to uniform baseline

full rationale

The paper's core argument is that uniform KV pruning assumes equal layer sensitivity (suboptimal) and that reallocating a fixed global budget by measured per-layer sensitivity yields better utilization. This is presented as an empirical proposal tested across models and tasks, with outperformance shown at matched global pruning ratios. No mathematical derivation, equations, or self-citation chain is visible in the abstract or claim structure. The sensitivity measurement is not defined circularly in terms of the final performance metric, nor is any result renamed or fitted-then-predicted. The load-bearing element is direct experimental comparison to the uniform baseline, which is independent of the proposed method. This is a standard empirical contribution with no reduction to inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Based solely on the abstract, which provides no specific technical details on implementation; the core domain assumption about layer sensitivity is noted but no free parameters or invented entities are identifiable.

axioms (1)

domain assumption Layers differ significantly in their sensitivity to KV cache pruning
Explicitly stated in the abstract as the reason uniform pruning is suboptimal.

pith-pipeline@v0.9.0 · 5462 in / 1264 out tokens · 81032 ms · 2026-05-08T03:43:30.751085+00:00 · methodology

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Shallow Prefill, Deep Decoding: Efficient Long-Context Inference via Layer-Asymmetric KV Visibility
cs.AI 2026-05 unverdicted novelty 6.0

SPEED uses layer-asymmetric KV visibility to process non-anchor prompt tokens only in lower layers during prefill, achieving near-baseline quality on Llama-3.1-8B with 33% better TTFT and 25% lower active KV memory at...

Reference graph

Works this paper leans on

7 extracted references · 3 canonical work pages · cited by 1 Pith paper

[1]

Efficient attentions for long document summarization,

Attention score is not all you need for token importance indicator in kv cache reduction: Value also matters. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Process- ing, pages 21158–21166. Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun S Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards ...

work page arXiv 2024
[2]

Retrievalattention: Accelerating long-context llm inference via vector retrieval.arXiv preprint arXiv:2409.10516. Meta AI. 2024. Meta llama 3.1. https://ai.meta. com/blog/meta-llama-3. Accessed 2024. Belinda Phipson and Gordon K Smyth. 2016. Permu- tation p-values should never be zero: calculating ex- act p-values when permutations are randomly drawn. arX...

work page arXiv 2024
[3]

arXiv, PubMed, GovReport:Accessed via ccdv on Hugging Face, these datasets are re- leased under the Apache-2.0 License
[4]

Supreme Court decisions and adheres to the terms of the original sources

LegalCase:Provided by the authors at Law- AI (GitHub), this dataset is constructed from publicly available Indian and U.K. Supreme Court decisions and adheres to the terms of the original sources
[5]

Qasper:Available via AllenAI on Hugging Face, the dataset is distributed under the CC- BY-4.0 License
[6]

HotpotQA:Obtained from HotpotQA on Hugging Face, the dataset is released under the CC-BY-SA-4.0 License
[7]

A.2 Model Licenses We employ open-weight language models accessed via the Hugging Face Transformers library (Wolf et al., 2020)

GSM-∞:Released by the authors at Infini- AI-Lab (GitHub), this synthetic dataset is gen- erated programmatically and is used in accor- dance with the terms specified in the reposi- tory. A.2 Model Licenses We employ open-weight language models accessed via the Hugging Face Transformers library (Wolf et al., 2020). Specifically, the models used are: •googl...

work page arXiv 2020