Who Should Score the Cache? Mid-Depth Layers Are the Best Judges of KV-Cache Eviction

Short version: this is a real result with good evidence, and a serious editor should send it out. The question of which layer should score the cache is genuinely unasked in the prior eviction literature, and the paper answers it with unusually careful measurements: depth sweeps on eighteen per-layer models across labs and scales, a parameter-free survival model computed from prefill attention on calibration prompts the evaluation never uses, a heads-versus-depth control, a masking-versus-deletion equivalence check, clustered bootstrap intervals, exact sign-flip tests, and honest negative results on LongBench and on models from 32B up. The survival model is a genuine predictor rather than a fit: its picks are best or tied on all eighteen per-layer models, and it reproduces the shared-cache reader ranking on Gemma-4-E2B within a few points. The shared-cache penalty story is also compelling, and the sibling-pair control against Gemma-4-E4B is a sensible natural experiment, even though the authors correctly say it rests on one pair.

The main soft spot is scope of the central claim. Every depth curve uses the question-window scoring convention; the kvpress-window comparison in Table 1 tests only one mid-depth pick, not a full depth sweep, and the paper itself says it never compares numbers across the two windows. The survival model is built under the same convention. So "depth governs eviction quality at every scale we test" is established for question-window scoring, not for attention-scoring conventions generally. The authors' own no-question control shows why this matters: without the question, layers are nearly equally blind to the answer. I would not call this fatal, because the window choice is disclosed, but the abstract and introduction should either run a second full sweep under a different window or restate the law as conditional on the scoring convention.

The second soft spot is the practical headline. The 18 to 43 point gains over SnapKV are measured at per-model binding budgets chosen with knowledge of the default's behavior, on an eight-task pool. All of that is disclosed, and the gains persist over the full thirteen-task pool but shrink by 3 to 6 points. So the practical claim is real but budget- and task-pool-dependent; readers should not extrapolate the headline number to arbitrary settings.

Minor: I did not see a public code repository. The reproducibility statement is unusually detailed, but a compute-heavy empirical claim like this would benefit from a shipped implementation.

Who this is for: people working on KV-cache compression and efficient LLM inference. If that is your group, bring it to the reading group. My recommendation: accept for peer review, and ask the authors to tighten the window-dependence before publication.

The paper's central claim ('the depth of the scoring layer governs eviction quality at every scale we test', [b15]) rests on depth sweeps that all use the question-block scoring window defined in Section 3. Section 5.1's own control shows the depth signal is question-driven: without the question, layers are nearly equally blind to the answer. That suggests the curve is a property of question-visible scoring, not of layer depth per se. The Table 1 comparison against SnapKV uses the kvpress window (last 64 prefilled tokens) but only tests one mid-depth pick, not a full depth sweep, and the paper states it never compares numbers across the two windows. The survival model (Equation 1) is computed under the same question-window convention, so its predictive success does not independently establish a window-independent law. This is a scope condition the paper discloses rather than a contradiction, but it is the least secured step in the universal claim: if a full depth sweep under the kvpress window flattened, the abstract's 'depth governs' would need to be restated as 'depth governs under question-window scoring.' A reader should not treat the 18-model universal curve as covering arbitrary attention-scoring conventions without that check.

The reader's weakest assumption identified question-window attention mass on the answer and question visibility during prefill as the load-bearing step; the stress-test agrees and sharpens it into a scoring-convention scope condition. The paper's internal evidence is strong: a parameter-free survival model that matches the measured depth curves, calibration-stable picks on the significant models, a no-question control that moves in the predicted direction, a shared-cache prediction that reproduces five reader accuracies, exact sign-flip checks on the primary comparisons, and disclosed negative results on LongBench, large models, and alternate task pools. No internal inconsistency or correctness red flag was found. The practical headline gains are conditional on binding budgets and question-visible prefill, exactly as the paper states. The remaining unsecured step is narrow: the universal depth claim has not been shown under a second standard scoring window, because the kvpress comparison tests a single selected layer rather than a full depth curve. That keeps the verdict at CONDITIONAL but does not move it further.

If the results hold, the paper identifies a previously unreported degree of freedom in KV-cache eviction, namely the scoring layer, with a large practical effect at tight budgets, and supplies both a mechanistic explanation and a constant-free predictor that transfers to another task family. The strengths are substantial: evaluation breadth (21 models across scales and release dates), the discipline of never comparing numbers across the two scoring windows (Section 3), and an unusually transparent statistical and reproducibility apparatus. Every number is generated by a script from shared JSON files, one statistics implementation is used throughout, masked eviction is verified equivalent to physical deletion, and cluster bootstraps are checked with exact sign-flip tests that attain the minimum possible p-value (0.008) on the joint test. The survival model is a genuine predictor rather than a fit: it has no adjustable constants, is computed on calibration prompts disjoint from evaluation, and transfers to DocFinQA.

Every KV-cache eviction method ranks cached tokens by attention scores, and every existing method lets each layer score its own cache. This paper asks a question nobody had varied: which layer should do the scoring, and finds the answer is almost always a mid-depth layer. Across twenty-one models, retrieval accuracy collapses when a shallow layer scores the eviction, peaks between 40% and 75% of network depth, and degrades again at the last layer. The paper explains the curve with a parameter-free survival model that computes, from the uncompressed model's prefill attention and known answer spans, the probability that a gold answer survives the scorer's top-$k$ cut; the model reproduces the measured curves and picks a layer that is the best measured on most models. On small and mid-size models the chosen scorer with one shared token set beats per-head SnapKV by 18 to 43 points under tight memory budgets, and the gain carries to financial-document extraction at 32K context.

What carries the argument. The load-bearing object is the survival model, Equation (1), which treats eviction as a ranking problem: layer $\ell$ is a good scorer when it ranks the tokens the answer needs above the $k$-th place of its top-$k$ cut. For a prompt, the model reconstructs the set $K_\ell$ of tokens layer $\ell$ would keep — including the always-kept sink and recency tokens — and counts a gold answer as surviving if some occurrence of it lies entirely inside $K_\ell$, matching the all-or-nothing credit of the recall metric. Averaged over gold answers and prompts, this gives $\hat{S}_\ell$, a prediction computed entirely from the uncompressed model's prefill attention and the locations of known answer spans, with no adjustable constants and no generation. The survival score does two jobs: it reproduces the measured depth curves, including the reader ranking on a shared cache, and it selects the scoring layer. The mechanism behind the curve is that attention on the answer, driven by question tokens, builds with depth and peaks at mid-depth; without the question, layers are nearly equally blind to the answer, so the whole predictive story depends on question visibility during prefill. The paper tests the alternative explanation that the curve merely reflects where retrieval heads sit, and finds that mid-depth heads carry the signal but a single mid-depth layer collects them without a head search.

Core claim. The paper's central discovery is a universal depth curve: when a single layer scores every cache's eviction, long-context retrieval accuracy is a function of that layer's depth — a cliff at shallow layers, a peak between 0.4 and 0.75 of depth, and a decline at the last layer — on all eighteen per-layer models tested, from 0.5B to 109B parameters, with the same shape on shared-cache architectures. The mechanism offered is that a layer becomes a good judge only once its attention has used the question to locate the answer, a job that takes several layers; removing the question flattens the depth signal almost completely. Modeling eviction as a top-$k$ cut, the paper computes the probability that each gold answer survives the scorer's kept set — Equation (1), an all-or-nothing survival probability with no adjustable constants, built from prefill attention of the uncompressed model and known answer spans on 25 calibration prompts per task — and shows this survival score tracks the measured accuracy curves (Spearman 0.90–1.00) and selects a layer that is the best measured on fourteen of the eighteen models and never significantly worse on the other four. The same survival score computed from each reader's attention reproduces the reader ranking on a shared cache. The applied claim is that scoring every cache from the selected mid-depth layer with a single kept set for all heads beats per-head SnapKV by 18 to 43 points on five models up to 14B under tight budgets, ties on two Llama models, and transfers to financial-filing extraction at 32K context without an extra forward pass; from 32B up the default is already close to the best mid-depth layer. The paper also documents where the gain ends: on open-book question answering (LongBench), no mid-depth layer beats the own-layer default, and the survival model, built for exact-match answers, does not predict that outcome.

The premise it rests on. The predictive story rests on treating question-window attention mass on the gold answer, measured on 25 calibration prompts per task with known answer spans from the uncompressed model, as the proxy for eviction quality on unseen prompts; the paper itself shows the signal collapses when the question is removed, and the measured gains also depend on the chosen binding budget at which each model is compared.

Scoring-layer depth becomes a tunable, measurable decision: an eviction method can keep its scoring rule and gain accuracy simply by scoring from a selected mid-depth layer rather than each cache's own layer — the default loses to the best mid-depth scorer on all seven primary models and never significantly beats it anywhere.

Layer selection is cheap and automatic: the survival model uses only the uncompressed model's prefill attention and known answer spans on 25 calibration prompts per task, so the layer can be chosen once per model without a sweep over layers and with under 1% added prefill time.

A single kept set for all heads plus mid-depth scoring is a ready improvement to SnapKV-style methods at tight budgets on models up to 14B: gains of +18 to +43 points, recovering 83 to 99% of uncompressed accuracy where SnapKV keeps 37 to 63%.

Shared-cache architectures carry a hidden cost when the writer is shallow: on Gemma-4-E2B the default writer needs roughly twice a mid-depth reader's budget, and scoring by a deeper reader or by rank consensus over the readers recovers most of the gap.

The boundary of the effect is documented: from 32B up the own-layer default already sits near the best mid-depth layer, and on open-book QA no mid-depth layer beats the default — so the practical advice is strongest for small and mid-size models on exact-answer retrieval.

The RULER-selected layer transfers to real documents: on financial-filing extraction at 32K it beats the default by 8 to 12 points, and the combined method beats per-head SnapKV by 14 to 16 points there, all at the same memory budget with no extra forward pass.

Beyond eviction, the same depth logic likely constrains any scheme that borrows one layer's attention to decide what other layers keep — prefill token pruning, cross-layer compression, and training-time sparse schedules that fix a preceding layer as selector — and the survival model offers those settings a ready selection rule, since it needs only prefill attention and a notion of the span that mu

Because the best scorer differs by task family, which the paper states explicitly, a per-prefix selector that recognizes the task family could beat any single fixed layer; the survival model is cheap enough to evaluate on several families at once.

The question-dependence of the signal implies that prefix caches, where the question arrives after the cache is built, cannot use this rule directly; a testable extension would be ranking tokens by context-tail attention combined with a learned question prior, since the paper shows context alone flattens the curve.

The writer-depth explanation rests on one sibling pair; the paper itself proposes training the same model with a deeper writer as the decisive test, and the survival model makes that test quantitative — it predicts the shared-cache penalty shrinks to near zero when the writer sits at mid-depth.

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review