{"id":"26e00df8-0a5b-40c4-a908-d5a7eaf2ed6f","arxiv_id":"2611.02230","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"KV-cache eviction quality peaks when a mid-depth layer, not the cache's own layer, scores the eviction, and a parameter-free survival model can pick that layer.","lead":"This paper finds that the layer chosen to score KV-cache eviction matters more than which eviction method is used, with mid-depth layers performing best across 18 to 21 language models. A smart generalist might read it because it identifies a simple, nearly free design choice that can recover large fractions of lost accuracy under tight memory budgets, and it explains a known penalty in shared-cache architectures.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"The universal depth curve is established only under question-window scoring, so the central claim remains conditional on scoring convention.","rationale":"The reader's weakest assumption identified question-window attention mass on the answer and question visibility during prefill as the load-bearing step; the stress-test agrees and sharpens it into a scoring-convention scope condition. The paper's internal evidence is strong: a parameter-free survival model that matches the measured depth curves, calibration-stable picks on the significant models, a no-question control that moves in the predicted direction, a shared-cache prediction that reproduces five reader accuracies, exact sign-flip checks on the primary comparisons, and disclosed negative results on LongBench, large models, and alternate task pools. No internal inconsistency or correctness red flag was found. The practical headline gains are conditional on binding budgets and question-visible prefill, exactly as the paper states. The remaining unsecured step is narrow: the universal depth claim has not been shown under a second standard scoring window, because the kvpress comparison tests a single selected layer rather than a full depth curve. That keeps the verdict at CONDITIONAL but does not move it further.","tokens_in":25275,"tokens_out":17792,"duration_ms":131089,"concrete_test":"Run the full scorer-depth sweep (all seven grid layers plus the survival model's pick) on Qwen2.5-1.5B and Llama-3.1-8B using the kvpress scoring window (last 64 prefilled tokens, 5-token smoothing, per-head sets) at the binding budgets of Table 1. If the mid-depth peak and the picked layer's advantage persist within one grid step, the concern is resolved; if the curve flattens or the best layer leaves the 0.4-0.75 depth band, the universal depth-curve claim must be restricted to the question-window convention.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim ('the depth of the scoring layer governs eviction quality at every scale we test', [b15]) rests on depth sweeps that all use the question-block scoring window defined in Section 3. Section 5.1's own control shows the depth signal is question-driven: without the question, layers are nearly equally blind to the answer. That suggests the curve is a property of question-visible scoring, not of layer depth per se. The Table 1 comparison against SnapKV uses the kvpress window (last 64 prefilled tokens) but only tests one mid-depth pick, not a full depth sweep, and the paper states it never compares numbers across the two windows. The survival model (Equation 1) is computed under the same question-window convention, so its predictive success does not independently establish a window-independent law. This is a scope condition the paper discloses rather than a contradiction, but it is the least secured step in the universal claim: if a full depth sweep under the kvpress window flattened, the abstract's 'depth governs' would need to be restated as 'depth governs under question-window scoring.' A reader should not treat the 18-model universal curve as covering arbitrary attention-scoring conventions without that check.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This paper asks which transformer layer should supply the attention scores used to evict tokens from the KV cache. On eighteen per-layer models (0.5B to 109B, four model families) and three shared-cache models, the authors measure retrieval accuracy on an eight-task RULER pool as a function of scorer depth, using the question-block scoring window defined in Section 3. They report a universal curve: shallow scorers collapse (25 to 56 percentage points below the best layer), the peak lies at 0.4 to 0.75 of depth, and the last layer degrades. The paper proposes a parameter-free survival model (Equation 1) that estimates, from the uncompressed model's prefill attention and gold-answer spans on calibration prompts disjoint from evaluation, the probability that an answer survives the scorer's top-k cut; it reproduces the measured depth curves (Spearman 0.90 to 1.00, MAE 2.7 to 7.4pp) and picks the best measured layer on 14 of 18 per-layer models, within noise on the other four. A mechanism test shows the depth signal is question-driven: removing the question flattens the curve. Practically, scoring every cache from the picked layer with one kept set for all heads beats per-head SnapKV by 18 to 43pp at tight budgets on models up to 14B and on DocFinQA at 32K, with no detectable gain on 32B+ models and an honestly reported negative on LongBench.","tokens_in":27641,"tokens_out":2381,"duration_ms":166124,"significance":"If the results hold, the paper identifies a previously unreported degree of freedom in KV-cache eviction, namely the scoring layer, with a large practical effect at tight budgets, and supplies both a mechanistic explanation and a constant-free predictor that transfers to another task family. The strengths are substantial: evaluation breadth (21 models across scales and release dates), the discipline of never comparing numbers across the two scoring windows (Section 3), and an unusually transparent statistical and reproducibility apparatus. Every number is generated by a script from shared JSON files, one statistics implementation is used throughout, masked eviction is verified equivalent to physical deletion, and cluster bootstraps are checked with exact sign-flip tests that attain the minimum possible p-value (0.008) on the joint test. The survival model is a genuine predictor rather than a fit: it has no adjustable constants, is computed on calibration prompts disjoint from evaluation, and transfers to DocFinQA.","major_comments":[{"comment":"The headline claim that 'the depth of the scoring layer governs eviction quality' is broader than the evidence. All depth sweeps (Figures 1 and 2, Table 2) use the question-window scoring convention defined in Section 3; the kvpress-window comparison in Table 1 tests only the survival model's single pick per model, not a full depth sweep; and Section 3 states that numbers are never compared across the two windows. Section 5.1's own control shows the depth signal is question-driven: without the question, the reader-writer attention-mass gap on Gemma-4 shrinks by a factor of 6.5, and the per-model question/context-tail ratios drop to 1.4 to 5.0. The paper discloses the window convention in Section 3 and the question-visibility limitation in Section 7, but the abstract and title state the universal claim without this scope condition. I ask the authors to restate the universal depth-curve claim as conditional on question-visible scoring, or preferably to run a depth sweep under the kvpress window on at least two models (for example Qwen2.5-1.5B and Llama-3.1-8B) to show whether the curve shape transfers to the second scoring convention. This is a scope correction rather than a contradiction, but it is load-bearing because the unqualified claim is the paper's central advertised finding.","section":"Abstract; §1; §3; §5.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The binding budgets at which every headline margin is reported appear to be selected on a subset of the evaluation set. Section A.1 states that 'the scan uses the first ten evaluation prompts per task,' and the main results use 25 prompts per task (n = 200) from the same pool; if the ten scan prompts are the first ten of those 25, the budget is chosen with knowledge of 40% of the evaluation data per task. This does not bias the fixed-budget comparison among scorers, and the multi-budget evidence in Table A4 (Qwen2.5-1.5B at 5 to 50%) shows the qualitative conclusions are robust. However, the magnitude claims in the abstract ('+18 to +43pp', '+26.6pp', 'loses 28 points') are conditional on budgets chosen partly from the evaluation set, and the protocol also selects budgets where the default demonstrably loses on those very prompts. Please state explicitly whether the scan prompts are disjoint from the n = 200 evaluation prompts; if they are not, verify the headline margins at a budget selected without access to evaluation prompts.","section":"§3; §A.1; Abstract","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The abstract's 'beats per-head SnapKV by 18 to 43 points on five models up to 14B' is verifiable only by combining Table 1 (four distinct models across five columns, with Qwen2.5-1.5B at two budgets) with Table A4 (which adds Qwen2.5-0.5B). A cross-reference in the Table 1 caption would make the count checkable at a glance.","section":"Abstract; Table 1; Table A4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The GemFilter head-to-head uses question-window scoring ('L20 under the question window, as in Figure 1') while the surrounding Table 1 uses the kvpress window. The text states this, but an explicit sentence that the GemFilter comparison is not comparable with the Table 1 columns would prevent a reader from treating the 89.3% versus 36.5% as part of the SnapKV comparison.","section":"§4.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The third of the four Table 1 combination-over-SnapKV comparisons (Qwen2.5-14B @5%) has p = 0.055 under the exact sign-flip test while its cluster bootstrap CI excludes zero. The p-value is disclosed, but the main text counts this column among the 'clustered-significant' gains; a footnote noting that this margin is marginal under the exact test in isolation, with the joint test giving p = 0.008, would sharpen the reporting.","section":"§A.10; Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The sentence 'The same scorer at the same budget shifts by several points between the two windows' justifies never comparing across windows; a single quantified example would make the caution concrete and would also help readers interpret the size of the Table 1 gains.","section":"§3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The abstract says the curve 'explains' the shared-cache penalty, while Section 7 correctly states that the writer-depth explanation rests on one sibling pair. 'Is consistent with' or 'predicts' would be the proportionate wording until the deeper-writer experiment suggested in Section 7 is run.","section":"Abstract; §6; §7","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"With 'No multiple-comparison correction is applied,' the 18-model pick-versus-best-other claims in Table A2 should be read descriptively: across 18 models and 7 to 10 measured layers per model, some 'within noise' and 'significant' margins are expected by chance. The joint sign-flip test on the six primary comparisons is reassuring; a sentence acknowledging the descriptive status of the per-model comparisons in Section A.10 would complete the picture.","section":"§3; §A.10","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Short version: this is a real result with good evidence, and a serious editor should send it out. The question of which layer should score the cache is genuinely unasked in the prior eviction literature, and the paper answers it with unusually careful measurements: depth sweeps on eighteen per-layer models across labs and scales, a parameter-free survival model computed from prefill attention on calibration prompts the evaluation never uses, a heads-versus-depth control, a masking-versus-deletion equivalence check, clustered bootstrap intervals, exact sign-flip tests, and honest negative results on LongBench and on models from 32B up. The survival model is a genuine predictor rather than a fit: its picks are best or tied on all eighteen per-layer models, and it reproduces the shared-cache reader ranking on Gemma-4-E2B within a few points. The shared-cache penalty story is also compelling, and the sibling-pair control against Gemma-4-E4B is a sensible natural experiment, even though the authors correctly say it rests on one pair.\n\nThe main soft spot is scope of the central claim. Every depth curve uses the question-window scoring convention; the kvpress-window comparison in Table 1 tests only one mid-depth pick, not a full depth sweep, and the paper itself says it never compares numbers across the two windows. The survival model is built under the same convention. So \"depth governs eviction quality at every scale we test\" is established for question-window scoring, not for attention-scoring conventions generally. The authors' own no-question control shows why this matters: without the question, layers are nearly equally blind to the answer. I would not call this fatal, because the window choice is disclosed, but the abstract and introduction should either run a second full sweep under a different window or restate the law as conditional on the scoring convention.\n\nThe second soft spot is the practical headline. The 18 to 43 point gains over SnapKV are measured at per-model binding budgets chosen with knowledge of the default's behavior, on an eight-task pool. All of that is disclosed, and the gains persist over the full thirteen-task pool but shrink by 3 to 6 points. So the practical claim is real but budget- and task-pool-dependent; readers should not extrapolate the headline number to arbitrary settings.\n\nMinor: I did not see a public code repository. The reproducibility statement is unusually detailed, but a compute-heavy empirical claim like this would benefit from a shipped implementation.\n\nWho this is for: people working on KV-cache compression and efficient LLM inference. If that is your group, bring it to the reading group. My recommendation: accept for peer review, and ask the authors to tighten the window-dependence before publication.","headline":"A genuinely useful measured result—scoring-layer depth matters for eviction—but the universal \"depth governs\" claim is only established under question-window scoring; worth a serious referee, not a desk reject.","tokens_in":26132,"tokens_out":3196,"duration_ms":31962},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Which layer scores KV-cache eviction governs retrieval quality: shallow scorers collapse, mid-depth wins across 21 models, and a parameter-free survival model from prefill attention selects the layer.","keywords":["KV cache eviction","scoring layer depth","attention scoring","long-context retrieval","survival model","shared KV cache","SnapKV","RULER benchmark"],"falsifier":"Give the survival model a model whose layers find the answer early, for instance a small model fine-tuned so that question-window attention on the answer peaks before depth 0.3, and measure the depth curve at the binding budget: the claim predicts the accuracy peak follows the attention peak, so a curve that still collapses at shallow layers and peaks at mid-depth would falsify the mechanism. On the shared-cache side, obtaining or training a Gemma-4-E2B-equivalent with the writer placed near depth 0.6 would settle the writer-depth explanation, because the claim predicts the penalty largely disappears, as it does on the E4B sibling.","tokens_in":29094,"feed_emoji":"🎯","tokens_out":2773,"duration_ms":112899,"temperature":0.7,"pith_summary":"Every KV-cache eviction method ranks cached tokens by attention scores, and every existing method lets each layer score its own cache. This paper asks a question nobody had varied: which layer should do the scoring, and finds the answer is almost always a mid-depth layer. Across twenty-one models, retrieval accuracy collapses when a shallow layer scores the eviction, peaks between 40% and 75% of network depth, and degrades again at the last layer. The paper explains the curve with a parameter-free survival model that computes, from the uncompressed model's prefill attention and known answer spans, the probability that a gold answer survives the scorer's top-$k$ cut; the model reproduces the measured curves and picks a layer that is the best measured on most models. On small and mid-size models the chosen scorer with one shared token set beats per-head SnapKV by 18 to 43 points under tight memory budgets, and the gain carries to financial-document extraction at 32K context.","feed_headline":"Which layer prunes the cache? Mid-depth wins on every model tested","feed_subtitle":"A parameter-free selection rule from prefill attention beats SnapKV by 18 to 43 points.","key_machinery":"The load-bearing object is the survival model, Equation (1), which treats eviction as a ranking problem: layer $\\ell$ is a good scorer when it ranks the tokens the answer needs above the $k$-th place of its top-$k$ cut. For a prompt, the model reconstructs the set $K_\\ell$ of tokens layer $\\ell$ would keep — including the always-kept sink and recency tokens — and counts a gold answer as surviving if some occurrence of it lies entirely inside $K_\\ell$, matching the all-or-nothing credit of the recall metric. Averaged over gold answers and prompts, this gives $\\hat{S}_\\ell$, a prediction computed entirely from the uncompressed model's prefill attention and the locations of known answer spans, with no adjustable constants and no generation. The survival score does two jobs: it reproduces the measured depth curves, including the reader ranking on a shared cache, and it selects the scoring layer. The mechanism behind the curve is that attention on the answer, driven by question tokens, builds with depth and peaks at mid-depth; without the question, layers are nearly equally blind to the answer, so the whole predictive story depends on question visibility during prefill. The paper tests the alternative explanation that the curve merely reflects where retrieval heads sit, and finds that mid-depth heads carry the signal but a single mid-depth layer collects them without a head search.","core_discovery":"The paper's central discovery is a universal depth curve: when a single layer scores every cache's eviction, long-context retrieval accuracy is a function of that layer's depth — a cliff at shallow layers, a peak between 0.4 and 0.75 of depth, and a decline at the last layer — on all eighteen per-layer models tested, from 0.5B to 109B parameters, with the same shape on shared-cache architectures. The mechanism offered is that a layer becomes a good judge only once its attention has used the question to locate the answer, a job that takes several layers; removing the question flattens the depth signal almost completely. Modeling eviction as a top-$k$ cut, the paper computes the probability that each gold answer survives the scorer's kept set — Equation (1), an all-or-nothing survival probability with no adjustable constants, built from prefill attention of the uncompressed model and known answer spans on 25 calibration prompts per task — and shows this survival score tracks the measured accuracy curves (Spearman 0.90–1.00) and selects a layer that is the best measured on fourteen of the eighteen models and never significantly worse on the other four. The same survival score computed from each reader's attention reproduces the reader ranking on a shared cache. The applied claim is that scoring every cache from the selected mid-depth layer with a single kept set for all heads beats per-head SnapKV by 18 to 43 points on five models up to 14B under tight budgets, ties on two Llama models, and transfers to financial-filing extraction at 32K context without an extra forward pass; from 32B up the default is already close to the best mid-depth layer. The paper also documents where the gain ends: on open-book question answering (LongBench), no mid-depth layer beats the own-layer default, and the survival model, built for exact-match answers, does not predict that outcome.","pith_inferences":["Beyond eviction, the same depth logic likely constrains any scheme that borrows one layer's attention to decide what other layers keep — prefill token pruning, cross-layer compression, and training-time sparse schedules that fix a preceding layer as selector — and the survival model offers those settings a ready selection rule, since it needs only prefill attention and a notion of the span that mu","Because the best scorer differs by task family, which the paper states explicitly, a per-prefix selector that recognizes the task family could beat any single fixed layer; the survival model is cheap enough to evaluate on several families at once.","The question-dependence of the signal implies that prefix caches, where the question arrives after the cache is built, cannot use this rule directly; a testable extension would be ranking tokens by context-tail attention combined with a learned question prior, since the paper shows context alone flattens the curve.","The writer-depth explanation rests on one sibling pair; the paper itself proposes training the same model with a deeper writer as the decisive test, and the survival model makes that test quantitative — it predicts the shared-cache penalty shrinks to near zero when the writer sits at mid-depth."],"forward_implications":["Scoring-layer depth becomes a tunable, measurable decision: an eviction method can keep its scoring rule and gain accuracy simply by scoring from a selected mid-depth layer rather than each cache's own layer — the default loses to the best mid-depth scorer on all seven primary models and never significantly beats it anywhere.","Layer selection is cheap and automatic: the survival model uses only the uncompressed model's prefill attention and known answer spans on 25 calibration prompts per task, so the layer can be chosen once per model without a sweep over layers and with under 1% added prefill time.","A single kept set for all heads plus mid-depth scoring is a ready improvement to SnapKV-style methods at tight budgets on models up to 14B: gains of +18 to +43 points, recovering 83 to 99% of uncompressed accuracy where SnapKV keeps 37 to 63%.","Shared-cache architectures carry a hidden cost when the writer is shallow: on Gemma-4-E2B the default writer needs roughly twice a mid-depth reader's budget, and scoring by a deeper reader or by rank consensus over the readers recovers most of the gap.","The boundary of the effect is documented: from 32B up the own-layer default already sits near the best mid-depth layer, and on open-book QA no mid-depth layer beats the default — so the practical advice is strongest for small and mid-size models on exact-answer retrieval.","The RULER-selected layer transfers to real documents: on financial-filing extraction at 32K it beats the default by 8 to 12 points, and the combined method beats per-head SnapKV by 14 to 16 points there, all at the same memory budget with no extra forward pass."],"supporting_citations":[{"why":"SnapKV: the per-head, own-layer-scored eviction baseline that Table 1 modifies; its two defaults, the scoring layer and the per-head token sets, are exactly what the paper changes.","marker":"(Li et al., 2024)"},{"why":"RULER: the eight-task long-context retrieval pool on which every depth curve, survival-model pick, and method comparison is measured.","marker":"(Hsieh et al., 2024)"},{"why":"DocFinQA: the financial-filing task that tests whether the RULER-selected layer and the Table 1 gains transfer to real documents at 32K.","marker":"(Reddy et al., 2024)"},{"why":"LongBench: the open-book QA benchmark that supplies the negative result bounding the claim — no mid-depth layer beats the default there.","marker":"(Bai et al., 2024)"},{"why":"GemFilter: the nearest prior work that moves selection off the own layer (in recompression); its layer ablation first showed a middle-layer peak and it provides the head-to-head of Section 4.1.","marker":"(Shi et al., 2026)"},{"why":"Retrieval heads: the mid-depth head concentration the paper tests as the alternative explanation for the depth curve and controls for in Section 5.1.","marker":"(Wu et al., 2025)"},{"why":"Gemma 4 technical report: the shared-cache architecture, one writer cache read by several layers, whose writer-depth penalty is documented in Section 6.","marker":"(Gemma Team, 2026)"},{"why":"Attention sinks: defines the always-kept sink and recency tokens that enter the eviction protocol and the survival model's kept set.","marker":"(Xiao et al., 2024)"},{"why":"Supplies the SnapKV, H2O, and PyramidKV baseline semantics the paper ports for Table 1; the paper verifies over 99.8% agreement in kept tokens on Llama-3.1-8B.","marker":"kvpress reference implementation (v0.5.4, https://github.com/NVIDIA/kvpress)"}],"fun_headline_variants":["Mid-depth layers are the best judges of KV-cache eviction","Which layer should score the KV cache? Mid-depth, on eighteen models","KV-cache eviction: scoring depth peaks at mid layers, not the last","Use a mid-depth layer to judge KV-cache eviction, not the writer","Answer survival from prefill attention picks the best scoring layer"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The predictive story rests on treating question-window attention mass on the gold answer, measured on 25 calibration prompts per task with known answer spans from the uncompressed model, as the proxy for eviction quality on unseen prompts; the paper itself shows the signal collapses when the question is removed, and the measured gains also depend on the chosen binding budget at which each model is compared.","fun_headline_variants_meta":{"raw":{"variants":["Mid-depth layers are the best judges of KV-cache eviction","Which layer should score the KV cache? Mid-depth, on eighteen models","KV-cache eviction: scoring depth peaks at mid layers, not the last","Use a mid-depth layer to judge KV-cache eviction, not the writer","Answer survival from prefill attention picks the best scoring layer"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000237,"raw_usage":{"cost":0.0002371824,"is_byok":false,"cost_details":{"upstream_inference_cost":0.0002371824,"upstream_inference_prompt_cost":4.44444e-05,"upstream_inference_completions_cost":0.000192738},"total_tokens":2445,"prompt_tokens":1386,"completion_tokens":1059,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":1122,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":976}},"tokens_in":1386,"tokens_out":1059,"cost_source":"openrouter_usage","duration_ms":10317,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":976,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-03T06:09:15.939107+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Give the survival model a model whose layers find the answer early, for instance a small model fine-tuned so that question-window attention on the answer peaks before depth 0.3, and measure the depth curve at the binding budget: the claim predicts the accuracy peak follows the attention peak, so a curve that still collapses at shallow layers and peaks at mid-depth would falsify the mechanism. On the shared-cache side, obtaining or training a Gemma-4-E2B-equivalent with the writer placed near depth 0.6 would settle the writer-depth explanation, because the claim predicts the penalty largely disappears, as it does on the E4B sibling.","supporting_citations":[{"cited_title":"Discovering the gems in early layers: Accelerating long-context LLMs with 1000x input token reduction","cited_arxiv_id":null,"evidence_quote":"GemFilter: the nearest prior work that moves selection off the own layer (in recompression); its layer ablation first showed a middle-layer peak and it provides the head-to-head of Section 4.1."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}