{"paper":{"title":"Who Should Score the Cache? Mid-Depth Layers Are the Best Judges of KV-Cache Eviction","arxiv_id":"2611.02230","primary_cat":"cs.LG","submitted_at":"2026-11-01T21:46:00+00:00","version_compiled":"v1"},"notice":"Every quoted sentence is the authors' text from the arXiv listing abstract. Labels and probabilities are the outputs of classifier-1.13.0 on the question bank abstract-p1-20260923, exactly as returned. No sentence in this record was written by a language model.","schema":"pith.compiled-record/v0","status":"compiled","abstract":{"source":"arXiv listing abstract v1","sentences":[{"i":0,"p":1.0,"id":"s0","own":false,"role":"background","text":"Key-value (KV) cache eviction methods rank tokens by attention scores, and each layer scores its own cache.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.07,"main_claim_p":0.02,"quantitative":false,"quantitative_p":0.07,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.0,"background":1.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":1,"p":1.0,"id":"s1","own":false,"role":"gap","text":"To our knowledge, no eviction method asks which layer should score.","hedge":{"p":0.91,"label":"asserted"},"own_p":0.09,"main_claim_p":0.03,"quantitative":false,"quantitative_p":0.09,"role_probabilities":{"gap":1.0,"method":0.0,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":2,"p":0.99,"id":"s2","own":true,"role":"result","text":"We find, across twenty-one models, that the depth of the scoring layer governs eviction quality.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.96,"main_claim_p":0.59,"quantitative":true,"quantitative_p":0.67,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.99,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.01,"interpretation":0.0}},{"i":3,"p":1.0,"id":"s3","own":true,"role":"result","text":"On the eighteen models with per-layer caches, spanning 0.5B to 109B, long-context retrieval accuracy collapses at shallow layers, peaks between 0.4 and 0.75 of depth, and degrades at the last layer.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.95,"main_claim_p":0.49,"quantitative":true,"quantitative_p":0.99,"role_probabilities":{"gap":0.0,"method":0.0,"result":1.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":4,"p":1.0,"id":"s4","own":true,"role":"interpretation","text":"The curve arises because a layer judges tokens well only once it has used the question to find the answer, which takes several layers.","hedge":{"p":0.98,"label":"asserted"},"own_p":0.76,"main_claim_p":0.1,"quantitative":false,"quantitative_p":0.11,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":1.0}},{"i":5,"p":0.8,"id":"s5","own":true,"role":"method","text":"Modeling eviction as a top-$k$ cut, we compute answer survival probability from prefill attention and reproduce the measured curves.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.94,"main_claim_p":0.05,"quantitative":false,"quantitative_p":0.07,"role_probabilities":{"gap":0.0,"method":0.8,"result":0.11,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.08,"interpretation":0.01}},{"i":6,"p":0.6,"id":"s6","own":true,"role":"method","text":"This approach selects the scoring layer using calibration prompts with known answers, and the selected layer is the best layer measured on most models and never significantly below it.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.95,"main_claim_p":0.21,"quantitative":true,"quantitative_p":0.64,"role_probabilities":{"gap":0.0,"method":0.6,"result":0.31,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.09,"interpretation":0.0}},{"i":7,"p":0.99,"id":"s7","own":true,"role":"result","text":"It improves on the own-layer default on sixteen of the eighteen models, with significant gains on seven, all at 14B or below, while on larger models the default is already close to the best mid-depth layer.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.96,"main_claim_p":0.13,"quantitative":true,"quantitative_p":0.99,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.99,"background":0.0,"limitation":0.01,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":8,"p":0.99,"id":"s8","own":true,"role":"result","text":"Scoring every cache from the selected layer while keeping one set of tokens for all heads also beats per-head SnapKV by 18 to 43 points on five models up to 14B under tight budgets.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.94,"main_claim_p":0.11,"quantitative":true,"quantitative_p":0.99,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.99,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.01,"interpretation":0.0}},{"i":9,"p":1.0,"id":"s9","own":true,"role":"result","text":"This gain carries over to financial filings at 32K context, all at the same memory budget, with no extra forward pass.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.92,"main_claim_p":0.06,"quantitative":true,"quantitative_p":0.81,"role_probabilities":{"gap":0.0,"method":0.0,"result":1.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":10,"p":0.61,"id":"s10","own":true,"role":"interpretation","text":"The same curve explains a penalty in shared-cache models, where several layers read one cache and the layer that writes it scores by default.","hedge":{"p":0.99,"label":"asserted"},"own_p":0.82,"main_claim_p":0.08,"quantitative":false,"quantitative_p":0.14,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.39,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.61}},{"i":11,"p":1.0,"id":"s11","own":true,"role":"result","text":"On Gemma-4-E2B the writer is shallow and loses 28 points to a mid-depth reader at a 10% budget.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.94,"main_claim_p":0.05,"quantitative":true,"quantitative_p":0.99,"role_probabilities":{"gap":0.0,"method":0.0,"result":1.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":12,"p":0.8,"id":"s12","own":true,"role":"result","text":"We also find that in open-book question answering (LongBench) no mid-depth layer beats the own-layer default.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.95,"main_claim_p":0.04,"quantitative":true,"quantitative_p":0.87,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.8,"background":0.0,"limitation":0.2,"future_work":0.0,"contribution":0.0,"interpretation":0.0}}]},"compiled":{"main_claim":{"p":0.86,"text":"We find, across twenty-one models, that the depth of the scoring layer governs eviction quality.","location":"abstract s2","hedge_strength":{"p":1.0,"label":"asserted"}},"provenance":{"route":{"via":"TypeSafe API","model":"classifier-1.13.0","endpoint":"https://api.typesafe.ai/v1/systemone","provider":"TypeSafe"},"packing":"one request per paper","requests":1,"input_tokens":9606,"cost_usd_reported":0.0},"compiled_at":"2026-11-03","bank_version":"abstract-p1-20260923","model_version":"classifier-1.13.0","abstract_states":{"claims_first":0.66,"states_limitation":0.92,"human_participants":0.03,"controlled_comparison":0.96,"randomized_assignment":0.02,"states_quantitative_result":0.99,"compares_with_prior_results":0.96,"mentions_code_or_data_availability":0.03},"evidential_basis":{"p":1.0,"value":"observation or experiment","probabilities":{"argument":0.0,"literature":0.0,"proof_or_derivation":0.0,"analysis_of_existing_data":0.0,"observation_or_experiment":1.0,"simulation_or_computation":0.0}},"contribution_type":{"p":0.55,"value":"method or tool","probabilities":{"review":0.0,"method_or_tool":0.55,"empirical_study":0.45,"simulation_study":0.0,"theoretical_result":0.0,"dataset_or_benchmark":0.0,"position_or_commentary":0.0}},"contribution_statement":{"p":0.7,"text":"This approach selects the scoring layer using calibration prompts with known answers, and the selected layer is the best layer measured on most models and never significantly below it.","location":"abstract s6"}},"coverage":{"tier":"abstract","levels_run":["P abstract"],"not_examined":["full text","figures and tables","references and citing sentences","relations to other papers"],"sources_read":["arXiv listing abstract v1"],"levels_not_run":["S sections","T body sentences","C citations","R cross-paper relations"],"sentences_examined":13,"sentences_in_abstract":13},"measured_accuracy":{"url":"https://pith.science/accuracy","measured_on":"2026-10-02","method":"two independent model families labelled a fresh random sample blind; the figure is how often the published label matches their consensus","sentence_role_agreement":0.92,"main_claim_agreement":0.908,"typed_citation_agreement":0.933,"typed_citation_agreement_95_interval":[0.917,0.945],"review_comments_accurate":0.758,"review_comments_accurate_95_interval":[0.684,0.82],"review_caught_admitted_error_found_or_related":0.448,"review_comments_judged":{"judged_by":["gpt-5.6-luna","grok-4.7"],"judged_against":"Pith's own text copy of each paper: text extracted from the PDF, the first 90,000 characters","measured_on":"2026-10-04","accurate":113,"decided":149,"sampled_reviews":200,"split_by_lane":false,"by_lane":"https://pith.science/accuracy#by-lane"}}}