Pith. sign in

REVIEW 3 major objections 4 minor 31 references

Attention-mass KV eviction is matched to the wrong criterion on schema-dense contexts: it retains structural noise and collapses JSON QA from 88% to 0% at a 5% cache budget, while a role-aware allocator recovers most of the gap.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 05:52 UTC pith:4RAP65AD

load-bearing objection A genuinely useful diagnostic paper with a clean counterfactual, but the headline numbers are partly fitted and the deployable label source is the weak link. the 3 major comments →

arxiv 2607.13205 v1 pith:4RAP65AD submitted 2026-07-14 cs.CL cs.AIcs.LG

Adaptive Filtering of the KV Cache: Diagnosing and Correcting Structural-Role Bias in LLM Inference

Soumil Mandal This is my paper
classification cs.CL cs.AIcs.LG
keywords KV cache evictionattention sinkheavy-hitter H2Ostructural-role biasschema-dense contextexact-match QArole-conditional allocationSnapKV max-pool
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the standard practice of keeping tokens with the highest accumulated attention mass—the heavy-hitter (H2O) rule—is matched to the wrong criterion when the context is schema-dense, such as deeply nested JSON. In that regime, structural tokens (delimiters, whitespace, JSON keys) absorb an order of magnitude more attention than the tokens that actually carry answers, so an energy-based filter keeps noise and evicts content, collapsing exact-match accuracy from 88% to 0% at a 5% cache budget. A counterfactual experiment shows that the operative mistake is over-retaining KEY tokens: evicting every KEY token recovers the full-cache baseline using less budget, while evicting delimiters or values is catastrophic. The proposed fix, a retraining-free role-conditional allocation layered on SnapKV's windowed score, closes 63–98% of the H2O gap at sub-20% budgets and at higher budgets matches or modestly exceeds full-cache accuracy. This matters because cache compression is the main lever for long-context inference, and real RAG/agent workloads are increasingly schema-dense.

Core claim

On schema-dense contexts (nested JSON, XML, markdown tables), a non-content 'structural sink'—mostly delimiter tokens—carries roughly 40x the per-token attention energy of answer-bearing VALUE tokens, and KEY tokens are retained at about 1.8x the rate of VALUES. H2O-style heavy-hitter eviction therefore acts as a non-stationary filter that retains noise, driving exact-match accuracy from 88% to 0% as the budget falls to 5%. The paper's central discovery is that this is a causal structural-role bias, not a recency or positional artifact: a counterfactual that evicts all KEY tokens (No-KEY) reaches the full-cache ceiling (0.88 EM) at B=0.75 using strictly less budget, whereas No-DELIM and No-V

What carries the argument

The central object is a role-conditional allocator: per (layer, head), compute SnapKV's windowed attention-mass score, apply its 1-D max-pool (kernel 7), then retain the top fraction of tokens within each structural-role bucket (KEY, VALUE, DELIM, PROSE, WS) according to fixed weights, with one tuned hyperparameter α_KEY. Its diagnostic counterpart is the counterfactual No-X eviction, which removes an entire role from the candidate set. The mechanism that carries the argument is that attention mass (signal energy) decouples from answer relevance (information content) on schema-dense input, and that the max-pool suppresses isolated sinks while the per-role top-k redistributes budget away from

Load-bearing premise

The whole pipeline depends on per-token structural-role labels being correct and available at inference time—the parser supplies them in all accuracy experiments, and when a cheap probe replaces the parser, the method's advantage largely disappears (18–32 EM points lost).

What would settle it

Permute the structural-role labels per token before applying the role-conditional allocator and measure EM at B=5–20% on synthetic json; if the result stays well above SnapKV-only, the role labels are not the source of the gain.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Heavy-hitter eviction (H2O) is not a safe default on schema-dense inputs: at 5% budget it can drive exact-match QA to 0%, so KV-compression benchmarks should include nested JSON/XML/table workloads.
  • Evicting KEY tokens is a better low-budget policy than keeping them on schema-dense data; the retention ordering DELIM ≈ VALUE >> KEY inverts the naive 'keys index, values are content' assumption.
  • SnapKV's gains in this regime come largely from its max-pool suppressing isolated structural sinks, not from its recency window; the pool is load-bearing (disabling it drops low-budget EM to 0.16).
  • The role-conditional allocator composes super-additively with SnapKV: at 5% budget the combination closes 63% of the H2O gap while each ingredient alone closes 21% and 0%.
  • At higher budgets (B ≥ 0.30) the method matches or modestly exceeds full-cache accuracy, but this margin is seed-sensitive; only the B=0.50 effect survives pooling (borderline, p≈0.046), not B=0.30.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the role-density boundary holds (policy gain vanishes when the over-attended role is rare, as on wikitable at 1.6% HEADER tokens), the method's value is concentrated on structurally dense schemas; real RAG corpora with sparse JSON nesting may see little benefit—a testable prediction.
  • The probe's failure mode—local token-shape classification mislabels question-path references as KEY—suggests that a block/segment feature or a two-level classifier could close the 18–32 point EM gap; this is a concrete extension the paper leaves open.
  • Because the bias appears in the non-instruct base model at similar magnitude, the structural-role bias is a property of pretrained transformers; that implies role-conditional allocation might transfer to other architectures without retuning, though the tokenizer coupling of the sink's identity complicates this.
  • The paper's masking-based harness isolates accuracy but not memory/latency; if a compacting implementation realizes the full 1−B reduction, the method's practical value is larger than demonstrated, but the seed-sensitive over-baseline effect should not be a headline for production use.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies KV-cache eviction on schema-dense contexts (nested JSON, XML, markdown tables) and argues that H2O-style heavy-hitter scoring, which ranks by accumulated attention mass, is a filter matched to the wrong criterion there: attention mass concentrates on structural tokens (delimiters, whitespace, JSON keys) rather than answer-bearing content. Using synthetic corpora and structural-role labels, the paper reports a DELIM attention sink on the order of 13.6 mean per-token mass vs. 0.34 for VALUE, an H2O retention bias of roughly 1.8x for KEY over VALUE, and an EM collapse from 0.88 at full budget to 0 at 5% budget on leaf-value QA. A counterfactual role-eviction experiment shows that banning KEY tokens from the candidate set recovers the full-cache baseline at B=0.75, while banning DELIM or VALUE collapses accuracy. The paper then composes a role-conditional allocator with SnapKV's windowed/max-pooled score, reports 63-98% gap closure at sub-20% budgets on seed-0, and replicates the diagnosis and ban-KEY recovery across Llama-3.1, Mistral-7B, Phi-3, and Qwen2.5. A 15 MB linear probe is proposed as a parser substitute, but replacing parser labels with probe labels causes an 18-32 point EM drop because the probe mislabels question path references as KEY, and the combined method then evicts those retrieval cues.

Significance. The diagnostic contribution is real and, if it holds, valuable: it identifies a concrete, previously underappreciated failure mode of attention-mass eviction on structured inputs, with a causal counterfactual and a training-free corrective that is transparently evaluated. The four-model-family replication, the Phi-3 tokenizer control that patterns with Qwen despite sharing Mistral's tokenizer, the fp32 check for decode-path artifacts, and the unusually honest reporting of the seed-sensitive over-baseline effect and the probe's downstream failure are all strengths. The main unresolved issue is that the paper's central corrective claim is contingent on accurate structural-role labels, and the only proposed real-world label source (the probe) is shown to fail in exactly the task-critical way. The synthetic-only evaluation also leaves external validity open. The paper is a solid diagnostic study, but the 'best deployable filter' claim is not yet supported.

major comments (3)
  1. [§4.1, Tables 3-4] The deployability claim is not supported by the probe experiments. Under α_KEY=0, every token labeled KEY is excluded from the candidate set; under α_KEY=0.02, KEY-labeled tokens receive very little budget. §6 shows the 15 MB linear probe labels the question's path reference as KEY on token shape, and swapping probe for parser labels drops EM from 0.98 to 0.66 at B=0.30 (an 18-32 point drop). This is the same failure mode as zeroing PROSE in §4.1, where the question text is evicted and accuracy collapses to 0.00. The paper reports this honestly as a limitation, but it means the headline 'closes 63-98% of the H2O gap' is currently demonstrated only with parser-oracle labels, not with the proposed deployment substitute. A noise-sensitivity analysis of the role-label source, or at least the block/segment-feature fix the paper identifies, is needed before the corrective claim can be accepted
  2. [§5, §7] The core quantitative result — 63-98% closure of the H2O gap at sub-20% budgets — is reported on a single seed-0 draw of n=50 prompts. The only multi-seed analysis (Table 4) concerns the over-baseline margin at B=0.30 and B=0.50, and it shows that seed 0 is a lucky draw: at B=0.30 the pooled mean is +0.01 with a 95% CI of [-0.03, +0.07] straddling zero. The same multi-seed or bootstrapped treatment is not provided for the low-budget combined-vs-SnapKV/H2O comparisons in Table 3, which are load-bearing for the abstract's central claim. Please provide multi-seed results or confidence intervals for the B=0.05/0.10/0.20 comparisons, or explicitly restrict the claim to the seed-0 draw.
  3. [§4, Table 2] All accuracy experiments use synthetic corpora (indented JSON, compact JSON, XML, wikitable) and parser-derived role labels. The paper's scope claim is about schema-dense workloads such as RAG, API logs, and table QA, yet §7 defers validation on real schema-dense corpora to future work. While the synthetic design is appropriate for controlled diagnosis, the corrective's external validity is not yet established: real-world JSON/XML has irregular formatting, mixed-content blocks, and label noise that the current parser/probe setup does not model. At least one real or realistically noisy schema-dense benchmark is needed to support the general claim.
minor comments (4)
  1. [Abstract] The phrase 'retention ordering is therefore DELIM≈VALUE≫KEY' is potentially confusing. Table 1 shows H2O's retention ordering is DELIM > KEY > VALUE; the phrase refers to the counterfactual-derived requiredness ordering, not to actual retention rates. Recommend rewording to avoid implying the two orderings are the same.
  2. [§5] The abstract says a 15 MB linear role probe 'supplies these labels at negligible inference cost, though matching parser-level downstream accuracy remains open.' The open-ness is understated: the probe loses 18-32 EM points and can delete retrieval cues. Consider making the qualification stronger in the abstract.
  3. [§4.1] The tokenizer-control description says Phi-3 shares Mistral's tokenizer 'with identical char-span counts.' The precise verification method for char-span counts is not described; please state how this was checked (e.g., a script that maps both tokenizers to the same raw characters).
  4. [§4.1] The paper says 'The method exposes one tuned knob over three diagnosis-fixed constants.' This is fair descriptively, but the optimal α_KEY differs by corpus (0 on indented JSON, 0.02 as safe default elsewhere), so 'single tuned hyperparameter' in the abstract should be read carefully — the safe default is a corpus-dependent choice, not a single global value.

Circularity Check

1 steps flagged

The role weights are read off the same counterfactual used to claim success, making the headline gap-closing partly in-sample; the causal diagnosis and cross-model core remain independent.

specific steps
  1. fitted input called prediction [§4.1, Hyperparameters; Table 2; Table 3]
    "the content-role weights {V:0.70, D:0.20, P:0.05, W:0.05} are read off the counterfactual ordering DELIM≈VALUE≫KEY (Table 2) rather than swept. Tuned: only αKEY∈[0,0.2], whose per-cell optimum is stable across all five budgets."

    The combined method's role policy is not an independent design: its weights are explicitly 'read off' the counterfactual in Table 2, run on the same synthetic JSON prompts and seed, and αKEY is tuned on those same cells. The paper then reports Table 3 (and the abstract's 'closes 63–98% of the H2O gap') as its headline result. Since αKEY=0 makes the method a softened version of the No-KEY condition already shown to recover toward the full-cache ceiling, part of the reported improvement is a restatement of the diagnostic input under a different score rather than an out-of-sample prediction. The non-circular remainder is the SnapKV-maxpool interaction and the cross-model replication of the ban-KEY effect.

full rationale

The central phenomenon — attention-mass concentration on structural tokens, KEY over-retention, and ban-KEY accuracy recovery — is established by counterfactual ablations and replicated across four model families, so it is not manufactured by the evaluation. The main circular element is confined to the quantitative 'closes 63–98% of the H2O gap' claim: the role weights are explicitly read off the same counterfactual on the same seed-0 synthetic JSON, and αKEY is tuned on those cells, so the in-sample gain is partly by construction. I do not count the probe/label sensitivity as circularity — it is a correctness risk on a dependent variable. There are no load-bearing self-citations and no imported uniqueness theorem. On balance the causal diagnosis is self-contained, but the headline gap-closing on the flagship corpus is an in-sample implementation of the diagnosis rather than an independent prediction, warranting a mid-range score.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The central result rests on two hand-set/fitted allocation parameters (α_KEY and the content-role weights), both tuned or read off the same synthetic benchmark used for evaluation. It also depends on parser-provided role labels being correct and available, on synthetic corpora representing real structured workloads, and on the adequacy of single-seed n=50 runs. No new physical or architectural entities are postulated; the 'DELIM sink' and 'structural-role bias' are re-characterizations of measured attention patterns.

free parameters (2)
  • α_KEY (KEY retention weight) = 0.02 (safe default; 0 on indented JSON)
    Tuned by sweeping {0, 0.02, 0.05, 0.1, 0.2}; the optimum is bimodal and corpus-dependent, and the value is chosen on the same synthetic benchmarks used for evaluation, so headline results partly reflect this fit (§4.1 'Hyperparameters', §4.1 'The α family').
  • Content-role weights {V, D, P, W} = V=0.70, D=0.20, P=0.05, W=0.05
    Hand-set from the counterfactual retention ordering DELIM≈VALUE≫KEY rather than optimized; robustness is checked with a ±0.1 sweep on the same corpus, and zeroing some roles changes results dramatically (§4.1).
axioms (4)
  • domain assumption Attention mass can be treated as per-token signal energy for diagnosing eviction bias.
    Section 3 states 'treating attention mass as per-token energy'; this underpins the SNR estimate and the 'wrong criterion' narrative, though the allocator itself uses only the ranking.
  • domain assumption A parser-based structural-role taxonomy (KEY/VALUE/DELIM/HEADER/PROSE/WS) correctly separates answer-carrying from structural tokens in schema-dense inputs.
    All EM experiments rely on these labels; §6 shows the learned probe's mislabels break downstream accuracy, and §4.1 zeroing PROSE collapses EM, so label correctness is load-bearing.
  • domain assumption The synthetic corpora (indented/compact JSON, XML, wikitable) are representative of schema-dense long-context workloads.
    All quantitative claims are measured on these generated datasets; no real-world RAG/log/API corpora are used, and the authors explicitly list validation on real corpora as future work.
  • domain assumption A single seed-0 prompt draw of n=50 is sufficient to compare eviction policies.
    Tables 1-3 and 5 use n=50 seed-0 without confidence intervals; the multi-seed analysis in Table 4 shows the over-baseline margin is seed-sensitive, so point estimates in the main tables carry unquantified sampling error.

pith-pipeline@v1.3.0-alltime-deepseek · 8904 in / 14697 out tokens · 144379 ms · 2026-08-02T05:52:47.212265+00:00 · methodology

0 comments
read the original abstract

Attention-based KV cache eviction (H2O and its descendants) compresses the memory-constrained state of a long-context model by ranking tokens on accumulated attention mass, treated here as signal energy, and keeping the heaviest. On schema-dense input streams such as nested JSON, this score acts as a non-stationary filter that disproportionately retains noise: a non-content sink role (delimiters or whitespace) carries an order of magnitude more energy than any content role, and structural KEY tokens are over-retained at roughly 1.8x the rate of the answer-carrying VALUE tokens, collapsing exact-match accuracy from 88% to 0% at a 5% budget as the signal-to-noise ratio of the retained state degrades. A counterfactual experiment establishes that suppressing KEY tokens is the best deployable filter. Our retraining-free, role-conditional allocation over SnapKV's windowed score, governed by a single tuned hyperparameter, closes 63-98% of the H2O gap at sub-20% budgets and, at higher budgets, modestly matches or exceeds full-cache accuracy -- a small, seed-sensitive denoising effect (borderline significant at B=0.50; not distinguishable from zero at B=0.30 over four seeds). A 15 MB linear role probe supplies these labels at negligible inference cost, though matching parser-level downstream accuracy remains open.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 9 linked inside Pith

  1. [1]

    attention mass

    INTRODUCTION KV cache eviction is the dominant lever for memory and latency in long-context LLM inference. The canonical method, H2O [1], ranks tokens by accumulated attention mass and keeps a fixed fraction (the “budget”) of heavy hitters—training-free, drop-in, grounded in the observation that a few tokens receive disproportionate attention. Through a s...

  2. [2]

    fair eviction

    BACKGROUND AND RELA TED WORK The H2O family.H2O [1] accumulates a heavy-hitter score s(t) = P q pt(q)during prefill and keeps the top-ktokens per (layer, head). Descendants include Scissorhands [3], FastGen [4], SnapKV [2] (windowed mass over the lastW queries with a 1-D max-pool over the KV axis), and Pyra- midKV [5]. Later methods refinewherebudget is s...

    Pith/arXiv arXiv 2026

  3. [3]

    Accuracy is exact-match (EM) overn=50prompts at seed0across budgetsB∈ {5,10,20,30,50,75,100}%(B=100%is the no-eviction full-cache baseline)

    DIAGNOSIS: A STRUCTURAL-ROUTING BIAS Setup.We evaluate on four synthetic corpora: in- dentedsynthetic json(nested objects, leaf-value queries),synthetic json compact(no whites- pace),synthetic xml(same content as XML), and wikitable(markdown tables with distractors). Accuracy is exact-match (EM) overn=50prompts at seed0across budgetsB∈ {5,10,20,30,50,75,1...

  4. [4]

    keys index, values are content

    COUNTERFACTUAL EVICTION AND THE COMBINED METHOD Counterfactual.Correlation between mass and role does not establish causation. For each (layer, head, prompt) we se- lect the H2O top-kfrom a candidate set excluding all tokens of one role, then decode (Table 2). Onsynthetic json Table 1. H2O retention rate per (corpus, role, budget) on Llama-3.1-8B. KEY/HEA...

  5. [5]

    degrades with tokenizer divergence

    CROSS-MODEL GENERALIZA TION We replicate the diagnosis, counterfactual, and combined method onMistral-7B-Instruct-v0.3,Phi- 3-mini(both SentencePiece), andQwen2.5-7B- Instruct(a divergent tiktoken-family BPE), all on synthetic jsonctx 16k with identical prompts (n=50). The claims split into auniversal coreand amodel-coupled residual(Table 5). The KEY-over...

  6. [6]

    DEPLOYMENT: A LINEAR ROLE PROBE The method needs per-token role labels, supplied by a struc- tural parser in every experiment above. Toward parser-free deployment we train a multinomial logistic regression over a 5-token window of token IDs (current±2):∼3.85M pa- rameters (∼15 MB at fp32), trained in8.6s on 200 prompts. In isolationit labels roles near-pe...

  7. [7]

    CONCLUSION Attention-mass scoring conflates signal energy (high on structural sinks) with information content (high on answer- relevant tokens), so heavy-hitter eviction is a filter matched to the wrong criterion. The contribution is three composable pieces: a diagnostic forwhyrecency-based methods help (implicit sink avoidance), an isolation ofwhichpart ...

  8. [8]

    H2O: Heavy-hitter oracle for ef- ficient generative inference of large language models,

    Zhenyu Zhang et al., “H2O: Heavy-hitter oracle for ef- ficient generative inference of large language models,” inAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  9. [9]

    SnapKV: LLM knows what you are looking for before generation,

    Yuhong Li et al., “SnapKV: LLM knows what you are looking for before generation,” inAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  10. [10]

    Scissorhands: Exploiting the per- sistence of importance hypothesis for LLM KV cache compression at test time,

    Zichang Liu et al., “Scissorhands: Exploiting the per- sistence of importance hypothesis for LLM KV cache compression at test time,” inAdvances in Neural Infor- mation Processing Systems (NeurIPS), 2023

    2023

  11. [11]

    Model tells you what to discard: Adap- tive KV cache compression for LLMs,

    Suyu Ge et al., “Model tells you what to discard: Adap- tive KV cache compression for LLMs,” inInternational Conference on Learning Representations (ICLR), 2024

    2024

  12. [12]

    PyramidKV: Dynamic KV cache compression based on pyramidal information funnel- ing,

    Zefan Cai et al., “PyramidKV: Dynamic KV cache compression based on pyramidal information funnel- ing,”arXiv:2406.02069, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    CAKE: Cascading and adaptive KV cache eviction with layer preferences,

    Ziran Qin, Yuchen Cao, Mingbao Lin, Wen Hu, Shixuan Fan, Ke Cheng, Weiyao Lin, and Jianguo Li, “CAKE: Cascading and adaptive KV cache eviction with layer preferences,” inInternational Conference on Learning Representations (ICLR), 2025

    2025

  14. [14]

    Quest: Query-aware sparsity for efficient long-context LLM inference,

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han, “Quest: Query-aware sparsity for efficient long-context LLM inference,” in International Conference on Machine Learning (ICML), 2024

    2024

  15. [15]

    RocketKV: Accelerating long-context LLM inference via two-stage KV cache compression,

    Payman Behnam, Yaosheng Fu, Ritchie Zhao, Po-An Tsai, Zhiding Yu, and Alexey Tumanov, “RocketKV: Accelerating long-context LLM inference via two-stage KV cache compression,” inInternational Conference on Machine Learning (ICML), 2025

    2025

  16. [16]

    LLMs know what to drop: Self- attention guided KV cache eviction for efficient long- context inference,

    Guangtao Wang, Shubhangi Upasani, Chen Wu, Dar- shan Gandhi, Jonathan Li, Changran Hu, Bo Li, and Urmish Thakker, “LLMs know what to drop: Self- attention guided KV cache eviction for efficient long- context inference,”arXiv:2503.08879, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    OBCache: Optimal brain KV cache pruning for effi- cient long-context LLM inference,

    Yuzhe Gu, Xiyu Liang, Jiaojiao Zhao, and Enmao Diao, “OBCache: Optimal brain KV cache pruning for effi- cient long-context LLM inference,”arXiv:2510.07651, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    Learning to evict from key-value cache,

    Luca Moschella, Laura Manduchi, and Ozan Sener, “Learning to evict from key-value cache,” arXiv:2602.10238, 2026

    Pith/arXiv arXiv 2026

  19. [19]

    SideQuest: Model-driven KV cache management for long-horizon agentic reasoning,

    Sanjay Kariyappa and G. Edward Suh, “SideQuest: Model-driven KV cache management for long-horizon agentic reasoning,”arXiv:2602.22603, 2026

    arXiv 2026

  20. [20]

    Efficient streaming language models with attention sinks,

    Guangxuan Xiao et al., “Efficient streaming language models with attention sinks,” inInternational Confer- ence on Learning Representations (ICLR), 2024

    2024

  21. [21]

    Massive activations in large language models,

    Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu, “Massive activations in large language models,” in Conference on Language Modeling (COLM), 2024

    2024

  22. [22]

    When attention sink emerges in language models: An empirical view,

    Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin, “When attention sink emerges in language models: An empirical view,” inInternational Conference on Learn- ing Representations (ICLR), 2025

    2025

  23. [23]

    Why do LLMs at- tend to the first token?,

    Federico Barbero, ´Alvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veliˇckovi´c, and Razvan Pascanu, “Why do LLMs at- tend to the first token?,”arXiv:2504.02732, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    What are you sinking? A geometric approach on atten- tion sink,

    Valeria Ruscio, Umberto Nanni, and Fabrizio Silvestri, “What are you sinking? A geometric approach on atten- tion sink,” inAdvances in Neural Information Process- ing Systems (NeurIPS), 2025

    2025

  25. [25]

    Ada-KV: Optimizing KV cache evic- tion by adaptive budget allocation for efficient LLM in- ference,

    Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S. Kevin Zhou, “Ada-KV: Optimizing KV cache evic- tion by adaptive budget allocation for efficient LLM in- ference,” inAdvances in Neural Information Processing Systems (NeurIPS), 2025

    2025

  26. [26]

    Not all heads matter: A head- level KV cache compression method with integrated re- trieval and reasoning,

    Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao, “Not all heads matter: A head- level KV cache compression method with integrated re- trieval and reasoning,” inInternational Conference on Learning Representations (ICLR), 2025

    2025

  27. [27]

    Retrieval head mechanistically explains long-context factuality,

    Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu, “Retrieval head mechanistically explains long-context factuality,”arXiv:2404.15574, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    CompressKV: Seman- tic retrieval heads know what tokens are not important before generation,

    Xiaolin Lin, Jingcun Wang, Olga Kondrateva, Yiyu Shi, Bing Li, and Grace Li Zhang, “CompressKV: Seman- tic retrieval heads know what tokens are not important before generation,”arXiv:2508.02401, 2025

    Pith/arXiv arXiv 2025

  29. [29]

    The pitfalls of KV cache compression,

    A. Chen, A. Geh, A. Grover, G. Van den Broeck, and A. Israel, “The pitfalls of KV cache compression,” arXiv:2510.00231, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    LongBench: A bilingual, multitask benchmark for long context understanding,

    Yushi Bai et al., “LongBench: A bilingual, multitask benchmark for long context understanding,” inAnnual Meeting of the Association for Computational Linguis- tics (ACL), 2024

    2024

  31. [31]

    RULER: What’s the real context size of your long-context language models?,

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shan- tanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg, “RULER: What’s the real context size of your long-context language models?,” inConference on Language Modeling (COLM), 2024

    2024