Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

This paper claims that KV-cache compression can pick its own per-request budget by sampling synthetic future queries from the Gaussian distribution of hidden states, cutting memory roughly in half without manual tuning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

GVote removes hand-set KV-cache budgets by synthesizing future queries from the hidden state distribution and keeping the union of keys they vote for, yielding around 2x memory savings at comparable accuracy.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection GVote has a genuinely new adaptive-budget idea, but the Gaussian-sampling premise and missing artifacts make the evidence too thin to support the claims yet. the 4 major comments →

arxiv 2509.03136 v1 pith:6XXT4URY submitted 2025-09-03 cs.DB cs.AI

Adaptive KV-Cache Compression without Manually Setting Budget

Chenxia Tang , Jianchun Liu , Hongli Xu , Liusheng Huang This is my paper
classification cs.DB cs.AI
keywords KV-cache compressionadaptive memory budgetMonte Carlo query samplingGaussian hidden statesattention sparsityLLM inferencelong-context benchmarksGrouped-Query Attention
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper targets a known pain point: existing KV-cache compression methods require a fixed compression ratio, but the optimal ratio differs wildly across workloads—simple QA tolerates aggressive pruning, while reasoning tasks collapse if too much is dropped. GVote removes the knob by treating the budget as an emergent quantity: the number of keys that a Monte-Carlo sample of synthetic future queries collectively needs. Synthetic queries come from a Gaussian fitted to the layer-normalized hidden states, projected through the model's own weights; each query votes for its top keys, and the union of votes defines the cache. Across GSM8K, RULER, and LongBench on four LLMs, the paper reports roughly 2x memory reduction at equal or better accuracy than fixed-budget SnapKV, StreamLLM, and AdaKV. The authors themselves flag that synthetic-versus-real attention overlap is high but not perfect (0.929 on an illustrative head), which is why the paper's load-bearing assumption—hidden-state Gaussianity—matters.

Core claim

GVote's central claim is that the optimal KV-cache budget is not a number to set in advance but an outcome of 'what will future queries need?'. Because future queries are unknown at prefill, GVote synthesizes them: it fits a diagonal Gaussian to the hidden states of the current prefix, samples S query vectors, projects them through the model's query projection, applies rotary embeddings at an averaged future position, and computes their attention against all cached keys. Each synthetic query keeps its top-k positions (with k fixed by the current query's nucleus-truncated attention), and the union of these selections is the compressed cache; its size, automatically, is the budget. The paper a

What carries the argument

The Monte-Carlo voting mechanism: S synthetic queries drawn from the fitted Gaussian N(μ, diag(σ²)) of hidden states are converted into real attention queries and scored against every cached key; each voter contributes its top-k key set, and the final cache is the union of all votes, with the union's cardinality serving as the per-head budget. The Gaussian fit is the load-bearing object: it turns the unknowable future-query distribution into a samplable one, and the union operation is what makes the budget adaptive rather than prescribed. A secondary design choice—top-k for synthetic voters instead of top-p—dampens noise from imperfectly estimated queries.

Load-bearing premise

The load-bearing premise is that hidden states along the sequence are close enough to Gaussian that queries sampled from the fitted distribution will attend to roughly the same key positions as the model's real future queries; the paper supports this with a single illustrative head (r = 0.7759, overlap 0.929) rather than a distribution test.

What would settle it

Run GVote on a held-out set of long contexts, record its keep-set, then decode fully and compute the keep-set of the actual future queries. If the mean overlap between the two—across all heads, layers, and several models—falls well below the 0.929 reported for the single illustrative head, the Gaussian sampling proxy is not carrying the argument. A second, cheaper test: apply a standard normality test (e.g., Shapiro-Wilk) to per-channel hidden states along the sequence axis; if most channels reject Gaussianity on the evaluation datasets, the fitted distribution is misspecified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Operators no longer need to tune a compression ratio per task; the same algorithm self-adjusts from short math prompts to 50K-token documents.
  • The resulting cache is non-uniform across heads and layers, with per-head budgets that track where attention actually concentrates, and this format is compatible with variable-length attention kernels like FlashAttention.
  • Because the budget is a statistic computed from hidden states, it inherits the distribution of the input: workload shifts that change attention also change the budget, avoiding both accuracy collapse and memory waste.
  • Roughly 2x memory reduction at equal or better accuracy compared with fixed-budget baselines, across four model families and eight benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The Gaussian proxy could be replaced or validated against a nonparametric alternative (e.g., sampling from the empirical history of hidden states); if the nonparametric version performs as well, Gaussianity is not load-bearing, and if it fails, the method's reliability is tied to the Gaussian claim.
  • The pattern of 'sample the distribution of future inputs to set a resource knob' may generalize beyond KV-caches—for example, to adaptive batch sizes, speculative-decoding window lengths, or sparse-attention block selection, wherever a downstream choice depends on unseen future states.
  • A natural stress test: generate long outputs and measure whether the union of synthetic-query key sets converges to the union over the real future queries as S grows; the reported single-head overlap (0.929) leaves head-wise and layer-wise variance unquantified.
  • The budget estimate from the current query's top-p set (|C0|) couples the two hyperparameters; deriving k directly from the Gaussian or from a separate statistic might yield a more fully parameter-free version.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes GVote, an adaptive KV-cache compression method that aims to eliminate the manual budget setting of fixed-ratio compressors such as SnapKV and AdaKV. The method first computes the current query's top-p attention set to obtain a per-request budget Bstep, then fits a diagonal Gaussian to the prompt hidden states, samples S synthetic queries, and retains the union of the top-k key sets selected by those synthetic queries. The authors argue that this union approximates the keys that future queries will need. Experiments are presented as accuracy-versus-memory plots on eight benchmarks and four models, and the paper claims a 2x memory reduction with comparable or better accuracy.

Significance. The underlying idea is timely and potentially useful: per-request, content-aware KV-cache budgets could indeed be preferable to fixed compression ratios across heterogeneous workloads. The paper is also honest in Section 3.3 about some limitations of synthetic-query quality. However, the central claims are currently supported almost entirely by qualitative accuracy-usage plots, with no numerical tables, error bars, code, or data release. The load-bearing Gaussian and synthetic-query approximation is validated on a single illustrative head/figure, and the method's 'automatic' budget depends on the manually chosen nucleus threshold pnuc. If properly validated, GVote could be a meaningful step toward adaptive KV-cache compression, but in its present form the evidence does not establish the mechanism or the quantitative advantages claimed.

major comments (4)
  1. [Section 3.2, Algorithm 1, line 2; Section 3.5] The claimed elimination of manual budget setting is not supported by the algorithm. Bstep is set to |C0|, where C0 = TopP(A0, pnuc). Every synthetic query then selects exactly k=|C0| keys, and the final cache size is the union of S such top-k sets. Thus pnuc acts as a manual budget hyperparameter: it directly controls the per-query selection size and hence, together with S, the final memory footprint. The recommendation pnuc=0.95 in Section 3.5 is a manual setting. The paper should either reposition pnuc as a quality/reliability threshold and show that the final cache size is insensitive to it, or acknowledge that GVote still requires a budget-related hyperparameter. As written, the central claim in the title and abstract is overstated.
  2. [Section 3.1, Section 3.3, Figure 3] The Gaussian assumption and the synthetic-query proxy are load-bearing but under-validated. The only direct evidence is Figure 3, which reports attention overlap 0.929 (below the paper's own 0.95 threshold) and a Pearson correlation r=0.7759 for a single attention head. No distributional test is provided for the claimed Gaussianity, and the validation is not carried over multiple heads, layers, datasets, or models. Moreover, the actual mechanism in Algorithm 1 uses the union of S top-k sets, not the single synthetic query tested in Figure 3. The paper never measures whether this union improves recall over the single-query overlap of 0.929, nor whether the union approximates the optimal keep-set for real future queries. Without such experiments, the claimed memory--accuracy advantage is not explained by the proposed mechanism.
  3. [Section 3.2, Algorithm 1, line 6] All synthetic queries share a single averaged RoPE position P = AverageCosSin(nf). Real future queries have distinct positions, and attention patterns can be strongly position-dependent. The paper provides no evidence that one averaged position is sufficient to generate synthetic queries whose top-k key sets cover the keys needed by future positions. This is a nontrivial component of the approximation and should be justified or tested, e.g., by comparing the union of synthetic top-k sets with the keep-set produced by actual future queries at their true positions.
  4. [Section 4, Figures 4 and 5] The central quantitative claim of '2x memory reduction with higher or comparable accuracy' is not supported by the evidence presented. The results are shown only as accuracy-usage plots, with no numerical values, no error bars, no standard deviations, and no specification of the number of runs or seeds. It is also unclear how the 2x figure is computed: GVote appears as a single point per dataset, while baselines are swept from 10% to 50%, so the comparison is not defined in a reproducible way. The paper should include a table with exact accuracy and cache-usage numbers, standard errors, and ideally release code/data so the claimed advantage can be verified independently.
minor comments (6)
  1. [Section 3.2, Algorithm 1; Section 3.5] The parameter ns (number of initial tokens ignored due to attention sinks) is mentioned in Step 2 but is not defined in the algorithm input list or in the parameters section. Similarly, nf is used but its role is not explained beyond 'average cos/sin for nf future positions'. Please define all free parameters explicitly and report their default values.
  2. [Section 2.1, Eq. (3)] The symbol S is used for both the score tensor (Eq. 3) and the number of synthetic samples (Algorithm 1). This is confusing. Consider using a different symbol, e.g., N or M, for the number of samples.
  3. [Section 3.5] Two unresolved placeholders appear: 'Detailed sensitivity analysis is presented in [ref]' and 'Further empirical results are discussed in the experiments section [ref]'. These must be fixed or the missing sensitivity analysis must be included in a revision.
  4. [Section 4.1] The abstract and introduction mention GSM8K, RULER and Longbench, while Figure 4 reports datasets such as 'Code Repo Understanding', 'Long-ICL', etc. Please clarify which Longbench subsets are used and how the eight benchmarks correspond to the three named datasets.
  5. [Section 1, Figure 1; Figures 6 and 7] Figure 1's legend appears not to include GVote (the text says it should), and Figures 6 and 7 are rendered as very small thumbnails without legible axes labels. Please ensure all figures are readable and consistent.
  6. [Throughout] There are typos, e.g., 'INTRODUCION' in the section heading, and some notation inconsistencies such as A0 ∈ R^{Shead×L} in Section 3.2 Step 1 despite A0 being defined for a single query. A careful proofread is recommended.

Circularity Check

0 steps flagged

No significant circularity: GVote's budget is an empirical proxy, not a constructional identity with the claimed target.

full rationale

The derivation chain is not circular. GVote computes its budget from the current query's top-p candidate set and from synthetic queries sampled from a Gaussian fit to the prompt's hidden states (Algorithm 1, lines 1-10). The claimed target is the union of keys required by future queries, defined as ∪_{u>t} argTop-p(A_u). These are not the same object by definition: the algorithm outputs K = ∪_s TopK(eQ_s K^⊤, |C0|), while the ideal target is an unknown future-query union. The gap is an empirical approximation, and the paper attempts to validate it directly in Section 3.3 by comparing synthetic queries against a real future query (overlap 0.929, r=0.7759). No parameter is fitted to the future-query target, so the prediction is not statistically forced. There are no load-bearing self-citations: the only external citation for the Gaussian observation (NVIDIA 2025) is independently corroborated by the authors' Figure 2. The admitted limitations—overlap falling below the 0.95 threshold, single-head/single-input validation, and pnuc remaining a manual threshold—weaken the strength of the prediction claim but do not make it circular. The memory-accuracy comparison is grounded in external benchmarks (GSM8K, RULER, LongBench) and fixed-budget baselines, providing independent evidence for the empirical claim. Therefore the paper is self-contained against external benchmarks and exhibits no significant circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The method rests on two statistically untested assumptions, Gaussianity of hidden states and fidelity of synthetic queries, plus two manually chosen hyperparameters (pnuc, S) and two unspecified configuration values (ns, nf). The claimed no-manual-budget property reduces to a manual choice of pnuc, which directly controls the initial budget.

free parameters (4)
  • pnuc = 0.95 (recommended)
    Nucleus threshold sets the starting budget via |C0|. It is manually chosen and tuned on RULER in Section 4.4, then applied across all tasks.
  • S = >= 8 (recommended)
    Number of synthetic queries. Larger values improve robustness but increase memory and compute; chosen empirically in Section 4.4.
  • nf = not specified
    Number of future positions used to average RoPE cosine and sine values in Algorithm 1 line 6; never given a default or tuned value in the text.
  • ns = not specified
    Number of initial tokens ignored due to attention sinks in Algorithm 1 Step 2; the variable is referenced but never defined or set.
axioms (5)
  • domain assumption Hidden states along the sequence axis are approximately Gaussian per channel
    Section 3.1 and Figure 2. The entire synthetic-query mechanism depends on this distributional assumption; no formal normality test is reported.
  • domain assumption Synthetic queries sampled from the fitted Gaussian have attention patterns close to real future queries
    Section 3.3. Validated on a single illustrative attention head with r=0.7759; not established across layers, heads, or benchmarks.
  • domain assumption The current query's top-p candidate set size |C0| is a reliable proxy for a single future query's token budget
    Algorithm 1 Steps 1-3. Bstep = |C0| bounds each synthetic query's top-k selection, making the final budget directly depend on the current query.
  • ad hoc to paper Preserving the minimal set of keys with cumulative top-p attention probability is the right optimality criterion
    Section 3.1 defines the top-p criterion as optimal without connecting it to downstream task loss; this is a modeling choice specific to the paper.
  • domain assumption Layer normalization drives hidden states toward Gaussianity
    Section 3.1 states this mechanism without proof; it is offered as the justification for the Gaussian observation.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Adaptive KV-Cache Compression without Manually Setting Budget." pith.science (2026). https://pith.science/paper/6XXT4URY

@misc{pith2026250903136,
  author       = {Pith},
  title        = {Pith review of: Adaptive KV-Cache Compression without Manually Setting Budget},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6XXT4URY}},
  note         = {Machine review of arXiv:2509.03136}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large language models (LLMs) inference relies heavily on KV-caches to accelerate autoregressive decoding, but the resulting memory footprint grows rapidly with sequence length, posing significant efficiency challenges. Current KV-cache compression methods suffer from a Procrustes' bed problem: they force diverse workloads into fixed compression ratios, leading to suboptimal resource allocation and inference performance. To this end, we present GVote, an adaptive KV-cache compression scheme that eliminates manual budget specification while achieving superior accuracy-efficiency trade-offs. GVote operates on the principle that the important keys are the aggregation of keys required by future queries. The method predicts future query attention demands by Monte-Carlo style sampling potential queries and aggregating selected keys to determine the optimal cache budget without manual specification. Experimental evaluation demonstrates GVote's effectiveness across multiple benchmarks, including GSM8K, RULER and Longbench. Compared to baselines, GVote exhibits 2$\times$ memory reduction while the accuracy maintains higher or comparable.

Figures

Figures reproduced from arXiv: 2509.03136 by Chenxia Tang, Hongli Xu, Jianchun Liu, Liusheng Huang.

Figure 1
Figure 1. Figure 1: Accuracy-usage analysis across diverse benchmarks using the Llama-3.1-8B-Instruct [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Hidden states exhibit explicit Gaussian structure. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comprehensive analysis of synthetic query approximation quality. (a) Layer-wise and [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Accuracy vs. Cache Usage across eight benchmarks using Qwen2.5-7B-Instruct. Each [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: GVote performance across different model architectures and sizes. All models are of [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Effect of sampling number S on GVote performance. Higher S would deliver better accuracy with moderately higher us￾age. However it would cost more memory and computation for compression [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilation

    cs.LG 2026-02 unverdicted novelty 6.0

    CompilerKV uses offline-compiled retention tables as portable priors to achieve SOTA prefill-only KV compression performance across backbones at low token budgets.

  2. CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilation

    cs.LG 2026-02 conditional novelty 6.0

    Offline-learned head-reliability and risk-threshold tables make prefill-only KV compression recover about 97.7% of uncompressed LongBench accuracy at a 512-token-per-layer memory budget.

Reference graph

Works this paper leans on

13 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gqa: Training generalized multi-query transformer models from multi-head check- points

    Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebr´on, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv:2305.13245,

    Pith/arXiv arXiv

  2. [5]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 ,

    Pith/arXiv arXiv

  3. [7]

    Ada-kv: Optimizing kv cache evic- tion by adaptive budget allocation for efficient llm inference

    Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou. Ada-kv: Optimizing kv cache evic- tion by adaptive budget allocation for efficient llm inference. arXiv preprint arXiv:2407.11550,

    Pith/arXiv arXiv

  4. [8]

    Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654,

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654,

    Pith/arXiv arXiv

  5. [10]

    Keep the cost down: A review on methods to optimize llm’s kv-cache consumption

    Luohe Shi, Hongyi Zhang, Yao Yao, Zuchao Li, and Hai Zhao. Keep the cost down: A review on methods to optimize llm’s kv-cache consumption. arXiv preprint arXiv:2407.18003,

    Pith/arXiv arXiv

  6. [11]

    Qwen2 technical report

    Qwen Team. Qwen2 technical report. arXiv preprint arXiv:2407.10671,

    Pith/arXiv arXiv

  7. [13]

    Big bird: Transformers for longer sequences

    10 Published as a conference paper at ICLR 2026 Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283–17297,

    work page 2026

  8. [2017]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453,

    Pith/arXiv arXiv

  9. [2020]

    Pyramidkv: Dynamic kv cache compression based on pyra- midal information funneling

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, et al. Pyramidkv: Dynamic kv cache compression based on pyra- midal information funneling. arXiv preprint arXiv:2406.02069,

    Pith/arXiv arXiv

  10. [2021]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691,

    Pith/arXiv arXiv

  11. [2023]

    Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks

    Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412.15204,

    Pith/arXiv arXiv

  12. [2024]

    Longformer: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150,

    Pith/arXiv arXiv 2004

  13. [2025]

    Ziran Qin, Yuchen Cao, Mingbao Lin, Wen Hu, Shixuan Fan, Ke Cheng, Weiyao Lin, and Jianguo Li

    Accessed: 2025-07-20. Ziran Qin, Yuchen Cao, Mingbao Lin, Wen Hu, Shixuan Fan, Ke Cheng, Weiyao Lin, and Jianguo Li. Cake: Cascading and adaptive kv cache eviction with layer preferences. arXiv preprint arXiv:2503.12491,

    arXiv 2025

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.