Pith. sign in

REVIEW 5 major objections 5 minor 30 references

TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruning

T0 review · 5 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TaskPress compresses KV cache once per task, and a single compressed cache then serves many unseen queries from the same task with near-full accuracy.

desk verdict A promising query-agnostic compression idea whose central reuse claim currently rests on a missing figure—worth reviewing, not yet proven. read the letter →

arxiv 2608.03276 v1 pith:QJRIA2CD submitted 2026-08-04 cs.AI

Wonpyo Park , Seung-won Hwang This is my paper
classification cs.AI
keywords KVcachecompressionquery-agnosticevictiontaskguidemeta-queryquantizationscalefactorlong-contextinferenceLLMservingtokenimportancescoring
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that long-context KV cache compression can be made query-agnostic by scoring tokens against a high-level task description rather than against each user query. The task guide acts as a meta-query during prefill; its attention to the context yields a key importance score, and the quantization scale factor of each value yields a value importance score, with the product selecting the tokens to keep. The expected payoff is that a single compressed cache serves many unseen queries from the same task, avoiding repeated prefill and query-dependent eviction. The paper reports that this works on long-context benchmarks at 50–75% eviction and improves end-to-end latency in multi-query serving.

What carries the argument

The task guide (meta-query): a 2–3 sentence natural-language description (<64 tokens) generated by an off-the-shelf LLM for known tasks, or discovered by evolutionary search over a calibration set for unknown tasks. It is appended to the context during prefill, its attention map $A \in \mathbb{R}^{L_t \times L_c}$ is averaged over the guide dimension and pooled (Eq. 1) to give $s^{\text{key}}_i$; the per-token quantization scale $\gamma_i = \max(|v_i|)/(2^{b-1}-1)$ (Eq. 2) gives $s^{\text{value}}_i$; the product $s_i = s^{\text{key}}_i \cdot s^{\text{value}}_i$ (Eq. 3) is the retention score. This product enforces that a kept token is both reachable by task-aligned attention and an outlier-r

What would settle it

On a set of long documents with multiple queries per document, compute the overlap between the top tokens kept by TaskPress using only the task guide and the top tokens actually attended by a held-out set of real queries. If the overlap is no higher than a random subset of the same size, the guide's attention is not carrying the information; alternatively, shuffle the words in the task guide and measure whether accuracy at 75% eviction drops far below the correct-guide result.

Watch

Extended reading notes

Core claim

The central discovery is that a natural-language task description, treated as a meta-query, carries enough information about where a whole family of queries will focus to drive KV cache compression on its own. During prefill, TaskPress appends the guide to the context, computes the guide-to-context attention map, averages over the guide dimension and pools it to get a key score per token (Eq. 1). It then multiplies this by the per-token quantization scale factor, which acts as a nearly free value-importance proxy because the scale is already computed for serving (Eqs. 2–3). The paper's experiments claim that the resulting cache, pruned to 50% or 75%, remains accurate across many unseen queri

Load-bearing premise

The method's success depends on the task guide's attention during prefill predicting which tokens a whole family of future queries will need; if that transfer is weak, the compressed cache omits the very tokens those queries rely on.

Editorial extensions

If this is right

  • Because the cache is built once per task, multi-query serving needs only one prefill and one eviction; the paper's measurements show growing wall-clock savings as query count increases.
  • At 75% eviction plus int4 quantization, total compression reaches 16x; TaskPress outperforms query-dependent and reconstruction-based baselines on LongBench averages while matching the full-cache baseline on several task groups.
  • Combining TaskPress with channel-wise key pruning (ThinK) gives higher accuracy than either method alone at 75% total eviction, indicating token-level and channel-level compression are complementary.
  • A zero-shot task guide generated by an off-the-shelf LLM already works well, and the evolutionary refinement further improves it on QA tasks, so task guides do not require task-specific hand-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The transfer premise implies a checkable corollary: the task guide's per-token attention should rank-correlate with the average attention of held-out queries from the same task; a strong correlation would confirm the mechanism, while a weak one would mean the accuracy gains come from something other than the guide's attention.
  • The same guide-as-meta-query pattern could apply to encoder-decoder models, where a fixed guide at the encoder input would make the compressed encoder state reusable across decoder queries; the paper only evaluates decoder-only LLMs.
  • The value-importance proxy relies on per-token quantization scales; applying the method to group-wise or channel-wise quantization would require a different zero-cost outlier signal, which the paper does not address.
  • The failure on needle-saturated contexts suggests that softmax-normalized attention itself, not the task-guide idea, is the bottleneck; replacing the pooled attention score with an unnormalized or entropy-corrected statistic might let the method handle dense-information settings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. TaskPress proposes query-agnostic KV-cache compression for long-context LLM inference. Instead of scoring token importance with the future user query, it appends a high-level task guide (a short meta-query) during prefill, computes the guide's cross-attention to context tokens as a key-importance score (Eq. 1), combines it multiplicatively with a quantization-scale-based value-importance score (Eqs. 2–3), and evicts low-scoring tokens. The resulting compressed cache is claimed to be reusable across multiple unseen queries from the same task. The authors also propose an evolutionary search to discover task guides when the task is unknown, using a small calibration set. Experiments on LongBench and RULER with LLaMA-3.1-8B, Qwen3-8B, and Qwen3-32B show competitive or better average accuracy versus query-dependent and query-agnostic baselines at 50% and 75% eviction, plus latency/memory gains in a multi-query serving scenario.

Significance. If the central claim holds, TaskPress addresses a real limitation of query-dependent KV pruning: it moves compression from query time to task time, enabling cache reuse across turns without the quadratic prefill overhead of reconstruction-based methods. The idea of repurposing quantized scale factors as a zero-cost outlier signal is elegant and well supported by the ablation (Table 8) and correlation analysis (§4.3.2). The paper is also careful to report limitations (task drift, dense needles, quantization reliance) and includes a controlled needle-density experiment (Table 14). However, the direct evidence for the central 'reusable cache' claim is currently missing: the only experiment designed to test cross-query reuse (Figure 3) is absent from the manuscript, and the efficiency experiment (Table 7) measures only latency, not accuracy. The evolutionary-guide experiments also use calibration samples repurposed from the test distribution, which is a leakage risk. These issues are fixable but currently leave the main contribution insufficiently verified.

major comments (5)
  1. [§4.1.4, Figure 3] The central claim — that one task-guide-compressed cache can be reused across multiple unseen queries with accuracy close to the full cache — is directly tested only in Figure 3, which compares query-dependent eviction, query-reuse, last-context proxy, and TaskPress on NarrativeQA and QMSum. This figure is referenced in the text but is not present in the manuscript. The LongBench tables evaluate a single query per context with a task-level guide; they demonstrate the guide as a pseudo-query but do not show reuse over successive queries on the same context. Table 7 reports latency and memory only, not accuracy. Please include the missing figure and report reuse accuracy (e.g., accuracy on queries 2..k when the cache is compressed once) for both datasets. Without this, the main contribution is unverified.
  2. [§4.1.2, Table 3] Evolved task guides are optimized using 50 calibration samples 'repurposed from the test set.' Although the authors state these samples are excluded from final evaluation, using test-distribution queries to select the guide is a form of information leakage: the evolutionary search can adapt to idiosyncrasies of the test set, inflating the reported gains. Please use a proper train/validation split of each dataset (or a separate development set) for calibration, and report results with the correspondingly selected guides. At minimum, quantify how much evolved-guide performance degrades when calibration and test samples are disjoint by construction.
  3. [§3.5.2 / Algorithm 1] The evolutionary search evaluates candidate guides by 'downstream task accuracy' on D_cal. Since the calibration set consists of queries, the fitness function presumably runs full generation for each candidate; the paper does not specify how many inference calls this requires per generation, nor how candidates are scored on summarization or synthetic tasks where accuracy is not a single clean number. The reported one-time cost of ~25 minutes (Section 4.2.2) is for one configuration, but the method's scalability across tasks and budget choices is unclear. Please specify the fitness metric, the number of inference calls per generation, and the variance of the evolved result across evolutionary runs.
  4. [§4.1.1, Table 1] Main accuracy tables (Tables 1, 2, 4) report point estimates without error bars or significance testing. Given the small differences between TaskPress and baselines on several tasks (e.g., Table 1 average 46.42 vs. 44.76 for KVzip at 50%, and many per-task differences near 0.5 points), the reader cannot assess whether the reported improvements are systematic or noise. Please include variance across at least 3 seeds or, for deterministic methods, at least report bootstrap confidence intervals on the averages.
  5. [§3.2, Eq. (1)] The central transfer assumption — that the task guide's cross-attention during prefill predicts which tokens future queries will need — is load-bearing and shown to be brittle in Table 13 (misaligned guides degrade accuracy) and Appendix B (dense needles cause attention dilution). The paper is honest about this, but the missing Figure 3 is the natural place to provide positive evidence. In addition, the 1D average pooling in Eq. (1) sums over indices i+k without defining boundary handling for k where i+k is outside [1, Lc]. Please specify the padding/clamping behavior.
minor comments (5)
  1. [Throughout] There is an inconsistency between 'ThinK' (the method name, also in References) and 'ThinkPress' used in Table 1 and possibly elsewhere. Please unify to one name.
  2. [Table 7] The table header is difficult to parse: 'Query-Dependent Ours E2E Gain (sec)' with columns 'Lat-Opt Mem-Opt Lat-Opt Mem-Opt' and then values like '-0.05 2.55'. Please clarify which columns correspond to which strategy and what the E2E gain is relative to.
  3. [§4.2.1] The '80× latency improvement' in the text references Fig. 4, but the figure shows two panels; it would be helpful to state the exact comparison protocol (context length, model, batch size) and whether the eviction latency includes the task-guide attention computation or just the eviction step.
  4. [§4.3.2] The Pearson correlation values are reported as '0.79±0.10' and '0.06±0.23' without stating over what set (tokens? layers? heads? datasets?) and how the standard deviation was computed. Please specify.
  5. [Appendix A] The task guides in Tables 15 and 16 are described as 'primarily generated by Gemini, with minimal human refinement.' Please describe the refinement process and whether the final guides were chosen based on validation performance, since this affects the zero-shot claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TaskPress's scoring equations are self-contained and the central claims are empirical evaluations, not derivations from the target.

full rationale

TaskPress defines token importance through three explicit equations: task-guided key importance from attention between a task guide and context (Eq. 1), value importance from quantization scale factors (Eq. 2), and a multiplicative combination (Eq. 3). None of these equations reference the downstream query answers, the evaluation metric, or the final accuracy; they are heuristic scoring rules evaluated empirically. The zero-shot task guide is generated from the task name, not from test labels, and the LongBench tables evaluate on real held-out queries. The evolutionary guide discovery (Section 4.1.2) does use 50 calibration samples repurposed from the test set, which is a data-leakage / overfitting concern, but it is not circularity: the final evaluation explicitly excludes those samples, and the reported accuracy is on different queries. The same applies to the choice of multiplication in Section 4.4, which is an empirical model-selection decision, not a derivation that reduces to the result it predicts. There are no load-bearing self-citations; the paper cites external works for baselines and code, not for its own core premise. The missing Figure 3 and the absence of accuracy measurements in the multi-query efficiency experiment (Table 7) are evidentiary gaps, but they do not make the derivation circular. Overall, the claimed prediction—that a task-guide-compressed cache serves unseen queries—is an empirical hypothesis tested against, not equated with, its inputs.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities. Its assumptions concern transformer attention geometry and quantization statistics. The main price paid is the task guide itself, which in the unknown-task setting is fitted to calibration samples, and the score-combination choice is benchmark-selected.

free parameters (5)
  • 1D average pooling kernel size K = not specified
    Eq (1) applies average pooling with kernel K, but the value of K is never given; changing it alters the key importance scores.
  • Score combination function = multiplication (s_key * s_value)
    Selected empirically: Table 9 shows multiplication beats normalized addition and harmonic mean on the reported tasks, so the design is validated on the benchmark rather than predicted.
  • Evolved task guide T = per-task guide string from evolutionary search
    In the unknown-task experiments, the guide is optimized against 50 calibration samples drawn from the test distribution and then evaluated on the remainder.
  • Evolution hyperparameters K=2, N=3, G=4 = K=2, N=3, G=4
    Chosen by hand; only one setting is reported, with no sensitivity analysis.
  • Task guide length = 2-3 sentences
    Table 11 shows moderate length is optimal; the final choice is made after benchmarking alternative granularities.
assumptions (4)
  • domain assumption Attention scores between task guide tokens and context tokens are a valid measure of key importance for downstream queries.
    Eq (1) defines the key score; the entire method relies on this transfer from the guide to real queries.
  • domain assumption Per-token quantization scale factor gamma_i = max(|v_i|) / (2^(b-1) - 1) is proportional to value importance.
    Section 3.3, Eq (2); supported by a reported 0.79 correlation with value norm, but the proxy is assumed for all layers and heads.
  • domain assumption An off-the-shelf LLM (Gemini) can generate a stable task guide that captures task-level intent.
    Section 3.5.1; zero-shot guides are the backbone of the LongBench results.
  • domain assumption Tokens with high key score and high value score form a sufficient subset for generation.
    Section 3.4; the multiplicative scoring assumes both structural retrievability and semantic richness are necessary.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruning." pith.science (2026). https://pith.science/paper/QJRIA2CD

@misc{pith2026260803276,
  author       = {Pith},
  title        = {Pith review of: TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QJRIA2CD}},
  note         = {Machine review of arXiv:2608.03276}
}
read the original abstract

Long-context inference with large language models is constrained by the linear growth of the key-value cache to sequence length. While pruning offers mitigation, prevailing methods determine query-specific token importance that cannot be reused across unseen queries. In contrast, we introduce TaskPress, a framework for task-guided, query-agnostic KV cache eviction. Instead of optimizing the cache for a single query, TaskPress constructs a reusable memory representation conditioned on a high-level task guide. The guide functions as a meta-query during prefill to filter irrelevant tokens before downstream queries are issued. In addition, TaskPress leverages quantization scale factors as a zero-cost signal for detecting influential representation outliers, providing an efficient proxy for token importance. Experiments on conducted on various tasks with long context input demonstrate that TaskPress efficiently creates a compact, reusable cache across diverse queries.

Figures

Figures reproduced from arXiv: 2608.03276 by the authors.

Figure 1
Figure 1. Comparison of attended context between spe [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of KV cache eviction processes. The upper and lower panels illustrates the attention maps for [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Accuracy comparison on the NarrativeQA and QMSum datasets with LLaMA-3.1-8B-Instruct. We evaluate four compression strategies: (1) query￾dependent, where the specific query guides eviction; (2) query-reuse, where a cache compressed for one query is applied to others; (3) last context, where the final con￾text segment serves as a proxy for query; and (4) Task guide (Ours), which utilizes a high-level task guide. 4.1.… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: (Left) the eviction latency of each method. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Pseudo-code comparison of KV cache evic [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

30 extracted references · 6 canonical work pages

  1. [1]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

  2. [2]

    arXiv preprint arXiv:2507.06261 , year=

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities , author=. arXiv preprint arXiv:2507.06261 , year=

    arXiv

  3. [3]

    arXiv e-prints , pages=

    The llama 3 herd of models , author=. arXiv e-prints , pages=

  4. [4]

    arXiv preprint arXiv:2402.04396 , year=

    Quip\#: Even better llm quantization with hadamard incoherence and lattice codebooks , author=. arXiv preprint arXiv:2402.04396 , year=

    arXiv

  5. [5]

    Advances in Neural Information Processing Systems , volume=

    Snapkv: Llm knows what you are looking for before generation , author=. Advances in Neural Information Processing Systems , volume=

  6. [6]

    arXiv preprint arXiv:2505.23416 , year=

    KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction , author=. arXiv preprint arXiv:2505.23416 , year=

    arXiv

  7. [7]

    Advances in Neural Information Processing Systems , volume=

    H2o: Heavy-hitter oracle for efficient generative inference of large language models , author=. Advances in Neural Information Processing Systems , volume=

  8. [8]

    Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

    Longbench: A bilingual, multitask benchmark for long context understanding , author=. Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

Show all 30 references
  1. [9]

    2024 , journal=

    RULER: What's the Real Context Size of Your Long-Context Language Models? , author=. 2024 , journal=

    2024

  2. [10]

    arXiv preprint arXiv:2305.13245 , year=

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints , author=. arXiv preprint arXiv:2305.13245 , year=

    arXiv

  3. [11]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  4. [12]

    Publications Manual , year = "1983", publisher =

    1983

  5. [13]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    1981

  6. [14]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  7. [15]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    arXiv

  8. [16]

    arXiv preprint arXiv:2309.00071 , year=

    Yarn: Efficient context window extension of large language models , author=. arXiv preprint arXiv:2309.00071 , year=

    arXiv

  9. [17]

    arXiv preprint arXiv:2402.02750 , year=

    Kivi: A tuning-free asymmetric 2bit quantization for kv cache , author=. arXiv preprint arXiv:2402.02750 , year=

    arXiv

  10. [18]

    Transactions of the Association for Computational Linguistics , volume=

    The narrativeqa reading comprehension challenge , author=. Transactions of the Association for Computational Linguistics , volume=. 2018 , publisher=

    2018

  11. [19]

    Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    QMSum: A new benchmark for query-based multi-domain meeting summarization , author=. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    2021

  12. [20]

    arXiv preprint arXiv:2105.03011 , year=

    A dataset of information-seeking questions and answers anchored in research papers , author=. arXiv preprint arXiv:2105.03011 , year=

    arXiv

  13. [21]

    arXiv preprint arXiv:2307.08691 , year=

    Flashattention-2: Faster attention with better parallelism and work partitioning , author=. arXiv preprint arXiv:2307.08691 , year=

    arXiv

  14. [22]

    arXiv preprint arXiv:2309.17453 , year=

    Efficient Streaming Language Models with Attention Sinks , author=. arXiv preprint arXiv:2309.17453 , year=

    arXiv

  15. [23]

    Cai, Zefan and Zhang, Yichi and Gao, Bofei and Liu, Yuliang and Li, Yucheng and Liu, Tianyu and Lu, Keming and Xiong, Wayne and Dong, Yue and Hu, Junjie and others , journal=

  16. [24]

    arXiv preprint arXiv:2406.10774 , year=

    Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference , author=. arXiv preprint arXiv:2406.10774 , year=

    arXiv

  17. [25]

    arXiv preprint arXiv:2510.00636 , year=

    Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution , author=. arXiv preprint arXiv:2510.00636 , year=

  18. [26]

    arXiv preprint arXiv:2405.16406 , year=

    SpinQuant: LLM quantization with learned rotations , author=. arXiv preprint arXiv:2405.16406 , year=

    arXiv

  19. [27]

    arXiv preprint arXiv:2509.15038 , year=

    Value-Guided KV Compression for LLMs via Approximated CUR Decomposition , author=. arXiv preprint arXiv:2509.15038 , year=

  20. [28]

    Agrawal, Lakshya A and Tan, Shangyin and Soylu, Dilara and Ziems, Noah and Khare, Rishi and Opsahl-Ong, Krista and Singhvi, Arnav and Shandilya, Herumb and Ryan, Michael J and Jiang, Meng and others , journal=

  21. [29]

    International Conference on Learning Representations , volume=

    Think: Thinner key cache by query-driven pruning , author=. International Conference on Learning Representations , volume=

  22. [30]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Attention score is not all you need for token importance indicator in kv cache reduction: Value also matters , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.