Pith. sign in

REVIEW 2 major objections 2 minor 7 cited by

CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation

T0 review · 2 major / 2 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CompressKV claims that, in GQA-based LLMs, a small set of per-layer semantic retrieval heads can decide before generation which KV-cache tokens are safe to evict, yielding consistent gains across memory budgets on LongBench and…

desk verdict The abstract sketches a plausible KV-cache compression idea, but the submitted full text is a SuperCDMS detector paper, so the actual manuscript cannot be reviewed at all. read the letter →

arxiv 2508.02401 v1 pith:5TXPIUIP submitted 2025-08-04 cs.CL cs.AI

Xiaolin Lin , Jingcun Wang , Olga Kondrateva , Yiyu Shi , Bing Li , Grace Li Zhang This is my paper
classification cs.CLcs.AI
keywords KVcachecompressionGroupedQueryAttentionsemanticretrievalheadslayer-adaptiveallocationlong-contextLLMstokenevictionLongBenchNeedle-in-a-Haystack
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes CompressKV, a way to shrink the key-value (KV) cache of long-context large language models by deciding which tokens to keep before generation starts. Its central claim is that, in each layer, a small set of attention heads—those able to retrieve the first and last tokens and important mid-text tokens together with their surrounding context—can identify the KV entries worth retaining. The method then splits the total cache budget across layers according to each layer's measured eviction error. On LongBench and Needle-in-a-Haystack, the authors report that CompressKV consistently outperforms existing KV-cache compression methods at the same memory budgets. If the claim holds, long-context inference becomes cheaper and faster without giving up accuracy.

What carries the argument

The central object is the semantic retrieval head: an attention head selected per layer because it retrieves the first tokens, the last tokens, and important internal tokens while attending to their surrounding context. These heads act as a pre-generation proxy for token importance, so their attention weights decide which key-value pairs are evicted. The second mechanism is layer-adaptive budget allocation, which sets each layer's cache allowance from that layer's eviction error rather than giving every layer the same budget. Together they convert KV-cache compression from a heuristic applied over all heads into a targeted selection guided by a few heads.

What would settle it

Run CompressKV on a long-context benchmark where the answer depends on a mid-text token that the selected semantic retrieval heads attend to weakly, and compare against an oracle that keeps all tokens; if the compressed model's accuracy drops sharply while a budget-equivalent all-head heuristic does not, the proxy assumption fails.

Watch

Extended reading notes

Core claim

The core discovery is that not all attention heads should vote on token importance when compressing KV caches in GQA-based LLMs. CompressKV first identifies, for each layer, semantic retrieval heads, defined as heads that simultaneously retrieve the initial and final tokens and attend to important mid-text tokens plus their semantic context. These heads' attention patterns, observed before generation, then determine which KV pairs are kept. In addition, the paper analyzes the cache eviction error of each layer and allocates a per-layer budget, so layers that lose more from eviction get a larger share. The authors claim this two-part scheme consistently outperforms state-of-the-art compression methods under various memory budgets on LongBench and Needle-in-a-Haystack.

Load-bearing premise

The method's load-bearing premise is that the attention patterns of a few selected heads are a faithful proxy for which tokens matter across diverse prompts, and that each layer's eviction error can be measured accurately enough to guide budget splits.

Editorial extensions

If this is right

  • KV cache memory can be lowered before generation starts, since token importance is decided during prefill rather than during decoding.
  • Compression quality no longer depends on averaging over all attention heads, so tokens attended to only by specialized heads can survive eviction.
  • Layer budgets that follow eviction error will outperform uniform per-layer budgets at the same total memory.
  • The same cache budget can support longer contexts or larger batch sizes on fixed hardware without extra accuracy loss.
  • The method is designed to work with the existing heads of GQA-based LLMs, so it can be applied without retraining the model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same few-heads-decide-importance idea could be extended to other compression targets, such as pruning activations or choosing which prompt tokens enter the context.
  • A natural test is whether the selected semantic retrieval heads transfer across different GQA model families; if they do, head selection becomes a one-time calibration, and if not, it must be redone per model.
  • Because selection happens before generation, CompressKV could be combined with chunked prefill or speculative decoding to hide the compression cost.
  • The benchmarks in the paper may not stress cases where the decisive token is deliberately unsalient; adversarial prompts would be a sharper test of the proxy assumption.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript, arXiv:2508.02401, presents CompressKV, a method for KV-cache compression in GQA-based LLMs. According to the abstract, the method selects per-layer 'semantic retrieval heads' before generation, uses these heads to score token importance, retains KV pairs for important tokens, and allocates cache budgets per layer based on eviction error. The claimed contribution is that CompressKV consistently outperforms state-of-the-art KV-cache compression approaches on LongBench and Needle-in-a-Haystack across memory budgets. However, the supplied full text is not the paper described in the abstract; it is an unrelated SuperCDMS detector-calibration paper (arXiv:2508.02402). Consequently, none of the method's technical content—head selection criteria, token-importance computation, eviction-error model, layer-adaptive allocation, or experimental details—is available for inspection.

Significance. If the claims in the abstract are correct, CompressKV would be a meaningful contribution to long-context LLM inference, as it proposes a more principled, head-selective alternative to heuristic all-head token eviction and adds per-layer budget adaptation. The public code release is a positive step for reproducibility. However, the significance cannot be evaluated from the material actually submitted, because the full text does not contain any of the technical apparatus or empirical evidence for CompressKV. The abstract alone is insufficient to establish either the novelty of the head-selection scheme or the validity of the performance claims.

major comments (2)
  1. [Full Text] The supplied full text is not the manuscript described in the abstract; it is 'Low-Energy Calibration of SuperCDMS HVeV Cryogenic Silicon Calorimeters Using Compton Steps' (arXiv:2508.02402), a physics detector-calibration paper. As a result, the entire technical content of the CompressKV submission—the definition of semantic retrieval heads, the token-importance scoring mechanism, the eviction-error computation, the layer-adaptive allocation algorithm, the exact experimental setup, and the results tables—is absent. This is a load-bearing deficiency: the central claim that CompressKV 'consistently outperforms state-of-the-art approaches' cannot be checked in any way. The submission cannot be reviewed in its current form.
  2. [Abstract / Methodology] Even taking the abstract on its own terms, the core assumption that a small set of retrieval heads selected for their ability to attend to initial tokens, final tokens, and 'important tokens within the text' can serve as a faithful proxy for which KV entries are safe to evict is unsupported. The abstract provides no definition of 'important tokens' or 'surrounding semantic context,' no evidence that head-level retrieval capability observed during selection generalizes across prompts, and no description of how the per-layer eviction error is measured or why that measurement is reliable enough to guide budget allocation. These gaps would need to be addressed in the actual full text; they cannot be resolved from the abstract alone.
minor comments (2)
  1. [Abstract] The abstract states that prior methods rely on 'all attention heads in Grouped Query Attention (GQA)-based LLMs' but does not name the specific baseline methods used in the comparisons; listing a few representative baselines would help contextualize the claimed improvements.
  2. [Abstract / Reproducibility] The GitHub repository link is given, but the abstract does not specify the exact LLMs, context lengths, or memory-budget configurations used in the LongBench and Needle-in-a-Haystack evaluations; these details are necessary for reproducibility and typically belong in the abstract or the first section of the full text.

Circularity Check

1 steps flagged · score 6.0 of 10

Abstract-level head selection is self-definitional: heads are picked for retrieving important tokens and then used to define which tokens are important; the supplied full text is a physics paper and cannot be inspected.

  1. self definitional [Abstract (method description; full-text manuscript not provided)]
    "we first identify the attention heads in each layer that are not only capable of retrieving the initial and final tokens of a prompt, but also capable of retrieving important tokens within the text and attending to their surrounding semantic context. Afterwards, we exploit such heads to determine the important tokens and retain their corresponding KV cache pairs."

    The phrase 'important tokens' appears both as the criterion for selecting retrieval heads and as the output produced by those heads. In the abstract, no independent ground-truth definition of 'important' is provided: heads are chosen because they retrieve important tokens, and then token importance is determined by those very heads. If the selection step has no external importance labels, the resulting importance scores are true by construction and cannot independently validate the eviction decision. Initial and final tokens are positionally objective, but the mid-text 'important tokens' and 'surrounding semantic context' are defined only through the selected heads, making the selection and the downstream determination mutually constitutive.

full rationale

The only described derivation in the abstract is self-definitional: attention heads are selected for being able to retrieve important tokens, and then those same heads are used to determine which tokens are important. Without an external, head-independent notion of token importance in the selection phase, the method's token-importance scores reduce to the heads' own attention preferences, so the 'prediction' of important tokens is forced by the construction. This is a central load-bearing step, as the entire compression decision rests on those scores. Separately, the supplied full text is arXiv:2508.02402, a SuperCDMS detector-calibration paper, not the CompressKV manuscript. Consequently, no equations, eviction-error model, head-selection details, or experimental protocols are available. This prevents independent verification of the empirical claim that CompressKV consistently outperforms state-of-the-art methods, but the mismatch itself is not an instance of circular reasoning under the review rules. No self-citations or imported uniqueness theorems are visible in the abstract. The partial circularity identified in the abstract warrants a score of 6: a central predictive component reduces by construction, while the reported benchmark superiority remains an unverifiable empirical assertion rather than a derivationally forced result.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

Since only the abstract is available, the logical dependencies of the actual paper cannot be audited. The assumptions listed below are the two load-bearing premises directly visible from the abstract: the existence of semantic retrieval heads as reliable token-importance signals, and the feasibility of per-layer error-driven cache allocation.

assumptions (2)
  • domain assumption GQA-based LLMs contain distinct semantic retrieval heads that can identify important tokens and their surrounding context.
    The core of the method: the abstract states these heads are identified and used to determine important tokens. If such heads do not generalize or lack sufficient coverage, the eviction would degrade quality.
  • domain assumption Per-layer cache eviction error can be measured and used to set layer-wise budgets.
    The abstract's layer-adaptive strategy presumes that the eviction error per layer is computable at small cost and is a reliable signal for budget allocation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation." pith.science (2026). https://pith.science/paper/5TXPIUIP

@misc{pith2026250802401,
  author       = {Pith},
  title        = {Pith review of: CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5TXPIUIP}},
  note         = {Machine review of arXiv:2508.02401}
}
read the original abstract

Recent advances in large language models (LLMs) have significantly boosted long-context processing. However, the increasing key-value (KV) cache size poses critical challenges to memory and execution efficiency. Most KV cache compression methods rely on heuristic token eviction using all attention heads in Grouped Query Attention (GQA)-based LLMs. This method ignores the different functionalities of attention heads, leading to the eviction of critical tokens and thus degrades the performance of LLMs. To address the issue above, instead of using all the attention heads in GQA-based LLMs to determine important tokens as in the previous work, we first identify the attention heads in each layer that are not only capable of retrieving the initial and final tokens of a prompt, but also capable of retrieving important tokens within the text and attending to their surrounding semantic context. Afterwards, we exploit such heads to determine the important tokens and retain their corresponding KV cache pairs. Furthermore, we analyze the cache eviction error of each layer individually and introduce a layer-adaptive KV cache allocation strategy. Experimental results demonstrate the proposed CompressKV consistently outperforms state-of-the-art approaches under various memory budgets on LongBench and Needle-in-a-Haystack benchmarks. Our code is publicly available at: https://github.com/TUDa-HWAI/CompressKV.git.

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Adaptive Filtering of the KV Cache: Diagnosing and Correcting Structural-Role Bias in LLM Inference

    cs.CL 2026-07 conditional novelty 7.0 of 10

    Heavy-hitter KV cache eviction over-retains structural delimiters and keys on schema-dense inputs; a role-conditional reallocation of SnapKV's score recovers most of the accuracy collapse.

  2. Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads

    cs.CL 2026-07 unverdicted novelty 7.0 of 10

    LOCOS scores attention heads via OV-circuit output projection onto answer-token unembedding directions and identifies non-literal retrieval heads whose ablation collapses performance on non-literal benchmarks more tha...

  3. Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    Dustin reports 27.85x self-attention and 9.17x end-to-end speedups at 32k length on Qwen2.5-72B using draft-augmented sparse verification with negligible accuracy loss on PG-19 and LongBench.

  4. RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention

    cs.AI 2026-06 reject novelty 6.0 of 10

    RedKnot reports up to 3.5× lower time-to-first-token and 4.7–7.8× more concurrent sessions by reusing KV cache at the granularity of attention heads rather than tokens, while preserving QA accuracy.

  5. RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    RedKnot decomposes the KV cache along attention heads to enable position-independent reuse, prefix compression, hot/cold separation, and distributed placement while preserving output fidelity.

  6. RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    RedKnot decomposes the KV cache by attention heads to enable position-independent reuse, prefix compression, hot/cold separation, and distributed placement for long-context LLM serving without model changes.

  7. TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference

    cs.LG 2025-08 conditional novelty 5.0 of 10

    TPLA splits the latent KV cache across tensor-parallel GPUs while keeping every head's full view, yielding 1.79x and 1.93x decode speedups on DeepSeek-V3 and Kimi-K2 at 32K context with modest accuracy loss.

Reference graph

Works this paper leans on

33 extracted references · 22 canonical work pages · cited by 5 Pith papers

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al

    Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2024. GPT-4 Technical Report. arXiv:2303.08774

    arXiv 2024

  4. [4]

    Ainslie, J.; Lee-Thorp, J.; de Jong, M.; Zemlyanskiy, Y.; Lebron, F.; and Sanghai, S. 2023. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. In The 2023 Conference on Empirical Methods in Natural Language Processing

    work page 2023

  5. [5]

    Anthropic. 2024. The Claude 3 Model Family: Opus, Sonnet, Haiku. Technical report, Anthropic. Accessed: 2024-07-09

    work page 2024

  6. [6]

    Bai, Y.; Lv, X.; Zhang, J.; Lyu, H.; Tang, J.; Huang, Z.; Du, Z.; Liu, X.; Zeng, A.; Hou, L.; Dong, Y.; Tang, J.; and Li, J. 2024. L ong B ench: A Bilingual, Multitask Benchmark for Long Context Understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    work page 2024

  7. [7]

    Cai, Z.; Zhang, Y.; Gao, B.; Liu, Y.; Li, Y.; Liu, T.; Lu, K.; Xiong, W.; Dong, Y.; Hu, J.; and Xiao, W. 2025. PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling. arXiv:2406.02069

    arXiv 2025

  8. [8]

    Dao, T. 2024. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. In The Twelfth International Conference on Learning Representations

    work page 2024

Show all 33 references
  1. [9]

    Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783

    2024 arXiv

  2. [10]

    Ge, S.; Zhang, Y.; Liu, L.; Zhang, M.; Han, J.; and Gao, J. 2024. Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs. In The Thirteenth International Conference on Learning Representations

    2024

  3. [11]

    Han, C.; Wang, Q.; Peng, H.; Xiong, W.; Chen, Y.; Ji, H.; and Wang, S. 2024. LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Hu...

    2024

  4. [12]

    Hui, B.; Yang, J.; Cui, Z.; Yang, J.; Liu, D.; Zhang, L.; Liu, T.; Zhang, J.; Yu, B.; Lu, K.; et al. 2025. Qwen2.5 Technical Report. arXiv:2412.15115

    2025 arXiv

  5. [13]

    Jiang, D.; Liu, Y.; Liu, S.; Zhao, J.; Zhang, H.; Gao, Z.; Zhang, X.; Li, J.; and Xiong, H. 2024. From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models. arXiv:2310.08825

    2024 arXiv

  6. [14]

    Kamradt, G. 2023. NeedleInAHaystack. https://github.com/gkamradt/LLMTest_NeedleInAHaystack. Accessed: 2025-07-13

    2023

  7. [15]

    Kang, H.; Zhang, Q.; Kundu, S.; Jeong, G.; Liu, Z.; Krishna, T.; and Zhao, T. 2024. GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM. arXiv:2403.05527

    2024 arXiv

  8. [16]

    W.; Hassoun, J.; Keutzer, K.; and Gholami, A

    Kwon, W.; Kim, S.; Mahoney, M. W.; Hassoun, J.; Keutzer, K.; and Gholami, A. 2022. A Fast Post-Training Pruning Framework for Transformers. In Oh, A. H.; Agarwal, A.; Belgrave, D.; and Cho, K., eds., Advances in Neural Information Processing Systems

    2022

  9. [17]

    Li, Y.; Huang, Y.; Yang, B.; Venkitesh, B.; Locatelli, A.; Ye, H.; Cai, T.; Lewis, P.; and Chen, D. 2024. Snap KV : LLM Knows What You are Looking for Before Generation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  10. [18]

    Liu, Z.; Desai, A.; Liao, F.; Wang, W.; Xie, V.; Xu, Z.; Kyrillidis, A.; and Shrivastava, A. 2023. Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time. In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  11. [19]

    Liu, Z.; Yuan, J.; Jin, H.; Zhong, S.; Xu, Z.; Braverman, V.; Chen, B.; and Hu, X. 2024. KIVI : A Tuning-Free Asymmetric 2bit Quantization for KV Cache. In Forty-first International Conference on Machine Learning

    2024

  12. [20]

    Olsson, C.; Elhage, N.; Nanda, N.; Joseph, N.; DasSarma, N.; Henighan, T.; Mann, B.; Askell, A.; Bai, Y.; Chen, A.; Conerly, T.; Drain, D.; Ganguli, D.; Hatfield-Dodds, Z.; Hernandez, D.; Johnston, S.; Jones, A.; Kernion, J.; Lovitt, L.; Ndousse, K.; Amodei, D.; Brown, T.; Cla...

    2022 arXiv

  13. [21]

    Oren, M.; Hassid, M.; Yarden, N.; Adi, Y.; and Schwartz, R. 2024. Transformers are Multi-State RNNs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 18724--18741

    2024

  14. [22]

    Qin, Z.; Cao, Y.; Lin, M.; Hu, W.; Fan, S.; Cheng, K.; Lin, W.; and Li, J. 2025. CAKE : Cascading and Adaptive KV Cache Eviction with Layer Preferences. In The Thirteenth International Conference on Learning Representations

    2025

  15. [23]

    Ren, J.; Guo, Q.; Yan, H.; Liu, D.; Zhang, Q.; Qiu, X.; and Lin, D. 2024. Identifying Semantic Induction Heads to Understand In-Context Learning. In Findings of the Association for Computational Linguistics: ACL 2024

    2024

  16. [24]

    S.; Mueller, A.; Wallace, B

    Todd, E.; Li, M.; Sharma, A. S.; Mueller, A.; Wallace, B. C.; and Bau, D. 2024. Function Vectors in Large Language Models. In The Twelfth International Conference on Learning Representations

    2024

  17. [25]

    Wan, Z.; Wu, X.; Zhang, Y.; Xin, Y.; Tao, C.; Zhu, Z.; Wang, X.; Luo, S.; Xiong, J.; Wang, L.; and Zhang, M. 2025. D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models. In The Thirteenth International Conference on Learning Repre...

    2025

  18. [26]

    Wang, J.; Chen, Y.-G.; Lin, I.-C.; Li, B.; and Zhang, G. L. 2025. Bsis S haring: Cross-Layer Parameter Sharing for Large Language Model Compression. In International Conference on Learning Representations

    2025

  19. [27]

    Wu, W.; Wang, Y.; Xiao, G.; Peng, H.; and Fu, Y. 2025. Retrieval Head Mechanistically Explains Long-Context Factuality. In The Thirteenth International Conference on Learning Representations

    2025

  20. [28]

    Xiao, G.; Tang, J.; Zuo, J.; junxian guo; Yang, S.; Tang, H.; Fu, Y.; and Han, S. 2025. DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads. In The Thirteenth International Conference on Learning Representations

    2025

  21. [29]

    Xiao, G.; Tian, Y.; Chen, B.; Han, S.; and Lewis, M. 2024. Efficient Streaming Language Models with Attention Sinks. In The Twelfth International Conference on Learning Representations

    2024

  22. [30]

    Yang, D.; Han, X.; Gao, Y.; Hu, Y.; Zhang, S.; and Zhao, H. 2024. PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference. In Findings of the Association for Computational Linguistics ACL 2024, 3258--3270

    2024

  23. [31]

    Yin, K.; and Steinhardt, J. 2025. Which Attention Heads Matter for In-Context Learning? arXiv:2502.14010

    2025 arXiv

  24. [32]

    Zhang, Z.; Sheng, Y.; Zhou, T.; Chen, T.; Zheng, L.; Cai, R.; Song, Z.; Tian, Y.; Re, C.; Barrett, C.; Wang, Z.; and Chen, B. 2023. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. In Thirty-seventh Conference on Neural Information Processi...

    2023

  25. [33]

    Zheng, Z.; Wang, Y.; Huang, Y.; Song, S.; Yang, M.; Tang, B.; Xiong, F.; and Li, Z. 2024. Attention Heads of Large Language Models: A Survey. arXiv:2409.03752

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.