Pith. sign in

REVIEW 4 major objections 6 minor 41 references

KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression

T0 review · 4 major / 6 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Kara compresses only the recent KV cache with bidirectional attention and flexible chunks, keeping nearly full reasoning accuracy at 20% retention while raising concurrent serving throughput.

desk verdict Solid systems paper: sliding-window bidirectional scoring + Token2Chunk + periodic PagedAttention compression; accuracy holds at low retention, throughput gains are real but regime-specific. read the letter →

arxiv 2607.01237 v2 pith:IQ3HSRYA submitted 2026-05-01 cs.CL cs.AI

Shen Han , Yuyang Wu , Junpu Yu , Olexandr Isayev This is my paper
classification cs.CLcs.AI
keywords KVcachecompressionslidingwindowbidirectionalattentionToken2ChunkreasoningLLMsPagedthroughputchain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reasoning models write long chain-of-thought traces, so the key-value cache balloons and concurrent serving stalls when memory is tight. Existing compression either fires on a global length threshold—often wiping whole blocks of context and even cutting throughput—or keeps isolated tokens or rigid fixed chunks that miss flexible semantic spans. Kara instead compresses only a sliding window of recently generated tokens, scores them with bidirectional attention, and expands selected tokens into variable-length chunks via a Token2Chunk module. The authors embed this in KvLLM, a paged-attention serving system that compresses trailing blocks on a fixed schedule rather than a threshold. On math reasoning and needle-in-a-haystack tasks the method holds nearly full-cache accuracy at 20% retention, and under memory-constrained concurrency KvLLM raises average output throughput by about 13% over vanilla serving.

What carries the argument

Sliding-window bidirectional attention scores keys in the compressible recent region by accumulating attention from all queries in the window; Token2Chunk then treats consecutive high-score tokens as endpoints of candidate flexible chunks and keeps the best under a length and budget cap. Periodic compression applies this only to trailing paged blocks every fixed number of decoding steps.

What would settle it

At 20% retention, place critical facts only in positions that leave the sliding window after the first compression and are never re-selected; if accuracy collapses versus full cache while methods that rescore the whole history hold up, the local-window importance claim is false.

Watch

Extended reading notes

Core claim

Kara preserves nearly 100% of full-KV accuracy while retaining only 20% of the KV cache by scoring and selecting informative pairs only inside a recent sliding window with bidirectional attention, then expanding a subset into flexible contiguous chunks. Wrapped as KvLLM with periodic trailing-block compression under PagedAttention, it improves average output throughput by 12.75% over vanilla vLLM in memory-constrained concurrent serving.

Load-bearing premise

What will matter later can be spotted and kept by looking only at attention inside the most recent window; once a token leaves that window and is not retained, it can be discarded forever.

Editorial extensions

If this is right

  • Paged serving can free trailing KV blocks on a schedule and admit more concurrent decoding sequences without waiting for a global length threshold.
  • Math reasoning and needle-in-a-haystack accuracy can stay near the full-cache baseline at 20–30% retention.
  • A single retention ratio across layers and heads stays compatible with tensor parallelism and paged attention.
  • Threshold-triggered whole-cache eviction is not required for high-quality decoding-time compression.
  • For many long chain-of-thought workloads, carefully scored recent context can substitute for keeping the entire history resident.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same windowed score could drive soft demotion into CPU/SSD retrieval instead of permanent eviction, matching the authors’ sketched future path.
  • Because compression is local and periodic, it may compose with block-level sparse attention or KV-merge schemes that already operate at similar granularity.
  • If long-range dependencies dominate a task more than recent CoT structure does, quality may degrade where global rescoring methods do not.
  • Operators gain explicit knobs—window size and compression period—for the latency–concurrency trade-off that pure threshold policies lack.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Kara, a decoding-time sliding-window KV cache compressor for reasoning LLMs, and KvLLM, a vLLM/PagedAttention integration with a periodic trailing-block compression policy. Kara scores KV pairs inside a recent window via accumulated bidirectional attention (Eqs. 4–5), then expands a subset of selected tokens into flexible-length chunks with Token2Chunk (Eqs. 6–8). The authors argue that threshold-triggered global compression can hurt concurrent throughput and erase contiguous spans, and that isolated or rigidly chunked retention misses flexible semantic units. Empirically, Kara is reported to retain near full-KV accuracy at ~20% retention on MATH-500/AMC23/AIME24 across three models (Fig. 5), with ablations (Table 1), NIAH (Fig. 8), and KvLLM throughput/latency gains under memory-constrained serving (Table 2; App. D.5).

Significance. If the results hold under fairer fixed-budget and multi-regime serving comparisons, the work is a useful systems contribution: it targets a real bottleneck (long CoT KV growth under concurrent serving), couples a training-free compressor to PagedAttention, and shows that trigger policy (periodic trailing-window vs threshold) can matter as much as the scoring rule. Strengths include multi-model/multi-benchmark accuracy curves, component ablations isolating bidirectional scoring and Token2Chunk, NIAH stress of permanent eviction, complexity analysis (App. D.6), and an explicit limitations/future-work section on offloading and recomputation cost. The paper is incremental relative to SnapKV/RKV/ChunkKV/DMS-style delayed eviction, but the combination of window-local bidirectional scoring, flexible Token2Chunk retention, and periodic PagedAttention compression is a coherent serving-oriented design.

major comments (4)
  1. Abstract / §1 / Table 2 / App. D.5: The dual headline claim (near-100% accuracy at 20% retention + 12.75% average throughput over vanilla vLLM) is not supported as a general serving improvement. Table 3 shows KvLLM is flat or slightly worse than vanilla vLLM at batch sizes 16–64 and only wins under tight memory (GPU util 0.5) at batch 128/256; SnapKV can win at batch 16. Please redefine the 12.75% figure (which cells, how averaged), report the full batch-size table in the main text, and qualify the abstract/intro claim as regime-specific rather than average concurrent-serving improvement.
  2. §5 Configurations: Accuracy baselines are budget-matched post hoc to Kara’s realized per-sequence compressed length after Kara is run. This is not an independent fixed-budget contest and can systematically favor Kara if Token2Chunk or window dynamics produce favorable length distributions. Add fixed-budget comparisons (same N for all methods, including Kara) and/or report mean±std retained length vs accuracy; without this, the “superior accuracy under the same budget” claim in §1 is only partially substantiated.
  3. §5.3 / Table 2: The throughput comparison mixes two design axes—scoring (Kara vs SnapKV) and trigger policy (periodic trailing-block vs threshold-triggered). Because the paper’s systems thesis is that threshold triggering causes concurrency–throughput inversion (Fig. 1), the main comparison should include (i) Kara under threshold triggering and (ii) SnapKV (or RKV) under the same periodic policy, or at least ablate period δ and sequences-per-event. Otherwise it is unclear whether gains come from Kara’s importance model or from controlling when compression fires.
  4. §5 / Fig. 5: No error bars, multi-seed variance, or multiple decoding temperatures are reported for pass@1 on MATH-500/AMC/AIME, and AIME24 is a small set. For a systems paper claiming “nearly 100% of full-KV accuracy,” please report variance over seeds or problem shuffles (or bootstrap CIs) at least for the 20–30% retention operating points that anchor the abstract.
minor comments (6)
  1. Title/abstract inconsistency: the arXiv title uses “KARA” / “Efficient Reasoning LLM Serving…”, while the manuscript body uses “Kara: Sliding-Window…”. Align naming and casing throughout.
  2. §4.1 / Fig. 3–4: The bidirectional-attention empirical analysis is suggestive but limited to short prefixes (~120 tokens) on one model; a brief note on whether the correlation holds deeper in long CoT traces would help readers assess Eq. (4).
  3. Eq. (5) and Token2Chunk: the interaction of retention ratio r, chunk budget α, and max chunk length γ is easy to misread; a short pseudocode or worked numerical example in the main text (beyond App. C) would clarify the effective budget.
  4. Fig. 5: AdaKV/PyramidKV use adaptive per-layer/head budgets while Kara uses a uniform ratio; the caption notes this, but the main text should state more clearly that this architectural choice favors deployability rather than claiming a pure accuracy win under identical allocation freedom.
  5. App. B.1: Experiments use nano-vLLM and RTX 5090 / H100–H200 mixes; please state any behavioral differences from production vLLM that could affect absolute throughput numbers.
  6. Typos/clarity: “grow-and-compress”, “concurrency–throughput inversion”, and occasional missing spaces around citations; also “nano-vLLM” footnote placement is dense—consider moving implementation notes fully to the appendix.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: empirical systems method with independent scoring and external benchmarks.

full rationale

Kara's load-bearing chain is (1) score KV pairs in a recent window via bidirectional attention (Eq. 4–5), (2) expand some discrete indices into flexible chunks via Token2Chunk (Eq. 6–8), (3) apply periodic trailing-block compression in KvLLM. Importance scores are computed from the model's own Q/K states inside the window; they are not fitted to MATH-500/AIME/AMC accuracy or to the throughput metric. Retention ratio r, window/buffer sizes, chunk budget α, and compression period δ are hyperparameters swept or fixed, not quantities defined by the reported outcomes. Accuracy is measured zero-shot pass@1 against full-KV and external baselines; throughput/latency are wall-clock measurements under a simulated memory constraint. Budget-matching baselines to Kara's realized per-sequence length (§5 Configurations) is a comparison design choice, not a fitted parameter renamed as a prediction: the accuracy claim is not forced by construction of that match. Related-work citations (SnapKV, StreamingLLM, ChunkKV, etc.) are external baselines, not self-authored uniqueness theorems that forbid alternatives. No equation equates the dual headline claim to its inputs. Regime-specific throughput gains and permanent-eviction risk are experimental/assumption concerns, not circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 3 invented entities

Load-bearing content is algorithmic and empirical. The method rests on standard Transformer attention math, the domain belief that recent-window attention scores predict which KV pairs can be permanently dropped, and several hand-chosen sizes (window, buffer, retention, chunk budget, period). Token2Chunk, Kara, and KvLLM are invented procedures, not new physical entities; they have no independent evidence outside the reported benchmarks.

free parameters (5)
  • retention ratio r
    Fraction of compressible-window KV pairs kept; swept in {20,30,40,50,60}% and is the main accuracy–memory knob.
  • window length |W| and buffer |U|
    Hand-chosen sliding window and protected tail (swept |W|∈{256,384,512}, |U|∈{32,64}); control what is scored and what is never compressed in the current step.
  • Token2Chunk budget α and max chunk length γ
    Cap extra contiguous tokens and max span (γ=8, α∈{16,32} in experiments); directly shape retained structure.
  • compression period δ and sequences-per-event
    Periodic policy triggers every δ steps (e.g. 128) and compresses a fixed subset (e.g. 30 sequences); chosen for throughput trade-off, not derived.
  • PagedAttention block size and per-window retained length
    Serving config (block 256; window 3 blocks / retain 257) sets when memory is freed and how fair baseline budgets are matched.
assumptions (4)
  • domain assumption Scaled dot-product multi-head attention with causal mask is the correct importance substrate for KV retention.
    Standard Transformer assumption used throughout §3–4; scoring is built on Pl,h and bidirectional variants.
  • ad hoc to paper Accumulated bidirectional attention inside a recent window identifies KV pairs that are both important and informative enough for permanent eviction of the rest.
    Motivated by Fig. 3–4 correlation analysis (§4.1); not a proved lemma, and permanent loss is admitted in Limitations.
  • domain assumption Threshold-triggered full-cache compression causes concurrency–throughput inversion under memory-constrained multi-sequence serving.
    Empirical premise from Fig. 1–2 motivating periodic policy; treated as general enough to redesign the trigger.
  • ad hoc to paper Matching each baseline’s per-sequence budget to Kara’s realized compressed length yields a fair accuracy comparison.
    Stated in §5 Configurations; procedural fairness choice that can still advantage the method that chose the lengths.
invented entities (3)
  • Token2Chunk module
    purpose: Expand pairs of discrete high-score indices into flexible-length contiguous KV chunks under budget α and max length γ.
    New retention operator (Eq. 6–8); only validated inside this paper’s ablations.
  • Kara sliding-window compressor
    purpose: Decode-time compression restricted to recent context via bidirectional attention scoring plus Token2Chunk.
    Core method of the paper; performance is the paper’s own measurements.
  • KvLLM periodic compression policy
    purpose: Schedule trailing-block Kara compression every δ steps on selected sequences under PagedAttention.
    Serving-layer invention (Alg. 1) to avoid threshold thrashing; evidence is internal throughput tables.

how reviews work

0 comments
Cite this review

Pith. "Pith review of KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression." pith.science (2026). https://pith.science/paper/IQ3HSRYA

@misc{pith2026260701237,
  author       = {Pith},
  title        = {Pith review of: KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IQ3HSRYA}},
  note         = {Machine review of arXiv:2607.01237}
}
read the original abstract

Reasoning language models often generate long chain-of-thought (CoT), which accumulates a massive KV cache during the decoding phase and incurs high decoding latency and limited throughput. To address these issues, KV cache compression has emerged as a promising technique for reducing memory overhead by selectively removing unimportant KV pairs while preserving useful ones for subsequent decoding. Nevertheless, we identify two key limitations in existing KV cache compression methods: 1) their threshold-triggered compression policy may provide limited throughput improvement or even reduce throughput, and may fully eliminate KV pairs from certain blocks of the sequence, potentially worsening information loss. 2) they typically retain either isolated KV pairs or fixed-size chunks with rigid boundaries, failing to preserve important flexible-sized chunks at arbitrary token positions. To overcome these limitations, we propose Kara, a sliding-window KV cache compression method that performs decoding-time compression by operating only on the recently generated context. Kara leverages bidirectional attention to score and select informative KV pairs in the window. To enable flexible preservation of important semantic information, we design a Token2Chunk module to expand a subset of selected KV pairs into chunks. Furthermore, we adapt Kara to PagedAttention and develop KvLLM, an inference framework built upon vLLM, which reduces KV cache memory usage and effectively improves output throughput. Extensive experiments demonstrate consistent performance improvements of proposed Kara and KvLLM.

Figures

Figures reproduced from arXiv: 2607.01237 by the authors.

Figure 1
Figure 1. (a) Average throughput under different batch sizes, where batch size denotes the predefined maximum decoding sequences. We observe that vLLM with SnapKV achieves lower throughput than vanilla vLLM as the batch size in￾creases. (b) The actual number of decoding sequences varies with decoding steps. The decoding step denotes the number of global decoding iterations and predefined maximum decoding sequences is 256. We … view at source ↗
Figure 3
Figure 3. (a) Distribution comparison between causal at￾tention and bidirectional attention of a specific token. (b) Average causal attention weight versus bidirectional atten￾tion percentile. We first compute the bidirectional attention weights for all token pairs (xi, xj ) with j > i, using xi as the query and xj as the key. We then sort these weights in ascending order and group the pairs by percentiles. Finally, for each … view at source ↗
Figure 5
Figure 5. Performance of different KV cache compression methods across varying retention levels. The dash line represents the accuracy of the vanilla LLM model without compression. concurrency-throughput inversion effect observed in [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Performance of different KV cache compression methods across varying retention levels. The dash line represents the accuracy of the vanilla LLM model without compression. Note that AdaKV and PyramidKV allocate adaptive memory budgets across layers and heads. Baselines.…
Figure 6
Figure 6. Figure 6: Needle-In-A-Haystack (NIAH) performance. The x-axis denotes the input context length and the y-axis denotes the needle insertion depth. Each cell reports the retrieval score for the corresponding (length, depth) setting, and we also report the mean accuracy averaged ov…
Figure 6
Figure 6. Figure 6: The top and bottom plots show the attention distributions of a token at Layer 10 and Layer 30, respectively. 0 10 20 30 40 50 60 70 80 Token Index 0.0 0.5 1.0 1.5 2.0 2.5 Importance Score Candidate Tokens Selected Tokens of Token2Chunk Unselected Tokens [PITH_FULL_IMA…
Figure 7
Figure 7. Figure 7: Visualization of the retained positions before and after applying Token2Chunk. We set the maximum chunk length and Token2Chunk budget to 7 and 10, respectively. D.2 Visualization of Tokens Retained by Token2Chunk We analyze several sequences sampled from MATH-500 and r…
Figure 8
Figure 8. Figure 8: Needle-In-A-Haystack (NIAH) performance. The x-axis denotes the input context length and the y-axis denotes the needle insertion depth. Each cell reports the retrieval score for the corresponding (length, depth) setting, and we also report the mean accuracy averaged ov…
Figure 9
Figure 9. Figure 9: Comparison of concurrent sequence numbers for different frameworks. D.6 Complexity Analysis of Kara We analyze the computation complexity of applying Kara once to a single sequence, ignoring memory access and KV movement. Let the current sequence length be S, the slidi…

Review history (2 revisions) →

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 8 linked inside Pith

  1. [1]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  2. [2]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  3. [3]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    arXiv 2025

  4. [4]

    CRANE: Reasoning with constrained LLM generation

    Debangshu Banerjee, Tarun Suresh, Shubham Ugare, Sasa Misailovic, and Gagandeep Singh. CRANE: Reasoning with constrained LLM generation. InF orty-second International Confer- ence on Machine Learning, 2025

    2025

  5. [5]

    A survey on large language model acceleration based on kv cache management.arXiv preprint arXiv:2412.19442, 2024

    Haoyang Li, Yiming Li, Anxin Tian, Tianhao Tang, Zhanchao Xu, Xuejia Chen, Nicole Hu, Wei Dong, Qing Li, and Lei Chen. A survey on large language model acceleration based on kv cache management.arXiv preprint arXiv:2412.19442, 2024

    arXiv 2024

  6. [6]

    Llm inference unveiled: Survey and roofline model insights.arXiv preprint arXiv:2402.16363, 2024

    Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, et al. Llm inference unveiled: Survey and roofline model insights.arXiv preprint arXiv:2402.16363, 2024

    arXiv 2024

  7. [7]

    R-KV: Redundancy-aware KV cache compression for reasoning models

    Zefan Cai, Wen Xiao, Hanshi Sun, Cheng Luo, Yikai Zhang, Ke Wan, Yucheng Li, Yeyang Zhou, Li-Wen Chang, Jiuxiang Gu, Zhen Dong, Anima Anandkumar, Abedelkadir Asi, and Junjie Hu. R-KV: Redundancy-aware KV cache compression for reasoning models. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026

    2026

  8. [8]

    Ada-KV: Optimizing KV cache eviction by adaptive budget allocation for efficient LLM inference

    Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou. Ada-KV: Optimizing KV cache eviction by adaptive budget allocation for efficient LLM inference. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026

    2026

Show all 41 references
  1. [9]

    SnapKV: LLM knows what you are looking for before generation

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 10

    2024

  2. [10]

    KV cache compression, but what must we give in return? a comprehensive benchmark of long context capable approaches

    Jiayi Yuan, Hongyi Liu, Shaochen Zhong, Yu-Neng Chuang, Songchen Li, Guanchu Wang, Duy Le, Hongye Jin, Vipin Chaudhary, Zhaozhuo Xu, Zirui Liu, and Xia Hu. KV cache compression, but what must we give in return? a comprehensive benchmark of long context capable approaches. In Y...

    2024

  3. [11]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  4. [12]

    Beyond text-visual attention: Exploiting visual cues for effective token pruning in vlms

    Qizhe Zhang, Aosong Cheng, Ming Lu, Renrui Zhang, Zhiyong Zhuo, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. Beyond text-visual attention: Exploiting visual cues for effective token pruning in vlms. InProceedings of the IEEE/CVF International Conference on Computer Vis...

    2025

  5. [13]

    Inference-time hyper-scaling with KV cache compression

    Adrian Ła ´ncucki, Konrad Staniszewski, Piotr Nawrot, and Edoardo Ponti. Inference-time hyper-scaling with KV cache compression. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026

    2026

  6. [14]

    Lee, Sangdoo Yun, and Hyun Oh Song

    Jang-Hyun Kim, Jinuk Kim, Sangwoo Kwon, Jae W. Lee, Sangdoo Yun, and Hyun Oh Song. KVzip: Query-agnostic KV cache compression with context reconstruction. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026

    2026

  7. [15]

    Expected attention: Kv cache compres- sion by estimating attention from future queries distribution.arXiv preprint arXiv:2510.00636, 2025

    Alessio Devoto, Maximilian Jeblick, and Simon Jégou. Expected attention: Kv cache compres- sion by estimating attention from future queries distribution.arXiv preprint arXiv:2510.00636, 2025

    2025

  8. [16]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Symposium on Operating System...

    2023

  9. [17]

    Criticbench: Benchmarking llms for critique-correct reasoning

    Zicheng Lin, Zhibin Gou, Tian Liang, Ruilin Luo, Haowei Liu, and Yujiu Yang. Criticbench: Benchmarking llms for critique-correct reasoning. InFindings of the Association for Computa- tional Linguistics: ACL 2024, pages 1552–1587, 2024

    2024

  10. [18]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  11. [19]

    ChunkKV: Semantic-preserving KV cache compression for efficient long-context LLM inference

    Xiang Liu, Zhenheng Tang, Peijie Dong, Zeyu Li, Liuyue, Bo Li, Xuming Hu, and Xiaowen Chu. ChunkKV: Semantic-preserving KV cache compression for efficient long-context LLM inference. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026

    2026

  12. [20]

    Efficient many- shot in-context learning with dynamic block-sparse attention

    Emily Xiao, Chin-Jou Li, Yilin Zhang, Graham Neubig, and Amanda Bertsch. Efficient many- shot in-context learning with dynamic block-sparse attention. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Proceedings of the 63rd Annual Meeting...

    2025

  13. [21]

    ThinKV: Thought-adaptive KV cache compression for efficient reasoning models

    Akshat Ramachandran, Marina Neseem, Charbel Sakr, Rangharajan Venkatesan, Brucek Khailany, and Tushar Krishna. ThinKV: Thought-adaptive KV cache compression for efficient reasoning models. InThe F ourteenth International Conference on Learning Representations, 2026

    2026

  14. [22]

    QuoKA: Query-oriented KV selection for efficient LLM prefill

    Dalton Jones, Junyoung Park, Matthew J Morse, Mingu Lee, Matthew Harper Langston, and Christopher Lott. QuoKA: Query-oriented KV selection for efficient LLM prefill. InThe F ourteenth International Conference on Learning Representations, 2026. 11

    2026

  15. [23]

    Icecache: Memory-efficient KV-cache management for long-sequence LLMs

    Yuzhen Mao, Qitong Wang, Martin Ester, and Ke Li. Icecache: Memory-efficient KV-cache management for long-sequence LLMs. InThe F ourteenth International Conference on Learning Representations, 2026

    2026

  16. [24]

    Cache what lasts: Token retention for memory-bounded KV cache in LLMs

    Ngoc Bui, Shubham Sharma, Simran Lamba, Saumitra Mishra, and Rex Ying. Cache what lasts: Token retention for memory-bounded KV cache in LLMs. InThe F ourteenth International Conference on Learning Representations, 2026

    2026

  17. [25]

    Keydiff: Key similarity-based KV cache eviction for long-context LLM inference in resource-constrained environments

    Junyoung Park, Dalton Jones, Matthew J Morse, Raghavv Goel, Mingu Lee, and Christopher Lott. Keydiff: Key similarity-based KV cache eviction for long-context LLM inference in resource-constrained environments. InThe Thirty-ninth Annual Conference on Neural Informa- tion Proces...

    2026

  18. [26]

    Spargeattention: Accurate and training-free sparse attention accelerating any model inference

    Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. Spargeattention: Accurate and training-free sparse attention accelerating any model inference. InF orty-second International Conference on Machine Learning, 2025

    2025

  19. [27]

    MoBA: Mixture of block attention for long-context LLMs

    Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Yutao Zhang, Zhilin Yang, Xinyu Zhou, Mi...

    2026

  20. [28]

    Twilight: Adaptive attention sparsity with hierarchical top-$p$ pruning

    Chaofan Lin, Jiaming Tang, Shuo Yang, Hanshuo Wang, Tian Tang, Boyu Tian, Ion Stoica, Song Han, and Mingyu Gao. Twilight: Adaptive attention sparsity with hierarchical top-$p$ pruning. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026

    2026

  21. [29]

    Xattention: Block sparse attention with antidiagonal scoring

    Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, and Song Han. Xattention: Block sparse attention with antidiagonal scoring. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors,F orty- secon...

    2025

  22. [30]

    Efficient attention mechanisms for large language models: A survey.arXiv preprint arXiv:2507.19595, 2025

    Yutao Sun, Zhenyu Li, Yike Zhang, Tengyu Pan, Bowen Dong, Yuyi Guo, and Jianyong Wang. Efficient attention mechanisms for large language models: A survey.arXiv preprint arXiv:2507.19595, 2025

    2025

  23. [31]

    Deepseek-v4: Towards highly efficient million-token context intelligence, 2026

    DeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence, 2026

    2026

  24. [32]

    Kwai summary attention technical report.arXiv preprint arXiv:2604.24432, 2026

    Chenglong Chu, Guorui Zhou, Guowang Zhang, Han Li, Hao Peng, Hongtao Cheng, Jian Liang, Jiangxia Cao, Kun Gai, Lingzhi Zhou, et al. Kwai summary attention technical report.arXiv preprint arXiv:2604.24432, 2026

    2026 arXiv

  25. [33]

    Where does in-context learning \\ happen in large language models? InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    Suzanna Sia, David Mueller, and Kevin Duh. Where does in-context learning \\ happen in large language models? InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  26. [34]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  27. [35]

    A survey on large language model acceleration based on KV cache management.Transactions on Machine Learning Research, 2025

    Haoyang LI, Yiming Li, Anxin Tian, Tianhao Tang, Zhanchao Xu, Xuejia Chen, Nicole HU, Wei Dong, Li Qing, and Lei Chen. A survey on large language model acceleration based on KV cache management.Transactions on Machine Learning Research, 2025

    2025

  28. [36]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: efficient execution of structured language model programs. NIPS ’24, Red Hook, NY , USA,

  29. [37]

    Curran Associates Inc. 12

  30. [38]

    CAKE: Cascading and adaptive KV cache eviction with layer preferences

    Ziran Qin, Yuchen Cao, Mingbao Lin, Wen Hu, Shixuan Fan, Ke Cheng, Weiyao Lin, and Jianguo Li. CAKE: Cascading and adaptive KV cache eviction with layer preferences. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  31. [39]

    American invitational mathematics examination (aime) 2024, 2024

    Yifan Zhang and Team Math-AI. American invitational mathematics examination (aime) 2024, 2024

    2024

  32. [40]

    Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling.arXiv preprint arXiv:2406.02069, 2024

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling.arXiv preprint arXiv:2406.02069, 2024

    2024 arXiv

  33. [41]

    Data engineering for scaling language models to 128k context

    Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. ICML’24. JMLR.org, 2024. A Limitations, Future Work and Impact Limitations.We acknowledge several limitations of Kara and K...

    2024

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.