Pith. sign in

REVIEW 5 cited by

RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2502.14051 v3 pith:CWGSNJML submitted 2025-02-19 cs.CL cs.LG

Payman Behnam , Yaosheng Fu , Ritchie Zhao , Po-An Tsai , Zhiding Yu , Alexey Tumanov This is my paper
classification cs.CLcs.LG
keywords cacherocketkvattentioncompressionaccuracydecodeinputlong-context
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Transformer-based Large Language Models rely critically on the KV cache to efficiently handle extended contexts during the decode phase. Yet, the size of the KV cache grows proportionally with the input length, burdening both memory bandwidth and capacity as decoding progresses. To address this challenge, we present RocketKV, a training-free KV cache compression strategy containing two consecutive stages. In the first stage, it performs coarse-grain permanent KV cache eviction on the input sequence tokens. In the second stage, it adopts a hybrid sparse attention method to conduct fine-grain top-k sparse attention, approximating the attention scores by leveraging both head and sequence dimensionality reductions. We show that RocketKV provides a compression ratio of up to 400$\times$, end-to-end speedup of up to 3.7$\times$ as well as peak memory reduction of up to 32.6% in the decode phase on an NVIDIA A100 GPU compared to the full KV cache baseline, while achieving negligible accuracy loss on a variety of long-context tasks. We also propose a variant of RocketKV for multi-turn scenarios, which consistently outperforms other existing methods and achieves accuracy nearly on par with an oracle top-k attention scheme. The source code is available here: https://github.com/NVlabs/RocketKV.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding

    cs.LG 2026-07 accept novelty 7.0 of 10

    Page-local rank-8 spectral key summaries let sparse decode selection track the exact mass oracle and match FullKV quality at ~2% attended tokens with 2× latency cut at 1M context.

  2. C$^2$KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference

    cs.CL 2026-07 conditional novelty 6.0 of 10

    C2KV trains a sidecar extractor to produce compressed, position-agnostic KV cache segments that can be concatenated at inference time, reducing prefill, storage, and memory-bandwidth costs without modifying the frozen...

  3. Slipstream: Trajectory-Grounded Compaction Validation for Long-Horizon Agents

    cs.MA 2026-05 unverdicted novelty 6.0 of 10

    Slipstream uses asynchronous compaction with trajectory-grounded judge validation to improve long-horizon agent accuracy by up to 8.8 percentage points and reduce latency by up to 39.7%.

  4. BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding

    cs.CL 2025-12 unverdicted novelty 6.0 of 10

    BLASST dynamically sparsifies attention by thresholding softmax scores to skip blocks, delivering 1.5x speedups at 70%+ sparsity while preserving benchmark accuracy.

  5. HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compression

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    HARD-KV bridges dynamic head-adaptive KV cache compression with static inference engine constraints via Cascade Cache and Logits Calibration, reporting up to 2x throughput gains on long-context math benchmarks.

Pith tools