Pith. sign in

REVIEW 11 cited by

SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.06219 v3 pith:RTVUFAF5 submitted 2024-05-10 cs.LG cs.CL

Haojie Duanmu , Zhihang Yuan , Xiuhong Li , Jiangfei Duan , Xingcheng Zhang , Dahua Lin This is my paper · ORCID
classification cs.LGcs.CL
keywords cachequantizationskvqaccuracyllmschannelscontexthigh
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Large language models (LLMs) can now handle longer sequences of tokens, enabling complex tasks like book understanding and generating lengthy novels. However, the key-value (KV) cache required for LLMs consumes substantial memory as context length increasing, becoming the bottleneck for deployment. In this paper, we present a strategy called SKVQ, which stands for sliding-window KV cache quantization, to address the issue of extremely low bitwidth KV cache quantization. To achieve this, SKVQ rearranges the channels of the KV cache in order to improve the similarity of channels in quantization groups, and applies clipped dynamic quantization at the group level. Additionally, SKVQ ensures that the most recent window tokens in the KV cache are preserved with high precision. This helps maintain the accuracy of a small but important portion of the KV cache.SKVQ achieves high compression ratios while maintaining accuracy. Our evaluation on LLMs demonstrates that SKVQ surpasses previous quantization approaches, allowing for quantization of the KV cache to 2-bit keys and 1.5-bit values with minimal loss of accuracy. With SKVQ, it is possible to process context lengths of up to 1M on an 80GB memory GPU for a 7b model and up to 7 times faster decoding.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Hidden Language Consistency Phenomena in Reasoning LLMs

    cs.CL 2026-08 conditional novelty 6.0 of 10

    Reasoning models often stop using the requested language as problems get harder, and this language breakdown can make accuracy look better than it is.

  2. Harmonia: Algorithm-Hardware Co-Design for Memory- and Compute-Efficient BFP-based LLM Inference

    cs.AR 2026-02 conditional novelty 6.0 of 10

    Harmonia runs LLM inference with all activations in block floating point (BFP) and a 4-bit KV cache, reporting 3.08x average speedup, 2.03x energy savings, and under 1% accuracy loss on LongBench.

  3. Which Heads Matter for Reasoning? RL-Guided KV Cache Compression

    cs.CL 2025-10 conditional novelty 6.0 of 10

    A small set of "reasoning heads" found by RL can keep full KV cache while other heads are compressed to a constant size, giving 20–50% cache savings with near-lossless accuracy.

  4. PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs

    cs.CL 2025-05 conditional novelty 6.0 of 10

    PM-KVQ introduces progressive bit-width shrinking, per-block memory allocation, and positional-interpolation calibration to make 2-bit KV cache quantization nearly lossless on long-CoT LLMs.

  5. MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design

    cs.LG 2025-05 conditional novelty 6.0 of 10

    MxMoE assigns mixed bitwidths at the linear-block level in MoE models using an ILP that balances measured quantization sensitivity and expert activation frequency, then auto-generates fused Group-GEMM kernels, improvi...

  6. Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models

    cs.LG 2025-01 conditional novelty 6.0 of 10

    Trained linear predictors across transformer layers plus residual quantization compress LLM KV caches to about 2 bits per value with near-lossless accuracy on tested Llama and Qwen models.

  7. AKVQ-VL: Attention-Aware KV Cache Adaptive 2-Bit Quantization for Vision-Language Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    AKVQ-VL quantizes VLM KV caches to mostly 2 bits with attention-aware token protection and Walsh-Hadamard outlier removal, staying near FP16 accuracy on MileBench.

  8. RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotations

    cs.LG 2025-01 conditional novelty 6.0 of 10

    RotateKV combines outlier-aware, pre-RoPE grouped-head Hadamard rotation with attention-sink-aware retention to make 2-bit KV cache quantization accurate on LLaMA-2, LLaMA-3, and Mistral models.

  9. Beyond Homogeneous Attention: Memory-Efficient LLMs via Fourier-Approximated KV Cache

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Compressing the less context-sensitive dimensions of the LLM KV cache with a truncated Fourier transform preserves long-context accuracy on LongBench and needle-in-a-haystack tests, while the claimed memory savings ar...

  10. Position: Episodic Memory is the Missing Piece for Long-Term LLM Agents

    cs.AI 2025-02 conditional novelty 5.0 of 10

    The authors propose episodic memory, with five defining properties, as the unifying framework needed for LLM agents to learn and remember over long time horizons.

  11. A Survey on Large Language Model Acceleration based on KV Cache Management

    cs.AI 2024-12 conditional novelty 4.0 of 10

    A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.

Pith tools