Pith. sign in

REVIEW 12 cited by

PQCache: Product Quantization-based KVCache for Long Context LLM Inference

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.12820 v2 pith:3N5BVIFT submitted 2024-07-01 cs.CL cs.AIcs.LG

Hailin Zhang , Xiaodong Ji , Yilin Chen , Fangcheng Fu , Xupeng Miao , Xiaonan Nie , Weipeng Chen , Bin Cui This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords kvcachecomputationduringinferencelatencypqcacheretrievaltokens
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

As the field of Large Language Models (LLMs) continues to evolve, the context length in inference is steadily growing. Key-Value Cache (KVCache), the intermediate representations of tokens within LLM inference, has now become the primary memory bottleneck due to limited GPU memory. Current methods selectively determine suitable keys and values for self-attention computation in LLMs to address the issue. However, they either fall short in maintaining model quality or result in high serving latency. Drawing inspiration from advanced embedding retrieval techniques prevalent in the data management community, we consider the storage and retrieval of KVCache as a typical embedding retrieval problem. We propose PQCache, which employs Product Quantization (PQ) to manage KVCache, maintaining model quality while ensuring low serving latency. During the prefilling phase, we apply PQ to tokens' keys for each LLM layer and head. During the autoregressive decoding phase, we use PQ codes and centroids to approximately identify important preceding tokens, then fetch the corresponding key-value pairs for self-attention computation. Through meticulous design of overlapping and caching, we minimize any additional computation and communication overhead during both phases. Extensive experiments demonstrate that PQCache achieves both effectiveness and efficiency, with 4.60% score improvement over existing methods on InfiniteBench and low system latency in both prefilling and decoding.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding

    cs.LG 2026-07 accept novelty 7.0 of 10

    Page-local rank-8 spectral key summaries let sparse decode selection track the exact mass oracle and match FullKV quality at ~2% attended tokens with 2× latency cut at 1M context.

  2. KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Models

    cs.AI 2026-08 accept novelty 6.0 of 10

    KVDiagnosis is a diagnostic benchmark that pairs 59,800 compressed runs with FullCache controls and attaches cache, likelihood, attention, and decoding measurements to 12,520 failure rows, finding 63.2% of failures ha...

  3. SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling

    cs.LG 2025-05 conditional novelty 6.0 of 10

    SALE is a training-free sparse attention method that uses 4-bit quantized query-key estimates and a relative attention score to skip unimportant blocks, achieving over 3.36x prefill speedup on 64K+ token contexts with...

  4. TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization

    cs.CL 2025-05 conditional novelty 6.0 of 10

    TailorKV combines 1-bit quantization in shallow attention layers with dynamic Top-K token retrieval in deeper layers to serve 128k-context Llama-3.1-8B on a single 24GB GPU with a small accuracy loss.

  5. Efficient Graph-Based Approximate Nearest Neighbor Search Achieving: Low Latency Without Throughput Loss

    cs.DC 2025-04 conditional novelty 6.0 of 10

    AverSearch, a fully asynchronous graph-based nearest neighbor search engine, achieves lower query latency at higher throughput than the previous state of the art, iQAN, on multicore CPUs.

  6. Breaking the Boundaries of Long-Context LLM Inference: Adaptive KV Management on a Single Commodity GPU

    cs.OS 2025-06 conditional novelty 5.0 of 10

    LeoAM reports a 3.46x average latency speedup for long-context LLM inference on one commodity GPU by adaptively chunking KV data and loading compact key abstracts from disk instead of full KV values.

  7. Learn from the Past: Fast Sparse Indexing for Large Language Model Decoding

    cs.LG 2025-05 reject novelty 5.0 of 10

    LFPS predicts which KV cache positions a decoder will attend to by reusing historical vertical and slash patterns plus local expansion, then runs exact Top-k only on the predicted candidate set.

  8. Exploiting Sparsity for Long Context Inference: Million Token Contexts on Commodity GPUs

    cs.CL 2025-02 conditional novelty 5.0 of 10

    A top-k attention mechanism backed by CPU vector search allows million-token LLM contexts to run on a 16GB GPU while keeping over 95% of dense-attention performance, though the 2% sparsity claim does not hold at the l...

  9. SeerAttention-R: Sparse Attention Adaptation for Long Reasoning

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A learned gate selects the important KV blocks during long decoding, preserving math reasoning accuracy while skipping up to 90% of attention work.

  10. A Survey on Large Language Model Acceleration based on KV Cache Management

    cs.AI 2024-12 conditional novelty 4.0 of 10

    A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.

  11. XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference

    cs.LG 2024-12 conditional novelty 4.0 of 10

    XKV gives each network layer a personalized KV cache budget based on attention retention, cutting memory 61.6% with comparable accuracy.

  12. Taming the Titans: A Survey of Efficient LLM Inference Serving

    cs.CL 2025-04 conditional novelty 3.0 of 10

    A survey that classifies LLM inference serving techniques into instance, cluster, emerging-scenario, and miscellaneous categories, with a companion GitHub index and four research directions.

Pith tools