REVIEW 19 cited by
GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Key-value (KV) caching has become the de-facto to accelerate generation speed for large language models (LLMs) inference. However, the growing cache demand with increasing sequence length has transformed LLM inference to be a memory bound problem, significantly constraining the system throughput. Existing methods rely on dropping unimportant tokens or quantizing all entries uniformly. Such methods, however, often incur high approximation errors to represent the compressed matrices. The autoregressive decoding process further compounds the error of each step, resulting in critical deviation in model generation and deterioration of performance. To tackle this challenge, we propose GEAR, an efficient KV cache compression framework that achieves near-lossless high-ratio compression. GEAR first applies quantization to majority of entries of similar magnitudes to ultra-low precision. It then employs a low rank matrix to approximate the quantization error, and a sparse matrix to remedy individual errors from outlier entries. By adeptly integrating three techniques, GEAR is able to fully exploit their synergistic potentials. Our experiments demonstrate that compared to alternatives, GEAR achieves near-lossless 4-bit KV cache compression with up to 2.38x throughput improvement, while reducing peak-memory size up to 2.29x. Our code is publicly available at https://github.com/HaoKang-Timmy/GEAR.
Forward citations
Cited by 19 Pith papers
-
Runtime Observability for Heterogeneous Attention Memory
A runtime observability framework with typed, Lean-checked error contracts for four kinds of attention memory, tested on six LLM serving setups and a production corruption case.
-
PolarQuant: Quantizing KV Caches with Polar Transformation
PolarQuant achieves around 4x KV cache compression for LLMs by quantizing angles after a recursive polar transform with random preconditioning, with LongBench scores close to the full-precision model and above prior c...
-
From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
KV-cache serving systems concentrate into five archetypes under a four-axis taxonomy, with ownership explaining residual distributed design variance and seven measurement gaps blocking next steps.
-
Practical Online KV Cache Compaction for LLM Agents: An Empirical Study
In LLM agent trajectories, delaying KV cache compaction by a few turns and using the agent's later queries as the compression guide preserves most accuracy while cutting KV cache by 80%.
-
What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents
KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.
-
CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation
A method that uses a handful of 'semantic retrieval heads' instead of all attention heads to decide which key-value cache entries can be dropped, plus layer-wise cache budgeting, reportedly beats prior KV compression ...
-
CaliDrop: KV Cache Compression with Calibration
CaliDrop adds a stale-query calibration term on top of token eviction, improving accuracy at high KV compression ratios with modest throughput overhead.
-
TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization
TailorKV combines 1-bit quantization in shallow attention layers with dynamic Top-K token retrieval in deeper layers to serve 128k-context Llama-3.1-8B on a single 24GB GPU with a small accuracy loss.
-
Inference-time sparse attention with asymmetric indexing
Saap approximates attention by clustering keys with k-means and learning a query classifier, reducing KV-cache lookups about 20x at 4-5% selectivity with small accuracy loss on several long-context benchmarks.
-
SHARP: Accelerating Language Model Inference by SHaring Adjacent layers with Recovery Parameters
Sharing one MLP layer's weights across several layers plus low-rank adapters recovers most of a pretrained LLM's quality with a fraction of the storage and faster phone inference.
-
QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache
QuantSpec accelerates long-context LLM decoding by giving the self-draft model a 4-bit hierarchical KV cache shared with an INT8-quality target cache, achieving up to ~2.5x speedup and mostly >90% acceptance.
-
PolarQuant: Leveraging Polar Transformation for Efficient Key Cache Quantization and Decoding Acceleration
Quantizing LLM key caches in polar coordinates after RoPE preserves accuracy at lower total bit width and turns query-key multiplication into a lookup-table operation.
-
Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
Trained linear predictors across transformer layers plus residual quantization compress LLM KV caches to about 2 bits per value with near-lossless accuracy on tested Llama and Qwen models.
-
TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centering
TaDA mean-centers key and value activations along the head dimension, quantizes the deviations with per-layer precision search, and matches 16-bit baseline accuracy while using about 27 to 35 percent of the cache memory.
-
Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression
ScaleKV cuts KV cache memory for Visual Autoregressive text-to-image generation to 10% by classifying layers as drafters or refiners per scale and pruning low-attention tokens while keeping benchmark scores nearly unchanged.
-
HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference
HACK performs attention matrix multiplications directly on 2-bit quantized key-value cache entries, eliminating per-iteration dequantization and cutting end-to-end latency in disaggregated LLM serving.
-
Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
A survey organizing serving-time KV cache optimization techniques into temporal, spatial, and structural system behaviors, analyzing cross-behavior co-design patterns and open challenges.
-
PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference
PagedEviction prunes the KV cache in whole blocks using a key-value norm ratio, speeding up long-context LLM inference in vLLM while staying close to full-cache accuracy.
-
Win Fast or Lose Slow: Balancing Speed and Accuracy in Latency-Sensitive Decisions of LLMs
Adaptively quantizing parts of an LLM's layers to FP4 can improve win rates and trading yields in latency-sensitive agent tasks, but the reported gains come from choosing the best compression level after seeing test results.
Discussion (0). Continue with ORCID to comment.