REVIEW 12 cited by
MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
A critical approach for efficiently deploying computationally demanding large language models (LLMs) is Key-Value (KV) caching. The KV cache stores key-value states of previously generated tokens, significantly reducing the need for repetitive computations and thereby lowering latency in autoregressive generation. However, the size of the KV cache grows linearly with sequence length, posing challenges for applications requiring long context input and extensive sequence generation. In this paper, we present a simple yet effective approach, called MiniCache, to compress the KV cache across layers from a novel depth perspective, significantly reducing the memory footprint for LLM inference. Our approach is based on the observation that KV cache states exhibit high similarity between the adjacent layers in the middle-to-deep portion of LLMs. To facilitate merging, we propose disentangling the states into the magnitude and direction components, interpolating the directions of the state vectors while preserving their lengths unchanged. Furthermore, we introduce a token retention strategy to keep highly distinct state pairs unmerged, thus preserving the information with minimal additional storage overhead. Our MiniCache is training-free and general, complementing existing KV cache compression strategies, such as quantization and sparsity. We conduct a comprehensive evaluation of MiniCache utilizing various models including LLaMA-2, LLaMA-3, Phi-3, Mistral, and Mixtral across multiple benchmarks, demonstrating its exceptional performance in achieving superior compression ratios and high throughput. On the ShareGPT dataset, LLaMA-2-7B with 4-bit MiniCache achieves a remarkable compression ratio of up to 5.02x, enhances inference throughput by approximately 5x, and reduces the memory footprint by 41% compared to the FP16 full cache baseline, all while maintaining near-lossless performance.
Forward citations
Cited by 12 Pith papers
-
Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers
In looped transformers, the recurrence-indexed KV cache is low-rank along the loop axis, and a latent codec fitted to teacher activations compresses it up to 21× with small quality loss.
-
What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents
KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.
-
DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression
DepthWeave-KV achieves 8.3x KV cache memory reduction with near-full-cache task quality by factorizing key-value states across transformer layers using shared bases and token-adaptive residuals.
-
TransMLA: Multi-Head Latent Attention Is All You Need
A framework that converts pretrained GQA models into MLA models using RoRoPE, FreqFold, and balanced KV low-rank compression, regaining baseline performance with only a few billion fine-tuning tokens.
-
Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
Trained linear predictors across transformer layers plus residual quantization compress LLM KV caches to about 2 bits per value with near-lossless accuracy on tested Llama and Qwen models.
-
Task-KV: Task-aware KV Cache Optimization via Semantic Differentiation of Attention Heads
Task-KV identifies 'heterogeneous' attention heads by distance from a per-task semantic center and allocates differentiated KV cache budgets, achieving modest average gains on long-context benchmarks at reduced memory.
-
FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference
Frequency-guided inter-layer KV sharing with logit-aware head routing nearly matches full-cache long-context accuracy at about 3.9× lower peak KV memory.
-
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.
-
SpindleKV: A Novel KV Cache Reduction Method Balancing Both Shallow and Deep Layers
SpindleKV compresses LLM KV cache by evicting low-attention tokens in deep layers and replacing near-duplicate key and value vectors in shallow layers with a shared codebook, while preserving benchmark accuracy at 15 ...
-
Semantic Scheduling for LLM Inference
A semantic scheduler for LLM inference uses urgency labels and estimated remaining compute to cut waiting times for urgent requests, tested on emergency medical data.
-
Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression
ScaleKV cuts KV cache memory for Visual Autoregressive text-to-image generation to 10% by classifying layers as drafters or refiners per scale and pruning low-attention tokens while keeping benchmark scores nearly unchanged.
-
Position: Episodic Memory is the Missing Piece for Long-Term LLM Agents
The authors propose episodic memory, with five defining properties, as the unifying framework needed for LLM agents to learn and remember over long time horizons.
Discussion (0). Continue with ORCID to comment.