REVIEW 9 cited by
MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Auto-regressive inference of transformers benefit greatly from Key-Value (KV) caching, but can lead to major memory bottlenecks as model size, batch size, and sequence length grow at scale. We introduce Multi-Layer Key-Value (MLKV) sharing, a novel approach extending KV sharing across transformer layers to reduce memory usage beyond what was possible with Multi-Query Attention (MQA) and Grouped-Query Attention (GQA). Evaluations on various NLP benchmarks and inference metrics using uptrained Pythia-160M variants demonstrate that MLKV significantly reduces memory usage with minimal performance loss, reducing KV cache size down to a factor of 6x compared to MQA. These results highlight MLKV's potential for efficient deployment of transformer models at scale. We provide code at https://github.com/zaydzuhri/pythia-mlkv
Forward citations
Cited by 9 Pith papers
-
Accurate KV Cache Quantization with Outlier Tokens Tracing
OTT improves 2-bit KV cache quantization by identifying low-magnitude key outlier tokens and keeping them full-precision, outperforming KIVI across several LLMs and benchmarks.
-
TransMLA: Multi-Head Latent Attention Is All You Need
A framework that converts pretrained GQA models into MLA models using RoRoPE, FreqFold, and balanced KV low-rank compression, regaining baseline performance with only a few billion fine-tuning tokens.
-
CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuning
Attention pairs (Q-K and V-O) are SVD-decomposed so pruning or fine-tuning touches only a small singular-factor matrix, yielding better pruning tolerance and small PEFT gains.
-
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.
-
Chain-of-Model Learning for Language Model
A nested Transformer with causally ordered hidden chains offers multiple sub-model sizes, chain-based expansion, and KV-cache sharing for faster prefilling.
-
Multi-matrix Factorization Attention
MFA and MFA-KR factorize the attention QK circuit with shared low-rank key and value projections, matching or beating MHA accuracy at a small fraction of the KV cache.
-
A Survey on Large Language Model Acceleration based on KV Cache Management
A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.
-
More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression
Storing 4x as many KV cache tokens at 4-bit precision outperforms storing 1x tokens at 16-bit precision on long-context benchmarks at equal data-memory budgets.
-
Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures
Memory-augmented Transformer research is organized into a three-axis taxonomy bridging neuroscience memory concepts to network designs, but no new result is produced.
Discussion (0). Continue with ORCID to comment.