Pith. sign in

REVIEW 6 cited by

MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.09297 v3 pith:QQQUQSKZ submitted 2024-06-13 cs.LG

Zayd Muhammad Kawakibi Zuhri , Muhammad Farid Adilazuarda , Ayu Purwarianti , Alham Fikri Aji This is my paper · ORCID
classification cs.LG
keywords memorymlkvkey-valuesizetransformerattentionefficientinference
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Auto-regressive inference of transformers benefit greatly from Key-Value (KV) caching, but can lead to major memory bottlenecks as model size, batch size, and sequence length grow at scale. We introduce Multi-Layer Key-Value (MLKV) sharing, a novel approach extending KV sharing across transformer layers to reduce memory usage beyond what was possible with Multi-Query Attention (MQA) and Grouped-Query Attention (GQA). Evaluations on various NLP benchmarks and inference metrics using uptrained Pythia-160M variants demonstrate that MLKV significantly reduces memory usage with minimal performance loss, reducing KV cache size down to a factor of 6x compared to MQA. These results highlight MLKV's potential for efficient deployment of transformer models at scale. We provide code at https://github.com/zaydzuhri/pythia-mlkv

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TransMLA: Multi-Head Latent Attention Is All You Need

    cs.LG 2025-02 conditional novelty 6.0 of 10

    A framework that converts pretrained GQA models into MLA models using RoRoPE, FreqFold, and balanced KV low-rank compression, regaining baseline performance with only a few billion fine-tuning tokens.

  2. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.

  3. Multi-matrix Factorization Attention

    cs.LG 2024-12 conditional novelty 5.0 of 10

    MFA and MFA-KR factorize the attention QK circuit with shared low-rank key and value projections, matching or beating MHA accuracy at a small fraction of the KV cache.

  4. A Survey on Large Language Model Acceleration based on KV Cache Management

    cs.AI 2024-12 conditional novelty 4.0 of 10

    A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.

  5. More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression

    cs.CL 2024-12 conditional novelty 4.0 of 10

    Storing 4x as many KV cache tokens at 4-bit precision outperforms storing 1x tokens at 16-bit precision on long-context benchmarks at equal data-memory budgets.

  6. Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures

    cs.LG 2025-08 unverdicted novelty 3.0 of 10

    Memory-augmented Transformer research is organized into a three-axis taxonomy bridging neuroscience memory concepts to network designs, but no new result is produced.

Pith tools