Pith Number

pith:DNNVKB36

pith:2024:DNNVKB36I2CKOB27E6HPNONOGE

not attested not anchored not stored refs resolved

DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads

Guangxuan Xiao, Haotian Tang, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Song Han, Yao Fu

Only retrieval heads need full key-value caches for long-context processing in large language models, while streaming heads can use short fixed caches.

arxiv:2410.10819 v1 · 2024-10-14 · cs.CL

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{DNNVKB36I2CKOB27E6HPNONOGE}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

DuoAttention significantly reduces long-context inference memory by up to 2.55x for MHA and 1.67x for GQA models while speeding up decoding by up to 2.18x and 1.50x and accelerating pre-filling by up to 1.73x and 1.63x for MHA and GQA models, respectively, with minimal accuracy loss compared to full attention.

C2weakest assumption

The lightweight optimization-based algorithm using synthetic data accurately identifies the subset of retrieval heads whose full KV cache is necessary and sufficient to preserve long-context capabilities, while streaming heads can be safely approximated with constant-length cache.

C3one line summary

DuoAttention identifies retrieval heads requiring full KV cache and streaming heads using constant-length cache to reduce memory and latency in long-context LLM inference.

References

57 extracted · 57 resolved · 13 Pith anchors

[1] Cold compress: A toolkit for benchmarking kv cache compression approaches, 8 2024 2024

[2] SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills 2023 · arXiv:2308.16369

[3] Gqa: Training generalized multi-query transformer models from multi-head checkpoints 2023

[4] Program Synthesis with Large Language Models 2021 · arXiv:2108.07732

[5] LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding 2023 · arXiv:2308.14508

Formal links

3 machine-checked theorem links

Cited by

36 papers in Pith

Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Free Reuse

How Much Dense Attention is Necessary? Oracle-Guided Sparse Prefill for Full/GQA Layers in Hybrid Long-Context Models

MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression

When Graph Tokens Sink: A Mechanistic Analysis of Graph Language Models

RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference

Receipt and verification

First computed	2026-07-05T09:20:21.218995Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

1b5b55077e4684a7075f278ef6b9ae3100cf8bee17e1e4a02e011d0d06fbbc2f

Aliases

arxiv: 2410.10819 · arxiv_version: 2410.10819v1 · doi: 10.48550/arxiv.2410.10819 · pith_short_12: DNNVKB36I2CK · pith_short_16: DNNVKB36I2CKOB27 · pith_short_8: DNNVKB36

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/DNNVKB36I2CKOB27E6HPNONOGE \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 1b5b55077e4684a7075f278ef6b9ae3100cf8bee17e1e4a02e011d0d06fbbc2f

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "1ee98cce630568e14c29c58d2e47c5ae95e0fc6db026ed50f91ebd97c87704c2",
    "cross_cats_sorted": [],
    "license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
    "primary_cat": "cs.CL",
    "submitted_at": "2024-10-14T17:59:58Z",
    "title_canon_sha256": "a758016bfb920268d0e1a76110b73f14576ce21fe3ded35158872e4c6393be51"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2410.10819",
    "kind": "arxiv",
    "version": 1
  }
}