{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2024:DNNVKB36I2CKOB27E6HPNONOGE","short_pith_number":"pith:DNNVKB36","schema_version":"1.0","canonical_sha256":"1b5b55077e4684a7075f278ef6b9ae3100cf8bee17e1e4a02e011d0d06fbbc2f","source":{"kind":"arxiv","id":"2410.10819","version":1},"attestation_state":"computed","paper":{"title":"DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"Only retrieval heads need full key-value caches for long-context processing in large language models, while streaming heads can use short fixed caches.","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Guangxuan Xiao, Haotian Tang, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Song Han, Yao Fu","submitted_at":"2024-10-14T17:59:58Z","abstract_excerpt":"Deploying long-context large language models (LLMs) is essential but poses significant computational and memory challenges. Caching all Key and Value (KV) states across all attention heads consumes substantial memory. Existing KV cache pruning methods either damage the long-context capabilities of LLMs or offer only limited efficiency improvements. In this paper, we identify that only a fraction of attention heads, a.k.a, Retrieval Heads, are critical for processing long contexts and require full attention across all tokens. In contrast, all other heads, which primarily focus on recent tokens "},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":true},"canonical_record":{"source":{"id":"2410.10819","kind":"arxiv","version":1},"metadata":{"license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","primary_cat":"cs.CL","submitted_at":"2024-10-14T17:59:58Z","cross_cats_sorted":[],"title_canon_sha256":"a758016bfb920268d0e1a76110b73f14576ce21fe3ded35158872e4c6393be51","abstract_canon_sha256":"1ee98cce630568e14c29c58d2e47c5ae95e0fc6db026ed50f91ebd97c87704c2"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-07-05T09:20:21.219508Z","signature_b64":"yYrgQYxGWQ61NM3TTkcEJgfyf+Ee7bySmWM5uK5JXvNP/cMsxOS74J/tT6SyjjwY7sgXrrCkXhXzOvmwdeXSDw==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"1b5b55077e4684a7075f278ef6b9ae3100cf8bee17e1e4a02e011d0d06fbbc2f","last_reissued_at":"2026-07-05T09:20:21.218995Z","signature_status":"signed_v1","first_computed_at":"2026-07-05T09:20:21.218995Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"Only retrieval heads need full key-value caches for long-context processing in large language models, while streaming heads can use short fixed caches.","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Guangxuan Xiao, Haotian Tang, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Song Han, Yao Fu","submitted_at":"2024-10-14T17:59:58Z","abstract_excerpt":"Deploying long-context large language models (LLMs) is essential but poses significant computational and memory challenges. Caching all Key and Value (KV) states across all attention heads consumes substantial memory. Existing KV cache pruning methods either damage the long-context capabilities of LLMs or offer only limited efficiency improvements. In this paper, we identify that only a fraction of attention heads, a.k.a, Retrieval Heads, are critical for processing long contexts and require full attention across all tokens. In contrast, all other heads, which primarily focus on recent tokens "},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"DuoAttention significantly reduces long-context inference memory by up to 2.55x for MHA and 1.67x for GQA models while speeding up decoding by up to 2.18x and 1.50x and accelerating pre-filling by up to 1.73x and 1.63x for MHA and GQA models, respectively, with minimal accuracy loss compared to full attention.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The lightweight optimization-based algorithm using synthetic data accurately identifies the subset of retrieval heads whose full KV cache is necessary and sufficient to preserve long-context capabilities, while streaming heads can be safely approximated with constant-length cache.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"DuoAttention identifies retrieval heads requiring full KV cache and streaming heads using constant-length cache to reduce memory and latency in long-context LLM inference.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Only retrieval heads need full key-value caches for long-context processing in large language models, while streaming heads can use short fixed caches.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"6f6b2d023f34f7cb8054020f7e05534253a595ffe4d14415fe00766a7d055e4e"},"source":{"id":"2410.10819","kind":"arxiv","version":1},"verdict":{"id":"f30ba178-dadd-4a6e-a9d3-e380bfe45691","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-18T11:44:47.663514Z","strongest_claim":"DuoAttention significantly reduces long-context inference memory by up to 2.55x for MHA and 1.67x for GQA models while speeding up decoding by up to 2.18x and 1.50x and accelerating pre-filling by up to 1.73x and 1.63x for MHA and GQA models, respectively, with minimal accuracy loss compared to full attention.","one_line_summary":"DuoAttention identifies retrieval heads requiring full KV cache and streaming heads using constant-length cache to reduce memory and latency in long-context LLM inference.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The lightweight optimization-based algorithm using synthetic data accurately identifies the subset of retrieval heads whose full KV cache is necessary and sufficient to preserve long-context capabilities, while streaming heads can be safely approximated with constant-length cache.","pith_extraction_headline":"Only retrieval heads need full key-value caches for long-context processing in large language models, while streaming heads can use short fixed caches."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2410.10819/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":57,"sample":[{"doi":"","year":2024,"title":"Cold compress: A toolkit for benchmarking kv cache compression approaches, 8 2024","work_id":"a29c4a2d-66c2-4f5d-ba36-ce70998c896d","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2023,"title":"SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills","work_id":"3dbdd757-ca01-436f-acfd-12ffcd6f64c6","ref_index":2,"cited_arxiv_id":"2308.16369","is_internal_anchor":true},{"doi":"","year":2023,"title":"Gqa: Training generalized multi-query transformer models from multi-head checkpoints","work_id":"fa96f94f-da03-4951-8bb7-8f1f81f2f2ff","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2021,"title":"Program Synthesis with Large Language Models","work_id":"fd241a05-03b9-4de2-9588-9d77ce176125","ref_index":4,"cited_arxiv_id":"2108.07732","is_internal_anchor":true},{"doi":"","year":2023,"title":"LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding","work_id":"ba7831c4-9427-4e0e-a5c1-4e98511f4b53","ref_index":5,"cited_arxiv_id":"2308.14508","is_internal_anchor":true}],"resolved_work":57,"snapshot_sha256":"1d5652b8c53a940998b269b073c6d09d040d12e2fabeb54c95824394bc53b392","internal_anchors":13},"formal_canon":{"evidence_count":3,"snapshot_sha256":"8c4085eccfce9f2f700ba166ececbfc9ff02c36d438f190e58b751786965c504"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2410.10819","created_at":"2026-07-05T09:20:21.219056+00:00"},{"alias_kind":"arxiv_version","alias_value":"2410.10819v1","created_at":"2026-07-05T09:20:21.219056+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2410.10819","created_at":"2026-07-05T09:20:21.219056+00:00"},{"alias_kind":"pith_short_12","alias_value":"DNNVKB36I2CK","created_at":"2026-07-05T09:20:21.219056+00:00"},{"alias_kind":"pith_short_16","alias_value":"DNNVKB36I2CKOB27","created_at":"2026-07-05T09:20:21.219056+00:00"},{"alias_kind":"pith_short_8","alias_value":"DNNVKB36","created_at":"2026-07-05T09:20:21.219056+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":36,"internal_anchor_count":36,"sample":[{"citing_arxiv_id":"2606.23581","citing_title":"Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Free Reuse","ref_index":29,"is_internal_anchor":true},{"citing_arxiv_id":"2606.07703","citing_title":"How Much Dense Attention is Necessary? Oracle-Guided Sparse Prefill for Full/GQA Layers in Hybrid Long-Context Models","ref_index":16,"is_internal_anchor":true},{"citing_arxiv_id":"2607.00760","citing_title":"MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression","ref_index":55,"is_internal_anchor":true},{"citing_arxiv_id":"2606.03712","citing_title":"When Graph Tokens Sink: A Mechanistic Analysis of Graph Language Models","ref_index":36,"is_internal_anchor":true},{"citing_arxiv_id":"2606.31519","citing_title":"RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference","ref_index":34,"is_internal_anchor":true},{"citing_arxiv_id":"2606.28831","citing_title":"HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compression","ref_index":26,"is_internal_anchor":true},{"citing_arxiv_id":"2606.29844","citing_title":"MATCH: Modulating Attention via In-Context Retrieval for Long-Context Transformers","ref_index":120,"is_internal_anchor":true},{"citing_arxiv_id":"2605.29535","citing_title":"AsymVLM: Asymmetric Token Pruning for Efficient Vision-Language Model Inference","ref_index":10,"is_internal_anchor":true},{"citing_arxiv_id":"2606.09508","citing_title":"From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs","ref_index":44,"is_internal_anchor":true},{"citing_arxiv_id":"2602.08686","citing_title":"CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilation","ref_index":18,"is_internal_anchor":true},{"citing_arxiv_id":"2602.18196","citing_title":"RAT+: Train Dense, Infer Sparse -- Recurrence Augmented Attention for Dilated Inference","ref_index":32,"is_internal_anchor":true},{"citing_arxiv_id":"2605.20600","citing_title":"Head-Aware Key-Value Compression for Efficient Autoregressive Image Generation","ref_index":35,"is_internal_anchor":true},{"citing_arxiv_id":"2605.17633","citing_title":"SparseSAM: Structured Sparsification of Activations in Segment Anything Models","ref_index":29,"is_internal_anchor":true},{"citing_arxiv_id":"2605.18753","citing_title":"DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention","ref_index":46,"is_internal_anchor":true},{"citing_arxiv_id":"2605.17613","citing_title":"VeriCache: Turning Lossy KV Cache into Lossless LLM Inference","ref_index":69,"is_internal_anchor":true},{"citing_arxiv_id":"2507.21526","citing_title":"Accelerating Prefilling via Decoding-time Contribution Sparsity","ref_index":13,"is_internal_anchor":true},{"citing_arxiv_id":"2508.16703","citing_title":"ShadowNPU: System and Algorithm Co-design for NPU-Centric On-Device LLM Inference","ref_index":63,"is_internal_anchor":true},{"citing_arxiv_id":"2510.00231","citing_title":"The Pitfalls of KV Cache Compression","ref_index":14,"is_internal_anchor":true},{"citing_arxiv_id":"2510.18245","citing_title":"Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs","ref_index":46,"is_internal_anchor":true},{"citing_arxiv_id":"2407.11550","citing_title":"Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference","ref_index":41,"is_internal_anchor":true},{"citing_arxiv_id":"2502.11089","citing_title":"Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention","ref_index":49,"is_internal_anchor":true},{"citing_arxiv_id":"2512.12087","citing_title":"BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding","ref_index":22,"is_internal_anchor":true},{"citing_arxiv_id":"2602.01203","citing_title":"Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse","ref_index":33,"is_internal_anchor":true},{"citing_arxiv_id":"2602.18196","citing_title":"RAT+: Train Dense, Infer Sparse -- Recurrence Augmented Attention for Dilated Inference","ref_index":32,"is_internal_anchor":true},{"citing_arxiv_id":"2605.13734","citing_title":"KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving","ref_index":48,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":3,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/DNNVKB36I2CKOB27E6HPNONOGE","json":"https://pith.science/pith/DNNVKB36I2CKOB27E6HPNONOGE.json","graph_json":"https://pith.science/api/pith-number/DNNVKB36I2CKOB27E6HPNONOGE/graph.json","events_json":"https://pith.science/api/pith-number/DNNVKB36I2CKOB27E6HPNONOGE/events.json","paper":"https://pith.science/paper/DNNVKB36"},"agent_actions":{"view_html":"https://pith.science/pith/DNNVKB36I2CKOB27E6HPNONOGE","download_json":"https://pith.science/pith/DNNVKB36I2CKOB27E6HPNONOGE.json","view_paper":"https://pith.science/paper/DNNVKB36","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2410.10819&json=true","fetch_graph":"https://pith.science/api/pith-number/DNNVKB36I2CKOB27E6HPNONOGE/graph.json","fetch_events":"https://pith.science/api/pith-number/DNNVKB36I2CKOB27E6HPNONOGE/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/DNNVKB36I2CKOB27E6HPNONOGE/action/timestamp_anchor","attest_storage":"https://pith.science/pith/DNNVKB36I2CKOB27E6HPNONOGE/action/storage_attestation","attest_author":"https://pith.science/pith/DNNVKB36I2CKOB27E6HPNONOGE/action/author_attestation","sign_citation":"https://pith.science/pith/DNNVKB36I2CKOB27E6HPNONOGE/action/citation_signature","submit_replication":"https://pith.science/pith/DNNVKB36I2CKOB27E6HPNONOGE/action/replication_record"}},"created_at":"2026-07-05T09:20:21.219056+00:00","updated_at":"2026-07-05T09:20:21.219056+00:00"}