{"work":{"id":"a8d25452-c237-48c9-88a4-682717c3979a","openalex_id":"https://openalex.org/W4387294617","doi":"10.48550/arxiv.2309.17453","arxiv_id":"2309.17453","raw_key":null,"title":"Efficient Streaming Language Models with Attention Sinks","authors":null,"authors_text":"Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis","year":2023,"venue":"cs.CL","abstract":"Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during the decoding stage, caching previous tokens' Key and Value states (KV) consumes extensive memory. Secondly, popular LLMs cannot generalize to longer texts than the training sequence length. Window attention, where only the most recent KVs are cached, is a natural approach -- but we show that it fails when the text length surpasses the cache size. We observe an interesting phenomenon, namely attention sink, that keeping the KV of initial tokens will largely recover the performance of window attention. In this paper, we first demonstrate that the emergence of attention sink is due to the strong attention scores towards initial tokens as a \"sink\" even if they are not semantically important. Based on the above analysis, we introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence lengths without any fine-tuning. We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more. In addition, we discover that adding a placeholder token as a dedicated attention sink during pre-training can further improve streaming deployment. In streaming settings, StreamingLLM outperforms the sliding window recomputation baseline by up to 22.2x speedup. Code and datasets are provided at https://github.com/mit-han-lab/streaming-llm.","external_url":"https://arxiv.org/abs/2309.17453","cited_by_count":32,"metadata_source":"pith","metadata_fetched_at":"2026-08-05T02:28:24.338817+00:00","pith_arxiv_id":"2309.17453","created_at":"2026-05-09T06:15:37.718831+00:00","updated_at":"2026-08-05T02:28:24.338817+00:00","title_quality_ok":true,"display_title":"Efficient Streaming Language Models with Attention Sinks","render_title":"Efficient Streaming Language Models with Attention Sinks"},"hub":{"state":{"work_id":"a8d25452-c237-48c9-88a4-682717c3979a","tier":"super_hub","tier_reason":"100+ Pith inbound or 10,000+ external citations","pith_inbound_count":210,"external_cited_by_count":32,"distinct_field_count":18,"first_pith_cited_at":"2023-06-24T20:11:14+00:00","last_pith_cited_at":"2026-07-09T01:15:03+00:00","author_build_status":"needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-08-23T00:39:31.838141+00:00","tier_text":"super_hub"},"tier":"super_hub","role_counts":[{"context_role":"background","n":36},{"context_role":"method","n":7},{"context_role":"baseline","n":2},{"context_role":"other","n":2}],"polarity_counts":[{"context_polarity":"background","n":35},{"context_polarity":"use_method","n":6},{"context_polarity":"unclear","n":3},{"context_polarity":"baseline","n":2},{"context_polarity":"support","n":1}],"runs":{"ask_index":{"job_type":"ask_index","status":"succeeded","result":{"title":"Efficient Streaming Language Models with Attention Sinks","claims":[{"claim_text":"Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during the decoding stage, caching previous tokens' Key and Value states (KV) consumes extensive memory. Secondly, popular LLMs cannot generalize to longer texts than the training sequence length. Window attention, where only the most recent KVs are cached, is a natural approach -- but we show that it fails when the text length surpasses the cache size. We observe an interesting phenomenon, namely attent","claim_type":"abstract","evidence_strength":"source_metadata"},{"claim_text":"through later rounds. The key result is that TokenDance does not introduce any additional accuracy degradation beyond what the underlying PIC method already produces. 11 Conference'17, July 2017, Washington, DC, USA Zhuohang Bian, Feiyang Wu, Chengrui Zhang, Hangcheng Dong, Yun Liang, and Youwei Zhuo 7 Related Work LLM Serving Systems.vLLM [ 12], SGLang [40], Orca [38], and Sarathi-Serve [ 1] improve batching, scheduling, and memory management for LLM inference. These systems op- timize executio","claim_type":"background","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"timodal benchmarks, effectively balancing global reasoning and precise local evidence. Keywords:Attention Sinks·Large Vision-Language Models 1 Introduction Transformers are widely known to exhibit theattention sinkphenomenon, where a small subset of tokens attracts disproportionately large attention weights de- spite possessing limited semantic content [6,10,42,45]. In Large Language Models (LLMs), these sinks often manifest in punctuation or special symbols (e.g.,[BOS] ⋆ Corresponding authors. ","claim_type":"background","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"Our training corpus comprises a heterogeneous mixture of datasets with substantial variation in sequence lengths, making naive padding prohibitively inefficient. We therefore pack multiple examples into fixed-length sequences, using the document- and conversation-level attention masking described in Sec. III, implemented efficiently via FlexAttention [127]. For conversation masking specifically, we apply it stochasti- cally rather than universally. Full conversation masking showed no consistent ","claim_type":"method","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"Trillion-parameter sequential transducers for generative recommendations.arXiv preprint arXiv:2402.17152(2024). [45] Zhaoqi Zhang, Haolei Pei, Jun Guo, Tianyu Wang, Yufei Feng, Hui Sun, Shaowei Liu, and Aixin Sun. 2026. Onetrans: Unified feature interaction and sequence modeling with one transformer in industrial recommender. InProceedings of the ACM Web Conference 2026. 8162-8170. [46] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie","claim_type":"background","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"Both methods rely on offline optimization before deployment, with HqeKV searching tier ratios and ARKV searching scoring thresholds. 2 Budget-Driven Methods.These methods take an active-cache budget as input and decide which units remain available during decoding. KV cache eviction methods score each cache unit and retain the top-ranked subset under a fixed token count. StreamingLLM [23] keeps sink tokens and a recent window. H2O [ 10], SnapKV [11], Scissorhands [24], and NaCl [ 25] use attentio","claim_type":"background","confidence":0.85,"evidence_strength":"citation_context"},{"claim_text":"by integrating the use of both approaches. Attention Manipulation in Vision-language Models.Our motivation for introducing saliency-map alignment rewards stems from prior findings that steering attention can en- hance LLM and VLM performance. LLMs often display biased attention over long contexts, for example, under- utilizing middle segments [42] or over-focusing on initial tokens [77]. To address this, works have explored fine- tuning [4], input-adaptive calibration [87], and position- specifi","claim_type":"background","confidence":0.85,"evidence_strength":"citation_context"}],"why_cited":"Pith tracks Efficient Streaming Language Models with Attention Sinks because it crossed a citation-hub threshold. Current citing contexts most often use it as background evidence (13 contexts).","role_counts":[{"n":13,"context_role":"background"},{"n":2,"context_role":"method"},{"n":1,"context_role":"other"}]},"error":null,"updated_at":"2026-05-16T18:19:04.432943+00:00"},"author_expand":{"job_type":"author_expand","status":"succeeded","result":{"authors_linked":[{"id":"ae2c19d2-7d09-496f-b14e-00a1f7887432","orcid":null,"display_name":"Guangxuan Xiao"},{"id":"a09283b9-dafb-469c-9912-72deea250331","orcid":null,"display_name":"Yuandong Tian"},{"id":"9d3070a3-7018-4d64-afb9-3d632610da83","orcid":null,"display_name":"Beidi Chen"},{"id":"66d5efe3-8fc3-4c89-8802-b6fd59ce3ef3","orcid":null,"display_name":"Song Han"},{"id":"2830e405-bc8f-410a-a2aa-eff20c45860b","orcid":null,"display_name":"Mike Lewis"}]},"error":null,"updated_at":"2026-05-16T18:19:04.428033+00:00"},"context_extract":{"job_type":"context_extract","status":"succeeded","result":{"enqueued_papers":25},"error":null,"updated_at":"2026-05-14T07:47:44.208800+00:00"},"graph_features":{"job_type":"graph_features","status":"succeeded","result":{"co_cited":[{"title":"Qwen3 Technical Report","work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e","shared_citers":17},{"title":"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning","work_id":"fff3953b-5efb-4753-bee4-002f59995810","shared_citers":15},{"title":"The Llama 3 Herd of Models","work_id":"1549a635-88af-4ac1-acfe-51ae7bb53345","shared_citers":15},{"title":"GPT-4 Technical Report","work_id":"b928e041-6991-4c08-8c81-0359e4097c7b","shared_citers":14},{"title":"RULER: What's the Real Context Size of Your Long-Context Language Models?","work_id":"c0bc4689-3ce8-4e3d-9442-bd74869445bb","shared_citers":14},{"title":"Longformer: The Long-Document Transformer","work_id":"abea7a44-6668-4de7-aab6-f53a6e5aa088","shared_citers":13},{"title":"arXiv preprint arXiv:2406.10774 , year=","work_id":"2cad64c9-e2d5-42b6-8db9-03fafde4bcb0","shared_citers":12},{"title":"PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling","work_id":"6317700d-f903-4ce1-8f53-b43cb146d48b","shared_citers":12},{"title":"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning","work_id":"e6b75ad5-2877-4168-97c8-710407094d20","shared_citers":10},{"title":"Generating Long Sequences with Sparse Transformers","work_id":"c5b81688-45ee-4a9a-b095-e6290f45cb6c","shared_citers":10},{"title":"GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints","work_id":"b73ad5b2-e553-4c71-b0c9-67e67ba7b158","shared_citers":10},{"title":"Training Verifiers to Solve Math Word Problems","work_id":"acab1aa8-b4d6-40e0-a3ee-25341701dca2","shared_citers":10},{"title":"When attention sink emerges in language models: An empirical view","work_id":"ab480e70-3517-4875-81ef-6cc6fa6db46d","shared_citers":10},{"title":"Ada-KV: Optimizing KV cache eviction by adaptive budget allocation for efficient LLM inference","work_id":"b41eb6db-193f-48ba-8278-6b1698524755","shared_citers":9},{"title":"Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free","work_id":"35cc586b-44f1-4948-a84b-866e8335e649","shared_citers":9},{"title":"KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache","work_id":"735737c3-24e5-41c3-ab4f-04edcb36731c","shared_citers":9},{"title":"LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding","work_id":"ba7831c4-9427-4e0e-a5c1-4e98511f4b53","shared_citers":9},{"title":"Z., and Liu, Z","work_id":"a8da768a-7f38-4db5-912b-4d5b7581c8dd","shared_citers":9},{"title":"DeepSeek-V3 Technical Report","work_id":"57d2791d-2219-4c31-a077-afc04b12a75c","shared_citers":8},{"title":"Duoattention: Efficient long-context LLM inference with retrieval and streaming heads","work_id":"dcb5f5ed-ec94-4386-bfa7-02fcaeb844f1","shared_citers":8},{"title":"Fast Transformer Decoding: One Write-Head is All You Need","work_id":"160ea164-b1d4-4adb-8ccb-a4655d8a0bb4","shared_citers":8},{"title":"Llama 2: Open Foundation and Fine-Tuned Chat Models","work_id":"68a5177f-d644-44c1-bd4f-4e5278c22f5d","shared_citers":8},{"title":"Mistral 7B","work_id":"eb5e1305-ad11-4875-ad8d-ad8b8f697599","shared_citers":8},{"title":"Model tells you what to discard: Adaptive kv cache compression for llms","work_id":"91379982-466d-4895-96f5-079b66259a73","shared_citers":8}],"time_series":[{"n":4,"year":2024},{"n":4,"year":2025},{"n":70,"year":2026}],"dependency_candidates":[]},"error":null,"updated_at":"2026-05-14T07:47:46.707997+00:00"},"identity_refresh":{"job_type":"identity_refresh","status":"succeeded","result":{"items":[{"title":"Qwen3 Technical Report","outcome":"unchanged","work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e","resolver":"local_arxiv","confidence":0.98,"old_work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e"}],"counts":{"fixed":0,"merged":0,"unchanged":1,"quarantined":0,"needs_external_resolution":0},"errors":[],"attempted":1},"error":null,"updated_at":"2026-05-14T07:47:48.993422+00:00"},"role_polarity":{"job_type":"role_polarity","status":"succeeded","result":{"title":"Efficient Streaming Language Models with Attention Sinks","claims":[{"claim_text":"Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during the decoding stage, caching previous tokens' Key and Value states (KV) consumes extensive memory. Secondly, popular LLMs cannot generalize to longer texts than the training sequence length. Window attention, where only the most recent KVs are cached, is a natural approach -- but we show that it fails when the text length surpasses the cache size. We observe an interesting phenomenon, namely attent","claim_type":"abstract","evidence_strength":"source_metadata"},{"claim_text":"through later rounds. The key result is that TokenDance does not introduce any additional accuracy degradation beyond what the underlying PIC method already produces. 11 Conference'17, July 2017, Washington, DC, USA Zhuohang Bian, Feiyang Wu, Chengrui Zhang, Hangcheng Dong, Yun Liang, and Youwei Zhuo 7 Related Work LLM Serving Systems.vLLM [ 12], SGLang [40], Orca [38], and Sarathi-Serve [ 1] improve batching, scheduling, and memory management for LLM inference. These systems op- timize executio","claim_type":"background","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"timodal benchmarks, effectively balancing global reasoning and precise local evidence. Keywords:Attention Sinks·Large Vision-Language Models 1 Introduction Transformers are widely known to exhibit theattention sinkphenomenon, where a small subset of tokens attracts disproportionately large attention weights de- spite possessing limited semantic content [6,10,42,45]. In Large Language Models (LLMs), these sinks often manifest in punctuation or special symbols (e.g.,[BOS] ⋆ Corresponding authors. ","claim_type":"background","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"Our training corpus comprises a heterogeneous mixture of datasets with substantial variation in sequence lengths, making naive padding prohibitively inefficient. We therefore pack multiple examples into fixed-length sequences, using the document- and conversation-level attention masking described in Sec. III, implemented efficiently via FlexAttention [127]. For conversation masking specifically, we apply it stochasti- cally rather than universally. Full conversation masking showed no consistent ","claim_type":"method","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"Trillion-parameter sequential transducers for generative recommendations.arXiv preprint arXiv:2402.17152(2024). [45] Zhaoqi Zhang, Haolei Pei, Jun Guo, Tianyu Wang, Yufei Feng, Hui Sun, Shaowei Liu, and Aixin Sun. 2026. Onetrans: Unified feature interaction and sequence modeling with one transformer in industrial recommender. InProceedings of the ACM Web Conference 2026. 8162-8170. [46] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie","claim_type":"background","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"Both methods rely on offline optimization before deployment, with HqeKV searching tier ratios and ARKV searching scoring thresholds. 2 Budget-Driven Methods.These methods take an active-cache budget as input and decide which units remain available during decoding. KV cache eviction methods score each cache unit and retain the top-ranked subset under a fixed token count. StreamingLLM [23] keeps sink tokens and a recent window. H2O [ 10], SnapKV [11], Scissorhands [24], and NaCl [ 25] use attentio","claim_type":"background","confidence":0.85,"evidence_strength":"citation_context"},{"claim_text":"by integrating the use of both approaches. Attention Manipulation in Vision-language Models.Our motivation for introducing saliency-map alignment rewards stems from prior findings that steering attention can en- hance LLM and VLM performance. LLMs often display biased attention over long contexts, for example, under- utilizing middle segments [42] or over-focusing on initial tokens [77]. To address this, works have explored fine- tuning [4], input-adaptive calibration [87], and position- specifi","claim_type":"background","confidence":0.85,"evidence_strength":"citation_context"}],"why_cited":"Pith tracks Efficient Streaming Language Models with Attention Sinks because it crossed a citation-hub threshold. Current citing contexts most often use it as background evidence (13 contexts).","role_counts":[{"n":13,"context_role":"background"},{"n":2,"context_role":"method"},{"n":1,"context_role":"other"}]},"error":null,"updated_at":"2026-05-16T18:19:04.436583+00:00"},"summary_claims":{"job_type":"summary_claims","status":"succeeded","result":{"title":"Efficient Streaming Language Models with Attention Sinks","claims":[{"claim_text":"Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during the decoding stage, caching previous tokens' Key and Value states (KV) consumes extensive memory. Secondly, popular LLMs cannot generalize to longer texts than the training sequence length. Window attention, where only the most recent KVs are cached, is a natural approach -- but we show that it fails when the text length surpasses the cache size. We observe an interesting phenomenon, namely attent","claim_type":"abstract","evidence_strength":"source_metadata"}],"why_cited":"Pith tracks Efficient Streaming Language Models with Attention Sinks because it crossed a citation-hub threshold.","role_counts":[]},"error":null,"updated_at":"2026-05-14T07:47:42.172528+00:00"}},"summary":{"title":"Efficient Streaming Language Models with Attention Sinks","claims":[{"claim_text":"Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during the decoding stage, caching previous tokens' Key and Value states (KV) consumes extensive memory. Secondly, popular LLMs cannot generalize to longer texts than the training sequence length. Window attention, where only the most recent KVs are cached, is a natural approach -- but we show that it fails when the text length surpasses the cache size. We observe an interesting phenomenon, namely attent","claim_type":"abstract","evidence_strength":"source_metadata"}],"why_cited":"Pith tracks Efficient Streaming Language Models with Attention Sinks because it crossed a citation-hub threshold.","role_counts":[]},"graph":{"co_cited":[{"title":"Qwen3 Technical Report","work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e","shared_citers":17},{"title":"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning","work_id":"fff3953b-5efb-4753-bee4-002f59995810","shared_citers":15},{"title":"The Llama 3 Herd of Models","work_id":"1549a635-88af-4ac1-acfe-51ae7bb53345","shared_citers":15},{"title":"GPT-4 Technical Report","work_id":"b928e041-6991-4c08-8c81-0359e4097c7b","shared_citers":14},{"title":"RULER: What's the Real Context Size of Your Long-Context Language Models?","work_id":"c0bc4689-3ce8-4e3d-9442-bd74869445bb","shared_citers":14},{"title":"Longformer: The Long-Document Transformer","work_id":"abea7a44-6668-4de7-aab6-f53a6e5aa088","shared_citers":13},{"title":"arXiv preprint arXiv:2406.10774 , year=","work_id":"2cad64c9-e2d5-42b6-8db9-03fafde4bcb0","shared_citers":12},{"title":"PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling","work_id":"6317700d-f903-4ce1-8f53-b43cb146d48b","shared_citers":12},{"title":"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning","work_id":"e6b75ad5-2877-4168-97c8-710407094d20","shared_citers":10},{"title":"Generating Long Sequences with Sparse Transformers","work_id":"c5b81688-45ee-4a9a-b095-e6290f45cb6c","shared_citers":10},{"title":"GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints","work_id":"b73ad5b2-e553-4c71-b0c9-67e67ba7b158","shared_citers":10},{"title":"Training Verifiers to Solve Math Word Problems","work_id":"acab1aa8-b4d6-40e0-a3ee-25341701dca2","shared_citers":10},{"title":"When attention sink emerges in language models: An empirical view","work_id":"ab480e70-3517-4875-81ef-6cc6fa6db46d","shared_citers":10},{"title":"Ada-KV: Optimizing KV cache eviction by adaptive budget allocation for efficient LLM inference","work_id":"b41eb6db-193f-48ba-8278-6b1698524755","shared_citers":9},{"title":"Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free","work_id":"35cc586b-44f1-4948-a84b-866e8335e649","shared_citers":9},{"title":"KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache","work_id":"735737c3-24e5-41c3-ab4f-04edcb36731c","shared_citers":9},{"title":"LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding","work_id":"ba7831c4-9427-4e0e-a5c1-4e98511f4b53","shared_citers":9},{"title":"Z., and Liu, Z","work_id":"a8da768a-7f38-4db5-912b-4d5b7581c8dd","shared_citers":9},{"title":"DeepSeek-V3 Technical Report","work_id":"57d2791d-2219-4c31-a077-afc04b12a75c","shared_citers":8},{"title":"Duoattention: Efficient long-context LLM inference with retrieval and streaming heads","work_id":"dcb5f5ed-ec94-4386-bfa7-02fcaeb844f1","shared_citers":8},{"title":"Fast Transformer Decoding: One Write-Head is All You Need","work_id":"160ea164-b1d4-4adb-8ccb-a4655d8a0bb4","shared_citers":8},{"title":"Llama 2: Open Foundation and Fine-Tuned Chat Models","work_id":"68a5177f-d644-44c1-bd4f-4e5278c22f5d","shared_citers":8},{"title":"Mistral 7B","work_id":"eb5e1305-ad11-4875-ad8d-ad8b8f697599","shared_citers":8},{"title":"Model tells you what to discard: Adaptive kv cache compression for llms","work_id":"91379982-466d-4895-96f5-079b66259a73","shared_citers":8}],"time_series":[{"n":4,"year":2024},{"n":4,"year":2025},{"n":70,"year":2026}],"dependency_candidates":[]},"authors":[{"id":"9d3070a3-7018-4d64-afb9-3d632610da83","orcid":null,"display_name":"Beidi Chen","source":"manual","import_confidence":0.72},{"id":"ae2c19d2-7d09-496f-b14e-00a1f7887432","orcid":null,"display_name":"Guangxuan Xiao","source":"manual","import_confidence":0.72},{"id":"2830e405-bc8f-410a-a2aa-eff20c45860b","orcid":null,"display_name":"Mike Lewis","source":"manual","import_confidence":0.72},{"id":"66d5efe3-8fc3-4c89-8802-b6fd59ce3ef3","orcid":null,"display_name":"Song Han","source":"manual","import_confidence":0.72},{"id":"a09283b9-dafb-469c-9912-72deea250331","orcid":null,"display_name":"Yuandong Tian","source":"manual","import_confidence":0.72}]}}