{"work":{"id":"0eb5eca2-2c11-4a77-a25a-18c331a50ed2","openalex_id":null,"doi":null,"arxiv_id":"2309.06180","raw_key":null,"title":"Efficient Memory Management for Large Language Model Serving with PagedAttention","authors":null,"authors_text":"Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu","year":2023,"venue":"cs.LG","abstract":"High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems. On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within and across requests to further reduce memory usage. Our evaluations show that vLLM improves the throughput of popular LLMs by 2-4$\\times$ with the same level of latency compared to the state-of-the-art systems, such as FasterTransformer and Orca. The improvement is more pronounced with longer sequences, larger models, and more complex decoding algorithms. vLLM's source code is publicly available at https://github.com/vllm-project/vllm","external_url":"https://arxiv.org/abs/2309.06180","cited_by_count":null,"metadata_source":"pith","metadata_fetched_at":"2026-05-20T12:13:15.957584+00:00","pith_arxiv_id":"2309.06180","created_at":"2026-05-08T17:28:42.030347+00:00","updated_at":"2026-05-20T12:13:15.957584+00:00","title_quality_ok":true,"display_title":"Efficient Memory Management for Large Language Model Serving with PagedAttention","render_title":"Efficient Memory Management for Large Language Model Serving with PagedAttention"},"hub":{"state":{"work_id":"0eb5eca2-2c11-4a77-a25a-18c331a50ed2","tier":"hub","tier_reason":"10+ Pith inbound or 1,000+ external citations","pith_inbound_count":56,"external_cited_by_count":null,"distinct_field_count":14,"first_pith_cited_at":"2023-10-03T05:17:08+00:00","last_pith_cited_at":"2026-05-19T14:39:54+00:00","author_build_status":"not_needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"not_needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-05-20T13:11:51.241731+00:00","tier_text":"hub"},"tier":"hub","role_counts":[{"context_role":"background","n":9},{"context_role":"method","n":4},{"context_role":"dataset","n":2}],"polarity_counts":[{"context_polarity":"background","n":9},{"context_polarity":"use_method","n":4},{"context_polarity":"use_dataset","n":2}],"runs":{"context_extract":{"job_type":"context_extract","status":"succeeded","result":{"enqueued_papers":25},"error":null,"updated_at":"2026-05-14T18:20:04.343407+00:00"},"graph_features":{"job_type":"graph_features","status":"succeeded","result":{"co_cited":[{"title":"DAPO: An Open-Source LLM Reinforcement Learning System at Scale","work_id":"64019d00-0b11-4bbd-b173-b46c8fad0157","shared_citers":6},{"title":"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning","work_id":"e6b75ad5-2877-4168-97c8-710407094d20","shared_citers":6},{"title":"SGLang: Efficient Execution of Structured Language Model Programs","work_id":"bcc79aed-93ef-4b91-94e6-e62b5fa15ce7","shared_citers":6},{"title":"DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models","work_id":"c5006563-f3ec-438a-9e35-b7b484f34828","shared_citers":5},{"title":"DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model","work_id":"1e1df141-cac8-47fd-b068-c4c96e51e331","shared_citers":5},{"title":"Evaluating Large Language Models Trained on Code","work_id":"042493e9-b26f-4b4e-bbde-382072ca9b08","shared_citers":5},{"title":"Qwen3 Technical Report","work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e","shared_citers":5},{"title":"The Llama 3 Herd of Models","work_id":"1549a635-88af-4ac1-acfe-51ae7bb53345","shared_citers":5},{"title":"Training Compute-Optimal Large Language Models","work_id":"b2faf28d-86b7-429c-bc42-469458efc246","shared_citers":5},{"title":"arXiv preprint arXiv:2306.14048 , year=","work_id":"1ee58eaa-8aa0-4dec-8e67-6e079ad5fc13","shared_citers":4},{"title":"Fast Transformer Decoding: One Write-Head is All You Need","work_id":"160ea164-b1d4-4adb-8ccb-a4655d8a0bb4","shared_citers":4},{"title":"Mamba: Linear-Time Sequence Modeling with Selective State Spaces","work_id":"4ee75248-1199-492c-a52f-6661e0f4adff","shared_citers":4},{"title":"RULER: What's the Real Context Size of Your Long-Context Language Models?","work_id":"c0bc4689-3ce8-4e3d-9442-bd74869445bb","shared_citers":4},{"title":"Scaling Laws for Neural Language Models","work_id":"b7dd8749-9c45-4977-ab9b-64478dce1ae8","shared_citers":4},{"title":"DeepSeek-V3 Technical Report","work_id":"57d2791d-2219-4c31-a077-afc04b12a75c","shared_citers":3},{"title":"Efficient Streaming Language Models with Attention Sinks","work_id":"a8d25452-c237-48c9-88a4-682717c3979a","shared_citers":3},{"title":"FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness","work_id":"efa96825-0830-4cfc-a250-fdaf6af302ab","shared_citers":3},{"title":"GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers","work_id":"19ed8c44-202a-48f6-8169-637d5a5f2408","shared_citers":3},{"title":"GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints","work_id":"b73ad5b2-e553-4c71-b0c9-67e67ba7b158","shared_citers":3},{"title":"Group Sequence Policy Optimization","work_id":"3a98b53b-9f52-4d95-adf7-89353c0a9a65","shared_citers":3},{"title":"LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding","work_id":"ba7831c4-9427-4e0e-a5c1-4e98511f4b53","shared_citers":3},{"title":"LoRA: Low-Rank Adaptation of Large Language Models","work_id":"0426219a-789e-4964-adc8-a04538510818","shared_citers":3},{"title":"Measuring Mathematical Problem Solving With the MATH Dataset","work_id":"50652ac6-fb7c-4675-a2c2-159c241feb17","shared_citers":3},{"title":"Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism","work_id":"c888e6d1-0b1d-43d6-9ef5-f0912a0efa1b","shared_citers":3}],"time_series":[{"n":1,"year":2023},{"n":1,"year":2024},{"n":1,"year":2025},{"n":29,"year":2026}],"dependency_candidates":[]},"error":null,"updated_at":"2026-05-14T18:19:20.078510+00:00"},"identity_refresh":{"job_type":"identity_refresh","status":"succeeded","result":{"items":[{"title":"Qwen3 Technical Report","outcome":"unchanged","work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e","resolver":"local_arxiv","confidence":0.98,"old_work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e"}],"counts":{"fixed":0,"merged":0,"unchanged":1,"quarantined":0,"needs_external_resolution":0},"errors":[],"attempted":1},"error":null,"updated_at":"2026-05-14T18:19:42.278719+00:00"},"summary_claims":{"job_type":"summary_claims","status":"succeeded","result":{"title":"Efficient Memory Management for Large Language Model Serving with PagedAttention","claims":[{"claim_text":"High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems. On top of it, we build vLLM, an LLM serving system","claim_type":"abstract","evidence_strength":"source_metadata"}],"why_cited":"Pith tracks Efficient Memory Management for Large Language Model Serving with PagedAttention because it crossed a citation-hub threshold.","role_counts":[]},"error":null,"updated_at":"2026-05-14T18:19:29.194558+00:00"}},"summary":{"title":"Efficient Memory Management for Large Language Model Serving with PagedAttention","claims":[{"claim_text":"High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems. On top of it, we build vLLM, an LLM serving system","claim_type":"abstract","evidence_strength":"source_metadata"}],"why_cited":"Pith tracks Efficient Memory Management for Large Language Model Serving with PagedAttention because it crossed a citation-hub threshold.","role_counts":[]},"graph":{"co_cited":[{"title":"DAPO: An Open-Source LLM Reinforcement Learning System at Scale","work_id":"64019d00-0b11-4bbd-b173-b46c8fad0157","shared_citers":6},{"title":"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning","work_id":"e6b75ad5-2877-4168-97c8-710407094d20","shared_citers":6},{"title":"SGLang: Efficient Execution of Structured Language Model Programs","work_id":"bcc79aed-93ef-4b91-94e6-e62b5fa15ce7","shared_citers":6},{"title":"DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models","work_id":"c5006563-f3ec-438a-9e35-b7b484f34828","shared_citers":5},{"title":"DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model","work_id":"1e1df141-cac8-47fd-b068-c4c96e51e331","shared_citers":5},{"title":"Evaluating Large Language Models Trained on Code","work_id":"042493e9-b26f-4b4e-bbde-382072ca9b08","shared_citers":5},{"title":"Qwen3 Technical Report","work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e","shared_citers":5},{"title":"The Llama 3 Herd of Models","work_id":"1549a635-88af-4ac1-acfe-51ae7bb53345","shared_citers":5},{"title":"Training Compute-Optimal Large Language Models","work_id":"b2faf28d-86b7-429c-bc42-469458efc246","shared_citers":5},{"title":"arXiv preprint arXiv:2306.14048 , year=","work_id":"1ee58eaa-8aa0-4dec-8e67-6e079ad5fc13","shared_citers":4},{"title":"Fast Transformer Decoding: One Write-Head is All You Need","work_id":"160ea164-b1d4-4adb-8ccb-a4655d8a0bb4","shared_citers":4},{"title":"Mamba: Linear-Time Sequence Modeling with Selective State Spaces","work_id":"4ee75248-1199-492c-a52f-6661e0f4adff","shared_citers":4},{"title":"RULER: What's the Real Context Size of Your Long-Context Language Models?","work_id":"c0bc4689-3ce8-4e3d-9442-bd74869445bb","shared_citers":4},{"title":"Scaling Laws for Neural Language Models","work_id":"b7dd8749-9c45-4977-ab9b-64478dce1ae8","shared_citers":4},{"title":"DeepSeek-V3 Technical Report","work_id":"57d2791d-2219-4c31-a077-afc04b12a75c","shared_citers":3},{"title":"Efficient Streaming Language Models with Attention Sinks","work_id":"a8d25452-c237-48c9-88a4-682717c3979a","shared_citers":3},{"title":"FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness","work_id":"efa96825-0830-4cfc-a250-fdaf6af302ab","shared_citers":3},{"title":"GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers","work_id":"19ed8c44-202a-48f6-8169-637d5a5f2408","shared_citers":3},{"title":"GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints","work_id":"b73ad5b2-e553-4c71-b0c9-67e67ba7b158","shared_citers":3},{"title":"Group Sequence Policy Optimization","work_id":"3a98b53b-9f52-4d95-adf7-89353c0a9a65","shared_citers":3},{"title":"LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding","work_id":"ba7831c4-9427-4e0e-a5c1-4e98511f4b53","shared_citers":3},{"title":"LoRA: Low-Rank Adaptation of Large Language Models","work_id":"0426219a-789e-4964-adc8-a04538510818","shared_citers":3},{"title":"Measuring Mathematical Problem Solving With the MATH Dataset","work_id":"50652ac6-fb7c-4675-a2c2-159c241feb17","shared_citers":3},{"title":"Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism","work_id":"c888e6d1-0b1d-43d6-9ef5-f0912a0efa1b","shared_citers":3}],"time_series":[{"n":1,"year":2023},{"n":1,"year":2024},{"n":1,"year":2025},{"n":29,"year":2026}],"dependency_candidates":[]},"authors":[]}}