{"work":{"id":"eee8f937-db6b-4c48-887b-960ca80c0d81","openalex_id":"https://openalex.org/W4400377444","doi":"10.48550/arxiv.2407.00079","arxiv_id":"2407.00079","raw_key":null,"title":"Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving","authors":null,"authors_text":"Ruoyu Qin, Zheming Li, Weiran He, et al","year":2024,"venue":"cs.DC","abstract":"Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI. It features a KVCache-centric disaggregated architecture that separates the prefill and decoding clusters. It also leverages the underutilized CPU, DRAM, and SSD resources of the GPU cluster to implement a disaggregated cache of KVCache. The core of Mooncake is its KVCache-centric scheduler, which balances maximizing overall effective throughput while meeting latency-related Service Level Objectives (SLOs). Unlike traditional studies that assume all requests will be processed, Mooncake faces challenges due to highly overloaded scenarios. To mitigate these, we developed a prediction-based early rejection policy. Experiments show that Mooncake excels in long-context scenarios. Compared to the baseline method, Mooncake can achieve up to a 525% increase in throughput in certain simulated scenarios while adhering to SLOs. Under real workloads, Mooncake's innovative architecture enables Kimi to handle 75% more requests.","external_url":"https://arxiv.org/abs/2407.00079","cited_by_count":13,"metadata_source":"pith","metadata_fetched_at":"2026-08-05T02:28:24.338817+00:00","pith_arxiv_id":"2407.00079","created_at":"2026-05-10T07:47:12.426587+00:00","updated_at":"2026-08-05T02:28:24.338817+00:00","title_quality_ok":true,"display_title":"Mooncake: A kvcache-centric disaggregated architecture for llm serving","render_title":"Mooncake: A kvcache-centric disaggregated architecture for llm serving"},"hub":{"state":{"work_id":"eee8f937-db6b-4c48-887b-960ca80c0d81","tier":"hub","tier_reason":"10+ Pith inbound or 1,000+ external citations","pith_inbound_count":33,"external_cited_by_count":13,"distinct_field_count":11,"first_pith_cited_at":"2024-09-16T17:59:52+00:00","last_pith_cited_at":"2026-07-09T02:11:18+00:00","author_build_status":"not_needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"not_needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-08-22T02:09:31.585294+00:00","tier_text":"hub"},"tier":"hub","role_counts":[{"context_role":"background","n":4},{"context_role":"method","n":2}],"polarity_counts":[{"context_polarity":"background","n":5},{"context_polarity":"use_method","n":1}],"runs":{},"summary":{},"graph":{},"authors":[]}}