Pith. sign in

REVIEW 2 major objections 5 minor 60 references

For agentic LLM serving, balancing only each session’s first request keeps the cluster load-balanced without giving up most local KV-cache reuse.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 05:15 UTC pith:MKXCA3D2

load-bearing objection Clean systems paper: agent session locality + two-tier KV store yields a simple first-turn/follow-up router with real 10–16% goodput gains on production traces. the 2 major comments →

arxiv 2607.08565 v1 pith:MKXCA3D2 submitted 2026-07-09 cs.DC cs.AI

SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling

Jiahao Wang , Kaizhan Lin , Kaixi Zhang , Jinbo Han , Xingda Wei , Sijie Shen , Chenguang Fang , Wenyuan Yu
show 2 more authors
Rong Chen Haibo Chen
This is my paper
classification cs.DC cs.AI
keywords LLM schedulingagentic servingKV cachesession-centric routingload balancetokens per secondprefill-decode disaggregationglobal KV store
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When LLMs serve agents rather than human chat, the goal shifts to cluster-wide tokens per second and KV-cache reuse becomes dominant—over 80 percent of tokens in production agent traces. Existing schedulers chase local cache hits so aggressively that they pin whole sessions onto a few machines, leaving others idle and capping throughput. The paper shows this trade-off is unnecessary: a global KV store can still supply reuse when a request lands elsewhere, and agent workloads have strong intra-session locality, so the placement of the first request of each session largely determines where the rest of the session sticks. SMetric therefore routes first-turn requests purely for load balance and routes follow-ups cache-aware, keeping the router stateless by reading the turn number from the conversation history already present in the API call. On real traces it raises cluster TPS by 10–16 percent under prefill–decode colocation and prefill TPS by 2–34 percent under disaggregation, while also improving most per-token latency numbers.

Core claim

Existing cache-aware schedulers overload a few instances under agentic workloads because first-turn requests share system prompts and subsequent turns stick to the first-turn instance. Load balance need not sacrifice KV reuse when a global-tier store is available, and balancing only the first request of each session is enough to balance the cluster while still harvesting most local-tier reuse on follow-ups. Realizing this differential policy yields higher tokens-per-second within latency SLOs than production and academic baselines under both colocated and disaggregated serving.

What carries the argument

Balanced session-centric scheduling (SMetric): route each session’s first request with pure load balancing; route follow-ups to the highest local KV hit unless the target is overloaded or the session’s cache has been evicted, in which case fall back to load balancing. The turn index is inferred from the message history already carried in the request, so the router stays stateless.

Load-bearing premise

That the turn number of a request can be read cheaply and reliably from the conversation history already present in a standard LLM API call, and that agents almost never drop that history.

What would settle it

Replay the same agent traces after agents strip or heavily rewrite conversation history on every turn, then measure whether first-turn detection and local stickiness collapse and whether SMetric’s TPS advantage disappears relative to pure load-balance and pure cache-aware baselines.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Providers can raise tokens-per-second for agent traffic without adding GPUs by changing only the cluster router.
  • Under-provisioned global KV stores remain usable because most reuse stays local after the first turn is balanced.
  • The same first-turn / follow-up split applies to both prefill–decode colocation and prefill-only clusters under disaggregation.
  • Because the router needs no per-session state table, it scales with request rate rather than with the number of live agent sessions.
  • Tail imbalance from long sessions can be corrected by migrating only when an instance exceeds a simple overload threshold relative to the cluster mean.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Any multi-turn workload whose reuse is dominated by same-session history—not only coding agents—could adopt the same first-request balance rule.
  • If future agents routinely fork sub-agents that share parent context, the scheduler may need an extra inter-session affinity hint beyond turn number alone.
  • As models grow and pressure on the global tier increases, the value of keeping follow-ups local after a balanced first turn should rise further.
  • The same turn-derived signal could feed autoscaling or instance warm-up decisions without extra client instrumentation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper studies request scheduling for agentic LLM serving, where agents (not humans) issue multi-turn sessions. Using two BAILIAN production traces, it shows KV$ reuse exceeds 80% and is dominated by intra-session locality, while existing cache-aware schedulers (including BAILIAN production and LMetric) over-pin sessions to a few instances and leave others idle, capping cluster TPS. The authors propose SMETRIC: route each session’s first request purely for load balance and follow-ups cache-aware (with overload and eviction guards), keeping the router stateless by inferring turn index from the conversation history already carried in the LLM API. On a 32-instance H20 cluster with Mooncake/LMCache, SMETRIC improves TPS within SLO by 10–16% under PD-colocation with a global store and prefill TPS by 2–34% under PD-disaggregation across global-tier provisionings, with better median TTFT and most TPOT percentiles.

Significance. If the results hold, the paper supplies a practical, low-complexity scheduling rule that is immediately deployable on production agent-serving stacks and that measurably improves the primary objective (cluster TPS within SLO) without requiring agent-side changes or a stateful session table. Strengths include two real production traces with session/turn metadata, careful replay that preserves recorded output lengths and KV reuse, evaluation under both PD-colocation and PD-disaggregation, two model sizes, global-tier capacity sweeps, and an ablation of the pre-filter guards. The design choice to keep the router stateless by deriving turn from the API payload is a clean systems contribution. The work is of clear interest to the LLM-serving and distributed-systems communities.

major comments (2)
  1. §4.2 and the evaluation setup: the central stickiness premise (Figure 12: 96.6% of follow-ups return to the first-turn instance under BAILIAN) and the est_hit guard both assume that agents almost never drop conversation history. The paper treats drops as rare and uneconomical, but the traces themselves are not used to quantify how often history is truncated or compacted in production. A short measurement of history-drop / compaction frequency (or a sensitivity experiment that randomly drops history) would make the claim robust rather than contingent on an unmeasured assumption.
  2. §5.1–5.3: the reported gains are measured under a carefully rewritten replay that forces the evaluated model to emit exactly the recorded number of tokens and rewrites follow-up prompts for KV consistency. While necessary, this removes any interaction between scheduling and real generation-length variance. The paper should either (a) report a secondary experiment with unconstrained generation (even if only on a subset) or (b) explicitly bound how much the forced-length rewrite can inflate the measured TPS gap relative to live traffic.
minor comments (5)
  1. Abstract and §1: the phrase “KV$-reuse exceeds 80% of request tokens” is slightly awkward; “reuse covers >80% of prefill tokens” would match the later definition more cleanly.
  2. Figure 11 vs. Figure 16: the same capacity-sweep experiment appears twice with slightly different absolute TPS numbers; a short note that one is the analysis setup and the other the full end-to-end run would avoid confusion.
  3. §4.2, line 5 of the pseudocode: “req.turn != 0” is clear, but the text never states whether turn is 0- or 1-indexed in the inferred message count; a one-sentence clarification would help implementers.
  4. Finding 6 and Figure 9: the simulation uses recorded execution times rather than re-simulating under the new schedule; the paper already notes the approximation, but a sentence on why residual bias cannot reverse the “balance is possible” conclusion would strengthen the finding.
  5. Related work: DualMap and Preble are cited; a one-sentence contrast with SMETRIC’s first-turn-only balancing (vs. continuous rebalancing or prompt partitioning) would sharpen the novelty claim.

Circularity Check

0 steps flagged

No circularity: empirical systems paper whose claims rest on measured trace properties and end-to-end evaluation, not on self-referential derivation or fitted-as-prediction steps.

full rationale

SMetric is a systems design paper. Its load-bearing chain is: (1) two production traces yield Findings 1–6 (high KV$ reuse, intra-session locality, first-turn system-prompt reuse, quick reuse intervals, session skew that is still balanceable); (2) those observations motivate the differential policy (first-turn load-balance, follow-up cache-aware, with two pre-filter guards); (3) the policy is implemented and measured against external baselines (BAILIAN production, LMetric, load-balance-only/Dynamo) on the same traces under PD-colocation and PD-disaggregation, reporting TPS-within-SLO and latency CDFs. No equation is fitted to a subset of data and then reported as a prediction of a related quantity; OVERLOAD and HIT_RATIO are hyperparameters whose sensitivity is explicitly swept and shown to sit on a plateau (Fig. 25). Turn index is recovered from the conversation history already present in the standard LLM API call—an operational convenience, not a definitional identity that forces the TPS numbers. Self-citations (e.g., LMetric) appear only as comparison baselines, not as uniqueness theorems or load-bearing premises. The headline gains are therefore external measurements, not constructions. Score 0 is the correct outcome.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

Systems paper whose central claim rests on measured workload regularities and two tunable thresholds rather than on free physical constants or invented particles. The free parameters are the two pre-filter thresholds; the axioms are the empirical workload findings and the standard two-tier KV architecture; no new physical entities are postulated.

free parameters (2)
  • OVERLOAD = 2 (default)
    Threshold (default 2) used by not_overloaded() to decide when to break session stickiness and rebalance; swept in §5.5 and shown robust but still a free design choice.
  • HIT_RATIO = default on plateau (exact default not numerically stated)
    Fraction of estimated history that must be present in the local cache before a follow-up is allowed to stick; swept 0-0.75 in §5.5.
axioms (4)
  • domain assumption Agentic production workloads exhibit >80% KV reuse, of which >65% is intra-session and ~90% of reuses occur within ~100 s (Findings 1,2,4).
    Load-bearing empirical premise measured on the two BAILIAN traces; if reuse were low or slow the first-turn-only balance policy would lose its advantage.
  • domain assumption A two-tier (local GPU + global CPU/RDMA) KV store is present and can supply missed blocks without becoming the permanent bottleneck when only first-turn traffic is load-balanced.
    Stated in §2 and used throughout §4; without a usable global tier the policy reduces to pure load-balance and loses the claimed reuse.
  • domain assumption Session turn index can be recovered from the message history already present in a standard OpenAI-style API request.
    Explicit design choice in §4.2 that keeps the router stateless; if agents strip history the detection fails.
  • standard math Standard LLM serving model: prefill then auto-regressive decode, KV cache of prefix tokens, PD colocation or disaggregation.
    Background architecture assumed throughout; not re-derived.

pith-pipeline@v1.1.0-grok45 · 28453 in / 2878 out tokens · 30137 ms · 2026-07-10T05:15:02.627890+00:00 · methodology

0 comments
read the original abstract

LLM scheduling is critical to serving, yet it remains unclear how well existing designs fit agentic serving--with LLM requests issued by agents instead of humans. This shifts the workload in two ways: (1) agents act only on complete responses, making the cluster's tokens per second (TPS) the primary goal and relaxing--not eliminating--per-token latency requirements; and (2) requests share much of their KV\$-reuse exceeds 80% of request tokens in a production trace from BAILIAN, versus 54-62% in chat. This paper first contributes a systematic study of request scheduling for agents on two real-world traces. We find that to increase KV\$ reuse, existing schedulers overly prioritize routing requests to instances caching their KV\$, overloading a few while leaving the rest idle, capping TPS. We thus present two key insights: (1) load balance need not sacrifice all KV\$ reuse, thanks to the global-tier KV\$ store and (2) by utilizing the workload's intra-session locality, balancing a small fraction of requests--the first request in each agent session--suffices to balance the cluster without sacrificing most KV\$ reuse on local instances. SMETRIC realizes these insights with balanced session-centric scheduling: it routes each session's first request purely for load balance and its follow-up requests in a cache-aware manner, preserving load balance and local reuse while keeping demand on the global tier low. Using the session turn information as the scheduling metric is deliberate: it is derived efficiently and accurately from the user inputs alone, so the scheduler stays clean and stateless. SMETRIC improves cluster TPS by 10-16% under prefill-decode colocation with a global store and prefill TPS by 2-34% under disaggregation over state-of-the-art schedulers, also with a better per-token latency.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

60 extracted references · 60 canonical work pages · 10 internal anchors

  1. [1]

    Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mo- han, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve. In18th USENIX Sympo- sium on Operating Systems Design and Implementation (OSDI 24), pages 117–134, Santa Clara, CA, July 2024. USENIX Association

    work page 2024

  2. [2]

    https://github.com/vllm-project/ aibrix, 2025

    Aibrix. https://github.com/vllm-project/ aibrix, 2025

    work page 2025

  3. [3]

    https://github.com/aigw-project/aigw, 2025

    Aigw. https://github.com/aigw-project/aigw, 2025

    work page 2025

  4. [4]

    Claude code: Sub-agents and agent teams

    Anthropic. Claude code: Sub-agents and agent teams. https://docs.anthropic.com/en/docs/ claude-code/sub-agents, 2024

    work page 2024

  5. [5]

    Building a C compiler with Claude

    Anthropic. Building a C compiler with Claude. https://www.anthropic.com/engineering/ building-c-compiler, 2025

    work page 2025

  6. [6]

    Claude code

    Anthropic. Claude code. https://code.claude.com/ docs/en/overview, 2026

    work page 2026

  7. [7]

    Elevated error rates on Opus 4.6

    Anthropic. Elevated error rates on Opus 4.6. Anthropic Status Page, 2026

    work page 2026

  8. [8]

    Impact of response latency on user behavior in web search

    Ioannis Arapakis, Xiao Bai, and Berkant Barla Cambazoglu. Impact of response latency on user behavior in web search. In Shlomo Geva, Andrew Trotman, Peter Bruza, Charles L. A. Clarke, and Kalervo Järvelin, editors,The 37th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’14, Gold Coast , QLD, Australia - July...

    work page 2014

  9. [9]

    How do ai agents spend your money? analyzing and predicting token consumption in agentic coding tasks, 2026

    Longju Bai, Zhemin Huang, Xingyao Wang, Jiao Sun, Rada Mihalcea, Erik Brynjolfsson, Alex Pentland, and Jiaxin Pei. How do ai agents spend your money? analyzing and predicting token consumption in agentic coding tasks, 2026

    work page 2026

  10. [10]

    PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. Pyramidkv: Dynamic KV cache com- pression based on pyramidal information funneling.CoRR, abs/2406.02069, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  11. [11]

    Siyuan Chen, Zhipeng Jia, Samira Khan, Arvind Krishna- murthy, and Phillip B. Gibbons. Slos-serve: Optimized serving of multi-slo llms.CoRR, abs/2504.08784, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  12. [12]

    DeepSeek-V3 Technical Report

    DeepSeek-AI. Deepseek-v3 technical report.CoRR, abs/2412.19437, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  13. [13]

    Deepseek-v4: Towards highly efficient million- token context intelligence

    DeepSeek-AI. Deepseek-v4: Towards highly efficient million- token context intelligence. Technical report, 2026. Available at Hugging Face model repository. Accessed: 2026-06-10

    work page 2026

  14. [14]

    Fm-agent: Scaling formal methods to large systems via llm-based hoare- style reasoning, 2026

    Haoran Ding, Zhaoguo Wang, and Haibo Chen. Fm-agent: Scaling formal methods to large systems via llm-based hoare- style reasoning, 2026

    work page 2026

  15. [15]

    Cost-efficient large language model serving for multi-turn conversations with cachedattention

    Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. Cost-efficient large language model serving for multi-turn conversations with cachedattention. In Saurabh Bagchi and Yiying Zhang, editors,Proceedings of the 2024 USENIX An- nual Technical Conference, USENIX ATC 2024, Santa Clara, CA, USA,...

    work page 2024

  16. [16]

    Prompt cache: Modular attention reuse for low-latency inference

    In Gim, Guojun Chen, Seung-Seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference. In Phillip B. Gibbons, Gen- nady Pekhimenko, and Christopher De Sa, editors,Proceed- ings of the Seventh Annual Conference on Machine Learning and Systems, MLSys 2024, Santa Clara, CA, USA, May 13-16,

    work page 2024

  17. [17]

    Google. llm-d. https://github.com/llm-d/llm-d, 2026

    work page 2026

  18. [18]

    Harvest: Opportunis- tic peer-to-peer GPU caching for LLM inference.CoRR, abs/2602.00328, 2026

    Nikhil Gopal and Kostis Kaffes. Harvest: Opportunis- tic peer-to-peer GPU caching for LLM inference.CoRR, abs/2602.00328, 2026

    work page arXiv 2026

  19. [19]

    Serv- ing dnns like clockwork: Performance predictability from the bottom up

    Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, An- toine Kaufmann, Ymir Vigfusson, and Jonathan Mace. Serv- ing dnns like clockwork: Performance predictability from the bottom up. In14th USENIX Symposium on Operating Sys- tems Design and Implementation, OSDI 2020, Virtual Event, November 4-6, 2020, pages 443–462. USENIX Association, 2020

    work page 2020

  20. [20]

    Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length LLM inference with KV cache quantization. In Amir Glober- sons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors,Ad- vances in Neural Inform...

    work page 2024

  21. [21]

    Turboattention: Efficient attention approximation for high throughputs llm

    Hao Kang, Srikant Bharadwaj, James Hensman, Tushar Kr- ishna, Victor Rühle, and Saravan Rajmohan. Turboattention: Efficient attention approximation for high throughputs llm. In Matei Zaharia, Gauri Joshi, and Yingyan (Celine) Lin, editors, Proceedings of the Eighth Conference on Machine Learning and Systems, MLSys 2025, Santa Clara, CA, USA, May 12-15,

    work page 2025

  22. [22]

    OpenReview.net/mlsys.org, 2025

    work page 2025

  23. [23]

    Efficient memory management for large lan- guage model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. InProceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, pages 611–

    work page 2023

  24. [24]

    John, and Neeraja J

    Ruihao Li, Shagnik Pal, Vineeth Narayan Pullu, Prasoon Sinha, Jeeho Ryoo, Lizy K. John, and Neeraja J. Yadwadkar. MI- RAGE: KV cache optimization through parameter remapping for multi-tenant LLM serving.CoRR, abs/2507.11507, 2025

    work page arXiv 2025

  25. [25]

    Snapkv: LLM knows what you are looking for before generation

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: LLM knows what you are looking for before generation. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tom- czak, and Cheng Zhang, editors,Advances in Neural Infor- mation Process...

    work page 2024

  26. [26]

    iServe: An Intent-based Serving System for LLMs

    Dimitrios Liakopoulos, Tianrui Hu, Prasoon Sinha, and Neer- aja J. Yadwadkar. iserve: An intent-based serving system for llms.CoRR, abs/2501.13111, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  27. [27]

    Parrot: Efficient serving of llm-based applications with semantic variable

    Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, and Lili Qiu. Parrot: Efficient serving of llm-based applications with semantic variable. In Ada Gavrilovska and Douglas B. Terry, editors,18th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2024, Santa Clara, CA, USA, July 10-12, 2024, pages 929–945. US...

    work page 2024

  28. [28]

    Cachegen: KV cache com- pression and streaming for fast large language model serving

    Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. Cachegen: KV cache com- pression and streaming for fast large language model serving. InProceedings of the ACM SIGCOMM 2024 Conference, ACM SIGCOMM 2024, Sydn...

    work page 2024

  29. [29]

    KIVI: A tuning-free asymmetric 2bit quantization for KV cache

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen (Henry) Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In Ruslan Salakhutdinov, Zico Kolter, Katherine A. Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Fe- lix Berkenkamp, editors,Forty-first International Conference on...

    work page 2024

  30. [30]

    https://github.com/LMCache/LMCache, 2026

    Lmcache: The best kv cache layer for enterprise-scale llm in- ference. https://github.com/LMCache/LMCache, 2026

    work page 2026

  31. [31]

    Autellix: An Efficient Serving Engine for LLM Agents as General Programs

    Michael Luo, Xiaoxiang Shi, Colin Cai, Tianjun Zhang, Justin Wong, Yichuan Wang, Chi Wang, Yanping Huang, Zhifeng Chen, Joseph E. Gonzalez, and Ion Stoica. Autellix: An efficient serving engine for LLM agents as general programs. CoRR, abs/2502.13965, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  32. [32]

    Spotserve: Serving generative large language models on preemptible instances

    Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia. Spotserve: Serving generative large language models on preemptible instances. In Rajiv Gupta, Nael B. Abu-Ghazaleh, Madan Musuvathi, and Dan Tsafrir, editors,Proceedings of the 29th ACM International Conference on Architectural Support for Programming Lan- guages and Op...

    work page 2024

  33. [33]

    MiniMax-M3

    MiniMax. MiniMax-M3. https://www.minimax.io/ blog/minimax-m3, 2026

    work page 2026

  34. [34]

    Kimi introduces agent swarm: Let 100 AI agents work for you

    Moonshot AI. Kimi introduces agent swarm: Let 100 AI agents work for you. https://www.kimi.com/blog/ agent-swarm, 2026

    work page 2026

  35. [35]

    ai-dynamo

    NVIDIA. ai-dynamo. https://github.com/ai- dynamo/dynamo, 2025

    work page 2025

  36. [36]

    OpenAI API

    OpenAI. OpenAI API. https://openai.com/api/, 2026

    work page 2026

  37. [37]

    Openclaw.https://openclaw.ai, 2026

    OpenClaw. Openclaw.https://openclaw.ai, 2026. 15

    work page 2026

  38. [38]

    Splitwise: Efficient generative LLM inference using phase splitting

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative LLM inference using phase splitting. In 51st ACM/IEEE Annual International Symposium on Com- puter Architecture, ISCA 2024, Buenos Aires, Argentina, June 29 - July 3, 2024, pages 118–132. IEEE, 2024

    work page 2024

  39. [39]

    Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot

    Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25), pages 155–170, Santa Clara, CA, February 2025. USENIX Association

    work page 2025

  40. [40]

    Qwen Team. Qwen3.7. https://qwen.ai/blog?id= qwen3.7, 2026

    work page 2026

  41. [41]

    DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving

    Chaoyi Ruan, Yinhe Chen, Dongqi Tian, Yandong Shi, Yongji Wu, Jialin Li, and Cheng Li. Dynaserve: Unified and elas- tic tandem-style execution for dynamic disaggregated LLM serving.CoRR, abs/2504.09285, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  42. [42]

    Traceupscaler: Upscaling traces to evaluate systems at high load

    Sultan Mahmud Sajal, Timothy Zhu, Bhuvan Urgaonkar, and Siddhartha Sen. Traceupscaler: Upscaling traces to evaluate systems at high load. InProceedings of the Nineteenth Euro- pean Conference on Computer Systems, EuroSys 2024, Athens, Greece, April 22-25, 2024, pages 942–961. ACM, 2024

    work page 2024

  43. [43]

    Preble: Efficient distributed prompt scheduling for LLM serving

    Vikranth Srivatsa, Zijian He, Reyna Abhyankar, Dongming Li, and Yiying Zhang. Preble: Efficient distributed prompt scheduling for LLM serving. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singa- pore, April 24-28, 2025. OpenReview.net, 2025

    work page 2025

  44. [44]

    Llumnix: Dynamic scheduling for large language model serving

    Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. Llumnix: Dynamic scheduling for large language model serving. In Ada Gavrilovska and Douglas B. Terry, editors,18th USENIX Symposium on Oper- ating Systems Design and Implementation, OSDI 2024, Santa Clara, CA, USA, July 10-12, 2024, pages 173–191. USENIX Association, 2024

    work page 2024

  45. [45]

    https://github.com/vllm- project/vllm/releases/tag/v0.12.0, 2025

    vllm v0.12.0 release. https://github.com/vllm- project/vllm/releases/tag/v0.12.0, 2025

    work page 2025

  46. [46]

    Kvcache cache in the wild: Characterizing and optimiz- ing kvcache cache at a large cloud provider

    Jiahao Wang, Jinbo Han, Xingda Wei, Sijie Shen, Dingyan Zhang, Chenguang Fang, Rong Chen, Wenyuan Yu, and Haibo Chen. Kvcache cache in the wild: Characterizing and optimiz- ing kvcache cache at a large cloud provider. In2025 USENIX Annual Technical Conference (USENIX ATC 25). USENIX Association, July 2025

    work page 2025

  47. [47]

    Loongserve: Efficiently serving long- context large language models with elastic sequence paral- lelism

    Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xu- anzhe Liu, and Xin Jin. Loongserve: Efficiently serving long- context large language models with elastic sequence paral- lelism. In Emmett Witchel, Christopher J. Rossbach, Andrea C. Arpaci-Dusseau, and Kimberly Keeton, editors,Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principle...

    work page 2024

  48. [48]

    Dualpath: Breaking the storage bandwidth bottleneck in agentic LLM inference.CoRR, abs/2602.21548, 2026

    Yongtong Wu, Shaoyuan Chen, Yinmin Zhong, Rilin Huang, Yixuan Tan, Wentao Zhang, Liyue Zhang, Shangyan Zhou, Yuxuan Liu, Shunfeng Zhou, Mingxing Zhang, Xin Jin, and Panpan Huang. Dualpath: Breaking the storage bandwidth bottleneck in agentic LLM inference.CoRR, abs/2602.21548, 2026

    work page arXiv 2026

  49. [49]

    Aegaeon: Effective GPU pooling for concurrent LLM serving on the market

    Yuxing Xiang, Xue Li, Kun Qian, Yufan Yang, Diwen Zhu, Wenyuan Yu, Ennan Zhai, Xuanzhe Liu, Xin Jin, and Jingren Zhou. Aegaeon: Effective GPU pooling for concurrent LLM serving on the market. In Youjip Won, Youngjin Kwon, Ding Yuan, and Rebecca Isaacs, editors,Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, SOSP 2025, Lotte H...

    work page 2025

  50. [50]

    Cacheblend: Fast large language model serving for RAG with cached knowledge fusion

    Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. Cacheblend: Fast large language model serving for RAG with cached knowledge fusion. InProceedings of the Twentieth European Conference on Computer Systems, EuroSys 2025, Rotterdam, The Netherlands, 30 March 2025 - 3 April 2025, pages 94–109. ...

    work page 2025

  51. [51]

    Chunkattention: Efficient self-attention with prefix-aware KV cache and two- phase partition

    Lu Ye, Ze Tao, Yong Huang, and Yang Li. Chunkattention: Efficient self-attention with prefix-aware KV cache and two- phase partition. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pag...

    work page 2024

  52. [52]

    Flashinfer: Efficient and customizable attention engine for LLM inference serving

    Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, and Luis Ceze. Flashinfer: Efficient and customizable attention engine for LLM inference serving. In Matei Zaharia, Gauri Joshi, and Yingyan (Celine) Lin, editors,Proceedings of the Eighth Conference on Machine Learni...

    work page 2025

  53. [53]

    Stateful Large Language Model Serving with Pensieve

    Lingfan Yu and Jinyang Li. Stateful large language model serving with pensieve.CoRR, abs/2312.05516, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  54. [54]

    Dualmap: Enabling both cache affin- ity and load balancing for distributed LLM serving.CoRR, abs/2602.06502, 2026

    Ying Yuan, Pengfei Zuo, Bo Wang, Zhangyu Chen, Zhipeng Tan, and Zhou Yu. Dualmap: Enabling both cache affin- ity and load balancing for distributed LLM serving.CoRR, abs/2602.06502, 2026

    work page arXiv 2026

  55. [55]

    Simple is better: Multiplication may be all you need for llm request scheduling

    Dingyan Zhang, Jinbo Han, Kaixi Zhang, Xingda Wei, Si- jie Shen, Chenguang Fang, Wenyuan Yu, Jingren Zhou, and Rong Chen. Simple is better: Multiplication may be all you need for llm request scheduling. InProceedings of the 20th USENIX Symposium on Operating Systems Design and Imple- mentation (OSDI ’26), Seattle, W A, USA, July 2026. USENIX Association

    work page 2026

  56. [56]

    Blitzscale: Fast and live large model autoscaling with O(1) host caching

    Dingyan Zhang, Haotian Wang, Yang Liu, Xingda Wei, Yizhou Shan, Rong Chen, and Haibo Chen. Blitzscale: Fast and live large model autoscaling with O(1) host caching. In Lidong Zhou and Yuanyuan Zhou, editors,19th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2025, Boston, MA, USA, July 7-9, 2025, pages 275–293. USENIX Association, 2025

    work page 2025

  57. [57]

    Yanqi Zhang, Yuwei Hu, Runyuan Zhao, John C. S. Lui, and Haibo Chen. Unifying KV cache compression for large lan- guage models with leankv.CoRR, abs/2412.03131, 2024. 16

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  58. [58]

    SGLang: Efficient Execution of Structured Language Model Programs

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. Efficiently programming large language models using sglang.CoRR, abs/2312.07104, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  59. [59]

    Distserve: Dis- aggregating prefill and decoding for goodput-optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Distserve: Dis- aggregating prefill and decoding for goodput-optimized large language model serving. In18th USENIX Symposium on Oper- ating Systems Design and Implementation, OSDI 2024, Santa Clara, CA, USA, July 10-12, 2024, pages 193–210. USENIX Association, 2024

    work page 2024

  60. [60]

    PolyServe: Efficient Multi-SLO Serving at Scale

    Kan Zhu, Haiyang Shi, Le Xu, Jiaxin Shan, Arvind Krishna- murthy, Baris Kasikci, and Liguang Xie. Polyserve: Efficient multi-slo serving at scale.CoRR, abs/2507.17769, 2025. 17

    work page internal anchor Pith review Pith/arXiv arXiv 2025