Pith. sign in

REVIEW 2 major objections 4 minor 47 references

Multiplying new prefill tokens by current batch size schedules LLM requests for both KV-cache hits and load balance with no hyperparameter tuning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 20:34 UTC pith:Z2IGS7P7

load-bearing objection A clean, practical observation that P-token × BS cancels the usual weights and beats tuned linear/simulators on real traces; the failure analysis is honest but un-stressed. the 2 major comments →

arxiv 2603.15202 v3 pith:Z2IGS7P7 submitted 2026-03-16 cs.DC cs.OS

Simple is Better: Multiplication May Be All You Need for LLM Request Scheduling

Dingyan Zhang , Jinbo Han , Kaixi Zhang , Xingda Wei , Sijie Shen , Chenguang Fang , Wenyuan Yu , Jingren Zhou
show 1 more author
Rong Chen
This is my paper
classification cs.DC cs.OS
keywords LLM request schedulingKV-cache awarenessload balancingmultiplicative scoreTTFTTPOTglobal routerhyperparameter-free
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

High-quality routing of LLM requests across a cluster must simultaneously prefer instances that already hold useful KV-cache (to cut prefill work) and keep load balanced (to avoid long queues). Existing combinators either need workload-specific weight tuning or expensive per-model simulators, and still often trade one goal for the other. This paper shows that the product of two carefully chosen indicators—new prefill tokens if the request is sent to an instance, times that instance’s current batch size—acts like a linear combination whose free parameters cancel when scores are compared, so no tuning is required. The resulting score, called LMETRIC, yields large reductions in time-to-first-token and time-per-output-token on production chatbot, API and coding-agent traces, and a simple two-phase detector catches the rare mathematical failure cases before they matter. The method has already been deployed.

Core claim

A scheduling score formed by multiplying the number of new prefill tokens (after accounting for KV-cache hits) by the instance’s current batch size simultaneously satisfies both KV-cache awareness and load balancing; the product behaves like a linear combination yet needs no hyperparameters because the weights cancel during pairwise comparison of scores.

What carries the argument

LMETRIC score = (new prefill tokens if routed to instance) × (current batch size of instance); the instance with the minimal product is chosen.

Load-bearing premise

The mathematical failure condition—a request class whose arrival fraction creates a persistent KV-cache hotspot that the product never corrects—is both extremely rare on real traces and reliably caught by a two-phase detector before imbalance becomes severe.

What would settle it

Replay a production trace containing a sustained request class whose arrival fraction satisfies x/¯x > |M|/|¯M| and measure whether consecutive product scores keep routing to the same hotspot instances after the two-phase filter is disabled; if TTFT or TPOT then degrades below a well-tuned linear baseline, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Production routers can drop all weight-tuning loops and simulator maintenance for PD-colocated clusters.
  • Mean TTFT drops by roughly 90 % and mean TPOT by 20–50 % versus pure load-balancing or tuned linear baselines on chatbot and coding workloads.
  • The same two indicators and product can be used as a drop-in replacement inside existing indicator factories without changing the rest of the serving stack.
  • Failure modes become first-class, detectable events rather than silent degradations that appear only under load.
  • Canary deployments can safely roll the policy out cluster-wide once the detector reports zero hotspots over a short window.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same product form may extend to PD-disaggregated deployments if the load indicator is redefined as the decode-only batch size on the decode pool.
  • If future models change the relative cost of prefill versus decode, the paper’s indicator-selection argument still supplies a systematic way to re-choose the two factors rather than re-tuning weights.
  • The cancellation property suggests a broader design pattern: any pair of positive indicators whose linear combination is already known to work can be replaced by their product to eliminate hyperparameters.
  • Online detectors of the form given in §5.2 could be reused as lightweight load-imbalance monitors even when a different combinator is employed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes LMETRIC, a global LLM request scheduler that scores each instance by the product of two indicators: new prefill tokens after KV-cache hits (P-token) and current batch size (BS). It argues that multiplication achieves the same trade-off as a linear combination of KV-awareness and load balancing while canceling free weights under pairwise comparison, eliminating workload-specific hyperparameter tuning and model-hardware simulators. Indicator choices are justified by ablations; a fluid model of batch-size growth yields an approximate failure condition (request-class fraction x/¯x > |M|/|¯M|) together with a two-phase detector that falls back to pure load balancing. End-to-end experiments on a 16-GPU H20 cluster with Qwen2/Qwen3 models and four real traces (chatbot, API, coding, tool-agent) report large TTFT/TPOT reductions versus re-implemented vLLM-v1, Dynamo, llm-d and a production linear scheduler; a production canary is mentioned.

Significance. If the product rule is robust, the result is practically important: production LLM routers currently rely on either carefully tuned linear weights or expensive per-model simulators, both of which are brittle under workload drift. A parameter-free score that matches or beats those baselines on real traces, together with an open analysis framework and a production deployment claim, would be a clear systems contribution. Strengths include the clean algebraic observation that weights cancel under comparison, systematic indicator ablations, a derived (if approximate) failure condition, and apples-to-apples re-implementations of baselines inside a high-performance Rust router.

major comments (2)
  1. §5.2, Eqs. (1)–(4) and Fig. 20: the central claim that multiplication “fails only under extremely rare conditions that can be detected and mitigated beforehand” rests on a fluid approximation that assumes every request of class c is routed to the hit set M, and on four traces that never violate x/¯x > |M|/|¯M|. Consequently the two-phase detector (consecutive-score threshold 2×|M|, 1-min window) is never activated in any reported experiment, and no ablation quantifies its false-positive cost or its ability to restore balance once a synthetic hotspot is injected. Without such stress evidence the “no-tuning + reliable mitigation” guarantee remains incompletely supported.
  2. §6 and abstract: the headline numbers (92 % TTFT / 24 % TPOT vs vLLM; 39 % / 51 % vs production) are means or selected percentiles on half-capacity loads; Fig. 22 shows that under higher rates and on ToolAgent the ranking versus llm-d is mixed (sometimes higher mean TTFT). The paper should report a consistent set of mean/P50/P99 metrics across all rates and traces, and clarify whether the production canary used the same detector parameters, so that the magnitude of the claimed gains can be assessed under the conditions where the product is most stressed.
minor comments (4)
  1. Abstract vs. body: abstract claims 39 % TTFT / 51 % TPOT vs production; §6 text and Fig. 21 report different relative numbers. Align the summary statistics.
  2. Fig. 11–12, 15, 21–22: axis labels and legends are dense; several panels lack units or use “TPOP” instead of “TPOT”. Clean for camera-ready.
  3. §3 programming model and indicator factory are useful but only sketched; a short appendix listing the exact symbolic indicators exposed would aid reproducibility.
  4. Discussion of PD-disaggregation is limited to a single paragraph; a sentence on whether the product score extends (or fails) under disaggregated prefill/decode would help scope the claim.

Circularity Check

0 steps flagged

No circularity: product score follows from algebraic cancellation of linear weights, indicator choice is empirical, and failure conditions are a fluid approximation checked on traces rather than fitted predictions.

full rationale

The central construction (score = P-token × BS) is obtained by rewriting a two-term linear combination so that the free weight λ cancels under pairwise comparison (Fig. 17 and surrounding text); the algebra is self-contained and does not embed the target performance numbers. Indicator selection (P-token over 1-KV-hit-ratio, BS over total tokens) is justified by direct ablation on the same traces used for end-to-end evaluation (§5.1, Figs. 18–19), not by fitting a free parameter that is later re-labeled a prediction. The failure-mode analysis (§5.2) starts from an explicit fluid approximation of batch-size growth under the extreme routing assumption that every request of a class lands on the hit set M (Eq. 1), rearranges to the inequality x/¯x ≤ |M|/|¯M| (Eqs. 2–3), and adds a consecutive-score filter (Eq. 4). These steps are ordinary first-order queueing reasoning; they do not import a uniqueness theorem, do not rely on a self-citation for the derivation itself, and do not fit parameters to the very quantity later claimed to be rare. Empirical rarity is simply observed on the four evaluation traces (Fig. 20); the detector is never activated in the reported experiments, but that is an empirical limitation, not a circular reduction of the claim to its inputs. Baselines that re-implement the authors’ own production linear scheduler are used only for comparison, not as load-bearing premises. Consequently the derivation chain does not reduce by construction to its own inputs.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 2 invented entities

The central claim rests on a small set of domain assumptions about LLM serving (KV-cache hit reduces prefill work, batch size dominates decode load, PD-colocation) plus two lightweight detector thresholds. No continuous free parameters are fitted to the evaluation traces; the product itself is parameter-free. The invented entities are the score and the two-phase detector, both of which have clear operational definitions.

free parameters (2)
  • consecutive-request threshold (2×|M|)
    Second-phase detector waits for more than 2×|M| consecutive multiplications that prefer a hotspot before filtering; the constant 2 is chosen by hand.
  • profiling window (1 min)
    Request-class fractions x/¯x are estimated inside a 1-minute sliding window; window length is a design choice.
axioms (3)
  • domain assumption Prefill work is accurately captured by the number of new (non-cached) tokens; decode work is accurately captured by running+queued batch size.
    Stated in §5.1 and used to justify the two indicators; supported by ablations but not proved for all model architectures.
  • domain assumption Under PD-colocation the same instance handles both prefill and decode of a request.
    Explicit scope restriction in the Discussion; the product rule is not claimed for disaggregated deployments.
  • ad hoc to paper Request classes that share a long common prefix can be treated as a single fluid arrival process for the purpose of the imbalance inequality.
    Introduced in §5.2 to derive Eqs. 1–3; standard fluid approximation but not independently validated for every trace.
invented entities (2)
  • LMETRIC product score (P-token × BS) independent evidence
    purpose: Single scalar that simultaneously encodes KV-cache awareness and load balance without free weights.
    Defined in §5; independent evidence is the end-to-end latency improvement and production canary.
  • Two-phase KV-hotspot detector no independent evidence
    purpose: Detect and mitigate the rare failure regime of the product before load imbalance occurs.
    Defined in §5.2; independent evidence is the mathematical derivation plus the claim that the regime never appears in the evaluated traces.

pith-pipeline@v1.1.0-grok45 · 26959 in / 2793 out tokens · 30709 ms · 2026-07-14T20:34:24.628452+00:00 · methodology

0 comments
read the original abstract

High-quality LLM request scheduling requires meeting two key objectives: ensuring the routed instance has KVCache to accelerate request execution, and ensuring that the workload is balanced across instances. Achieving both objectives is challenging because pursuing one may compromise the other. Current approaches use various combinators (e.g., linear combinations) to compute a scheduling score that combines indicators for the two objectives. These approaches are complex: they either require significant workload-specific hyperparameter tuning or model-hardware-aware simulator development, yet could still lead to suboptimal performance. In this paper, we show that using a simple multiplication of two carefully chosen indicators: one KVCache-aware (new prefill tokens if routed to an instance) and one load-balancing-aware (current batch size of the instance), as the scheduling score (LMETRIC) can achieve both objectives simultaneously without any hyperparameter tuning. The key idea is that the simply multiplied score considers both objectives in a manner similar to a linear combination, but the original hyperparameters cancel out during comparison, so no tuning is needed to find the best parameters. The two indicators are chosen based on our analysis of LLM characteristics. Our extensive experiments show that this simple approach can reduce TTFT by 92% and 39%, and TPOT by 24% and 51%, compared to vLLM-v1 and an in-production scheduler on real-world workloads covering chatbots and coding agents. We also derive the mathematical conditions under which multiplication may fail, and find that such conditions are extremely rare in practice and can be detected (and mitigated) beforehand. LMETRIC has been deployed in production and canary release confirms its effectiveness

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 4 linked inside Pith

  1. [1]

    S., RAMJEE, R.,ANDTUMANOV, A

    AGRAWAL, A., KEDIA, N., MOHAN, J., PANWAR, A., KWA- TRA, N., GULAVANI, B. S., RAMJEE, R.,ANDTUMANOV, A. VIDUR: A large-scale simulation framework for LLM in- ference. InProceedings of the Seventh Annual Conference on Machine Learning and Systems, MLSys 2024, Santa Clara, CA, USA, May 13-16, 2024(2024), P. B. Gibbons, G. Pekhimenko, and C. D. Sa, Eds., mlsys.org

    2024

  2. [2]

    Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve

    AGRAWAL, A., KEDIA, N., PANWAR, A., MOHAN, J., KWA- TRA, N., GULAVANI, B., TUMANOV, A.,ANDRAMJEE, R. Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve. In18th USENIX Symposium on Operating Sys- tems Design and Implementation (OSDI 24)(Santa Clara, CA, July 2024), USENIX Association, pp. 117–134

    2024

  3. [3]

    https://github.com/vllm-project/ aibrix, 2025

    Aibrix. https://github.com/vllm-project/ aibrix, 2025

    2025

  4. [4]

    Aigw.https://github.com/aigw-project/aigw, 2025

    2025

  5. [5]

    Opti- mizing inference serving on serverless platforms.Proc

    ALI, A., PINCIROLI, R., YAN, F.,ANDSMIRNI, E. Opti- mizing inference serving on serverless platforms.Proc. VLDB Endow. 15, 10 (2022), 2071–2084

    2022

  6. [6]

    Claude api

    ANTHROPIC. Claude api. https://www.anthropic. com/api, 2025

    2025

  7. [7]

    ARAPAKIS, I., BAI, X.,ANDCAMBAZOGLU, B. B. Impact of response latency on user behavior in web search. InThe 37th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’14, Gold Coast , QLD, Australia - July 06 - 11, 2014(2014), S. Geva, A. Trot- man, P. Bruza, C. L. A. Clarke, and K. Järvelin, Eds., ACM, pp. 103–112

    2014

  8. [8]

    Azure llm inference traces

    AZURE. Azure llm inference traces. https://github. com/Azure/AzurePublicDataset/blob/master/ AzureLLMInferenceDataset2024.md, 2024

    2024

  9. [9]

    https://github

    Qwen-bailian anonymous dataset. https://github. com/alibaba-edu/qwen-bailian-usagetraces- anon, 2025

    2025

  10. [10]

    CHEN, S., JIA, Z., KHAN, S., KRISHNAMURTHY, A.,AND GIBBONS, P. B. Slos-serve: Optimized serving of multi-slo llms.CoRR abs/2504.08784(2025)

    Pith/arXiv arXiv 2025

  11. [11]

    Conditional memory via scalable lookup: A new axis of sparsity for large language models, 2026

    CHENG, X., ZENG, W., DAI, D., CHEN, Q., WANG, B., XIE, Z., HUANG, K., YU, X., HAO, Z., LI, Y., ZHANG, H., ZHANG, H., ZHAO, D.,ANDLIANG, W. Conditional memory via scalable lookup: A new axis of sparsity for large language models, 2026

    2026

  12. [12]

    Quasar: resource- efficient and qos-aware cluster management

    DELIMITROU, C.,ANDKOZYRAKIS, C. Quasar: resource- efficient and qos-aware cluster management. InArchitectural Support for Programming Languages and Operating Systems, ASPLOS 2014, Salt Lake City, UT, USA, March 1-5, 2014 (2014), R. Balasubramonian, A. Davis, and S. V . Adve, Eds., ACM, pp. 127–144

    2014

  13. [13]

    https:// github.com/flashinfer-ai/flashinfer, 2025

    FlashInfer: Kernel Library for LLM Serving. https:// github.com/flashinfer-ai/flashinfer, 2025

    2025

  14. [14]

    Cost-Efficient large language model serving for multi-turn conversations with CachedAttention

    GAO, B., HE, Z., SHARMA, P., KANG, Q., JEVDJIC, D., DENG, J., YANG, X., YU, Z.,ANDZUO, P. Cost-Efficient large language model serving for multi-turn conversations with CachedAttention. In2024 USENIX Annual Technical Confer- ence (USENIX ATC 24)(Santa Clara, CA, July 2024), USENIX Association, pp. 111–126

    2024

  15. [15]

    Amazon found every 100ms of latency cost them 1% in sales

    GIGASPACES. Amazon found every 100ms of latency cost them 1% in sales. https://www.gigaspaces. com/blog/amazon-found-every-100ms-of- latency-cost-them-1-in-sales, 2024

    2024

  16. [16]

    Accelerate your development speed with copilot

    GITHUB. Accelerate your development speed with copilot. https://copilot.github.com, 2024

    2024

  17. [17]

    GOG, I., SCHWARZKOPF, M., GLEAVE, A., WATSON, R. N. M.,ANDHAND, S. Firmament: Fast, centralized cluster scheduling at scale. In12th USENIX Symposium on Operat- ing Systems Design and Implementation, OSDI 2016, Savan- nah, GA, USA, November 2-4, 2016(2016), K. Keeton and T. Roscoe, Eds., USENIX Association, pp. 99–115

    2016

  18. [18]

    Gemini api

    GOOGLE. Gemini api. https://ai.google.dev/api, 2025

    2025

  19. [19]

    GOOGLE. llm-d. https://github.com/llm-d/llm- d, 2025

    2025

  20. [20]

    Serving dnns like clockwork: Performance predictability from the bottom up

    GUJARATI, A., KARIMI, R., ALZAYAT, S., HAO, W., KAUF- MANN, A., VIGFUSSON, Y.,ANDMACE, J. Serving dnns like clockwork: Performance predictability from the bottom up. In14th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2020, Virtual Event, November 4-6, 2020(2020), USENIX Association, pp. 443–462

    2020

  21. [21]

    Bestserve: Serving strategies with optimal good- put in collocation and disaggregation architectures.CoRR abs/2506.05871(2025)

    HU, X., ZENG, T., YUAN, X., SONG, L., ZHANG, G.,AND HE, B. Bestserve: Serving strategies with optimal good- put in collocation and disaggregation architectures.CoRR abs/2506.05871(2025)

    Pith/arXiv arXiv 2025

  22. [22]

    K., PRABHU, R., MOHAN, J., PETER, S., RAM- JEE, R.,ANDPANWAR, A

    KAMATH, A. K., PRABHU, R., MOHAN, J., PETER, S., RAM- JEE, R.,ANDPANWAR, A. Pod-attention: Unlocking full prefill-decode overlap for faster LLM inference. InProceed- ings of the 30th ACM International Conference on Architec- tural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS 2025, Rotterdam, Netherlands, 30 March 2025 - 3 Apri...

    2025

  23. [23]

    H., GONZALEZ, J., ZHANG, H.,ANDSTOICA, I

    KWON, W., LI, Z., ZHUANG, S., SHENG, Y., ZHENG, L., YU, C. H., GONZALEZ, J., ZHANG, H.,ANDSTOICA, I. Ef- ficient memory management for large language model serving with pagedattention. InProceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023(2023), J. Flinn, M. I. Seltzer, P. Druschel, A. Kaufma...

    2023

  24. [24]

    Fairbatching: Fairness-aware batch formation for LLM inference.CoRR abs/2510.14392(2025)

    LYU, H., LIU, B., WU, M.,ANDCHEN, H. Fairbatching: Fairness-aware batch formation for LLM inference.CoRR abs/2510.14392(2025)

    arXiv 2025

  25. [25]

    Spotserve: Serving generative large language models on preemptible instances

    MIAO, X., SHI, C., DUAN, J., XI, X., LIN, D., CUI, B., ANDJIA, Z. Spotserve: Serving generative large language models on preemptible instances. InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS 2024, La Jolla, CA, USA, 27 April 2024- 1 May 2024 (2024), R. Gupta...

    2024

  26. [26]

    https://github.com/kvcache- ai/Mooncake/blob/main/FAST25-release/ traces/toolagent_trace.jsonl, 2025

    Mooncake trace. https://github.com/kvcache- ai/Mooncake/blob/main/FAST25-release/ traces/toolagent_trace.jsonl, 2025

    2025

  27. [27]

    ai-dynamo

    NVIDIA. ai-dynamo. https://github.com/ai- dynamo/dynamo, 2025

    2025

  28. [28]

    Openai developer platform

    OPENAI. Openai developer platform. https:// platform.openai.com/docs/overview

  29. [29]

    Chatgpt.https://chatgpt.com, 2025

    OPENAI. Chatgpt.https://chatgpt.com, 2025

    2025

  30. [30]

    Splitwise: Efficient gener- ative LLM inference using phase splitting

    PATEL, P., CHOUKSE, E., ZHANG, C., SHAH, A., GOIRI, Í., MALEKI, S.,ANDBIANCHINI, R. Splitwise: Efficient gener- ative LLM inference using phase splitting. In51st ACM/IEEE Annual International Symposium on Computer Architecture, ISCA 2024, Buenos Aires, Argentina, June 29 - July 3, 2024 (2024), IEEE, pp. 118–132

    2024

  31. [31]

    https://en.wikipedia

    Pollaczek–khinchine formula. https://en.wikipedia. org/wiki/Pollaczekâ˘A¸ SKhinchine_formula# cite_note-2, 2025

    2025

  32. [32]

    Mooncake: Trading more stor- age for less computation — a KVCache-centric architecture for serving LLM chatbot

    QIN, R., LI, Z., HE, W., CUI, J., REN, F., ZHANG, M., WU, Y., ZHENG, W.,ANDXU, X. Mooncake: Trading more stor- age for less computation — a KVCache-centric architecture for serving LLM chatbot. In23rd USENIX Conference on File and Storage Technologies (FAST 25)(Santa Clara, CA, Feb. 2025), USENIX Association, pp. 155–170

    2025

  33. [33]

    https://qwen.ai/blog?id= 4074cca80393150c248e508aa62983f9cb7d27cd& from=research.latest-advancements-list, 2026

    Qwen3-next. https://qwen.ai/blog?id= 4074cca80393150c248e508aa62983f9cb7d27cd& from=research.latest-advancements-list, 2026

    2026

  34. [34]

    M., ZHU, T., URGAONKAR, B.,ANDSEN, S

    SAJAL, S. M., ZHU, T., URGAONKAR, B.,ANDSEN, S. Traceupscaler: Upscaling traces to evaluate systems at high load. InProceedings of the Nineteenth European Conference on Computer Systems, EuroSys 2024, Athens, Greece, April 22-25, 2024(2024), ACM, pp. 942–961

    2024

  35. [35]

    E.,ANDSTOICA, I

    SHENG, Y., CAO, S., LI, D., ZHU, B., LI, Z., ZHUO, D., GONZALEZ, J. E.,ANDSTOICA, I. Fairness in serving large language models. In18th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2024, Santa Clara, CA, USA, July 10-12, 2024(2024), A. Gavrilovska and D. B. Terry, Eds., USENIX Association, pp. 965–988

    2024

  36. [36]

    Preble: Efficient distributed prompt scheduling for LLM serving

    SRIVATSA, V., HE, Z., ABHYANKAR, R., LI, D.,AND ZHANG, Y. Preble: Efficient distributed prompt scheduling for LLM serving. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025(2025), OpenReview.net

    2025

  37. [37]

    https://github.com/vllm- project/vllm/releases/tag/v0.12.0, 2025

    vllm v0.12.0 release. https://github.com/vllm- project/vllm/releases/tag/v0.12.0, 2025

    2025

  38. [38]

    https://github.com/vllm-project/vllm/ issues/24461, 2026

    [bugfix]: Avoid unnecessary coordination for non-moe data par- allel. https://github.com/vllm-project/vllm/ issues/24461, 2026

    2026

  39. [39]

    Kvcache cache in the wild: characterizing and optimizing kvcache cache at a large cloud provider

    WANG, J., HAN, J., WEI, X., SHEN, S., ZHANG, D., FANG, C., CHEN, R., YU, W.,ANDCHEN, H. Kvcache cache in the wild: characterizing and optimizing kvcache cache at a large cloud provider. InProceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference(USA, 2025), USENIX ATC ’25, USENIX Association

    2025

  40. [40]

    Kvcache cache in the wild: Characterizing and optimizing kvcache cache at a large cloud provider

    WANG, J., HAN, J., WEI, X., SHEN, S., ZHANG, D., FANG, C., CHEN, R., YU, W.,ANDCHEN, H. Kvcache cache in the wild: Characterizing and optimizing kvcache cache at a large cloud provider. In2025 USENIX Annual Technical Conference (USENIX ATC 25)(July 2025), USENIX Association

    2025

  41. [41]

    Burst- gpt: A real-world workload dataset to optimize llm serving systems

    WANG, Y., CHEN, Y., LI, Z., KANG, X., FANG, Y., ZHOU, Y., ZHENG, Y., TANG, Z., HE, X., GUO, R.,ET AL. Burst- gpt: A real-world workload dataset to optimize llm serving systems. InProceedings of the 31st ACM SIGKDD Confer- ence on Knowledge Discovery and Data Mining V . 2(2025), pp. 5831–5841

    2025

  42. [42]

    Aegaeon: Effective GPU pooling for concurrent LLM serving on the market

    XIANG, Y., LI, X., QIAN, K., YANG, Y., ZHU, D., YU, W., ZHAI, E., LIU, X., JIN, X.,ANDZHOU, J. Aegaeon: Effective GPU pooling for concurrent LLM serving on the market. InProceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, SOSP 2025, Lotte Hotel World, Seoul, Republic of Korea, October 13-16, 2025(2025), Y . Won, Y . Kwon, D. Yuan...

    2025

  43. [43]

    Servegen: Workload characterization and generation of large language model serving in production.CoRR abs/2505.09999 (2025)

    XIANG, Y., LI, X., QIAN, K., YU, W., ZHAI, E.,ANDJIN, X. Servegen: Workload characterization and generation of large language model serving in production.CoRR abs/2505.09999 (2025)

    Pith/arXiv arXiv 2025

  44. [44]

    Blitzscale: Fast and live large model au- toscaling with O(1) host caching

    ZHANG, D., WANG, H., LIU, Y., WEI, X., SHAN, Y., CHEN, R.,ANDCHEN, H. Blitzscale: Fast and live large model au- toscaling with O(1) host caching. In19th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2025, Boston, MA, USA, July 7-9, 2025(2025), L. Zhou and Y . Zhou, Eds., USENIX Association, pp. 275–293

    2025

  45. [45]

    Cpi 2: CPU performance isolation for shared compute clusters

    ZHANG, X., TUNE, E., HAGMANN, R., JNAGAL, R., GOKHALE, V.,ANDWILKES, J. Cpi 2: CPU performance isolation for shared compute clusters. InEighth Eurosys Con- ference 2013, EuroSys ’13, Prague, Czech Republic, April 14-17, 2013(2013), Z. Hanzálek, H. Härtig, M. Castro, and M. F. Kaashoek, Eds., ACM, pp. 379–391

    2013

  46. [46]

    Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving

    ZHONG, Y., LIU, S., CHEN, J., HU, J., ZHU, Y., LIU, X., JIN, X.,ANDZHANG, H. Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving. In18th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2024, Santa Clara, CA, USA, July 10-12, 2024(2024), A. Gavrilovska and D. B. Terry, Eds., USENIX Associa...

    2024

  47. [47]

    Polyserve: Efficient multi-slo serving at scale.CoRR abs/2507.17769(2025)

    ZHU, K., SHI, H., XU, L., SHAN, J., KRISHNAMURTHY, A., KASIKCI, B.,ANDXIE, L. Polyserve: Efficient multi-slo serving at scale.CoRR abs/2507.17769(2025). 15

    Pith/arXiv arXiv 2025