Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MELL claims that migrating running LLM requests between GPUs—choosing between token transfer and KV-cache transfer on the fly—serves the same workload with up to 31% fewer GPUs, backed by an online algorithm it claims is within 4/3 of…

desk verdict Promising LLM serving system with a fixable sign error in the competitive-ratio lemma; worth engaging, but the headline proof and simulation numbers need work before acceptance. read the letter →

arxiv 2501.06709 v1 pith:VZO5F6QN submitted 2025-01-12 cs.DC

Liu Qianli , Hong Zicong , Chen Fahao , Li Peng , Guo Song This is my paper · ORCID
classification cs.DC
keywords LLMservingKVcacheGPUschedulingrequestmigrationonlinealgorithmcompetitiveratioloadimbalancememoryefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MELL is a serving system that tries to lower the number of GPUs needed for LLM inference by treating the key-value cache—the per-request memory that grows as tokens are generated—as movable load. Instead of compressing the cache or pushing it to CPU memory, it migrates running requests between GPUs when one GPU is overloaded and another has spare memory, and it chooses on the fly whether to transfer tokens or the KV cache to balance compute and communication cost. The central claim is that an online scheduler using this mechanism needs at most $4/3$ times as many GPUs as an optimal offline schedule while limiting the number of migrations. In experiments, a prototype reduces the required GPUs by 9% to 31% and raises average GPU memory utilization to 88% to 95%, compared with 65% to 82% for the baselines.

What carries the argument

The load-bearing object is an online, priority-aware scheduling algorithm over GPU categories. Requests are bucketed by KV cache size into L, M, S, and T classes relative to the per-GPU cache capacity $C$; GPUs are labelled by the largest class they host; and allocate/depart/update operations are designed to preserve packing invariants such as two M-requests per M-GPU and three S-requests per S-GPU. A companion weight function assigns values $1$, $5/6$, $1/2$, $1/3$, and $0$ to single-large, combined-large, medium, small, and tiny requests, and the proof attempts to use these weights to bound the number of GPUs needed. Around this scheduler sits the adaptive request migration mechanism, which profiles communication and computation boundaries and chooses between two migration modes—transferring tokens and re-prefilling, or transferring the KV cache directly—so that migrations stay within available link and compute capacity.

What would settle it

Take one Large KV cache of size $0.9C$ and one Medium cache of size $0.49C$; they cannot share a GPU because $0.9C+0.49C> C$, so the optimal schedule uses two GPUs, but the paper's weights give $W(I)=3/2$ and Lemma 2.2 would require $OPT(I) \le 9/8$. Checking this two-request instance against the lemma settles whether the bound holds.

Watch

Extended reading notes

Core claim

On its own terms, MELL claims that the KV cache's transient, growing footprint can be smoothed across a GPU cluster by moving running requests, and that this smoothing can be planned online. Requests are classified into four size bands, each GPU is labelled by the largest request it hosts, and the scheduler maintains packing invariants (an M-GPU holds two M-requests, an S-GPU holds three, and a T-GPU is at least 75% full) through allocate, depart, and update operations. The claimed theoretical result is that this invariant-maintaining algorithm uses at most $4/3$ times the GPU count of the optimal schedule, with a constant number of exceptions, and that each operation triggers at most ten migrations. The claimed empirical result is a prototype that serves the same workload with 9% to 31% fewer GPUs than best-fit, worst-fit, and migration-based load-balancing schedulers, with average GPU memory utilization reaching 88% to 95%.

Load-bearing premise

The 4/3 competitive-ratio claim rests on Lemma 2.2, an inequality saying the optimal schedule needs at most three-quarters of a weighted request total; the paper's own enumeration of GPU packings gives the opposite inequality, so the guarantee is not established as written.

Editorial extensions

If this is right

  • If the $4/3$ competitive ratio holds, an operator does not need future knowledge of arrivals or response lengths to stay within one-third of the offline-optimal GPU count.
  • Bounding migrations to at most ten per arrival/completion/growth event keeps load balancing affordable on PCIe and 10 Gbps interconnects.
  • Under the tested workloads, 9% to 31% fewer GPUs serve the same traffic, with the largest savings under high-frequency arrivals and on GPUs with small memory.
  • Average GPU memory utilization rises to 88% to 95%, versus 65% to 82% for the baselines, so the saved GPUs are not merely idle capacity.
  • The gain over a load-balancing scheduler that migrates without long-horizon packing is up to 15% in GPU count, showing that migration alone is not enough.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural follow-up experiment is to vary the inter-GPU fabric (fast NVLink versus commodity Ethernet) to measure how much of the reported GPU savings comes from cheap migration.
  • Because the scheduler only reads KV cache sizes, it can be layered with KV cache compression or CPU offloading; the resulting memory savings would multiply rather than compete.
  • The same migrate-or-recompute choice applies to any long-lived inference state, such as LoRA adapters or multi-turn conversation history, where moving state is cheaper than rebuilding it.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents MELL, a GPU KV-cache management system for LLM serving that opportunistically migrates requests across GPUs to reduce the number of GPUs needed. The contribution is an adaptive migration mechanism that switches between token transfer and KV-cache transfer, plus an online scheduling algorithm claimed to achieve a 4/3 competitive ratio against the optimal offline strategy while bounding the number of migrations per operation. The system is implemented on top of vLLM and evaluated via trace-driven simulation based on a small testbed, reporting 9--31% GPU reduction and 10--43% GPU utilization improvement over baselines.

Significance. If the theoretical claim of a 4/3 competitive ratio holds, it would be a valuable formal performance guarantee for online KV-cache scheduling, which most prior work treats heuristically. The adaptive migration mechanism also addresses a real system bottleneck. However, the theoretical proof contains a clear error in Lemma 2.2, and the empirical evaluation rests on simulation without confidence intervals. The core idea and system design are promising, but the central proof and experimental rigor need revision.

major comments (3)
  1. [§VII, Lemma 2.2] Lemma 2.2 states OPT(I) ≤ 3/4 W(I), but this is contradicted by Table I. Table I shows that every feasible GPU combination in an optimal solution has total weight at most 4/3, so W(I) ≤ (4/3)OPT(I), i.e., OPT(I) ≥ (3/4)W(I). The printed lemma is also false for a single L-request, where W=1 and OPT=1 violates OPT ≤ 3/4. The proof of Theorem 2 Case 2 needs W(I) ≤ (4/3)OPT(I), which follows from the corrected inequality, not from the stated one. Please correct the lemma and its proof, and verify the direction of the inequality throughout.
  2. [§VII, Lemma 2.1] The proof of Lemma 2.1 is only a sketch. The key step, "it is easy to see that at least ⌊C/2⌋ combined L-requests can fit with ⌊C/2⌋ M/S-requests," is not justified, and the constant c is never made explicit. Since Lemma 2.1 is used directly in the competitive ratio proof, a complete proof accounting for all GPU types and the constant is necessary.
  3. [§VIII] The empirical GPU-reduction and utilization claims are based on simulation of a large cluster using traces collected from a small testbed (Section VIII-B). No confidence intervals, standard deviations, or number of simulation runs are reported. Please provide statistical significance and describe how the simulation scales the testbed traces to the large cluster, so that the reported 9--31% and 10--43% ranges can be assessed.
minor comments (5)
  1. [Figure 13 caption] The caption reads "LLaMA-13B on NVIDIA V100," but the experimental setup in Section VIII-B uses A100 GPUs for LLaMA-13B; this appears to be a typo.
  2. [§III] In the sentence "Lluminx adopts a load balancing strategy," the system name should be "Llumnix" for consistency with the rest of the paper.
  3. [§VII] The symbol C is used both for GPU memory capacity in Section VI and for the number of combined L-requests in Section VII, which creates ambiguity; please use distinct notation.
  4. [§VII, Theorem 2 proof] In Case 1, "|PA(I)| ≤ 4/3 S(I)" appears to be a typo for "|A(I)| ≤ 4/3 S(I)"; please correct.
  5. [§VII, Theorem 3] Theorem 3 gives a bound of ten migrations per operation, but the proof is informal and would benefit from a more rigorous counting argument, especially for the Update case.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the 4/3 competitive-ratio claim is an analytical bound derived from structural properties, and the Lemma 2.2 direction error is a mathematical correctness issue, not a circular reduction.

full rationale

The paper's central theoretical claim is the 4/3 competitive ratio in Theorem 2, obtained from Lemma 2.1 and Lemma 2.2. The weights assigned to request types are introduced as an 'Intermediate value weight' for the proof and are bounded via Table I; this is a standard proof technique and the competitive ratio is not assumed in the definition of the algorithm or the weights. The experiments compare MELL against external baselines (BF, WF, LB) on real workloads (LMSYS-Chat-1M, WildChat, Azure traces) and report GPU counts, migration frequency, and GPU utilization; no reported prediction is defined in terms of the algorithm's own fitted parameters. The only self-citation in the paper (e.g., reference [7]) appears in a general background sentence about serving systems and does not carry any load-bearing step of the derivation or evaluation. The printed Lemma 2.2 states OPT(I) <= 3/4 W(I), while Table I actually supports the reverse inequality, so the proof as written contains a correctness defect; however, that is a flaw in the mathematical argument, not a case where a conclusion is equivalent to its inputs by construction. Therefore, no significant circularity is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

MELL introduces no new physical or abstract entities. Its design choices include manually selected request size thresholds, tunable priority weights, and profiled boundaries, all of which are parameters of the system rather than fitted constants. The main axiomatic load is the idealized system model and the unproven structural invariant maintenance.

free parameters (3)
  • Request category thresholds = C/2, C/3, C/4, C/8
    Chosen by hand to classify requests into L, M, S, and T types. The competitive ratio proof and the assigned weights depend on these specific thresholds.
  • GPU priority weight factors
    Section VI-C states that the weights for workload, idle memory, and distance in the GPU priority function are set by the LLM service provider, leaving them as tunable parameters that affect allocation decisions.
  • Offline boundary profiling values
    Section V requires profiling the communication boundary of each link and the computation boundary of each instance. These values shape the hybrid migration decision and are hardware-specific, but no measured values are reported.
assumptions (4)
  • domain assumption The GPU cluster is homogeneous and each GPU has a fixed memory capacity C for the KV cache.
    Section VI-A defines the system model this way. Real clusters often have heterogeneous GPUs, and GPU memory also holds model weights, activations, and other state, which the model ignores.
  • domain assumption The KV cache size S_t_i of each request is known to the scheduler and grows monotonically with token count.
    Section VI-A assumes S_t_i is available and non-decreasing. Finding 2 acknowledges response length is unpredictable, so the scheduler must rely on observed current sizes rather than future values.
  • domain assumption Live request migration can be performed with near-zero downtime and with bounded communication and computation overhead as described by the boundary mechanism.
    Section V builds on Llumnix-style KV cache transfer and ServerlessLLM-style token transfer, assuming these migrations do not disrupt normal serving beyond the profiled boundaries.
  • ad hoc to paper The algorithm in Figure 10 can maintain all five structural properties in Theorem 1 with only a constant number of exceptions.
    The proof of Theorem 1 asserts this with brief bullet points and does not provide a complete case analysis. This maintenance property is load-bearing for the competitive ratio bound.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management." pith.science (2026). https://pith.science/paper/VZO5F6QN

@misc{pith2026250106709,
  author       = {Pith},
  title        = {Pith review of: Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VZO5F6QN}},
  note         = {Machine review of arXiv:2501.06709}
}
read the original abstract

Serving large language models (LLMs) for massive users is challenged by the significant memory footprint of the transient state, known as the key-value (KV) cache, which scales with sequence length and number of requests. Instead of renting or buying more expensive GPUs, the load imbalance of the KV cache across GPUs, coupled with recent advances in inter-GPU communication, provides an opportunity to serve more requests via request migration. However, high migration overhead and unpredictable request patterns make it challenging. Therefore, this paper proposes MELL, a memory-efficient LLM serving system via multi-GPU KV cache management. It saves the number of GPUs needed in the system by considering the dynamic KV cache load and the costly request migration. Specifically, we first develop an adaptive request migration mechanism to balance the computational and communication overheads and adapt to diverse resource conditions. Then, we design an online algorithm tailored to a multi-LLM request and multi-GPU scheduling problem with migration enabled. It aims to minimise the required GPUs while limiting the number of migrations. Finally, we implement a prototype of MELL and demonstrate that it reduces the number of GPUs by 31% and increases the GPU utilization by 43% at most compared to existing LLM serving systems.

Figures

Figures reproduced from arXiv: 2501.06709 by the authors.

Figure 2
Figure 2. The memory cost of processing a request with [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 5
Figure 5. The distribution of response lengths under various [PITH_FULL_IMAGE:figures/full_fig_p003_5.png] view at source ↗
Figure 6
Figure 6. The request serving ratio of an LLM serving system [PITH_FULL_IMAGE:figures/full_fig_p003_6.png] view at source ↗
Figures from the paper (7 more)
Figure 9
Figure 9. Figure 9: An example of adaptive request migration. [PITH_FULL_IMAGE:figures/full_fig_p004_9.png]
Figure 8
Figure 8. Figure 8: System architecture of MELL. Finding 4. Existing migrations for the KV cache are either compute-intensive or communication-intensive [PITH_FULL_IMAGE:figures/full_fig_p004_8.png]
Figure 10
Figure 10. Figure 10: Three request operations for request i on GPU cluster J for multi-GPU KV cache scheduling. Algorithm 1: Overall Workflow Input: LLM request set I, GPU cluster J for t ∈ T do for i ∈ {i | S t i > 0 ∨ S t−1 i > 0} do if request i arrives at t then J.Allocate(i) else if …
Figure 11
Figure 11. Figure 11: The number of GPUs needed by different systems [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]
Figure 12
Figure 12. Figure 12: The migration frequency in different systems under [PITH_FULL_IMAGE:figures/full_fig_p008_12.png]
Figure 13
Figure 13. Figure 13: The performance improvement caused by request [PITH_FULL_IMAGE:figures/full_fig_p009_13.png]
Figure 15
Figure 15. Figure 15: The number of GPUs at different times in each [PITH_FULL_IMAGE:figures/full_fig_p009_15.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning

    cs.DC 2025-05 conditional novelty 7.0 of 10

    Prism uses GPU memory ballooning to dynamically reallocate memory among co-located LLMs, unifying space and time sharing and improving SLO attainment and cost efficiency.

  2. LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs

    cs.SE 2026-08 reject novelty 6.0 of 10

    An empirical GitHub mining study finds vLLM is the most adopted LLM serving framework, parallel and memory optimizations dominate, and multi-framework use is rare.

  3. A Survey on Large Language Model Acceleration based on KV Cache Management

    cs.AI 2024-12 conditional novelty 4.0 of 10

    A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.

Reference graph

Works this paper leans on

44 extracted references · 38 canonical work pages · cited by 3 Pith papers

  1. [1]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amode...

    work page 2020

  2. [2]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report,” 2024. [Online]. Available: https: //arxiv.org/abs/2303.08774

    arXiv 2024

  3. [3]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” 2023. [Online]. Available: https: //arxiv.org/abs/2302.13971

    arXiv 2023

  4. [4]

    Characterization of large language model development in the datacenter,

    Q. Hu, Z. Ye, Z. Wang, G. Wang, M. Zhang, Q. Chen, P. Sun, D. Lin, X. Wang, Y . Luo, Y . Wen, and T. Zhang, “Characterization of large language model development in the datacenter,” inNSDI, 2024, pp. 709– 729

    work page 2024

  5. [5]

    Orca: A distributed serving system for Transformer-Based generative models,

    G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for Transformer-Based generative models,” in OSDI, 2022, pp. 521–538

    work page 2022

  6. [6]

    Splitwise: Efficient generative llm inference using phase splitting,

    P. Patel, E. Choukse, C. Zhang, A. Shah, I. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative llm inference using phase splitting,” in ISCA, June 2024

    work page 2024

  7. [7]

    Optimus: Warming serverless ml inference via inter-function model transformation,

    Z. Hong, J. Lin, S. Guo, S. Luo, W. Chen, R. Wattenhofer, and Y . Yu, “Optimus: Warming serverless ml inference via inter-function model transformation,” in EuroSys. Association for Computing Machinery, 2024, p. 1039–1053

    work page 2024

  8. [8]

    Otas: An elastic transformer serving system via token adaptation,

    J. Chen, W. Xu, Z. Hong, S. Guo, H. Wang, J. Zhang, and D. Zeng, “Otas: An elastic transformer serving system via token adaptation,” in INFOCOM, 2024, pp. 1–10

    work page 2024

Show all 44 references
  1. [9]

    Galaxy: A resource-efficient collaborative edge ai system for in-situ transformer inference,

    S. Ye, J. Du, L. Zeng, W. Ou, X. Chu, Y . Lu, and X. Chen, “Galaxy: A resource-efficient collaborative edge ai system for in-situ transformer inference,” in INFOCOM, 2024, pp. 1–10

    2024

  2. [10]

    Efficiently scaling transformer inference,

    R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean, “Efficiently scaling transformer inference,” in MLSys, 2023, pp. 606–624

    2023

  3. [11]

    LongloRA: Efficient fine-tuning of long-context large language models,

    Y . Chen, S. Qian, H. Tang, X. Lai, Z. Liu, S. Han, and J. Jia, “LongloRA: Efficient fine-tuning of long-context large language models,” in ICLR, 2024

    2024

  4. [12]

    Efficient memory management for large lan- guage model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large lan- guage model serving with pagedattention,” in SOSP, 2023, p. 611–626

    2023

  5. [13]

    Keyformer: Kv cache reduction through key tokens selection for efficient generative inference,

    M. Adnan, A. Arunkumar, G. Jain, P. Nair, I. Soloveychik, and P. Ka- math, “Keyformer: Kv cache reduction through key tokens selection for efficient generative inference,” in MLSys, 2024, pp. 114–127

    2024

  6. [14]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models,

    Z. Zhang, Y . Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y . Tian, C. Re, C. Barrett, Z. Wang, and B. Chen, “H2o: Heavy-hitter oracle for efficient generative inference of large language models,” in NIPS, 2023

    2023

  7. [15]

    Efficient streaming language models with attention sinks,

    G. Xiao, Y . Tian, B. Chen, S. Han, and M. Lewis, “Efficient streaming language models with attention sinks,” in ICLR, 2024

    2024

  8. [16]

    Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time,

    Z. Liu, A. Desai, F. Liao, W. Wang, V . Xie, Z. Xu, A. Kyrillidis, and A. Shrivastava, “Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time,” in NIPS, 2023

    2023

  9. [17]

    Kivi: A tuning-free asymmetric 2bit quantization for kv cache,

    Z. Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, V . Braverman, B. Chen, and X. Hu, “Kivi: A tuning-free asymmetric 2bit quantization for kv cache,” arXiv preprint arXiv:2402.02750 , 2024

    2024 arXiv

  10. [18]

    Model tells you what to discard: Adaptive KV cache compression for LLMs,

    S. Ge, Y . Zhang, L. Liu, M. Zhang, J. Han, and J. Gao, “Model tells you what to discard: Adaptive KV cache compression for LLMs,” in ICLR, 2024

    2024

  11. [19]

    Flexgen: high-throughput generative inference of large language models with a single gpu,

    Y . Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. R ´e, I. Stoica, and C. Zhang, “Flexgen: high-throughput generative inference of large language models with a single gpu,” in ICML, 2023

    2023

  12. [20]

    Infinigen: Efficient generative in- ference of large language models with dynamic kv cache management,

    W. Lee, J. Lee, J. Seo, and J. Sim, “Infinigen: Efficient generative in- ference of large language models with dynamic kv cache management,” in OSDI, 2024, pp. 155–172

    2024

  13. [21]

    Cost-Efficient large language model serving for multi-turn conversations with CachedAttention,

    B. Gao, Z. He, P. Sharma, Q. Kang, D. Jevdjic, J. Deng, X. Yang, Z. Yu, and P. Zuo, “Cost-Efficient large language model serving for multi-turn conversations with CachedAttention,” in ATC, 2024, pp. 111–126

    2024

  14. [22]

    Deepspeed- inference: enabling efficient inference of transformer models at unprece- dented scale,

    R. Y . Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng, O. Ruwase, S. Smith, M. Zhang, J. Rasley, and Y . He, “Deepspeed- inference: enabling efficient inference of transformer models at unprece- dented scale,” in SC, 2022

    2022

  15. [23]

    Llumnix: Dynamic scheduling for large language model serving,

    B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, Y . Li, and W. Lin, “Llumnix: Dynamic scheduling for large language model serving,” in OSDI, 2024, pp. 173–191

    2024

  16. [24]

    Serverlessllm: Low-latency serverless inference for large language models,

    Y . Fu, L. Xue, Y . Huang, A.-O. Brabete, D. Ustiugov, Y . Patel, and L. Mai, “Serverlessllm: Low-latency serverless inference for large language models,” in OSDI, 2024, pp. 135–153

    2024

  17. [25]

    Turbotransformers: an efficient gpu serving system for transformer models,

    J. Fang, Y . Yu, C. Zhao, and J. Zhou, “Turbotransformers: an efficient gpu serving system for transformer models,” in PPoPP, 2021, p. 389–402

    2021

  18. [26]

    Taming throughput-latency tradeoff in llm inference with sarathi-serve,

    A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. Gulavani, A. Tumanov, and R. Ramjee, “Taming throughput-latency tradeoff in llm inference with sarathi-serve,” in OSDI, 2024, pp. 117–134

    2024

  19. [27]

    Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving,

    Y . Zhong, S. Liu, J. Chen, J. Hu, Y . Zhu, X. Liu, and X. Jin, “Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving,” in OSDI, 2024, pp. 193–210

    2024

  20. [28]

    Inference without interference: Disaggregate llm inference for mixed downstream workloads,

    C. Hu, H. Huang, L. Xu, X. Chen, J. Xu, S. Chen, H. Feng, C. Wang, S. Wang, Y . Bao, N. Sun, and Y . Shan, “Inference without interference: Disaggregate llm inference for mixed downstream workloads,” 2024. [Online]. Available: https://arxiv.org/abs/2401.11181

    2024 arXiv

  21. [29]

    dLoRA: Dynam- ically orchestrating requests and adapters for LoRA LLM serving,

    B. Wu, R. Zhu, Z. Zhang, P. Sun, X. Liu, and X. Jin, “dLoRA: Dynam- ically orchestrating requests and adapters for LoRA LLM serving,” in OSDI, 2024, pp. 911–927

    2024

  22. [30]

    LMSYS- chat-1m: A large-scale real-world LLM conversation dataset,

    L. Zheng, W.-L. Chiang, Y . Sheng, T. Li, S. Zhuang, Z. Wu, Y . Zhuang, Z. Li, Z. Lin, E. Xing, J. E. Gonzalez, I. Stoica, and H. Zhang, “LMSYS- chat-1m: A large-scale real-world LLM conversation dataset,” in ICLR, 2024

    2024

  23. [31]

    Wildchat: 1m chatGPT interaction logs in the wild,

    W. Zhao, X. Ren, J. Hessel, C. Cardie, Y . Choi, and Y . Deng, “Wildchat: 1m chatGPT interaction logs in the wild,” in ICLR, 2024

    2024

  24. [32]

    Judging LLM-as-a-judge with MT-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-judge with MT-bench and chatbot arena,” in NIPS Datasets and Benchmarks Track, 2023

    2023

  25. [33]

    Koala: A dialogue model for academic research,

    X. Geng, A. Gudibande, H. Liu, E. Wallace, P. Abbeel, S. Levine, and D. Song, “Koala: A dialogue model for academic research,” Blog post, April 2023. [Online]. Available: https://bair.berkeley.edu/blog/2023/04/ 03/koala/

    2023

  26. [34]

    Response length perception and sequence scheduling: An LLM-empowered LLM inference pipeline,

    Z. Zheng, X. Ren, F. Xue, Y . Luo, X. Jiang, and Y . You, “Response length perception and sequence scheduling: An LLM-empowered LLM inference pipeline,” in NIPS, 2023

    2023

  27. [35]

    R. M. Karp, Reducibility among Combinatorial Problems . Springer US, 1972, pp. 85–103. [Online]. Available: https://doi.org/10.1007/ 978-1-4684-2001-2 9

    1972

  28. [36]

    Adaptive resource provi- sioning for the cloud using online bin packing,

    W. Song, Z. Xiao, Q. Chen, and H. Luo, “Adaptive resource provi- sioning for the cloud using online bin packing,” IEEE Transactions on Computers, vol. 63, no. 11, pp. 2647–2660, 2014

    2014

  29. [37]

    Efficient online strategies for renting servers in the cloud,

    S. Kamali and A. L ´opez-Ortiz, “Efficient online strategies for renting servers in the cloud,” in SOFSEM, 2015, pp. 277–288

    2015

  30. [38]

    Powernap: eliminating server idle power,

    D. Meisner, B. T. Gold, and T. F. Wenisch, “Powernap: eliminating server idle power,” SIGARCH Comput. Archit. News , vol. 37, no. 1, p. 205–216, mar 2009. [Online]. Available: https://doi.org/10.1145/ 2528521.1508269

    2009

  31. [39]

    Easy, fast, and cheap llm serving for everyone,

    vLLM, “Easy, fast, and cheap llm serving for everyone,” 2024. [Online]. Available: https://github.com/vllm-project/vllm

    2024

  32. [40]

    Ray: a unified framework for scaling ai and python applications,

    Ray, “Ray: a unified framework for scaling ai and python applications,”

  33. [41]

    Gloo: Collective communications library with various primitives for multi-machine training,

    Meta, “Gloo: Collective communications library with various primitives for multi-machine training,” 2024. [Online]. Available: https://github. com/facebookincubator/gloo

    2024

  34. [42]

    Openai platform document,

    OpenAI, “Openai platform document,” 2024. [Online]. Available: https://platform.openai.com/docs/models

    2024

  35. [43]

    Anthropic platform document,

    Anthropic, “Anthropic platform document,” 2024. [Online]. Available: https://docs.anthropic.com/en/docs/about-claude/models

    2024

  36. [2024]

    Available: https://github.com/ray-project/ray

    [Online]. Available: https://github.com/ray-project/ray

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.