Pith. sign in

REVIEW 2 minor 101 references

Breaking the Ice: Analyzing Cold Start Latency in vLLM

T0 review · 0 major / 2 minor · reviewed 2026-06-30 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read vLLM cold-start latency decomposes into six CPU-bound steps whose times a lightweight model predicts from hardware details.

desk verdict This paper gives the first systematic breakdown of vLLM startup latency into six steps along with an open-sourced analytical predictor. read the letter →

arxiv 2606.07362 v3 pith:ZV6ZK6LX submitted 2026-06-05 cs.LG

Huzaifa Shaaban Kabakibo , Animesh Trivedi , Lin Wang This is my paper · ORCID
classification cs.LG
keywords vLLMcoldstartlatencyinferenceengineanalyticalmodelperformancecharacterizationstartupCPU-bound
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper measures the time for vLLM to become ready after first loading a model. It splits that time into six distinct steps and shows each one grows in a regular way with model size and the number of CPU cores or memory available. The measurements establish that the delays come mostly from CPU operations rather than GPU work. From the observed patterns the authors build a simple formula that estimates total startup time on any given machine. The formula supplies concrete numbers for deciding how many servers to provision when running many inference instances at once.

What carries the argument

The six-step breakdown of vLLM initialization together with the lightweight analytical model built from the observed scaling trends.

What would settle it

Measure actual startup latency on a hardware configuration and model size not used to build the model, then check whether the analytical prediction matches the measured value within a small error bound.

Watch

Extended reading notes

Core claim

vLLM's startup process decomposes into six foundational steps that are predominantly CPU-bound, each exhibiting consistent and interpretable scaling trends with model- and system-level parameters, enabling a lightweight analytical model to predict startup latency accurately for a given hardware configuration.

Load-bearing premise

The startup process can be decomposed into six foundational steps that exhibit consistent and interpretable scaling trends with respect to model- and system-level parameters.

Editorial extensions

If this is right

  • Resource planners can use the model to estimate how long new inference instances will take to become ready.
  • Latency contributions can be traced to individual steps, guiding targeted hardware or software changes.
  • Large inference services gain concrete numbers for deciding server counts and model placement.
  • The same scaling trends support repeated predictions when models or machines change.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The step-wise measurement method could be applied to other LLM serving systems to produce comparable prediction models.
  • Forecasts from the model could inform autoscaling policies that preload engines before demand spikes.
  • In serverless inference the same predictions would quantify the cost penalty of cold starts.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 2 minor

Summary. The manuscript presents the first detailed performance characterization of vLLM startup latency. It decomposes the startup process into six foundational steps, demonstrates that the process is predominantly CPU-bound, shows consistent and interpretable scaling trends with respect to model- and system-level parameters, and develops a lightweight analytical model that accurately predicts vLLM's startup latency for a given hardware configuration. All benchmarking datasets, analysis tools, and prediction scripts are open-sourced.

Significance. If the predictive accuracy holds as asserted via the open-sourced artifacts, the work supplies actionable guidance for resource planning in large-scale inference environments through fine-grained latency attribution. The open-sourcing of datasets, tools, and scripts is a clear strength that enables reproducibility and independent verification.

minor comments (2)
  1. [Abstract] Abstract: while the claim of accurate prediction is central, the abstract does not include any quantitative error metrics or validation details; consider adding a short statement on measured prediction accuracy (e.g., average error across tested configurations) to make the claim more self-contained.
  2. The six-step decomposition is described as foundational; a summary table listing each step, its dominant resource (CPU/GPU), and key scaling parameter would improve readability and allow quicker cross-reference with the scaling trends.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive assessment of our manuscript on vLLM cold-start latency characterization, the recognition of its novelty in decomposing the process into six steps, confirming its CPU-bound nature, identifying scaling trends, and developing a predictive model, as well as the value placed on our open-sourced artifacts. The minor revision recommendation is noted. No specific major comments were raised in the report.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; model derives from independent empirical decomposition

full rationale

The paper decomposes startup into six steps, reports observed scaling trends with model/system parameters, and builds a lightweight analytical model on those trends. No equations, self-citations, or uniqueness claims are provided that reduce the predictor to a re-expression of fitted inputs by construction. The derivation remains self-contained against external benchmarks because the scaling observations and model accuracy are asserted via open-sourced datasets and scripts rather than internal tautology. This is the normal case of an empirical study whose central claim does not collapse to its own inputs.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Abstract-only review yields no explicit free parameters, axioms, or invented entities; the model is described as 'lightweight analytical' built on scaling trends, but the concrete parameters or assumptions inside it are not visible.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Breaking the Ice: Analyzing Cold Start Latency in vLLM." pith.science (2026). https://pith.science/paper/ZV6ZK6LX

@misc{pith2026260607362,
  author       = {Pith},
  title        = {Pith review of: Breaking the Ice: Analyzing Cold Start Latency in vLLM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZV6ZK6LX}},
  note         = {Machine review of arXiv:2606.07362}
}
read the original abstract

As scalable inference services become popular, the cold start latency of an inference engine becomes important. Today, vLLM has evolved into the de-facto inference engine of choice for many inference workloads. Although popular, due to its complexity and rapid evolution, there has not been a systematic study on the startup latency of its engine. With major architectural innovations under it (e.g., the V1 API, introduction of torch.compile), in this paper, we present the first detailed performance characterization of vLLM startup latency. We break down the startup process into six foundational steps and demonstrate that this process is predominantly CPU-bound. Each step exhibits consistent and interpretable scaling trends with respect to model- and system-level parameters, enabling fine-grained attribution of latency sources. Building on these insights, we develop a lightweight analytical model that accurately predicts vLLM's startup latency for a given hardware configuration, providing actionable guidance for resource planning in large-scale inference environments. All our benchmarking datasets, analysis tools, and prediction scripts are open-sourced at https://github.com/upb-cn/vllm-startup-profiler

Figures

Figures reproduced from arXiv: 2606.07362 by the authors.

Figure 1
Figure 1. Startup times of different vLLM versions using the OPT￾6.7B model on an H100 GPU (lower is better). widely used inference engines, evolving quickly through frequent, community-driven releases (Kwon et al., 2023; Nar et al., 2025). It delivers a highly optimized infer￾ence path (Gordic, 2025) with techniques such as Page￾dAttention and prefix caching (Kwon et al., 2023), chun￾ked prefill (Agrawal et al., 2023), disag… view at source ↗
Figure 2
Figure 2. vLLM startup latency breakdown with Llama3.2-3B. isolate the core system behavior. We decompose the startup process into six foundational steps and identify that the overall process is largely CPU-bounded. By examining the scaling characteristics of each step, we uncover consistent and interpretable relationships among model configuration, system environment, and startup latency. Leveraging these insights, we develo… view at source ↗
Figure 4
Figure 4. Strong linear relationship between the model size and loading weights time. pretrained parameters (the weights for the attention block, FFN, etc.) from checkpoint files to GPU memory. Fig￾ure 4 shows the latency of this step across a variety of models loaded with FP16 format. The size of the model depends primarily on the number of model parameters and the numeric precision used to store them. In the figure, the par… view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: Strong linear relationship between compiled graphs size (shown in parentheses, in KB) and Dynamo transformation time. a major optimization milestone. It leverages PyTorch’s com￾pilation infrastructure to convert Python-level execution into optimized, low-level kernels,…
Figure 8
Figure 8. Figure 8: Strong linear relationship between model size and CUDA graph capturing time. with model size (PCC = 0.92), except for MoE models. Non￾MoE models show a strong linear dependency on model size, consistent with expectations, since this step performs a dummy forward pass w…
Figure 9
Figure 9. Figure 9: CUDA graph capturing time for different batch sizes using the Llama2-7B (Touvron et al., 2023) model. where we measure CUDA capturing time for different mod￾els with different sizes. We observe that as the model size in￾creases, so does the capturing time confirming a …
Figure 12
Figure 12. Figure 12: CPU usage per core over time during vLLM startup for the Qwen-4B model. Sampling interval: 100 ms. Overall, while these results confirm that CPU choice has a substantial effect on the vLLM startup latency, a pre￾cise attribution of the performance differences to the C…
Figure 13
Figure 13. Figure 13: Impact of running the startup process while the model weights are retrieved from storage (SSD). Llama2-13B Yi-6B Llama2-7B Falcon-7B 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Model Loading Time (s) 3.73 1.80 1.98 2.03 2.83 1.70 1.50 1.84 1.98 0.98 1.07 1.22 Safetensors Coreweav…
Figure 14
Figure 14. Figure 14: Model loading times across four models using different loading backends. model tensors as pre-serialized binary files that are memory￾mapped and loaded directly into CPU memory before GPU transfers. (ii) Run:ai Model Streamer enables concurrent reading and streaming o…
Figure 16
Figure 16. Figure 16: Workflow of the proposed predictor. Falcon-11B Mistral-7B Gemma-7B Llama3-3B Qwen-7B 0 5 10 15 20 25 30 vLLM's Startup Latency (s) 31.33 24.53 21.91 21.54 23.81 33.01 23.65 23.74 23.62 24.74 Predicted Truth [PITH_FULL_IMAGE:figures/full_fig_p011_16.png]
Figure 17
Figure 17. Figure 17: Validation of the predictor against measured startup latency across different models. of optimizations that reuse parts of the initialization path. For instance, we confirm that the predictor can be used to estimate the performance of fast model re-initialization mech…
Figure 1
Figure 1. Figure 1 [PITH_FULL_IMAGE:figures/full_fig_p018_1.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

101 extracted references · 101 canonical work pages

  1. [1]

    Proceedings of the 22nd USENIX Symposium on Networked Systems Design and Implementation , articleno =

    Khare, Alind and Garg, Dhruv and Kalra, Sukrit and Grandhi, Snigdha and Stoica, Ion and Tumanov, Alexey , title =. Proceedings of the 22nd USENIX Symposium on Networked Systems Design and Implementation , articleno =. 2025 , isbn =

    work page 2025

  2. [2]

    7 Top Large Language Model Use Cases And Applications , year =

    work page

  3. [3]

    Best applications of large language models , year =

    work page

  4. [4]

    Real-World Use Cases for Large Language Models (LLMs) , year =

    work page

  5. [5]

    Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 , pages=

    Medusa: Accelerating serverless LLM inference with materialization , author=. Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 , pages=

    work page

  6. [6]

    Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V

    Burstgpt: A real-world workload dataset to optimize llm serving systems , author=. Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2 , pages=

    work page

  7. [7]

    2025 , howpublished=

    Towards Swift Serverless LLM Cold Starts with ParaServe , author=. 2025 , howpublished=

    work page 2025

  8. [8]

    23rd USENIX Conference on File and Storage Technologies (FAST 25) , year =

    Ruoyu Qin and Zheming Li and Weiran He and Jialei Cui and Feng Ren and Mingxing Zhang and Yongwei Wu and Weimin Zheng and Xinran Xu , title =. 23rd USENIX Conference on File and Storage Technologies (FAST 25) , year =

    work page

Show all 101 references
  1. [9]

    18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , pages=

    ServerlessLLM: Low-Latency serverless inference for large language models , author=. 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , pages=

  2. [10]

    Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference , articleno =

    Hu, Junhao and Xu, Jiang and Liu, Zhixia and He, Yulong and Chen, Yuetao and Xu, Hao and Liu, Jiang and Meng, Jie and Zhang, Baoquan and Wan, Shining and Dan, Gengyuan and Dong, Zhiyu and Ren, Zhihao and Liu, Changhong and Xie, Tao and Lin, Dayun and Zhang, Qin and Yu, Yue and...

    2025

  3. [11]

    Faastube: Optimizing gpu-oriented data transfer for serverless computing , author=

  4. [12]

    2018 USENIX annual technical conference (USENIX ATC 18) , pages=

    SOCK: Rapid task provisioning with Serverless-Optimized containers , author=. 2018 USENIX annual technical conference (USENIX ATC 18) , pages=

    2018

  5. [13]

    Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems , pages=

    Catalyzer: Sub-millisecond startup for serverless computing with initialization-less booting , author=. Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems , pages=

  6. [14]

    18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , pages=

    Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve , author=. 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , pages=

  7. [15]

    2018 USENIX annual technical conference (USENIX ATC 18) , pages=

    SAND: towards High-Performance serverless computing , author=. 2018 USENIX annual technical conference (USENIX ATC 18) , pages=

    2018

  8. [16]

    Proceedings of the 26th ACM international conference on architectural support for programming languages and operating systems , pages=

    Faascache: keeping serverless computing alive with greedy-dual caching , author=. Proceedings of the 26th ACM international conference on architectural support for programming languages and operating systems , pages=

  9. [17]

    2022 USENIX annual technical conference (USENIX ATC 22) , pages=

    Help rather than recycle: Alleviating cold startup in serverless computing through Inter-Function container sharing , author=. 2022 USENIX annual technical conference (USENIX ATC 22) , pages=

    2022

  10. [18]

    Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems , pages=

    Icebreaker: Warming serverless functions better with heterogeneity , author=. Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems , pages=

  11. [19]

    Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 , pages=

    Rainbowcake: Mitigating cold-starts in serverless with layer-wise container caching and sharing , author=. Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 , pages=

  12. [20]

    Proceedings of the Twentieth European Conference on Computer Systems , pages=

    Stateful large language model serving with pensieve , author=. Proceedings of the Twentieth European Conference on Computer Systems , pages=

  13. [21]

    Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=

    Efficient Memory Management for Large Language Model Serving with PagedAttention , author=. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=

  14. [22]

    2025 , note =

    vLLM 2024 Retrospective and 2025 Vision , author =. 2025 , note =

    2024

  15. [23]

    Pre-Deployment Profiling — NVIDIA Dynamo Documentation , howpublished =

  16. [24]

    2025 , howpublished =

    LLM-D , title =. 2025 , howpublished =

    2025

  17. [25]

    2025 , note =

    vLLM V1: A Major Upgrade to vLLM's Core Architecture , author =. 2025 , note =

    2025

  18. [26]

    2025 , note =

    Introduction to torch.compile and How It Works with vLLM , author =. 2025 , note =

    2025

  19. [27]

    Deprecation of vLLM V0 , author =

  20. [28]

    Improve startup time UX in vLLM , author =

  21. [29]

    2022 , eprint=

    OPT: Open Pre-trained Transformer Language Models , author=. 2022 , eprint=

    2022

  22. [30]

    2025 , howpublished =

    vLLM Team , title =. 2025 , howpublished =

    2025

  23. [31]

    Aleksa Gordic , howpublished =

  24. [32]

    NVIDIA H100 Tensor Core GPU , author =

  25. [33]

    NVIDIA L40S GPU , author =

  26. [34]

    2023 , note =

    LLaMA 3.2 - 3B Model , howpublished =. 2023 , note =

    2023

  27. [35]

    Llama 2: Open foundation and fine-tuned chat models , author=

  28. [36]

    2023 , howpublished =

    tiiuae , title =. 2023 , howpublished =

    2023

  29. [37]

    Falcon2-11b technical report , author=

  30. [38]

    Qwen Technical Report , author=

  31. [39]

    2024 , howpublished =

    Qwen , title =. 2024 , howpublished =

    2024

  32. [40]

    2024 , howpublished =

    Google , title =. 2024 , howpublished =

    2024

  33. [41]

    2024 , eprint=

    Yi: Open Foundation Models by 01.AI , author=. 2024 , eprint=

    2024

  34. [42]

    2023 , eprint=

    Mistral 7B , author=. 2023 , eprint=

    2023

  35. [43]

    2023 , howpublished =

    MosaicML NLP Team , title =. 2023 , howpublished =

    2023

  36. [44]

    2025 , eprint=

    gpt-oss-120b and gpt-oss-20b Model Card , author=. 2025 , eprint=

    2025

  37. [45]

    2024 , eprint=

    OLMoE: Open Mixture-of-Experts Language Models , author=. 2024 , eprint=

    2024

  38. [46]

    2024 , eprint=

    DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model , author=. 2024 , eprint=

    2024

  39. [47]

    2025 , eprint=

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning , author=. 2025 , eprint=

    2025

  40. [48]

    2024 , eprint=

    Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs , author=. 2024 , eprint=

    2024

  41. [49]

    vLLM Project , howpublished =

  42. [50]

    torch.compiler overview , year =

  43. [51]

    Introduction to torch.compile , year =

  44. [52]

    torch.compile integration , year =

  45. [53]

    torch.compiler , year =

  46. [54]

    Dynamo Overview , year =

  47. [55]

    What’s Behind PyTorch 2.0? TorchDynamo and TorchInductor (primarily for developers) , year =

  48. [56]

    Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 , pages=

    Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation , author=. Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 , pages=

  49. [57]

    Dynamo Deep-Dive , year =

  50. [58]

    Writing Graph Transformations on ATen IR , year =

  51. [59]

    NVIDIA: Getting Started with CUDA Graphs , year =

  52. [60]

    A100 vs H100 vs L40S: A simple side-by-side and how to decide , year =

  53. [61]

    L40s: Power Meets Versatility , year =

    NVIDIA H100 vs. L40s: Power Meets Versatility , year =

  54. [62]

    Safetesnors: ML Safer for All , year =

  55. [63]

    Run:ai Model Streamer , year =

  56. [64]

    CoreWeave's Tensorizer: Module, Model, and Tensor Serialization/Deserialization , year =

  57. [65]

    Proceedings of the 26th Symposium on Operating Systems Principles , pages=

    Resource central: Understanding and predicting workloads for improved resource management in large cloud platforms , author=. Proceedings of the 26th Symposium on Operating Systems Principles , pages=

  58. [66]

    21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24) , pages=

    Characterization of large language model development in the datacenter , author=. 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24) , pages=

  59. [67]

    Gyges: Dynamic Cross-Instance Parallelism Transformation for Efficient LLM Inference , author=

  60. [68]

    Efficient Function-as-a-Service for Large Language Models with TIDAL , author=

  61. [69]

    CSGO: Generalized Optimization for Cold Start in Wireless Collaborative Edge LLM Systems , author=

  62. [70]

    2025 , howpublished =

    NVIDIA CUDA C Programming Guide , author =. 2025 , howpublished =

    2025

  63. [71]

    13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18) , pages =

    Ray: A Distributed Framework for Emerging AI Applications , author =. 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18) , pages =. 2018 , address =

    2018

  64. [72]

    ACM Computing Surveys , volume=

    Cold start latency in serverless computing: A systematic review, taxonomy, and future directions , author=. ACM Computing Surveys , volume=. 2024 , publisher=

    2024

  65. [73]

    Applied Sciences , volume=

    The Hybrid Model: Prediction-Based Scheduling and Efficient Resource Management in a Serverless Environment , author=. Applied Sciences , volume=. 2025 , publisher=

    2025

  66. [74]

    2020 , publisher=

    Interpretable machine learning , author=. 2020 , publisher=

    2020

  67. [75]

    , author=

    The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. , author=. Queue , volume=. 2018 , publisher=

    2018

  68. [76]

    2022 IEEE International black sea conference on communications and networking (BlackSeaCom) , pages=

    A time series forecasting approach to minimize cold start time in cloud-serverless platform , author=. 2022 IEEE International black sea conference on communications and networking (BlackSeaCom) , pages=. 2022 , organization=

    2022

  69. [77]

    Taming Cold Starts: Proactive Serverless Scheduling with Model Predictive Control , author=

  70. [78]

    2025 , note =

    pepy.tech , title =. 2025 , note =

    2025

  71. [79]

    NVIDIA Dynamo: Adaptive Load Planning and GPU Worker Autoscaling , year =

  72. [80]

    NVIDIA Dynamo Platform , year =

  73. [81]

    LLM-D: Distributed Large Language Model Deployment Framework , year =

  74. [82]

    2025 , howpublished =

    AIBrix , title =. 2025 , howpublished =

    2025

  75. [83]

    2025 , howpublished =

    vLLM Production Stack , title =. 2025 , howpublished =

    2025

  76. [84]

    Advances in Neural Information Processing Systems , volume=

    Toward efficient inference for mixture of experts , author=. Advances in Neural Information Processing Systems , volume=

  77. [85]

    A comprehensive survey of mixture-of-experts: Algorithms, theory, and applications , author=

  78. [86]

    2025 , howpublished =

    Baladithya Balamurugan and Anton Alexander and Arun Raman and Kshitiz Gupta and Wenhan Tan and Brian Kreitzer and Eliuth Triana Isaza and Harish Rao and Jiahong Liu , title =. 2025 , howpublished =

    2025

  79. [87]

    Performance Analysis , author =

  80. [88]

    First call to llama model takes too much time , author =

  81. [89]

    vLLM-compile warm-start time should be close to zero , author =

  82. [90]

    Improve startup time UX , author =

  83. [91]

    Add opentelemetry tracing for vLLM start up phases , author =

  84. [92]

    Add a script to benchmark compilation time , author =

  85. [93]

    Zero-Reload Model Switching with vLLM Sleep Mode , year =

  86. [94]

    2025 , howpublished =

    LLM-D Incubation , title =. 2025 , howpublished =

    2025

  87. [95]

    2026 , howpublished =

    Hex , title =. 2026 , howpublished =

    2026

  88. [96]

    Prism: Unleashing gpu sharing for cost-efficient multi-llm serving , author=

  89. [97]

    2024 , eprint=

    DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving , author=. 2024 , eprint=

    2024

  90. [98]

    18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , year=

    Llumnix: Dynamic Scheduling for Large Language Model Serving , author=. 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , year=

  91. [99]

    Nar and Greg Pereira and Yuan Tang and Robert Shaw and Anish Asthana , title =

    Fatih E. Nar and Greg Pereira and Yuan Tang and Robert Shaw and Anish Asthana , title =. 2025 , url =

    2025

  92. [100]

    Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills , author=

  93. [101]

    16th USENIX symposium on operating systems design and implementation (OSDI 22) , pages=

    Orca: A distributed serving system for \ Transformer-Based \ generative models , author=. 16th USENIX symposium on operating systems design and implementation (OSDI 22) , pages=

Pith tools

Reviewed June 30, 2026 · model on record in the stance chip above.