Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloud

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read This review argues that ServerlessLLM's multi-tier checkpoint loading, token-only live migration, and startup-time-aware scheduling cut LLM cold starts by up to 8.2x and bring OPT-6.7B startup down to 0.8 seconds.

desk verdict A faithful but unoriginal reading-note on ServerlessLLM; the headline numbers are borrowed, and the paper never pins down what it counts as cold start. read the letter →

arxiv 2411.15664 v1 pith:JJP4RD4X submitted 2024-11-23 cs.DC cs.LG

classification cs.DCcs.LG
keywords serverlesscomputingcoldstartlatencylargelanguagemodelsLLMcheckpointloadinglivemigrationmodelschedulingFaaS
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is a review of approaches to the cold-start problem in serverless LLM inference, and its central contention is that ServerlessLLM's combination of mechanisms makes the problem tractable. Large LLM checkpoints, which can be hundreds of gigabytes, take tens of seconds to download and load, destroying interactive response times. ServerlessLLM attacks this by caching checkpoints in the GPU server's underused DRAM and SSDs, loading them in parallel chunks with direct I/O and pinned memory, and, when a model must move, migrating only the token stream instead of the large KV-cache. The review reports that this cuts model loading up to 8.2x for LLaMA-2-70B and starts OPT-6.7B in 0.8 seconds, which would make serverless LLM serving practical at interactive latencies without keeping many idle GPUs warm.

What carries the argument

The load-bearing machinery is ServerlessLLM's multi-tier storage hierarchy: model checkpoints sit on NVMe or SATA SSDs, are staged through DRAM, and land in GPU memory, with a loading-optimized checkpoint format that stores tensors in sequential, chunk-readable partitions. Parallel chunk-based loading, direct I/O that bypasses the OS cache, pinned memory, and pre-computed tensor addresses maximize PCIe bandwidth. The live-migration mechanism transfers only the token sequence, on the order of tens to hundreds of kilobytes, and recomputes the KV-cache on the destination GPU instead of shipping a multi-gigabyte cache. The scheduler estimates startup as $q + n/b$ for loading and as $a(t_{\text{in}}+t_{\text{out}})+b$ for migration resuming, then selects the server with the lowest estimate. These three mechanisms together carry the cold-start reduction claim.

What would settle it

Rerun ServerlessLLM on the hardware described in the review (8 NVIDIA A5000 GPUs with 1 TB DDR4 and MinIO storage; 4 A40 servers at 10 Gbps) and measure end-to-end startup for OPT-6.7B, OPT-30B, and LLaMA-2-70B under an Azure-like trace; if OPT-6.7B does not start near 0.8 seconds, OPT-30B not near 7.5 seconds, or LLaMA-2-70B loading not near 8.2x faster than PyTorch and SafeTensors, the central claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that ServerlessLLM's three mechanisms together remove the dominant cold-start cost of serverless LLM inference. A loading-optimized checkpoint format with parallel chunk-based reads and pre-computed tensor addresses lets checkpoints stream from NVMe SSDs through DRAM into GPU memory at near storage bandwidth. Live migration transfers only the intermediate tokens and recomputes the KV-cache on the destination GPU, so a model can move servers without a full checkpoint reload. A startup-time-optimized scheduler estimates loading time as $q + n/b$ and resuming time as $a(t_{\text{in}}+t_{\text{out}})+b$, then picks the server with the lowest estimate. The review transcribes the reported results: up to 6x faster loading for OPT-2.7B, up to 8.2x for LLaMA-2-70B, an end-to-end startup of 0.8 seconds for OPT-6.7B, and 28x faster startup for OPT-30B versus Ray Serve.

Load-bearing premise

The performance case is inherited entirely from the cited ServerlessLLM evaluation; if the original testbed results do not transfer to real, shared, multitenant serverless clusters, the review's central message about 6-8x speedups and 0.8-second startups does not stand.

Editorial extensions

If this is right

  • Serverless LLM deployments could start small models in under a second and large models in seconds, making interactive chatbots and real-time assistants viable without pre-warmed GPU pools.
  • Loading LLaMA-2-70B up to 8.2x faster than PyTorch and SafeTensors means checkpoint transfer no longer dominates cold start; storage bandwidth becomes the effective limit.
  • Locality-aware scheduling with live migration cuts P99 latency by roughly 1.95x versus the default scheduler and 2x versus preemption-based Shepherd*, so concurrent models can share a cluster without expensive reloads.
  • The 28x end-to-end startup speedup for OPT-30B suggests serverless serving can approach dedicated serving performance while keeping pay-per-use cost.
  • Resource efficiency improves sharply: ServerlessLLM reaches 4-second latency with one GPU per server, while a cached baseline needs four GPUs for 12 seconds.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If these results hold on shared multitenant clusters, the same three mechanisms could be combined with predictive warm-keeping so the first request of a burst rarely sees a cold start; this review does not test that combination.
  • The token-only migration strategy may generalize to any stateful serverless workload whose working state is much smaller than its loaded program, though the review only demonstrates it for autoregressive generation.
  • The review's own caveats, including higher energy use from pinned memory and direct I/O and the risk that many-hundred-gigabyte model mixes exhaust the DRAM and SSD tiers, point to a natural next experiment: measure ServerlessLLM under mixed-model, memory-constrained, multitenant workloads.
  • A head-to-head comparison against production-grade continuous LLM serving systems would clarify how much of the advantage is serverless-specific, but this review does not include such a benchmark.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This manuscript is a review report on cold-start latency in serverless LLM inference, centered on ServerlessLLM. It describes the cold-start problem, summarizes ServerlessLLM's multi-tier checkpoint loading, token-level live migration, and startup-time-aware scheduling, reports performance numbers from the ServerlessLLM paper, and then discusses other techniques (container pools, warmed containers, pinging, RainbowCake), FaaS provider mitigations, and future directions.

Significance. If taken as a review, the paper offers a readable and mostly faithful synthesis of ServerlessLLM's architecture and a compact overview of general cold-start mitigation. Its strengths are that it explicitly identifies its primary source, lays out the multi-tier loading, live migration, and scheduler in enough detail for a newcomer, and includes some critical discussion, e.g., Section 3.6 on migration overhead. However, the paper would not change the field's understanding: it adds no experiments, no datasets, no artifact, and no new synthesis beyond the source papers. Its usefulness as a survey is limited by the missing measurement definition and the selective coverage of related work. The primary value is pedagogical, not scientific.

major comments (4)
  1. [Abstract; §3.2; §4.1; §4.3] The paper never defines the interval over which the reported 'cold start' numbers are measured. Section 2 defines cold start as comprising both checkpoint loading and GPU resource activation, but Section 3.2 and Section 4.1 report only checkpoint loading, and Section 4.3 says ServerlessLLM 'initiates in 0.8 seconds' without specifying whether this includes function-container creation, remote-to-local checkpoint transfer on an empty local cache, GPU context initialization, or first-token latency. Because the abstract's headline claim is about cold-start reduction, this ambiguity is load-bearing; the limitation paragraph in Section 3.3 addresses checkpoint layout assumptions but not this measurement boundary. The authors should either define an end-to-end cold-start interval and confirm the numbers cover it, or explicitly re-scope all claims to model-loading time.
  2. [§4.1; §4.3] All quantitative results (6x, 8.2x, 28x, 212x, 0.8 s, 7.5 s) are transcribed from [31] without independent verification, error bars, number of trials, or an artifact. As a review paper this is acceptable if clearly attributed, but the text presents these numbers as established facts. For example, Section 4.3 reports 'up to 212X lower latency' without defining whether the latency is mean, median, or P99, and Section 4.1 reports a single speedup factor without variance. Add a prominent caveat that these figures are as reported by ServerlessLLM's authors, and add the metric definitions and workload conditions.
  3. [§3.7] The startup-time estimator in Section 3.7 depends on model-specific parameters a and b ('resuming time = a x (tin + tout) + b'), but the review does not state how a and b are fitted, what their units are, or whether the source paper validates the estimator's accuracy. Since scheduling decisions are only as good as this estimator, the review should flag the calibration requirement as an open issue rather than presenting the estimator as a solved component.
  4. [§1; §5] The paper is framed as a review of current trends, but Section 5 presents only a small, undocumented subset of mitigation techniques (persistent containers, pools, warmed containers, pinging, and RainbowCake) and omits a systematic inclusion criterion. A survey should either state its search strategy and selection criteria or narrow its scope to ServerlessLLM plus directly comparable systems; otherwise the 'current trends' claim in the abstract is not supported.
minor comments (6)
  1. [§6] Several provider-specific claims (e.g., AWS provisioned concurrency, Google pre-warmed instances, Azure Premium Plan) are stated without citations; add references or mark them as summaries of vendor documentation.
  2. [References] Reference [28] has a URL inserted mid-citation, [13] cites Wikipedia, and [25] lacks publication venue and page numbers; the reference list needs cleanup.
  3. [Throughout] The manuscript has numerous spacing and capitalization issues, e.g., 'A WS', 'T echnische', 'focusses', 'Y et'; the text should be proofread.
  4. [Figure 2] Figure 2 is referenced only by citation [1] and is not discussed in the text; either integrate it or remove it.
  5. [§5] The sentence introducing methods 'outlined in [29]' is followed by a list that mixes categories from [29] with additional methods without clear attribution; clarify which items come from [29] and which are added by this review.
  6. [§3.1] The figures for GPU server capacity (4 TB memory, 64 TB NVMe, etc.) are not cited at the point of use; add a citation to [31].

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is a review that transcribes results from an external source without attempting a derivation, and the author has no overlap with the cited work.

full rationale

This manuscript is a literature review of ServerlessLLM and related cold-start mitigation work. It does not present a novel derivation, equation chain, or fitted model. All performance claims (e.g., 'up to 8.2X faster model loading', '0.8 seconds' startup, '28X speedup') are explicitly attributed to reference [31] (Fu et al., ServerlessLLM), an external paper by different authors. The review repeatedly says 'ServerlessLLM achieves...' and cites [31], and Section 3.2 explicitly marks the 6X/8.2X figures as from [31]. There is no self-citation chain because Himel Ghosh is not an author of [31], [32], [33], or [34]. The dependence on [31] for empirical numbers is a citation-dependence or verifiability issue—not circularity, because the review is not claiming to derive those numbers from its own assumptions. The review's own limitations (e.g., 'The set of assumptions made by the authors that the checkpoints have model execution files and model parameter files, may not always hold') are about the original system's assumptions, not about a circular derivation in this review. A reviewer concern about whether the 0.8 s and 8.2x numbers measure full cold start or only checkpoint loading is a scope/clarity concern about the cited evaluation, not a circularity pattern. No step in this paper reduces by construction to its own inputs; the central claim is inherited, not derived. Therefore the circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper is a review, so the ledger records assumptions inherited from the systems it summarizes. The only explicit free parameters are a and b in the migration resume time estimator, taken from [31] without values. No new entities are introduced.

free parameters (2)
  • a (model-specific coefficient in migration resume time estimator) = not reported; fitted per model and batch in [31]
    Section 3.7 defines resume time as a*(t_in+t_out)+b, so a is a free parameter that must be fit from observations; this review gives no value.
  • b (model-specific intercept in migration resume time estimator) = not reported; fitted
    Same formula as above; b is an intercept fitted to measured resume times and is not derived from first principles.
assumptions (3)
  • domain assumption LLM checkpoints contain separate model execution files and model parameter files, and a load-optimized checkpoint format can be assumed.
    Section 3.3 lists this as an assumption of ServerlessLLM; if a checkpoint does not have this structure, the fast chunked loading pipeline must be adapted.
  • domain assumption Autoregressive language models can resume inference on a destination GPU by recomputing the KV cache from intermediate tokens faster than transferring the full KV cache.
    Section 3.5 designs live migration around this property; it holds only for autoregressive decoding and only when model weights are already resident on the destination.
  • domain assumption Underutilized host DRAM and NVMe storage in GPU servers can be treated as spare checkpoint cache without harming co-located workloads.
    Section 3.1 assumes idle storage is free to reuse; contention, fairness, and interference in shared clusters are not analyzed in this review.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloud." pith.science (2026). https://pith.science/paper/JJP4RD4X

@misc{pith2026241115664,
  author       = {Pith},
  title        = {Pith review of: Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloud},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JJP4RD4X}},
  note         = {Machine review of arXiv:2411.15664}
}
read the original abstract

This review report discusses the cold start latency in serverless inference and existing solutions. It particularly reviews the ServerlessLLM method, a system designed to address the cold start problem in serverless inference for large language models. Traditional serverless approaches struggle with high latency due to the size of LLM checkpoints and the overhead of initializing GPU resources. ServerlessLLM introduces a multitier checkpoint loading system, leveraging underutilized GPU memory and storage to reduce startup times by 6--8x compared to existing methods. It also proposes live inference migration and a startup-time-optimized model scheduler, ensuring efficient resource allocation and minimizing delays. This system significantly improves performance and scalability in serverless environments for LLM workloads. Besides ServerlessLLM, several other methods from recent research literature, including Rainbowcake, are reviewed in this paper. Further discussions explore how FaaS providers tackle cold starts and the possible future scopes.

Figures

Figures reproduced from arXiv: 2411.15664 by the authors.

Figure 2
Figure 2. Cold Start [1] 2.1 Cold Start mitigation approaches Some of the prevalent solutions to tackle the cold start issues are the following: 1. Over-subscription of the GPUs to cater to peak demand scenarios. AWS serverless inference keeps a number of GPUs in the warmed state to avoid cold starts. Although it is effective for comparatively smaller models like BERT, it is not enough for LLM for their requirement of sub￾sta… view at source ↗
Figure 1
Figure 1. Serverless Computing Architecture [21] shared container layers, RainbowCake allows effi￾cient, fine-grained resource management across func￾tions, making it well-suited to high-concurrency en￾vironments. Section 6 offers an insight on how the Function-as-a-Service (FaaS) providers deal with cold starts. Finally, Section 7 offers potential future direc￾tions, including the integration of memory manage￾ment optimizati… view at source ↗
Figure 3
Figure 3. Flowchart to illustrate the methods of the ServerlessLLM to mitigate Cold start problem. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The multitier storage design to mitigate [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Diagram that illustrates the Live Migration process. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Estimation times for scheduling. loading time estimator and model migration time es￾timator. • Model loading Time estimation: The model queuing time (q), model size (n) and the band￾width (b) are considered. ServerlessLLM tracks bandwidth for network, SSD and DRAM and …
Figure 7
Figure 7. Figure 7: Illustration of the performance of ServerlessLLM compared to baselines. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Addressing the sustainable AI trilemma: a case study on LLM agents and RAG

    cs.CY 2025-01 conditional novelty 4.0 of 10

    LLM-dependent memory operations in agents and RAG consume orders of magnitude more energy than vector methods, and resource-constrained hardware pays higher energy for lower quality.

Reference graph

Works this paper leans on

37 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [31]

    Rise of the Planet of Serverless Computing: A Systematic Review

    Jinfeng Wen, Zhenpeng Chen, Xin Jin, and Xu- anzhe Liu. Rise of the planet of serverless com- puting: A systematic review https://arxiv. org/abs/2206.12275, 2022

  2. [1]

    slideshare

    https: // www. slideshare. net/ slideshow/ become-a-serverless-black-belt-optimizing-your-serverless-applications-srv401-reinvent-2017/ 840310167

  3. [2]

    https: // aws. amazon. com/ cloudwatch/

  4. [3]

    https: // dashbird. io/

  5. [4]

    com/ jeremydaly/ lambda-warmer

    https: // github. com/ jeremydaly/ lambda-warmer

  6. [5]

    com/ google-cloud/ 3-solutions-to-mitigate-the-cold-starts-on-cloud-run-8c60f0ae7894

    https: // medium. com/ google-cloud/ 3-solutions-to-mitigate-the-cold-starts-on-cloud-run-8c60f0ae7894

  7. [6]

    https: // cloud. google. com/ appengine/ docs/ legacy/ standard/ go111/ configuring-warmup-requests . 10

  8. [7]

    https: // cloud. google. com/ run/ docs/ about-instance-autoscaling

Show all 37 references
  1. [8]

    microsoft

    https: // techcommunity. microsoft. com/ t5/ apps-on-azure-blog/ our-latest-work-to-improve-azure-functions-cold-starts/ ba-p/ 4164500

  2. [9]

    https: // microsoft. github. io/ AzureTipsAndTricks/ blog/ tip260. html

  3. [10]

    https: // www

    Anyscale Technical Blog. https: // www. anyscale. com/ blog/ loading-llama-2-70b-20x-faster-with-anyscale-endpoints

  4. [11]

    SedAI ( https: // www

    Aws autonomous concurrency. SedAI ( https: // www. sedai. io/ blog/ the-end-of-cold-starts-autonomous-concurrency-for-aws-lambda)

  5. [12]

    Snowflake ( https: // www

    Llm inference. Snowflake ( https: // www. snowflake. com/ guides/ llm-inference/ )

  6. [13]

    ( https: // en

    Serverless computing. ( https: // en. wikipedia. org/ wiki/ Serverless_ computing)

  7. [14]

    Bingyang Wu, et. al. Fast Distributed Infer- ence Serving for Large Language Models, 2023. https://arxiv.org/pdf/2305.05920.pdf

  8. [15]

    Anup Mohan et. al. Agile cold starts for scal- able serverless. In 11th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud

  9. [16]

    Catalyzer: Sub-millisecond startup for serverless computing with initialization-less booting

    Dong Du et al. Catalyzer: Sub-millisecond startup for serverless computing with initialization-less booting. Associa- tion for Computing Machinery (ACM) https://doi.org/10.1145/3373376.3378512, 2020

  10. [17]

    Istemi Ekin Akkus et. al. SAND: Towards High-Performance serverless computing. In 2018 USENIX Annual Technical Conference (USENIX ATC 18) , pages 923–935. USENIX Association, July 2018

  11. [18]

    Centralized core-granular scheduling for serverless functions

    Kostis Kaffes et al. Centralized core-granular scheduling for serverless functions. In Proceed- ings of the ACM Symposium on Cloud Comput- ing, page 158–164. Association for Computing Machinery (ACM) https://doi.org/10.1145/ 3357223.3362709, 2019

  12. [19]

    https: // www. usenix. org/ conference/ hotcloud19/ presentation/ mohan. USENIX Association, July 2019

  13. [20]

    Faaslight: General application-level cold-start latency optimization for function-as-a-service in serverless comput- ing

    Xuanzhe Liu et al. Faaslight: General application-level cold-start latency optimization for function-as-a-service in serverless comput- ing. ACM Trans. Softw. Eng. Methodol. https: // doi. org/ 10. 1145/ 3585007, July 2023

  14. [21]

    Rapid task provision- ing with Serverless-Optimized containers

    Edward Oakes et.al. Rapid task provision- ing with Serverless-Optimized containers. In 2018 USENIX Annual Technical Con- ference (USENIX ATC 18) , pages 57–70. USENIX Association https://www.usenix. org/conference/atc18/presentation/oakes, July 2018

  15. [22]

    Ghobaei-Arani

    M Ghorbian, M. Ghobaei-Arani. A sur- vey on the cold start latency approaches in serverless computing: an optimization-based perspective. https: // doi. org/ 10. 1007/ s00607-024-01335-5 , August 2024

  16. [23]

    Cold start latency in serverless com- puting: A systematic review, taxonomy, and future directions

    Muhammed GOLEC, GUNEET KAUR W ALIA, MOHIT KUMAR, FELIX CUADRADO, Sukhpal Singh Gill, and STEVE UHLIG. Cold start latency in serverless com- puting: A systematic review, taxonomy, and future directions. ACM Computing Surveys http: // dx. doi. org/ 10. 1145/ 3700875, October 2024

  17. [24]

    What is serverless computing? IBM ( https: // www

    IBM. What is serverless computing? IBM ( https: // www. ibm. com/ topics/ serverless)

  18. [25]

    Mitigat- ing cold starts in serverless platforms: A pool-based approach https://arxiv.org/abs/ 1903.12221, 2019

    Ping-Min Lin and Alex Glikson. Mitigat- ing cold starts in serverless platforms: A pool-based approach https://arxiv.org/abs/ 1903.12221, 2019

  19. [26]

    Garrett McGrath and Paul R. Brenner. Server- less computing: Design, implementation, and performance. In 2017 IEEE 37th International Conference on Distributed Computing Systems Workshops (ICDCSW) , pages 405–410, 2017

  20. [27]

    Persson and W

    G. Persson and W. Branth Sj¨ oberg. Miti- gating serverless cold starts through predict- ing computational resource demand: Predict- ing function invocations based on real-time user navigation. https: // doi. org/ 10. 1007/ s00607-024-01335-5 , August 2023

  21. [28]

    Samyam Rajbhandari, et. al. ZeRO-Infinity: Breaking the GPU MemoryWall for Extreme Scale Deep Learning, 2021. https://doi.org/ 10.1145/3458817.3476205

  22. [29]

    Pietzuch https: //api.semanticscholar.org/CorpusID: 11 51997872

    Simon Shillaker and Peter R. Pietzuch https: //api.semanticscholar.org/CorpusID: 11 51997872. A provider-friendly serverless framework for latency-critical applications. 2018

  23. [30]

    Parichehr et. al. Vahidinia. Cold start in server- less computing: Current trends and mitigation strategies. In 2020 International Conference on Omni-layer Intelligent Systems (COINS) , pages 1–7, 2020

  24. [32]

    Yao Fu, et. al. ServerlessLLM: Low-Latency Serverless Inference for Large Language Models,

  25. [33]

    Ying Sheng, et. al. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU, 2023. https://arxiv.org/ abs/2303.06865

  26. [34]

    Yiding Wang, et. al. Tabi: An Efficient Multi- Level Inference System for Large Language Models, 2023. https://dl.acm.org/doi/10. 1145/3552326.3587438

  27. [35]

    Taming serverless cold start of cloud model inference with edge comput- ing

    Kongyange Zhao, Zhi Zhou, Lei Jiao, Shen Cai, Fei Xu, and Xu Chen. Taming serverless cold start of cloud model inference with edge comput- ing. IEEE Transactions on Mobile Computing , 23(8):8111–8128, 2024. 12

  28. [36]

    Hanfei Yu and Rohan et. al. Basu Roy. Rain- bowcake: Mitigating cold-starts in serverless with layer-wise container caching and shar- ing. ASPLOS ’24. Association for Com- puting Machinery https://doi.org/10.1145/ 3617232.3624871, 2024

  29. [2024]

    https://arxiv.org/abs/2401.14351

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.