REVIEW 4 major objections 6 minor 1 cited by
Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloud
T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This review argues that ServerlessLLM's multi-tier checkpoint loading, token-only live migration, and startup-time-aware scheduling cut LLM cold starts by up to 8.2x and bring OPT-6.7B startup down to 0.8 seconds.
desk verdict A faithful but unoriginal reading-note on ServerlessLLM; the headline numbers are borrowed, and the paper never pins down what it counts as cold start. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is ServerlessLLM's multi-tier storage hierarchy: model checkpoints sit on NVMe or SATA SSDs, are staged through DRAM, and land in GPU memory, with a loading-optimized checkpoint format that stores tensors in sequential, chunk-readable partitions. Parallel chunk-based loading, direct I/O that bypasses the OS cache, pinned memory, and pre-computed tensor addresses maximize PCIe bandwidth. The live-migration mechanism transfers only the token sequence, on the order of tens to hundreds of kilobytes, and recomputes the KV-cache on the destination GPU instead of shipping a multi-gigabyte cache. The scheduler estimates startup as $q + n/b$ for loading and as $a(t_{\text{in}}+t_{\text{out}})+b$ for migration resuming, then selects the server with the lowest estimate. These three mechanisms together carry the cold-start reduction claim.
What would settle it
Rerun ServerlessLLM on the hardware described in the review (8 NVIDIA A5000 GPUs with 1 TB DDR4 and MinIO storage; 4 A40 servers at 10 Gbps) and measure end-to-end startup for OPT-6.7B, OPT-30B, and LLaMA-2-70B under an Azure-like trace; if OPT-6.7B does not start near 0.8 seconds, OPT-30B not near 7.5 seconds, or LLaMA-2-70B loading not near 8.2x faster than PyTorch and SafeTensors, the central claim fails.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that ServerlessLLM's three mechanisms together remove the dominant cold-start cost of serverless LLM inference. A loading-optimized checkpoint format with parallel chunk-based reads and pre-computed tensor addresses lets checkpoints stream from NVMe SSDs through DRAM into GPU memory at near storage bandwidth. Live migration transfers only the intermediate tokens and recomputes the KV-cache on the destination GPU, so a model can move servers without a full checkpoint reload. A startup-time-optimized scheduler estimates loading time as $q + n/b$ and resuming time as $a(t_{\text{in}}+t_{\text{out}})+b$, then picks the server with the lowest estimate. The review transcribes the reported results: up to 6x faster loading for OPT-2.7B, up to 8.2x for LLaMA-2-70B, an end-to-end startup of 0.8 seconds for OPT-6.7B, and 28x faster startup for OPT-30B versus Ray Serve.
Load-bearing premise
The performance case is inherited entirely from the cited ServerlessLLM evaluation; if the original testbed results do not transfer to real, shared, multitenant serverless clusters, the review's central message about 6-8x speedups and 0.8-second startups does not stand.
Editorial extensions
If this is right
- Serverless LLM deployments could start small models in under a second and large models in seconds, making interactive chatbots and real-time assistants viable without pre-warmed GPU pools.
- Loading LLaMA-2-70B up to 8.2x faster than PyTorch and SafeTensors means checkpoint transfer no longer dominates cold start; storage bandwidth becomes the effective limit.
- Locality-aware scheduling with live migration cuts P99 latency by roughly 1.95x versus the default scheduler and 2x versus preemption-based Shepherd*, so concurrent models can share a cluster without expensive reloads.
- The 28x end-to-end startup speedup for OPT-30B suggests serverless serving can approach dedicated serving performance while keeping pay-per-use cost.
- Resource efficiency improves sharply: ServerlessLLM reaches 4-second latency with one GPU per server, while a cached baseline needs four GPUs for 12 seconds.
Reading between the lines
- If these results hold on shared multitenant clusters, the same three mechanisms could be combined with predictive warm-keeping so the first request of a burst rarely sees a cold start; this review does not test that combination.
- The token-only migration strategy may generalize to any stateful serverless workload whose working state is much smaller than its loaded program, though the review only demonstrates it for autoregressive generation.
- The review's own caveats, including higher energy use from pinned memory and direct I/O and the risk that many-hundred-gigabyte model mixes exhaust the DRAM and SSD tiers, point to a natural next experiment: measure ServerlessLLM under mixed-model, memory-constrained, multitenant workloads.
- A head-to-head comparison against production-grade continuous LLM serving systems would clarify how much of the advantage is serverless-specific, but this review does not include such a benchmark.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript is a review report on cold-start latency in serverless LLM inference, centered on ServerlessLLM. It describes the cold-start problem, summarizes ServerlessLLM's multi-tier checkpoint loading, token-level live migration, and startup-time-aware scheduling, reports performance numbers from the ServerlessLLM paper, and then discusses other techniques (container pools, warmed containers, pinging, RainbowCake), FaaS provider mitigations, and future directions.
Significance. If taken as a review, the paper offers a readable and mostly faithful synthesis of ServerlessLLM's architecture and a compact overview of general cold-start mitigation. Its strengths are that it explicitly identifies its primary source, lays out the multi-tier loading, live migration, and scheduler in enough detail for a newcomer, and includes some critical discussion, e.g., Section 3.6 on migration overhead. However, the paper would not change the field's understanding: it adds no experiments, no datasets, no artifact, and no new synthesis beyond the source papers. Its usefulness as a survey is limited by the missing measurement definition and the selective coverage of related work. The primary value is pedagogical, not scientific.
major comments (4)
- [Abstract; §3.2; §4.1; §4.3] The paper never defines the interval over which the reported 'cold start' numbers are measured. Section 2 defines cold start as comprising both checkpoint loading and GPU resource activation, but Section 3.2 and Section 4.1 report only checkpoint loading, and Section 4.3 says ServerlessLLM 'initiates in 0.8 seconds' without specifying whether this includes function-container creation, remote-to-local checkpoint transfer on an empty local cache, GPU context initialization, or first-token latency. Because the abstract's headline claim is about cold-start reduction, this ambiguity is load-bearing; the limitation paragraph in Section 3.3 addresses checkpoint layout assumptions but not this measurement boundary. The authors should either define an end-to-end cold-start interval and confirm the numbers cover it, or explicitly re-scope all claims to model-loading time.
- [§4.1; §4.3] All quantitative results (6x, 8.2x, 28x, 212x, 0.8 s, 7.5 s) are transcribed from [31] without independent verification, error bars, number of trials, or an artifact. As a review paper this is acceptable if clearly attributed, but the text presents these numbers as established facts. For example, Section 4.3 reports 'up to 212X lower latency' without defining whether the latency is mean, median, or P99, and Section 4.1 reports a single speedup factor without variance. Add a prominent caveat that these figures are as reported by ServerlessLLM's authors, and add the metric definitions and workload conditions.
- [§3.7] The startup-time estimator in Section 3.7 depends on model-specific parameters a and b ('resuming time = a x (tin + tout) + b'), but the review does not state how a and b are fitted, what their units are, or whether the source paper validates the estimator's accuracy. Since scheduling decisions are only as good as this estimator, the review should flag the calibration requirement as an open issue rather than presenting the estimator as a solved component.
- [§1; §5] The paper is framed as a review of current trends, but Section 5 presents only a small, undocumented subset of mitigation techniques (persistent containers, pools, warmed containers, pinging, and RainbowCake) and omits a systematic inclusion criterion. A survey should either state its search strategy and selection criteria or narrow its scope to ServerlessLLM plus directly comparable systems; otherwise the 'current trends' claim in the abstract is not supported.
minor comments (6)
- [§6] Several provider-specific claims (e.g., AWS provisioned concurrency, Google pre-warmed instances, Azure Premium Plan) are stated without citations; add references or mark them as summaries of vendor documentation.
- [References] Reference [28] has a URL inserted mid-citation, [13] cites Wikipedia, and [25] lacks publication venue and page numbers; the reference list needs cleanup.
- [Throughout] The manuscript has numerous spacing and capitalization issues, e.g., 'A WS', 'T echnische', 'focusses', 'Y et'; the text should be proofread.
- [Figure 2] Figure 2 is referenced only by citation [1] and is not discussed in the text; either integrate it or remove it.
- [§5] The sentence introducing methods 'outlined in [29]' is followed by a list that mixes categories from [29] with additional methods without clear attribution; clarify which items come from [29] and which are added by this review.
- [§3.1] The figures for GPU server capacity (4 TB memory, 64 TB NVMe, etc.) are not cited at the point of use; add a citation to [31].
Circularity Check
No circularity: the paper is a review that transcribes results from an external source without attempting a derivation, and the author has no overlap with the cited work.
full rationale
This manuscript is a literature review of ServerlessLLM and related cold-start mitigation work. It does not present a novel derivation, equation chain, or fitted model. All performance claims (e.g., 'up to 8.2X faster model loading', '0.8 seconds' startup, '28X speedup') are explicitly attributed to reference [31] (Fu et al., ServerlessLLM), an external paper by different authors. The review repeatedly says 'ServerlessLLM achieves...' and cites [31], and Section 3.2 explicitly marks the 6X/8.2X figures as from [31]. There is no self-citation chain because Himel Ghosh is not an author of [31], [32], [33], or [34]. The dependence on [31] for empirical numbers is a citation-dependence or verifiability issue—not circularity, because the review is not claiming to derive those numbers from its own assumptions. The review's own limitations (e.g., 'The set of assumptions made by the authors that the checkpoints have model execution files and model parameter files, may not always hold') are about the original system's assumptions, not about a circular derivation in this review. A reviewer concern about whether the 0.8 s and 8.2x numbers measure full cold start or only checkpoint loading is a scope/clarity concern about the cited evaluation, not a circularity pattern. No step in this paper reduces by construction to its own inputs; the central claim is inherited, not derived. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (2)
- a (model-specific coefficient in migration resume time estimator) =
not reported; fitted per model and batch in [31]
- b (model-specific intercept in migration resume time estimator) =
not reported; fitted
assumptions (3)
- domain assumption LLM checkpoints contain separate model execution files and model parameter files, and a load-optimized checkpoint format can be assumed.
- domain assumption Autoregressive language models can resume inference on a destination GPU by recomputing the KV cache from intermediate tokens faster than transferring the full KV cache.
- domain assumption Underutilized host DRAM and NVMe storage in GPU servers can be treated as spare checkpoint cache without harming co-located workloads.
Cite this review
Pith. "Pith review of Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloud." pith.science (2026). https://pith.science/paper/JJP4RD4X
@misc{pith2026241115664,
author = {Pith},
title = {Pith review of: Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloud},
year = {2026},
howpublished = {\url{https://pith.science/paper/JJP4RD4X}},
note = {Machine review of arXiv:2411.15664}
}
read the original abstract
This review report discusses the cold start latency in serverless inference and existing solutions. It particularly reviews the ServerlessLLM method, a system designed to address the cold start problem in serverless inference for large language models. Traditional serverless approaches struggle with high latency due to the size of LLM checkpoints and the overhead of initializing GPU resources. ServerlessLLM introduces a multitier checkpoint loading system, leveraging underutilized GPU memory and storage to reduce startup times by 6--8x compared to existing methods. It also proposes live inference migration and a startup-time-optimized model scheduler, ensuring efficient resource allocation and minimizing delays. This system significantly improves performance and scalability in serverless environments for LLM workloads. Besides ServerlessLLM, several other methods from recent research literature, including Rainbowcake, are reviewed in this paper. Further discussions explore how FaaS providers tackle cold starts and the possible future scopes.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 1 Pith paper
-
Addressing the sustainable AI trilemma: a case study on LLM agents and RAG
LLM-dependent memory operations in agents and RAG consume orders of magnitude more energy than vector methods, and resource-constrained hardware pays higher energy for lower quality.
Reference graph
Works this paper leans on
-
[31]
Rise of the Planet of Serverless Computing: A Systematic Review
Jinfeng Wen, Zhenpeng Chen, Xin Jin, and Xu- anzhe Liu. Rise of the planet of serverless com- puting: A systematic review https://arxiv. org/abs/2206.12275, 2022
work page Pith review arXiv 2022
-
[1]
https: // www. slideshare. net/ slideshow/ become-a-serverless-black-belt-optimizing-your-serverless-applications-srv401-reinvent-2017/ 840310167
work page 2017
-
[2]
https: // aws. amazon. com/ cloudwatch/
-
[3]
https: // dashbird. io/
- [4]
-
[5]
com/ google-cloud/ 3-solutions-to-mitigate-the-cold-starts-on-cloud-run-8c60f0ae7894
https: // medium. com/ google-cloud/ 3-solutions-to-mitigate-the-cold-starts-on-cloud-run-8c60f0ae7894
-
[6]
https: // cloud. google. com/ appengine/ docs/ legacy/ standard/ go111/ configuring-warmup-requests . 10
-
[7]
https: // cloud. google. com/ run/ docs/ about-instance-autoscaling
Show all 37 references
-
[8]
microsoft
https: // techcommunity. microsoft. com/ t5/ apps-on-azure-blog/ our-latest-work-to-improve-azure-functions-cold-starts/ ba-p/ 4164500
-
[9]
https: // microsoft. github. io/ AzureTipsAndTricks/ blog/ tip260. html
-
[10]
https: // www
Anyscale Technical Blog. https: // www. anyscale. com/ blog/ loading-llama-2-70b-20x-faster-with-anyscale-endpoints
-
[11]
SedAI ( https: // www
Aws autonomous concurrency. SedAI ( https: // www. sedai. io/ blog/ the-end-of-cold-starts-autonomous-concurrency-for-aws-lambda)
-
[12]
Snowflake ( https: // www
Llm inference. Snowflake ( https: // www. snowflake. com/ guides/ llm-inference/ )
-
[13]
( https: // en
Serverless computing. ( https: // en. wikipedia. org/ wiki/ Serverless_ computing)
-
[14]
Bingyang Wu, et. al. Fast Distributed Infer- ence Serving for Large Language Models, 2023. https://arxiv.org/pdf/2305.05920.pdf
2023 arXiv
-
[15]
Anup Mohan et. al. Agile cold starts for scal- able serverless. In 11th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud
-
[16]
Catalyzer: Sub-millisecond startup for serverless computing with initialization-less booting
Dong Du et al. Catalyzer: Sub-millisecond startup for serverless computing with initialization-less booting. Associa- tion for Computing Machinery (ACM) https://doi.org/10.1145/3373376.3378512, 2020
2020
-
[17]
Istemi Ekin Akkus et. al. SAND: Towards High-Performance serverless computing. In 2018 USENIX Annual Technical Conference (USENIX ATC 18) , pages 923–935. USENIX Association, July 2018
2018
-
[18]
Centralized core-granular scheduling for serverless functions
Kostis Kaffes et al. Centralized core-granular scheduling for serverless functions. In Proceed- ings of the ACM Symposium on Cloud Comput- ing, page 158–164. Association for Computing Machinery (ACM) https://doi.org/10.1145/ 3357223.3362709, 2019
-
[19]
https: // www. usenix. org/ conference/ hotcloud19/ presentation/ mohan. USENIX Association, July 2019
2019
-
[20]
Faaslight: General application-level cold-start latency optimization for function-as-a-service in serverless comput- ing
Xuanzhe Liu et al. Faaslight: General application-level cold-start latency optimization for function-as-a-service in serverless comput- ing. ACM Trans. Softw. Eng. Methodol. https: // doi. org/ 10. 1145/ 3585007, July 2023
2023
-
[21]
Rapid task provision- ing with Serverless-Optimized containers
Edward Oakes et.al. Rapid task provision- ing with Serverless-Optimized containers. In 2018 USENIX Annual Technical Con- ference (USENIX ATC 18) , pages 57–70. USENIX Association https://www.usenix. org/conference/atc18/presentation/oakes, July 2018
2018
-
[22]
Ghobaei-Arani
M Ghorbian, M. Ghobaei-Arani. A sur- vey on the cold start latency approaches in serverless computing: an optimization-based perspective. https: // doi. org/ 10. 1007/ s00607-024-01335-5 , August 2024
2024
-
[23]
Cold start latency in serverless com- puting: A systematic review, taxonomy, and future directions
Muhammed GOLEC, GUNEET KAUR W ALIA, MOHIT KUMAR, FELIX CUADRADO, Sukhpal Singh Gill, and STEVE UHLIG. Cold start latency in serverless com- puting: A systematic review, taxonomy, and future directions. ACM Computing Surveys http: // dx. doi. org/ 10. 1145/ 3700875, October 2024
-
[24]
What is serverless computing? IBM ( https: // www
IBM. What is serverless computing? IBM ( https: // www. ibm. com/ topics/ serverless)
-
[25]
Mitigat- ing cold starts in serverless platforms: A pool-based approach https://arxiv.org/abs/ 1903.12221, 2019
Ping-Min Lin and Alex Glikson. Mitigat- ing cold starts in serverless platforms: A pool-based approach https://arxiv.org/abs/ 1903.12221, 2019
1903 arXiv
-
[26]
Garrett McGrath and Paul R. Brenner. Server- less computing: Design, implementation, and performance. In 2017 IEEE 37th International Conference on Distributed Computing Systems Workshops (ICDCSW) , pages 405–410, 2017
2017
-
[27]
Persson and W
G. Persson and W. Branth Sj¨ oberg. Miti- gating serverless cold starts through predict- ing computational resource demand: Predict- ing function invocations based on real-time user navigation. https: // doi. org/ 10. 1007/ s00607-024-01335-5 , August 2023
-
[28]
Samyam Rajbhandari, et. al. ZeRO-Infinity: Breaking the GPU MemoryWall for Extreme Scale Deep Learning, 2021. https://doi.org/ 10.1145/3458817.3476205
2021
-
[29]
Pietzuch https: //api.semanticscholar.org/CorpusID: 11 51997872
Simon Shillaker and Peter R. Pietzuch https: //api.semanticscholar.org/CorpusID: 11 51997872. A provider-friendly serverless framework for latency-critical applications. 2018
2018
-
[30]
Parichehr et. al. Vahidinia. Cold start in server- less computing: Current trends and mitigation strategies. In 2020 International Conference on Omni-layer Intelligent Systems (COINS) , pages 1–7, 2020
2020
-
[32]
Yao Fu, et. al. ServerlessLLM: Low-Latency Serverless Inference for Large Language Models,
-
[33]
Ying Sheng, et. al. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU, 2023. https://arxiv.org/ abs/2303.06865
2023 arXiv
-
[34]
Yiding Wang, et. al. Tabi: An Efficient Multi- Level Inference System for Large Language Models, 2023. https://dl.acm.org/doi/10. 1145/3552326.3587438
2023
-
[35]
Taming serverless cold start of cloud model inference with edge comput- ing
Kongyange Zhao, Zhi Zhou, Lei Jiao, Shen Cai, Fei Xu, and Xu Chen. Taming serverless cold start of cloud model inference with edge comput- ing. IEEE Transactions on Mobile Computing , 23(8):8111–8128, 2024. 12
2024
-
[36]
Hanfei Yu and Rohan et. al. Basu Roy. Rain- bowcake: Mitigating cold-starts in serverless with layer-wise container caching and shar- ing. ASPLOS ’24. Association for Com- puting Machinery https://doi.org/10.1145/ 3617232.3624871, 2024
-
[2024]
https://arxiv.org/abs/2401.14351
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.