Pith. sign in

REVIEW 4 major objections 4 minor 49 references

Refactoring a serving engine's sequential cold-start program into communicating finite automata lets independent initialization steps run concurrently and merge I/O, cutting time to first token by up to 7.2× while preserving program semanti

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 13:46 UTC pith:LQ7ODTPM

load-bearing objection Solid systems result with a formal correctness claim that doesn't cover its own biggest optimization; read it for the engineering, not the proof. the 4 major comments →

arxiv 2607.18957 v1 pith:LQ7ODTPM submitted 2026-07-21 cs.DC

InstantInfer: Enabling Fast LLM Cold Start with Communicating Finite Automata

Yitao Yuan (1 , 2) , Yongchao He (2) , Shaoke Fang (1) , Wenfei Wu (1) ((1) Peking University , (2) Scitix AI) This is my paper
classification cs.DC
keywords LLM cold startCommunicating Finite Automataprogram refactoringconcurrent initializationI/O mergingmodel loading pipelinetime to first tokenmodel switching
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to show that an LLM engine's cold-start path can be treated not as one monolithic sequential program but as a set of small finite automata whose states and dependencies form a directed acyclic graph. Each component—process, tensor, or data chunk—declares its monotonic state transitions and the cross-component states it waits on, so independent transitions can safely overlap and fine-grained tensor reads can be merged into chunk-sized I/O. The paper proves that any refactored execution terminates in exactly the same final states as the original sequential program, as long as each component still follows its own state sequence. On that basis it builds InstantInfer, refactoring process-tree creation, tensor loading, and model switching, and reports up to 7.2× lower cold-start time to first token, up to 32.3× faster model loading, and up to 11.8× lower service stall during model switching. The contribution matters because it recasts cold-start optimization as a dependency-declaration problem rather than a hand-rewriting problem.

Core claim

The central claim is that the cold-start bottleneck is not the initialization work itself but the ordering forced by sequential control flow. The paper defines each physical or logical component as a finite automaton with monotonic, irreversible states; dependencies are declared as (component, state) prerequisites, and a channel runtime publishes states and wakes waiters. Because transitions are monotonic, the whole cold start forms a DAG, and the paper proves (Theorems 1 and 2) that any DAG CFA terminates at the same final states as the original chain, provided each automaton's own transition order is unchanged. InstantInfer instantiates this in three places: parent and child processes gain

What carries the argument

The Communicating Finite Automaton (CFA) abstraction: each initialization component is an automaton with a monotonic, finite, ordered state variable, and a dependency (c_i, s_i) → (c_j, s_j) means c_j's transition out of s_j waits until c_i reaches s_i. A channel daemon maintains wait queues and published-state flags, exposing set_state and wait_state primitives so developers keep the original code and insert declarations rather than rewriting components. The proof machinery is the DAG/Chain equivalence: sequential execution is a Chain CFA, and any DAG CFA in which every automaton preserves its per-FA transition sequence terminates in the same final states. This equivalence is what licenses

Load-bearing premise

The argument assumes that every real ordering constraint between initialization code blocks is declared as a state dependency, so any shared state that is not mentioned—globals, GPU context creation, IPC handles, inherited file descriptors—is safe under arbitrary interleaving; if one undeclared dependency exists, the concurrently refactored program can race or crash.

What would settle it

Instrument a refactored component pair that shares a hidden resource—for example, two processes that both write to the same configuration file or one process that uses a GPU context created by another—without declaring a state dependency, run the pipeline many times, and check whether final states ever differ or the process crashes.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Cold-start time stops being the sum of all process initialization times; with an intermediate state per process, startup becomes the max over root-to-leaf paths plus the ordered dependent stages.
  • Model loading can become a hardware-bound transfer: a chunked disk-to-host-to-GPU pipeline with cross-GPU AllGather reaches near-link storage throughput, so remaining gains require faster storage or interconnects.
  • Model switching can overlap teardown and initialization safely by declaring GPU-memory release as the one blocking dependency, shrinking the window in which no requests are served.
  • The approach is incremental: because the original code structure is preserved and only state declarations are inserted, the same CFA framework can be applied phase by phase to other startup stages.
  • If the claimed speedups hold, cold-start TTFT no longer needs to dominate user-perceived latency in serverless LLM serving; bursts can be absorbed by quickly bringing new engines online.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The equivalence theorem covers final states of declared automata, not data-race-freedom of every shared object; extending InstantInfer to a new component safely requires tooling that verifies all real shared-state interactions are captured by declared dependencies, or the safety guarantee may not transfer.
  • The chunk loading strategy assumes tensor-to-chunk mappings are precomputed offline; an untested extension is adaptive chunk sizing or dynamic repartitioning based on measured storage and interconnect bandwidth.
  • The model-switch schedule uses profiled predictions of new-model environment init time and old-model GPU release time; prediction error would shift the overlap point, suggesting a feedback or online-control variant as a natural follow-up.
  • The CFA abstraction likely generalizes beyond LLM cold starts to any multi-component initialization pipeline with monotonic progress, such as database startup or container image materialization, though the paper only evaluates LLM serving.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes the Communicating Finite Automata (CFA) abstraction to model LLM cold-start components as monotonic state machines with explicit state dependencies, and uses it to refactor vLLM's sequential startup into concurrent, I/O-merged execution. Three cold-start procedures are refactored: process-tree materialization, tensor loading, and model switching. The authors claim a rigorous proof that CFA-based refactoring is equivalent to the original sequential program (Theorems 1–2, §3.3 and §8), and report end-to-end TTFT speedups up to 7.2×, model-loading speedups up to 32.3×, and model-switching stall reductions up to 11.8× across two GPU clusters and four models each.

Significance. If the correctness claim were established, this would be a valuable contribution: a unified abstraction that exposes cross-component concurrency and I/O-granularity mismatches in LLM cold start, with a practical framework integrated into vLLM. The empirical evaluation is broad and credible: two clusters, multiple dense and MoE models, burst workloads, multi-instance scaling, ablations, and hardware utilization. The paper also measures CPU and memory overhead transparently. However, the formal correctness argument, which is a central advertised contribution, is much weaker than claimed: the main theorem is near-tautological and does not cover the chunk-based tensor-loading refactor that yields the largest speedup. The data-safety arguments for all three refactorings are informal prose. The empirical results are likely to stand on their own, but the paper currently overstates its formal guarantees.

major comments (4)
  1. [§3.3, Theorem 2] Theorem 2 is essentially a tautology. It assumes that each FA in the DAG CFA has an internal transition sequence identical to that in the Chain CFA, then concludes the final states are the same. With that hypothesis, the conclusion follows by definition; the proof in §8 simply observes that both terminate at final states. The theorem does not show that respecting the declared state dependencies is sufficient to preserve per-FA transition sequences in a real concurrent execution, nor does it address undeclared shared-state interactions (e.g., global variables, CUDA context creation, IPC endpoints, file descriptors). The claim in §1 and §3.3 of a 'rigorous proof' for the refactoring is therefore materially overstated.
  2. [§4.2, refined tensor CFA] The chunk-based tensor-loading refactor, which drives the largest speedup in the ablation (2.4×–3.7× in §6.5), is not covered by Theorems 1–2. In the refined model, tensor state spaces are redefined from {InDisk, InMem, InGPU} to {Alloc, Loaded}, and new chunk FAs with states {InDisk, InMem, InGPU, Destroyed} (plus Gathered in the distributed variant) are introduced. The original sequential vLLM loader has no chunk components and uses a different tensor state space, so there is no Chain CFA containing the same per-FA sequences. Consequently Theorem 2 cannot be instantiated to prove equivalence for this refactoring. The correctness-and-safety paragraph ("each logical tensor is assigned values after its own allocation and its associated chunks' readiness") is informal prose, not a proof. The formal guarantee therefore has a gap exactly where the main optimization lives.
  3. [§4.1–§4.3, data-safety arguments] The correctness of all three refactorings rests on the unstated axiom that the only ordering constraints are those the programmer explicitly declares as CFA state dependencies. The framework provides no static check or proof that the original vLLM initialization code has no other shared-state dependencies between the concurrent code blocks. The data-safety paragraphs in §4.1–4.3 are informal and non-quantitative; for instance, the chunk-to-tensor dependency is written as (C_i, InGPU) → (T_j, Alloc) and (T_i, Loaded) → (C_j, InGPU), but the text says a chunk is destroyed only after *all* associated tensors are Loaded. The formal model does not define conjunctive (AND) dependencies, so the written dependencies do not express the intended safety condition. If any undeclared dependency exists, the refactored program can race or crash, and neither Theorem 1 nor Theorem 2 rules this out.
  4. [§3.1, FA merging] The FA-merging operation described in §3.1 ('multiple related FAs can be safely merged and simplified... reduce state transitions while maintaining the same semantics') is not formalized or proved. The refined tensor model is not a simple merge of existing FAs; it changes state spaces and introduces new component types. The paper should either provide a precise semantics-preservation theorem for the merge/refinement operation or explicitly present the tensor-loading correctness argument as an engineering claim supported by testing, not as a consequence of Theorems 1–2.
minor comments (4)
  1. [§7, Related Work] The text mentions 'InstaInfer' instead of 'InstantInfer' in the sentence about loading-oriented systems. Please fix the typo.
  2. [§6.5, Ablation] The reported compounded speedup (5.0×–7.8×) is not obviously consistent with the individual contributions (2.4×–3.7×, 1.2×–1.5×, 1.6×–2.1×) if they are intended to be multiplicative. The text should clarify whether the later numbers are additional normalized reductions on the already-optimized baseline, or whether the total is computed differently.
  3. [§4.2, dependencies] The notation for chunk-to-tensor dependencies is ambiguous: a single dependency (T_i, Loaded) → (C_j, InGPU) suggests one tensor can trigger destruction, whereas the prose requires all associated tensors to be Loaded. Please define AND-dependencies explicitly in the CFA model.
  4. [§8, Theorem 1 proof] The proof of Theorem 1 states that any incomplete DAG has a state with in-degree 0 and out-degree > 0. This is true for a DAG but should be stated as a lemma with a short justification, since the graph contains both internal transition edges and cross-FA dependency edges that can interact.

Circularity Check

1 steps flagged

Theorem 2's equivalence proof is definitional and does not cover the chunk-merged tensor refactor producing the main speedup; the empirical results remain independent.

specific steps
  1. self definitional [§3.3 Theorem 2; §8 proof; §4.2 'Correctness and Safety']
    "Theorem 2. If the internal state transition sequence of each FA in a DAG CFA is identical to that in a Chain CFA, then when both CFAs terminate, the final states of all FAs will be exactly the same. ... In both CFAs, the partial order of states for each FA is identical. With (1), the final states of all FAs in both the DAG CFA and the Chain CFA are guaranteed to be identical when they terminate. ... By theorems in §3.3, the CFA-refined program converges to the same state as the original program."

    The claimed proof of refactoring correctness reduces to the theorem's own hypothesis: if every FA has the same internal transition sequence, then each FA's final state is already determined by that sequence, so 'same final states' is a restatement of the premise rather than an independently derived guarantee. The theorem therefore cannot by itself establish that the actual refactored vLLM code is equivalent to the sequential original. For the main speedup, tensor materialization, the refined model redefines tensor state spaces as {Alloc, Loaded} and introduces chunk FAs with states {InDisk, InMem, InGPU, Destroyed}; the original sequential loader has no such components, so no Chain CFA with identical per-FA sequences exists and Theorem 2 is not instantiable. The 'By theorems in §3.3' asser

full rationale

The empirical evaluation is self-contained and not circular: InstantInfer is benchmarked against external systems (vLLM, SGLang, ServerlessLLM, Safetensors, fastsafetensors, Run:ai), and the headline speedups are measured end-to-end rather than derived from the CFA theorem. There is no load-bearing self-citation and no fitted parameter is renamed as a prediction; the model-switching schedule uses offline profiling (t'_1, t'_2) only as a scheduling heuristic. The notable circularity-like issue is confined to the formal correctness claim: Theorem 2 is definitionally true and is invoked for the tensor-loading refactor even though the refactor changes per-FA state spaces and adds new chunk FAs, so the theorem's key hypothesis is not satisfied. This makes the formal guarantee overstate what is proved, but it does not contaminate the experimental comparisons. Accordingly, a modest score of 3 is appropriate: partial circularity in the formal derivation chain, independent empirical content.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 2 invented entities

The central scientific content of the paper is empirical (a new system with measured speedups), so the axiom ledger is dominated by software-design assumptions rather than fitted physical constants. The main free parameters are engineering choices (chunk size, buffer sizes, profiling-based schedule) that affect performance but not the correctness of the framework. The key unproven assumption is that the declared dependencies capture all ordering constraints; this is the real risk.

free parameters (3)
  • chunk_size
    Equal-length physical file chunks; size is chosen by the implementer and directly controls I/O merging efficiency, but no formula or value is given in the paper.
  • CPU/GPU buffer sizes
    'Sized according to tensor dimensions, storage types, and GPU count' (§6.6); examples given (4 GB) but no general rule, and they affect throughput and memory usage.
  • model-switch schedule predictions t'_1, t'_2
    Obtained 'using historical measurements or offline profiling' (§4.3); these are fitted to prior runs and used to set the old model's shutdown time, a heuristic parameter.
axioms (3)
  • domain assumption All component state transitions are monotonic and irreversible
    stated in §3.1; real initialization code (retries, conditional branches, dynamic behavior) may not fit this, so it restricts the applicability of the framework.
  • standard math Every state transition finishes in finite time
    stated as the assumption in §8; a standard liveness condition for termination.
  • ad hoc to paper Declared state dependencies are sufficient to guarantee data safety
    the framework's central safety premise; the proofs in §3.3/§8 abstract away the actual program code, and the per-case data-safety discussions in §4.1-4.3 are informal, so this is load-bearing.
invented entities (2)
  • channel daemon (runtime) independent evidence
    purpose: background process/thread that handles set_state/wait_state events, maintains waiter queues, and wakes/blocking components
    implemented in InstantInfer and exercised in the evaluation; it is a proposed system component, not an externally observable physical entity, but its effect is measured.
  • data chunk independent evidence
    purpose: fixed-size physical file chunk used as an I/O unit to merge tensor loads and enable pipelining across disk, host, and GPU
    introduced in §4.2; its performance benefit is directly measurable and reflected in the tensor-loading speedups.

pith-pipeline@v1.3.0-alltime-deepseek · 21316 in / 9231 out tokens · 84921 ms · 2026-08-01T13:46:37.560490+00:00 · methodology

0 comments
read the original abstract

Cold starts in large language model (LLM) inference services significantly affect user experience, yet they remain inefficient due to sequential initialization and a massive number of fine-grained I/O requests issued by complex software components. Although refactoring the program can yield advantages such as concurrent execution and I/O merging, this approach is error-prone and carries correctness risks when dealing with massive, heterogeneous components. We propose the Communicating Finite Automata (CFA) abstraction to systematically analyze cross-component optimization opportunities, and design a programming framework to enable CFA-based component program refactoring. This framework preserves the original sequential program structure while enabling safe concurrent component execution. We prove the correctness of the program refactoring. We apply the CFA abstraction and framework to refactor process tree creation, tensor loading, and model switching in vLLM, forming a new cold-start system named InstantInfer. Extensive experiments demonstrate that InstantInfer substantially accelerates LLM cold starts (achieving up to 7.2 times speedup) and exhibits robustness across diverse GPUs, workloads, and scales.

Figures

Figures reproduced from arXiv: 2607.18957 by 2), (2) ScitiX AI), Shaoke Fang (1), Wenfei Wu (1) ((1) Peking University, Yitao Yuan (1, Yongchao He (2).

Figure 1
Figure 1. Figure 1: Startup latency vs model size. The circle area is proportional to the model parameter count. component, execution progress is described by monotonic state transitions; across components, logical dependencies are described by component-state dependencies. The CFA ab￾straction enables direct application of the two optimizations: independent state transitions across components can safely run concurrently, and… view at source ↗
Figure 3
Figure 3. Figure 3: Process/Thread Examples of CFA Programming reorganize these state transitions and their associated I/O op￾erations using FA merging. This can align the refactored I/O granularity with hardware characteristics, thereby achieving higher bandwidth saturation. 3.2 Programming Framework and Runtime We provide a programming framework to refactor the LLM cold-start program with the CFA abstraction. Components in … view at source ↗
Figure 2
Figure 2. Figure 2: Core event loop of a channel. state 𝑠𝑗 to state 𝑠𝑗+1 depends on component 𝑐𝑖 having already reached state 𝑠𝑖 . In other words, only when 𝑐𝑖 successfully arrives at state 𝑠𝑖 can it trigger the specific operations of com￾ponent 𝑐𝑗 from state 𝑠𝑗 to 𝑠𝑗+1. Because the state transition of each component is monotonic in cold start, for the sake of conciseness, we simplify the state transition as the starting stat… view at source ↗
Figure 4
Figure 4. Figure 4: Process-tree materialization in conventional engines and under CFA-guided execution. (lines 11–16), it either creates (line 12) or appends (line 14) to the corresponding queue, or resumes the waiter immediately if the required state has already been published (line 16). An Example. The CFA programming framework can be ap￾plied to concurrent processes, threads, and coroutines. In the example shown in [PITH… view at source ↗
Figure 5
Figure 5. Figure 5: CFA-guided model loading. decomposed into an independent stage and a parent-dependent stage, so that 𝑇𝑖 = 𝑇 𝐼 𝑖 +𝑇 𝐷 𝑖 , where 𝑇 𝐷 1 = 0 because the fron￾tend has no parent-dependent stage. The total initialization time becomes the maximum time over all root-to-leaf paths: 𝑇CFA = max𝑖 {𝑇 𝐼 𝑖 + Í3 𝑗=𝑖 𝑇 𝐷 𝑗 } ≤ max𝑖 {𝑇 𝐼 𝑖 } + Í3 𝑖=1 𝑇 𝐷 𝑖 < 𝑇vLLM. The first inequality holds because the independent stage is… view at source ↗
Figure 6
Figure 6. Figure 6: CFA-guided runtime model switching. paths: each chunk is divided into multiple equal-length “seg￾ments” according to the number of ranks. Within each rank, the assigned segment is loaded following the aforementioned pipeline process. Once these segments become locally ready in their respective GPUs, the system per￾forms an additional AllGather collective communication operation, after which the correspondi… view at source ↗
Figure 7
Figure 7. Figure 7: Cold-start TTFT CDFs under different hardware and request settings. resource overhead of InstantInfer? (§6.6) 6.1 Evaluation Settings Testbed. We evaluate InstantInfer on two representative GPU clusters, each comprising 8 NVIDIA GPUs, to capture both high-end and more commonly deployed configurations. The first testbed uses NVLink-connected H20 (141 GB) GPUs with 50 GB/s networked storage (GPFS [2]), repre… view at source ↗
Figure 10
Figure 10. Figure 10: [H20] Service stall time during model switching. Qwen3- 30B-A3B Llama￾3.1-70B Qwen3- 235B-A22B DeepSeek￾R1 0.00 0.25 0.50 0.75 1.00 Normalized time vLLM + Process CFA + Tensor CFA + Switching CFA [PITH_FULL_IMAGE:figures/full_fig_p009_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: [H20] Startup time improvement from each de￾sign. increases the post-startup portion of TTFT, especially request scheduling and prefill queuing overheads. Across the evalu￾ated settings, increasing the burst size by one raises the TTFT by approximately 0.27 s on average. These results show that InstantInfer can still bring the model online quickly enough to absorb substantial traffic bursts. Impact of col… view at source ↗
Figure 12
Figure 12. Figure 12: [H20 and L40] Model loading time comparison across disk and memory. into host memory and then copying it to each GPU is substan￾tially more efficient than issuing repeated storage reads for each instance. We therefore enable in-memory model caching, as all evaluated frameworks support this mechanism: Server￾lessLLM loads the model into its custom caching layer, vLLM and SGLang load it into the page cache,… view at source ↗
Figure 13
Figure 13. Figure 13: [H20] Breakdown of Frontend/Core/Worker process creation. 0 2 4 Time (s) 0 20 40 Throughput (GB/s) GPU 0 GPU 3 GPU 1 Storage GPU 2 [PITH_FULL_IMAGE:figures/full_fig_p011_13.png] view at source ↗
Figure 17
Figure 17. Figure 17: [H20] Cold-start TTFT under a request burst (con￾currency=16). ServerlessLLM does not support DeepSeek-R1 because it is integrated with an older engine. vLLM SGLang SLLM InstantInfer 0 60 120 Qwen3- 30B-A3B 0.00 0.25 0.50 0.75 1.00 0 200 400 Llama￾3.1-70B 0 200 400 Qwen3- 235B-A22B 0 200 400 DeepSeek￾R1 CDF of TTFT (s) [PITH_FULL_IMAGE:figures/full_fig_p015_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: [H20] Cold-start TTFT under a request burst (con￾currency=64). ServerlessLLM does not support DeepSeek-R1 because it is integrated with an older engine. 1 async def coro_A(channel): # Component A 2 await A_init1() 3 channel.set_state("A", "A1") 4 await A_init2() 5 channel.set_state("A", "A2") 6 7 async def coro_B(channel): # Component B 8 await B_init1() 9 channel.set_state("B", "B1") 10 await channel.wai… view at source ↗
Figure 16
Figure 16. Figure 16: Coroutine Example of CFA Programming A Composability InstantInfer’s CFA abstraction is composable with comple￾mentary optimizations that target other components of cold￾start latency. For example, InstantInfer does not currently optimize the Optimize stage discussed in §2.2 (e.g., com￾pilation and CUDA graph capture). Medusa [42] addresses precisely this stage by serializing captured CUDA graphs and resto… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

49 extracted references · 12 linked inside Pith

  1. [1]

    ShareGPT Datasets.https://huggingface.co/datasets/Ryok oAI/ShareGPT52K

    2023. ShareGPT Datasets.https://huggingface.co/datasets/Ryok oAI/ShareGPT52K

    2023

  2. [2]

    IBM Storage Scale.https://www.ibm.com/docs/storage- scale

    2026. IBM Storage Scale.https://www.ibm.com/docs/storage- scale

    2026

  3. [3]

    Run:ai Model Streamer.https://github.com/run-ai/runai- model-streamer

    2026. Run:ai Model Streamer.https://github.com/run-ai/runai- model-streamer

    2026

  4. [4]

    Anthropic. 2026. Claude Code.https://claude.com/product/claude- code

    2026

  5. [5]

    Jiabin Chen, Fei Xu, Yikun Gu, Li Chen, Fangming Liu, and Zhi Zhou. 2024. HarmonyBatch: Batching multi-SLO DNN Inference with Heterogeneous Serverless Functions. In2024 IEEE/ACM 32nd International Symposium on Quality of Service (IWQoS). 1–10. doi: 10.1109/IWQoS61813.2024.10682915

    arXiv 2024

  6. [6]

    Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, and Arvind Krishnamurthy. 2024. Punica: Multi-Tenant LoRA Serving. In Proceedings of Machine Learning and Systems, P. Gibbons, G. Pekhi- menko, and C. De Sa (Eds.), V ol. 6. 1–13.https://proceedings.mlsy s.org/paper_files/paper/2024/file/054de805fcceb78a201f5e9d 53c85908-Paper-Conference.pdf

    2024

  7. [7]

    Jiangfei Duan, Runyu Lu, Haojie Duanmu, Xiuhong Li, Xingcheng Zhang, Dahua Lin, Ion Stoica, and Hao Zhang. 2024. MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving. arXiv:2404.02015 [cs.DC]https://arxiv.org/abs/2404.02015

    Pith/arXiv arXiv 2024

  8. [8]

    Hugging Face. 2023. Safetensors.https://github.com/huggingface /safetensors

    2023

  9. [9]

    Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. 2024. ServerlessLLM: Low- Latency Serverless Inference for Large Language Models. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 135–153.https: //www.usenix.org/conference/osdi24/pres...

    2024

  10. [10]

    GitHub. 2026. GitHub Copilot.https://github.com/features/copilot

    2026

  11. [11]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, ...

    2025

  12. [12]

    Zicong Hong, Jian Lin, Song Guo, Sifu Luo, Wuhui Chen, Roger Wattenhofer, and Yue Yu. 2024. Optimus: Warming Serverless ML Inference via Inter-Function Model Transformation. InProceedings of the Nineteenth European Conference on Computer Systems(Athens, Greece)(EuroSys ’24). Association for Computing Machinery, New York, NY , USA, 1039–1053. doi:10.1145/3...

    arXiv 2024

  13. [13]

    Junhao Hu, Jiang Xu, Zhixia Liu, Yulong He, Yuetao Chen, Hao Xu, Jiang Liu, Jie Meng, Baoquan Zhang, Shining Wan, Gengyuan Dan, Zhiyu Dong, Zhihao Ren, Changhong Liu, Tao Xie, Dayun Lin, Qin Zhang, Yue Yu, Hao Feng, Xusheng Chen, and Yizhou Shan

  14. [14]

    Nikoleta Iliakopoulou, Jovan Stojkovic, Chloe Alverti, Tianyin Xu, Hubertus Franke, and Josep Torrellas. 2025. Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environ- ments. InProceedings of the 58th IEEE/ACM International Sympo- sium on Microarchitecture (MICRO 2025). ACM, 217–231. doi: 10.1145/3725843.3756083

    arXiv 2025

  15. [15]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica

  16. [16]

    Baolin Li, Yankai Jiang, Vijay Gadepally, and Devesh Tiwari. 2024. LLM Inference Serving: Survey of Recent Advances and Opportunities. arXiv:2407.12391 [cs.DC]https://arxiv.org/abs/2407.12391

    Pith/arXiv arXiv 2024

  17. [17]

    Suyi Li, Hanfeng Lu, Tianyuan Wu, Minchen Yu, Qizhen Weng, Xusheng Chen, Yizhou Shan, Binhang Yuan, and Wei Wang. 2024. CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Genera- tive LLM Inference. arXiv:2401.11240 [cs.DC]https://arxiv.org/ab s/2401.11240

    Pith/arXiv arXiv 2024

  18. [18]

    Suyi Li, Hanfeng Lu, Tianyuan Wu, Minchen Yu, Qizhen Weng, Xusheng Chen, Yizhou Shan, Binhang Yuan, and Wei Wang. 2025. TOPPINGS: CPU-assisted, rank-aware adapter serving for LLM in- ference. InProceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference(Boston, MA, USA)(USENIX ATC ’25). USENIX Association, USA, Article 37, 17 pages.htt...

    arXiv 2025

  19. [19]

    Chongpeng Liu, Xiaojian Liao, Hancheng Liu, Limin Xiao, and Jianxin Li. 2025. PipeBoost: Resilient Pipelined Architecture for Fast Serverless LLM Scaling. arXiv:2503.17707 [cs.DC]https: //arxiv.org/abs/2503.17707

    Pith/arXiv arXiv 2025

  20. [20]

    Morley Mao

    Xueshen Liu, Yongji Wu, Yuncheng Yao, Danyang Zhuo, Ion Sto- ica, and Z. Morley Mao. 2026. Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Start. arXiv:2604.06664 [cs.DC]https://arxiv.org/abs/2604.06664

    Pith/arXiv arXiv 2026

  21. [21]

    Chiheng Lou, Sheng Qi, Chao Jin, Dapeng Nie, Haoran Yang, Yu Ding, Xuanzhe Liu, and Xin Jin. 2025. HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds. arXiv:2502.15524 [cs.DC]https://arxiv.org/abs/2502.15524

    arXiv 2025

  22. [22]

    Chiheng Lou, Sheng Qi, Rui Kang, Yong Zhang, Chen Sun, Pengcheng Wang, Bingyang Liu, Xuanzhe Liu, and Xin Jin. 2025. WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving. arXiv:2512.09472 [cs.DC]https://arxiv.org/abs/2512.09472

    Pith/arXiv arXiv 2025

  23. [24]

    Ziming Mao, Tian Xia, Zhanghao Wu, Wei-Lin Chiang, Tyler Griggs, Romil Bhardwaj, Zongheng Yang, Scott Shenker, and Ion Stoica

  24. [25]

    Meta. 2024. Introducing Llama 3.1: Our most capable models to date. https://ai.meta.com/blog/meta-llama-3-1

    2024

  25. [26]

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Hongyi Jin, Tianqi Chen, and Zhihao Jia. 2025. Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems. Comput. Surveys58, 1 (Sept. 2025), 1–37. doi:10.1145/3754448

    doi:10.1145/3754448 2025

  26. [27]

    InProceedings of the Twentieth European Con- ference on Computer Systems (EuroSys’25)

    SkyServe: Serving AI Models across Regions and Clouds with Spot Instances. InProceedings of the Twentieth European Con- ference on Computer Systems (EuroSys’25). ACM, 159–175. doi: 10.1145/3689031.3717459

    arXiv

  27. [28]

    Yinan Ni, Xiao Yang, Yuqi Tang, Zhimin Qiu, Chen Wang, and Tingzhou Yuan. 2025. Predictive-LoRA: A Proactive and Fragmentation-Aware Serverless Inference System for LLMs. arXiv:2512.20210 [cs.DC]https://arxiv.org/abs/2512.20210

    arXiv 2025

  28. [29]

    OpenAI. 2026. OpenAI Codex.https://openai.com/codex/

    2026

  29. [30]

    Microsoft. 2026. Microsoft 365 Copilot.https://www.microsoft.co m/en-us/microsoft-365/copilot

    2026

  30. [31]

    Gonzalez, and Ion Stoica

    Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, and Ion Stoica. 2024. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. arXiv:2311.03285 [cs.LG] https://arxiv.org/abs/2311.03285

    Pith/arXiv arXiv 2024

  31. [32]

    Yifan Sui, Hao Wang, Hanfei Yu, Yitao Hu, Jianxun Li, and Hao Wang

  32. [33]

    OpenClaw. 2026. OpenClaw.https://openclaw.ai/

    2026

  33. [34]

    PyTorch Team. 2026. Compile Time Caching in torch.compile. https://docs.pytorch.org/tutorials/recipes/torch_compile_cachi ng_tutorial.html

    2026

  34. [35]

    vLLM Team. 2026. Parallelism and Scaling of vLLM.https://docs.v llm.ai/en/latest/serving/parallelism_scaling/

    2026

  35. [36]

    arXiv:2505.14468 [cs.LG]https: //arxiv.org/abs/2505.14468

    ServerlessLoRA: Minimizing Latency and Cost in Serverless Inference for LoRA-Based LLMs. arXiv:2505.14468 [cs.LG]https: //arxiv.org/abs/2505.14468

    arXiv

  36. [37]

    Yifan Sui, Hanfei Yu, Yitao Hu, Jianxun Li, and Hao Wang. 2024. Pre-Warming is Not Enough: Accelerating Serverless Inference With Opportunistic Pre-Loading. InProceedings of the 2024 ACM Sympo- sium on Cloud Computing(Redmond, W A, USA)(SoCC ’24). Associa- tion for Computing Machinery, New York, NY , USA, 178–195. doi: 10.1145/3698038.3698509

    arXiv 2024

  37. [38]

    Takeshi Yoshimura, Tatsuhiro Chiba, Manish Sethi, Daniel Waddington, and Swaminathan Sundararaman. 2025. Speeding up Model Loading with fastsafetensors. arXiv:2505.23072 [cs.DC]https://arxiv.org/ab s/2505.23072

    Pith/arXiv arXiv 2025

  38. [39]

    Minchen Yu, Ao Wang, Dong Chen, Haoxuan Yu, Xiaonan Luo, Zhuo- hao Li, Wei Wang, Ruichuan Chen, Dapeng Nie, Haoran Yang, and Yu Ding. 2025. Torpor: GPU-Enabled Serverless Computing for Low- Latency, Resource-Efficient Inference. arXiv:2306.03622 [cs.DC] https://arxiv.org/abs/2306.03622

    Pith/arXiv arXiv 2025

  39. [40]

    Yuxing Xiang, Xue Li, Kun Qian, Yufan Yang, Diwen Zhu, Wenyuan Yu, Ennan Zhai, Xuanzhe Liu, Xin Jin, and Jingren Zhou. 2025. Ae- gaeon: Effective GPU Pooling for Concurrent LLM Serving on the Market. InProceedings of the ACM SIGOPS 31st Symposium on Oper- ating Systems Principles(Lotte Hotel World, Seoul, Republic of Korea) (SOSP ’25). Association for Com...

    arXiv 2025

  40. [41]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chu- jie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang...

    Pith/arXiv arXiv 2025

  41. [42]

    Shaoxun Zeng, Minhui Xie, Shiwei Gao, Youmin Chen, and Youyou Lu. 2025. Medusa: Accelerating Serverless LLM Inference with Mate- rialization. InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1(Rotterdam, Netherlands)(ASPLOS ’25). Associa- tion for Computing Machinery, N...

    arXiv 2025

  42. [43]

    Dingyan Zhang, Haotian Wang, Yang Liu, Xingda Wei, Yizhou Shan, Rong Chen, and Haibo Chen. 2025. BlitzScale: Fast and Live Large Model Autoscaling with O (1) Host Caching. In19th USENIX Sym- posium on Operating Systems Design and Implementation (OSDI 25). 275–293.https://www.usenix.org/conference/osdi25/presentati on/zhang-dingyan

    2025

  43. [44]

    Minchen Yu, Rui Yang, Chaobo Jia, Zhaoyuan Su, Sheng Yao, Tingfeng Lan, Yuchen Yang, Zirui Wang, Yue Cheng, Wei Wang, Ao Wang, and Ruichuan Chen. 2026. 𝜆Scale: Enabling Fast Scaling for Serverless Large Language Model Inference. arXiv:2502.09922 [cs.DC]https: //arxiv.org/abs/2502.09922

    arXiv 2026

  44. [45]

    Shan Yu, Jiarong Xing, Yifan Qiao, Mingyuan Ma, Yangmin Li, Yang Wang, Shuo Yang, Zhiqiang Xie, Shiyi Cao, Ke Bao, Ion Stoica, Harry Xu, and Ying Sheng. 2025. Prism: Unleashing GPU Sharing for Cost- Efficient Multi-LLM Serving. arXiv:2505.04021 [cs.DC]https: //arxiv.org/abs/2505.04021

    Pith/arXiv arXiv 2025

  45. [48]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Ef- ficient Execution of Structured Language Model Programs. InAdvances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paqu...

    2024

  46. [49]

    Wenbin Zhu, Zhaoyan Shen, Zili Shao, Hongjun Dai, and Feng Chen

  47. [50]

    A", "A1") 4await A_init2() 5channel.set_state(

    Tangram: Accelerating Serverless LLM Loading through GPU Memory Reuse and Affinity. arXiv:2512.01357 [cs.DC]https://arxiv. org/abs/2512.01357 14 vLLM SGLang SLLM InstantInfer 0 50100 Qwen3- 30B-A3B 0.00 0.25 0.50 0.75 1.00 0 150300 Llama- 3.1-70B 0 150300 Qwen3- 235B-A22B 0 150300 DeepSeek- R1 CDF of TTFT (s) Figure 17.[H20] Cold-start TTFT under a reques...

    arXiv

  48. [2023]

    InProceedings of the 29th Symposium on Operat- ing Systems Principles

    Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th Symposium on Operat- ing Systems Principles. 611–626.https://doi.org/10.1145/3600006. 3613165

    doi:10.1145/3600006

  49. [2025]

    arXiv:2501.14417 [cs.DC]https://arxiv.org/abs/2501.14417

    DeepServe: Serverless Large Language Model Serving at Scale. arXiv:2501.14417 [cs.DC]https://arxiv.org/abs/2501.14417

    Pith/arXiv arXiv