Pith. sign in

REVIEW 4 major objections 6 minor 64 references

Unlock the Potential of Fine-grained LLM Serving via Dynamic Module Scaling

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-grained module-level scaling lets an LLM serving system track fluctuating traffic by replicating and migrating individual decoder layers, cutting cost by over 46% while holding availability.

desk verdict A genuinely new scaling granularity with a plausible system, but the evaluation's unspecified GPU counts and unfair baselines make the headline gains unreliable; worth a serious referee, not acceptance as-is. read the letter →

arxiv 2507.18006 v1 pith:LCUY72C2 submitted 2025-07-24 cs.DC

Jingfeng Wu , Yiyuan He , Minxian Xu , Xitong Gao , Kejiang Ye , Chengzhong Xu This is my paper
classification cs.DC
keywords LLMservingmodule-levelscalinglayerreplicationmigrationauto-scalingGPUmemoryKVcacheelastic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

What is the right unit of scaling for an LLM serving system? This paper's answer is the module, not the whole instance: decoder layers, attention and feed-forward projections, and KV caches. It presents CoCoServe, which replicates hot modules onto idle GPU capacity to raise throughput and migrates overloaded modules off saturated devices to prevent out-of-memory failures and SLO violations. The reported result is that this module-level mechanism adapts to fluctuating request rates in about 0.3 seconds, reduces operational cost by over 46% for near-equivalent performance, and beats whole-instance serving baselines by 14%-75% in latency and 1.16x-4x in throughput. The upshot is that elastic LLM serving becomes a fine-grained resource-allocation problem rather than an all-or-nothing instance provisioning problem.

What carries the argument

The central mechanism is module-level replication and migration, implemented as pre- and post-hook data redirection so that the original model graph is untouched. Replication duplicates selected decoder layers on other devices, adding scatter and all-gather only at the boundaries of the replicated sequence; migration moves a layer, a projection, or the KV cache, letting compute-heavy modules (attention and FFN) and memory-heavy modules (KV cache) be placed on devices that match their resource profiles. The auto-scaling controller wraps these primitives in a generalized Amdahl-style speedup estimator $S_{\mathrm{homo}}(P) = 1/(\gamma + (1-\gamma)/n \cdot \sum_{i=1}^n 1/p_i)$ and a continuity-preserving greedy scale-up search, with a three-phase scale-down fallback that goes from migration to replica eviction to batch-size reduction.

What would settle it

Count the GPUs actually used per logical instance in the Section 6.2 single-instance experiments; if CoCoServe's instance spans more A100s than the baseline's one, re-running the same workload with identical GPU counts and observing whether the 14%-75% latency and 1.16x-4x throughput gaps persist would settle the claim.

Watch

Extended reading notes

Core claim

The central claim is that scaling at the module level changes the economics of LLM serving. CoCoServe copies a subset of a model's decoder layers onto other GPUs to create partial data parallelism, splitting batches across replicas while keeping most layers sequential; it also migrates layers, projection matrices, or KV caches to redistribute compute and memory pressure. The paper's measurements attribute to this mechanism a 17x reduction in OOM occurrence, SLO attainment held to the paper's threshold up to roughly 50 RPS, a memory footprint only about 53.5% of a 4-instance baseline deployment (hence the 46% cost reduction), and consistent latency and throughput advantages over both standard serving baselines across 13B and 70B models.

Load-bearing premise

The reported efficiency gains assume CoCoServe and the baselines are compared under equal GPU resources, and the paper never states how many GPUs each configuration uses in the single-instance evaluation.

Editorial extensions

If this is right

  • Autoscaling no longer needs a spare full model: replicating a handful of decoder layers onto a partially idle GPU can absorb a workload spike at sub-second timescales.
  • A single migrated layer or KV cache can be enough to pull a serving instance back from the OOM cliff, extending the request rate at which SLOs hold.
  • Deployments can be right-sized to a fraction of a model, so GPU fragments that would otherwise be idle become usable capacity; at the measured scale this is a greater-than-46% memory and cost reduction for near-equivalent performance.
  • Because the primitives are hook-based and operate without modifying the model architecture, the mechanism can be layered onto existing serving engines that already provide continuous batching.
  • The speedup model implies a predictable trade-off curve among replication depth, parallelism degree, and communication overhead, which a controller can search at runtime instead of committing to a static parallelism plan.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same primitives could support multi-tenant packing: several logical models could share one GPU by co-locating complementary modules, something the paper's single-cluster evaluation does not measure.
  • The per-module compute and memory classification suggests a natural next experiment: on a heterogeneous cluster, route compute-heavy modules to fast devices and memory-heavy modules to memory-rich devices; the paper states heterogeneous-device compatibility but does not test it.
  • The reported 0.3-second operation time implies the control loop could run far faster than typical instance-level autoscalers, possibly adapting to bursty traffic within one or two scheduling intervals; a trace-driven study of burst recovery time would make this explicit.
  • The Amdahl-style model could be inverted to estimate the marginal value of each additional replica, turning the continuity heuristic into a principled marginal-gain ranking that also accounts for heterogeneous device speeds.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript proposes CoCoServe, an elastic LLM serving system that performs fine-grained module-level replication and migration of decoder layers, projections, and KV caches across GPUs, together with an auto-scaling controller based on a speedup model and a graduated scale-down algorithm. The paper claims sub-second scaling operations, a 46% cost reduction, 14%-75% latency reduction, and 1.16x-4x throughput improvements over Hugging Face Transformers (HFT) and vLLM on LLaMA-13B and LLaMA-70B under request rates from 3 to 61 RPS.

Significance. If the reported results hold under fair resource accounting, module-level scaling is a promising complement to instance-level autoscaling: it can exploit idle resource fragments, mitigate SLO violations and OOM failures, and reduce operational cost. The paper contributes a concrete system design, open-source code, operation-level cost measurements, and an extensive RPS-sweep evaluation. However, the central efficiency claims currently hinge on an unstated GPU allocation across configurations, and the speedup model has a structural monotonicity issue that makes the scale-up algorithm's greedy condition vacuous. These issues must be resolved before the headline advantages can be accepted.

major comments (4)
  1. [§6.2, Fig. 8; §6.1] The paper never states how many GPUs each configuration uses. Figure 8's caption claims a single LLaMA-13B and LLaMA-70B instance deployed on an A100 GPU, but a BF16 LLaMA-70B has roughly 140 GB of weights, which exceeds one A100-40GB GPU, and Section 3.2 describes layer replication experiments on 4 NVIDIA A100 GPUs. The reported 14%-75% latency reduction and 1.16x-4x throughput gains are only meaningful if CoCoServe and the baselines use equal GPU resources; as written, the advantages could reflect additional hardware rather than module scaling. Please report exact GPU counts, tensor/pipeline/data parallelism settings, per-device memory usage, and any CPU offload for every configuration in Sections 6.2 and 6.3.
  2. [§6.3, Fig. 10] The multi-instance cost comparison is not apples-to-apples. The 46% cost reduction is computed by comparing CoCoServe's 2 instances (64,015 MiB) against HFT's 4 instances (119,573 MiB), but the paper does not state whether CoCoServe's 2 logical instances span all 4 GPUs while HFT's 2 instances span only 2 GPUs. If CoCoServe occupies all 4 GPUs, the cost claim collapses and the comparison should instead credit CoCoServe with 4 GPU instances. Please provide a per-configuration table of GPU counts and per-GPU memory, and include a matched-resource comparison in which all systems have the same total GPU allocation.
  3. [§4.1, Eq. (4); Algorithm 1] The speedup model has a built-in monotonicity problem: for any 0<γ<1, decreasing any p_i strictly decreases ∥1⊘P∥1 and hence strictly increases S_homo(P), so Algorithm 1's condition sp > sp_best is satisfied by every admissible replica addition. This makes the greedy filter vacuous and does not encode the continuity-versus-communication trade-off that Section 4.1 motivates. The constant γ is a free parameter and is never calibrated against the Section 3.2 measurements; the paper also does not explain how Eq. (4) is derived from Eq. (3). Please calibrate γ or replace the objective with one that penalizes non-contiguous replication, and show that the resulting algorithm can reject some candidate strategies.
  4. [§6.1, Figs. 6, 8, 10] Although Section 6.1 states that each request rate was repeated five times, no error bars, confidence intervals, or variance statistics are shown for any throughput or latency point. Several reported differences are small (for example, 13.65% latency advantage over vLLM for the 70B model under low load), so without variance information it is impossible to assess whether these differences are statistically meaningful. Please report the full distribution or at least standard deviations for all repeated measurements.
minor comments (6)
  1. [Table 1] The entry 'ffn.gate/up/dwon_proj' contains a typo; it should be 'down_proj'.
  2. [Fig. 9] The metric 'External Frag.' is not defined; please state how memory fragmentation is measured and why the unused memory in HFT and vLLM is classified as external fragmentation.
  3. [§6.3] The text states that CoCoServe delivers 'approximately 90% of its performance' compared with the 4-instance HFT configuration, but the reported differences are 11.28% latency and 9.59% throughput; please clarify what the 90% figure refers to.
  4. [Reference [39]] Reference [39] still contains the placeholder arXiv identifier 2402.XXXXX; a complete citation is needed.
  5. [§6.2 and Algorithm 2] The thresholds T_up and T_down described in Section 5 and the SLO threshold θ used in Algorithm 2 are never given values in the experimental setup; please state them.
  6. [§3.2, Fig. 6] The baseline configuration in Fig. 6 is described as 'a completely unmodified serial execution environment without any parallelism or layer replication,' but it is unclear whether it runs on one GPU or on all four A100 GPUs; please clarify.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the speedup model is an explicit Amdahl-style model stated before evaluation, the headline results are measured against external baselines, and the only self-citations are peripheral.

full rationale

CoCoServe's derivation chain is not circular in the sense defined here. Section 3.2 empirically measures the effect of layer replication on throughput and latency; Section 4.1 then builds an explicit model with W(P), T(P), and S(P), using Amdahl's law as an external starting point. The model contains no parameter fitted to the replication measurements that it is later said to 'align with.' Equation 4 is monotonically increasing in the p_i by construction (adding a replica always reduces the denominator), but the paper does not present this monotonicity as an empirical prediction; it states that the formulation is 'consistent' with the Section 3.2 observations, which is a validation statement, not a fitted-input-as-prediction step. Algorithm 1's greedy check is a design choice based on the model, and it is the end-to-end measurements of Section 6 (latency, throughput, memory, cost against HFT and vLLM) that carry the headline claims. Those measurements are external and not produced by Equation 4. The author self-citations [16] and [51] support prior SLO-related work and batching/scheduling mechanisms inherited from backend engines; they are not load-bearing, and no uniqueness theorem or prior result by the same authors is invoked to force the design. The serious resource-accounting ambiguity in Section 6.2 (a BF16 LLaMA-70B instance 'deployed on an A100 GPU' cannot fit in 40 GB, and the GPU count per configuration is never fixed) is a correctness and fairness threat to the experimental comparison, but it is not a circularity under the required definitions: it does not make the claimed results equivalent to the paper's inputs by definition or by self-citation.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the speedup model's uncalibrated constant gamma, unspecified control thresholds, and an assumed equivalence between replicated-module execution and serial semantics. No new physical entities are introduced.

free parameters (5)
  • gamma (configuration coefficient) = not given (input to Algorithm 1)
    In Eq (4), Shomo = 1/(gamma + (1-gamma)/n * sum_i 1/p_i), where gamma = delta*C/(d*B). The paper calls it a constant related to the cluster configuration but does not state how delta is determined, how gamma is measured or calibrated, or whether it is fitted to the Figure 6 observations. The auto-scaler's decisions depend on this value.
  • T_up and T_down thresholds = not specified
    The Auto-Scaling Controller triggers scale-up when resource vacancy exceeds T_up and scale-down when SLO violation rate surpasses T_down (Section 5). The values are not reported, yet they determine when scaling occurs and therefore shape the cost and latency results.
  • SLO threshold theta = not specified
    Algorithm 2 takes SLO threshold theta as input. The SLO attainment results in Figure 11b depend on this threshold, which is never stated.
  • batch size adjustment step Δbs = e.g., 5
    Algorithm 2 reduces batch size in steps of Δbs. The value is a design choice that affects how gracefully the system degrades under persistent overload.
  • replica size r = estimated per layer (e.g., 605 MB for a LLaMA-13B decoder layer from Table 1)
    Algorithm 1 computes max_replicas per device as available memory divided by replica size r. The exact memory accounting during replication (including KV cache and activations) is not specified.
assumptions (5)
  • standard math Amdahl's law applies to module replication speedup.
    Section 4.1 builds S(P) on Amdahl's law. This is acceptable background, but it requires the workload to be cleanly separable into serial and parallel fractions, which is an approximation for transformer inference.
  • domain assumption Each decoder layer has identical compute time and evenly splitting batches is optimal.
    Eq (4) removes the max function and variables based on even batch splitting and homogeneous devices. If layer latencies are heterogeneous, the formula is invalid.
  • domain assumption The cluster is homogeneous with uniform bandwidth and compute.
    Eq (4) is derived for homogeneous clusters. The paper claims support for heterogeneous devices but does not model them.
  • domain assumption Module replication and migration preserve model semantics and correctness.
    Section 3.1 states hooks transfer inputs and outputs and preserve sequential consistency, but correctness under concurrent replica execution is asserted, not formally proven or stress-tested.
  • ad hoc to paper The communication constant delta is independent of replica contiguity.
    Eq (2) uses a single delta for all communication events, while Section 3.2 argues that continuity between replicas affects communication overhead. This assumption is needed to make Eq (4) tractable, but it undercuts the continuity-based sorting in Algorithm 1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unlock the Potential of Fine-grained LLM Serving via Dynamic Module Scaling." pith.science (2026). https://pith.science/paper/LCUY72C2

@misc{pith2026250718006,
  author       = {Pith},
  title        = {Pith review of: Unlock the Potential of Fine-grained LLM Serving via Dynamic Module Scaling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LCUY72C2}},
  note         = {Machine review of arXiv:2507.18006}
}
read the original abstract

The rise of large language models (LLMs) has created new opportunities across various fields but has also introduced significant challenges in resource management. Current LLM serving systems face a fundamental tension: balancing serving demands with limited resources while adapting to unpredictable traffic patterns. Static deployments lead to suboptimal resource utilization and performance degradation under dynamic workloads. Furthermore, the high cost of adjusting instances hinders dynamic scaling, limiting the true potential of efficient LLM serving. To address this, we propose CoCoServe, an elastic system that facilitates dynamic and fine-grained scaling. Its key innovation lies in the module-level operations for the replication and migration of LLM modules, such as decoder layers and projections. Through a comprehensive analysis of the trade-offs associated with these operations, we develop an auto-scaling mechanism that dynamically regulates module-level resource allocation and performance optimization, enabling a more cost-effective deployment of LLMs. Our evaluation demonstrates that the scaling operations employed by CoCoServe exhibit excellent scalability and can reduce costs by 46% while maintaining availability. Compared to state-of-the-art LLM serving systems (e.g., Hugging Face Transformers and vLLM), our approach reduces latency by 14%-75% and achieves 1.16x-4x throughput on average across different model sizes and workloads.

Figures

Figures reproduced from arXiv: 2507.18006 by the authors.

Figure 1
Figure 1. Illustration of the scaling mechanism in CoCoServe. Yellow and green blocks represent different instances deployed across three devices labeled A, B, and C. Blocks filled with left-slanted and right-slanted lines represent replicated and migrated modules respectively. In recent years, an increasing number of MaaS products have entered the market, covering a wide range of application scenar￾ios, including conversatio… view at source ↗
Figure 2
Figure 2. GPU resource utilization comparison between HFT and vLLM serving frameworks across different request rates, conducted with a single LLaMA-13B instance deployed on an A100 GPU, with each test repeated five times. Suboptimal Utilization. Modern LLM serving systems often suffer from suboptimal resource utilization due to two primary factors. First, at low request rates (RPS ≤ 10), sig￾nificant computational and memory … view at source ↗
Figure 4
Figure 4. Illustration of the replication for decoder layers. Yellow blocks represent the yellow layers deployed on the main device, while green blocks represent replicas deployed on another device. Replication. The replication operation creates replicas of selected model modules across available devices, enabling parallel processing of requests through shared data compu￾tation. As shown in [PITH_FULL_IMAGE:figures/full_fig_… view at source ↗
Figures from the paper (8 more)
Figure 3
Figure 3. Figure 3: Performance comparison between the default con￾figuration and migrating 1 layer to another device under high load conditions, conducted with a single LLM-13B instance deployed on an A100 GPU across varying RPS rates. Costly Instance Adjustments. Runtime adjustment of m…
Figure 5
Figure 5. Figure 5: Illustration of the migration for modules in a decoder layer. Yellow modules are computation-intensive, while green modules are memory-intensive. The migration operation al￾lows them to be transferred to different devices respectively. Migration. The migration operatio…
Figure 6
Figure 6. Figure 6: Performance analysis of layer replication and paral￾lelism strategies under varying request rates In Figs 6a and 6b, we maintain a fixed parallelism degree of 2 while progressively increasing the layer replication count from baseline to 30 layers. The throughput result…
Figure 7
Figure 7. Figure 7: CoCoServe’s component interaction sequence. Metrics Monitor. The monitor collects metrics including GPU utilization, memory utilization, tokens per second, end￾to-end latency, etc., and feeds these metrics back to the Controller. For utilization metrics, the monitor ga…
Figure 8
Figure 8. Figure 8: Throughput and latency comparison between CoCoServe, HFT, and vLLM for a single LLaMA-13B and LLaMA-70B instance deployed on an A100 GPU under different load conditions. 36.89% higher throughput on average. The larger LLaMA￾70B model shows even more impressive results …
Figure 9
Figure 9. Figure 9: Memory utilization comparison. These results demonstrate that CoCoServe consistently outperforms both HFT and vLLM systems across all tested scenarios. In summary, under various RPS test scenarios, CoCoServe demonstrates significant performance advantages. The combined…
Figure 11
Figure 11. Figure 11: Performance comparison between serving systems: (a) shows CoCoServe reduces OOM occurrences by 17× compared to HFT, while (b) demonstrates superior SLO attainment of CoCoServe compared to HFT and vLLM. Our experiments also revealed significant differences in memory us…
Figure 10
Figure 10. Figure 10: Throughput and latency comparison between CoCoServe (2 LLM instances) and HFT (2 and 4 LLM instances) for LLaMA-13B and LLaMA-70B models deployed on an A100 GPU under different load conditions. The multi-instance evaluation clearly demonstrates Co￾CoServe’s advantages…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

64 extracted references · 50 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, and Florencia Leoni et al. Aleman. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Llama 2: Open foundation and fine- tuned chat models.https://arxiv.org/abs/2307.09288, 2023

    HugoTouvron,ThibautLavril,GautierIzacard,XavierMartinet,Marie- AnneLachaux,TimothéeLacroix,BaptisteRozière,NamanGoyal,Eric Hambro, and Faisal Azhar et al. Llama 2: Open foundation and fine- tuned chat models.https://arxiv.org/abs/2307.09288, 2023. Accessed: 2025-05-04

    arXiv 2023

  3. [3]

    Deepseek-v3 technical report.ArXiv, abs/2412.19437, 2024

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, and Bing-Li Wang et al. Deepseek-v3 technical report.ArXiv, abs/2412.19437, 2024

    arXiv 2024

  4. [4]

    Black-box tuning for language-model-as-a-service.ArXiv, abs/2201.03514, 2022

    Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for language-model-as-a-service.ArXiv, abs/2201.03514, 2022

    arXiv 2022

  5. [5]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhang- hao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as- a-judge with mt-bench and chatbot arena. InProceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, U...

    work page 2023

  6. [6]

    Yu, and Lichao Sun

    Yihan Cao, Siyu Li, Yixin Liu, Zhiling Yan, Yutong Dai, Philip S. Yu, and Lichao Sun. A comprehensive survey of ai-generated content aigc: A history of generative ai from gan to chatgpt.ArXiv, abs/2303.04226, 2023

    arXiv 2023

  7. [7]

    Evaluatinglarge language models trained on code.ArXiv, abs/2107.03374, 2021

    MarkChen,JerryTworek,HeewooJun,QimingYuan,HenriquePondé, JaredKaplan,HarrisonEdwards,andYuraBurdaetal. Evaluatinglarge language models trained on code.ArXiv, abs/2107.03374, 2021

    arXiv 2021

  8. [8]

    Code llama: Open foundation models for code.ArXiv, abs/2308.12950, 2023

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Tan, Yossi Adi, Jingyu Liu, and Tal Remez et al. Code llama: Open foundation models for code.ArXiv, abs/2308.12950, 2023

    arXiv 2023

Show all 64 references
  1. [9]

    Accessed: Apr

    Dall·e 3.https://openai.com/dall-e-3, 2025. Accessed: Apr. 23, 2025

    2025

  2. [10]

    Accessed: Apr

    Openai api.https://openai.com/blog/openai-api, 2025. Accessed: Apr. 23, 2025

    2025

  3. [11]

    Smoothquant: Accurate and efficient post-training quantization for large language models, 2024

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models, 2024

    2024

  4. [12]

    Plug-and-play: An efficient post-training pruningmethodforlargelanguagemodels

    Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. Plug-and-play: An efficient post-training pruningmethodforlargelanguagemodels. In TheTwelfthInternational Conference on Learning Representations, 2024

    2024

  5. [13]

    Awq: Activation-aware weight quantization for llm compression and acceleration, 2024

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei- Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration, 2024

    2024

  6. [14]

    Zico Kolter

    Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models, 2024

    2024

  7. [15]

    Multiplexing dynamic deep learning workloads with slo-awareness in gpu clusters

    Wenyan Chen, Chengzhi Lu, Huanle Xu, Kejiang Ye, and Chengzhong Xu. Multiplexing dynamic deep learning workloads with slo-awareness in gpu clusters. InProceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, page 589–604, New York, NY, USA, 2025. Asso...

    2025

  8. [16]

    Cloudnativesim: A toolkit for modeling and simulation of cloud-native applications

    JingfengWu,MinxianXu,YiyuanHe,KejiangYe,andChengzhongXu. Cloudnativesim: A toolkit for modeling and simulation of cloud-native applications. Software: Practice and experience, 2025

    2025

  9. [17]

    Llminaflash:Efficientlargelanguagemodelinferencewith limited memory, 2024

    KeivanAlizadeh,ImanMirzadeh,DmitryBelenko,KarenKhatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. Llminaflash:Efficientlargelanguagemodelinferencewith limited memory, 2024

    2024

  10. [18]

    Spotserve: Serving generative large language models on preemptible instances

    Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia. Spotserve: Serving generative large language models on preemptible instances. InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Op...

    2024

  11. [19]

    Llumnix:Dynamicschedulingforlargelanguage model serving

    Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, YongLi,andWeiLin. Llumnix:Dynamicschedulingforlargelanguage model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 173–191, Santa Clara, CA, July 2024. USENIX Association

    2024

  12. [20]

    Serving heterogeneous machine learning models on multi-gpu servers with spatio-temporal sharing

    Seungbeom Choi, Sunho Lee, Yeonjae Kim, Jongse Park, Youngjin Kwon, and Jaehyuk Huh. Serving heterogeneous machine learning models on multi-gpu servers with spatio-temporal sharing. In2022 USENIX Annual Technical Conference (USENIX ATC 22), pages 199– 216, 2022

    2022

  13. [21]

    Inferline:latency-aware provisioningandscalingforpredictionservingpipelines.In Proceedings of the 11th ACM Symposium on Cloud Computing, pages 477–491, 2020

    Daniel Crankshaw, Gur-Eyal Sela, Xiangxi Mo, Corey Zumar, Ion Stoica,JosephGonzalez,andAlexeyTumanov. Inferline:latency-aware provisioningandscalingforpredictionservingpipelines.In Proceedings of the 11th ACM Symposium on Cloud Computing, pages 477–491, 2020

    2020

  14. [22]

    Optimizing llm inference throughput via memory-aware and sla-constrained dynamic batching, 2025

    Bowen Pang, Kai Li, and Feifan Wang. Optimizing llm inference throughput via memory-aware and sla-constrained dynamic batching, 2025

    2025

  15. [23]

    Alloystack: A library operating system for serverless workflow applications.Pro- ceedings of the Twentieth European Conference on Computer Systems, 2025

    JianingYou,KangChen,LaipingZhao,YimingLi,YichiChen,Yuxuan Du, Yanjie Wang, Luhang Wen, Keyang Hu, and Keqiu Li. Alloystack: A library operating system for serverless workflow applications.Pro- ceedings of the Twentieth European Conference on Computer Systems, 2025

    2025

  16. [24]

    Lora-flow: Dynamic lora fusion for large lan- guage models in generative tasks

    Hanqing Wang, Bowen Ping, Shuo Wang, Xu Han, Yun Chen, Zhiyuan Liu, and Maosong Sun. Lora-flow: Dynamic lora fusion for large lan- guage models in generative tasks. InAnnual Meeting of the Association for Computational Linguistics, 2024

    2024

  17. [25]

    Accessed: Apr

    Ollama - get up and running with large language models.https: //ollama.com/, 2025. Accessed: Apr. 23, 2025

    2025

  18. [26]

    ThomasWolf,LysandreDebut,VictorSanh,JulienChaumond,Clement Delangue,AnthonyMoi,PierricCistac,TimRault,RémiLouf,Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quenti...

    2020

  19. [27]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, and Lianmin et al. Zheng. Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th Symposium 13 EuroSys ’26, April 13–April 16, 2026, Edinburgh, UK Jingfeng Wu ∗, Yiyuan He∗,...

    2026

  20. [28]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  21. [29]

    Llm inference serving: Survey of recent advances and opportunities, 2024

    Baolin Li, Yankai Jiang, Vijay Gadepally, and Devesh Tiwari. Llm inference serving: Survey of recent advances and opportunities, 2024

    2024

  22. [30]

    Optimizing mixture-of-experts inference time combining model deployment and communication scheduling, 2024

    Jialong Li, Shreyansh Tripathi, Lakshay Rastogi, Yiming Lei, Rui Pan, and Yiting Xia. Optimizing mixture-of-experts inference time combining model deployment and communication scheduling, 2024

    2024

  23. [31]

    Mooncake: A kvcache-centric disag- gregated architecture for llm serving, 2024

    Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: A kvcache-centric disag- gregated architecture for llm serving, 2024

    2024

  24. [32]

    Spinfer: Leveraging low-level sparsity for efficient large language model inference on gpus

    Ruibo Fan, Xiangrui Yu, Peijie Dong, Zeyu Li, Gu Gong, Qiang Wang, Wei Wang, and Xiaowen Chu. Spinfer: Leveraging low-level sparsity for efficient large language model inference on gpus. InProceedings of theTwentiethEuropeanConferenceonComputerSystems ,EuroSys’25, page 243–260,...

    2025

  25. [33]

    H2o:Heavy-hitteroracleforefficientgenerativeinference of large language models.Advances in Neural Information Processing Systems, 36:34661–34710, 2023

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett,etal. H2o:Heavy-hitteroracleforefficientgenerativeinference of large language models.Advances in Neural Information Processing Systems, 36:34...

    2023

  26. [34]

    Splitwise: Efficient gen- erative llm inference using phase splitting.2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 118–132, 2023

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Inigo Goiri, Aashaka Shah, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient gen- erative llm inference using phase splitting.2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 118–132, 2023

    2024

  27. [35]

    Piotr Nawrot, Adrian Lncucki, Marcin Chochowski, David Tarjan, and Edoardo M. Ponti. Dynamic memory compression: retrofitting llms for accelerated inference. InProceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024

    2024

  28. [36]

    Inference without interference: Disaggregate llm inference for mixed downstream workloads, 2024

    Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, ShuangChen,HaoFeng,ChenxiWang,SaWang,YungangBao,Ninghui Sun, and Yizhou Shan. Inference without interference: Disaggregate llm inference for mixed downstream workloads, 2024

    2024

  29. [37]

    Dynamollm:Designingllminferenceclustersforperformance and energy efficiency, 2024

    Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Josep Torrellas, and Esha Choukse. Dynamollm:Designingllminferenceclustersforperformance and energy efficiency, 2024

    2024

  30. [38]

    Skyserve:Servingaimodels across regions and clouds with spot instances

    Ziming Mao, Tian Xia, Zhanghao Wu, Wei-Lin Chiang, Tyler Griggs, RomilBhardwaj,andZonghengYangetal. Skyserve:Servingaimodels across regions and clouds with spot instances. InProceedings of the 20thEuropeanConferenceonComputerSystems(EuroSys2025) ,pages 159–175, 2025

    2025

  31. [39]

    Towardsefficientandreliablellmserving: A real-world workload study.arXiv preprint arXiv:2402.XXXXX, 2024

    Yuxin Wang, Yuhan Chen, Zeyu Li, Zhenheng Tang, Rui Guo, Xin Wang,andQiangWangetal. Towardsefficientandreliablellmserving: A real-world workload study.arXiv preprint arXiv:2402.XXXXX, 2024

    2024

  32. [40]

    Usher: Holistic interference avoidance for resource optimized ML inference

    Sudipta Saha Shubha, Haiying Shen, and Anand Iyer. Usher: Holistic interference avoidance for resource optimized ML inference. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 947–964, Santa Clara, CA, July 2024. USENIX Association

    2024

  33. [41]

    Bingya Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. Loongserve: Efficiently serving long-context large language models with elastic sequence parallelism.Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, 2024

    2024

  34. [42]

    Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, and Xuanzhe Liu et al. Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 193–210...

    2024

  35. [43]

    Alpaserve: Statistical multiplexing with model parallelism for deep learning serving

    Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, and Yanping Huang et al. Alpaserve: Statistical multiplexing with model parallelism for deep learning serving. In17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23), pages 663...

    2023

  36. [44]

    Aditya Dhakal, Sameer G Kulkarni, and K. K. Ramakrishnan. Gslice: controlled spatial sharing of gpus for a scalable inference platform. In Proceedings of the 11th ACM Symposium on Cloud Computing, SoCC ’20, page 492–506, New York, NY, USA, 2020. Association for Computing Machinery

    2020

  37. [45]

    Yadwadkar, and Christos Kozyrakis

    Francisco Romero, Qian Li, Neeraja J. Yadwadkar, and Christos Kozyrakis. Infaas: Automated model-less inference serving. In2021 USENIX Annual Technical Conference (USENIX ATC 21), pages 397–

  38. [46]

    McAfee,MichaelAndersch,MohammadShoeybi,andBryanCatanzaro

    Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence C. McAfee,MichaelAndersch,MohammadShoeybi,andBryanCatanzaro. Reducing activation recomputation in large transformer models.ArXiv, abs/2205.05198, 2022

    2022 arXiv

  39. [47]

    On parallel processing systems: Amdahl’s law generalized and some results on optimal design

    Leonard Kleinrock and Jau-Hsiung Huang. On parallel processing systems: Amdahl’s law generalized and some results on optimal design. IEEE Trans. Software Eng., 18:434–447, 1992

    1992

  40. [48]

    xformers: A modular and hackable transformer modelling library.https://github.com/facebookresearch/ xformers, 2022

    Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong,VittorioCaggiano,SeanNaren,MinXu,JieruHu,MartaTintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: A modular and hackable transformer modelling ...

    2022

  41. [49]

    https://developer.nvidia.com/management-library-nvml, 2025

    Nvml. https://developer.nvidia.com/management-library-nvml, 2025. Accessed: Apr. 23, 2025

    2025

  42. [50]

    Orca: A distributed serving system for transformer- based generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for transformer- based generative models. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, Carls- bad, CA, July 202...

    2022

  43. [51]

    Uellm: A unified and efficient approach for large language model inference serving

    Yiyuan He, Minxian Xu, Jingfeng Wu, Wanyi Zheng, Kejiang Ye, and Chengzhong Xu. Uellm: A unified and efficient approach for large language model inference serving. InService-Oriented Computing: 22nd International Conference, ICSOC 2024, Tunis, Tunisia, December 3–6,2024,Procee...

    2024

  44. [52]

    Stanford alpaca: An instruction- following llama model.https://github.com/tatsu-lab/stanford_alpaca,

    RohanTaori,IshaanGulrajani,TianyiZhang,YannDubois,XuechenLi, Carlos Guestrin, and Percy Liang et al. Stanford alpaca: An instruction- following llama model.https://github.com/tatsu-lab/stanford_alpaca,

  45. [53]

    Mepipe: Democratizing llm training with memory- efficient slice-level pipeline scheduling on cost-effective accelerators

    ZhenboSun,ShengqiChen,YuanweiWang,JianSha,GuanyuFeng,and Wenguang Chen. Mepipe: Democratizing llm training with memory- efficient slice-level pipeline scheduling on cost-effective accelerators. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’2...

    2025

  46. [54]

    Le, and Z

    Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, and Z. Chen. Gpipe: Efficient training of giantneuralnetworksusingpipelineparallelism. In NeuralInformation Processing Systems, 2018

    2018

  47. [55]

    Mist:Efficientdistributedtrainingoflargelanguagemodelsviamemory- parallelism co-optimization

    Zhanda Zhu, Christina Giannoula, Muralidhar Andoorveedu, Qidong Su, Karttikeya Mangalam, Bojian Zheng, and Gennady Pekhimenko. Mist:Efficientdistributedtrainingoflargelanguagemodelsviamemory- parallelism co-optimization. InProceedings of the Twentieth European Conference on Co...

    2025

  48. [56]

    14 Unlock the Potential of Fine-grained LLM Serving via Dynamic Module Scaling EuroSys ’26, April 13–April 16, 2026, Edinburgh, UK

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, JaredCasper,andBryanCatanzaro.Megatron-lm:Trainingmulti-billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053, 2019. 14 Unlock the Potential of Fine-grained LLM Serving via Dyn...

    1909 arXiv

  49. [57]

    Alpa: Automating inter and intra- operator parallelism for distributed deep learning

    Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, and Yanping Huang et al. Alpa: Automating inter and intra- operator parallelism for distributed deep learning. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 559–578, C...

    2022

  50. [58]

    Fast state restoration in llm serving with hcache

    Shiwei Gao, Youmin Chen, and Jiwu Shu. Fast state restoration in llm serving with hcache. InProceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, page 128–143, New York, NY, USA, 2025. Association for Computing Machinery

    2025

  51. [59]

    Fast and live model auto scaling with o(1) host caching, 2024

    Dingyan Zhang, Haotian Wang, Yang Liu, Xingda Wei, Yizhou Shan, Rong Chen, and Haibo Chen. Fast and live model auto scaling with o(1) host caching, 2024

    2024

  52. [60]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. InProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20,...

    2020

  53. [61]

    Infinigen: Efficientgenerativeinferenceoflargelanguagemodelswithdynamickv cache management

    WonbeomLee,JungiLee,JunghwanSeo,andJaewoongSim. Infinigen: Efficientgenerativeinferenceoflargelanguagemodelswithdynamickv cache management. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 155–172, Santa Clara, CA, July 2024. USENIX Assoc...

    2024

  54. [411]

    USENIX Association, July 2021

    2021

  55. [2020]

    Association for Computational Linguistics

  56. [2023]

    GitHub repository, Accessed: 2025-05-04

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.