Pith. sign in

REVIEW 4 major objections 4 minor 71 references

OpScale: Operator-level Provisioning and Autoscaling for LLM Serving

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Operator-level scaling, not whole-model replicas, meets LLM latency SLOs with up to 36.3% fewer GPUs.

desk verdict A promising operator-level autoscaling system with a strong controlled evaluation, but the routing/KV-cache affinity gap is a load-bearing unspecified mechanism. read the letter →

arxiv 2608.13499 v1 pith:PSSB7EWO submitted 2026-08-13 cs.DC

Xingqi Cui , Chieh-Jan Mike Liang , Ziang Tang , Jiarong Xing , Haoran Qiu This is my paper · ORCID
classification cs.DC
keywords operator-levelautoscalingLLMinferenceservingGPUresourceprovisioningSLO-awarescalingqueueingmodelinterference-awareplacementoperatorheterogeneity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the right unit of scaling for LLM serving is the individual operator, not the whole model replica. Model-level autoscaling is slow because loading a multi-billion-parameter model takes around ten seconds, and coarse because it duplicates every operator even when only one is the bottleneck. The paper builds OPSCALE, a cluster orchestration system that profiles each operator's sensitivity, computes per-operator replica counts with a queueing model, and packs replicas onto GPUs while accounting for interference. On production traces replayed across up to 40 A100s and 24 GB200s, the claimed payoff is meeting time-to-first-token SLOs with up to 36.3% fewer GPUs and 28% less power, or up to 44% higher token throughput under a fixed GPU budget. The central claim is that fine-grained elasticity is a practical, cluster-level efficiency win, not just a theoretical possibility.

What carries the argument

The load-bearing object is the operator-level deployment plan: for each operator $v$ in the model DAG, OPSCALE chooses batch size $B_v$, replica count $R_v$, tensor-parallel shards $P_v$, device assignment $A_v$, and SM allocation $S_v$. The core identity is the SLO decomposition $T^{itr}=\sum_{v\in V}(T_v+C_v+W_v)$, where each operator is modeled as an $M/M/R$ queue so $W_v$ follows the Erlang-C formula. A greedy critical-path algorithm adjusts $R_v$ and $B_v$ to push total latency under the SLO, and placement is done by a best-fit-decreasing bin-packing heuristic using a measured interference factor $I_{d,v}(b,p)=\tilde{T}_v(b,p)/T_v(b,p)\ge 1$ to reject placements that would inflate latency beyond the SLO. The machinery is what converts the heterogeneity observation into a concrete provisioning plan at millisecond timescales.

What would settle it

Replay a held-out production trace through OPSCALE's provisioning and placement path, comparing the queueing model's predicted per-operator waiting times and P99 TTFT against measured values; if the P90 relative error exceeds the reported 1.9% on a trace not used for calibration, or if SLO attainment drops materially below the reported 97–98% at the same GPU count, the Poisson/Erlang-C foundation is the weak link.

Watch

Extended reading notes

Core claim

The central discovery the paper argues for is operator heterogeneity: operators within one model differ widely in how their compute time, memory footprint, and SM usage respond to sequence length, batch size, and arrival rate, so scaling them uniformly wastes resources. OPSCALE treats the model as an operator DAG and scales only the operators that sit on the current critical path, using an SLO constraint of the form $T^{itr}=\sum_{v}(T_v+C_v+W_v)$, with waiting $W_v$ from an Erlang-C queueing model. The paper reports that this yields SLO attainment of 98.4% for a dense model and 98.1% for a mixture-of-experts model, while model-level baselines achieve 84–97%, and that the savings grow with model size and operator heterogeneity. It further claims that op-level elasticity helps even in static deployments, extracting up to 44% more throughput from the same GPUs.

Load-bearing premise

The load-bearing assumption is that each operator behaves as an independent $M/M/R$ queue with Poisson arrivals and exponential service times, even though the paper's own Figure 2 shows production traffic is bursty within ten-second windows.

Editorial extensions

If this is right

  • Sub-second elasticity becomes the default: scaling only a bottleneck operator takes under half a second instead of the ten-plus seconds needed to load a full model replica.
  • GPU and power budgets for LLM serving can shrink by roughly a third at the same SLO attainment, with the largest gains on mixture-of-experts models and on clusters with fast interconnects.
  • Static GPU clusters can serve more tokens under the same latency SLO by directing the fixed budget to bottleneck operators; the paper reports up to 44% higher throughput at 40 GPUs.
  • Operator-level scaling is complementary to existing model-level autoscaling policies and to finer kernel-level spatial multiplexing, so it can be adopted as a mechanism beneath those policies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: prefill-decode disaggregation should multiply OPSCALE's gains, since the paper's own appendix shows prefill stages benefit two to three times more than decode stages; a split system could apply operator-level elasticity mainly to prefill.
  • Editorial inference: the savings should track operator heterogeneity across model families, so a testable prediction is that multimodal and mixture-of-experts models at the same parameter count will show larger GPU savings than dense text models; the paper's MoE results are consistent with this.
  • Editorial inference: the Poisson/$M/M/R$ assumption is the likeliest point of failure in a new deployment; swapping the Erlang-C waiting formula for a batch-aware or state-dependent arrival model would preserve the architecture while potentially tightening the savings estimate.
  • Editorial inference: if the mechanism generalizes, the same operator-DAG provisioning logic could apply to other generative model families whose operators also have heterogeneous sensitivity profiles.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes OpScale, a cluster-scale orchestration framework that changes the unit of autoscaling for LLM serving from whole model replicas to individual operators. It motivates this with an operator characterization study showing heterogeneity in compute, memory, and SM-allocation sensitivity, and then presents a control plane that profiles operators, provisions replicas via a queueing-theoretic model (M/M/R with Erlang-C), and places replicas with a contention-aware best-fit heuristic. The execution plane extends nano-vLLM with dynamic operator replicas, multi-stream management, and request routing. On up to 40 A100s and 24 GB200s with production traces, the paper reports up to 36.3% fewer GPUs, 28% lower power, and up to 44% higher throughput compared with model-level autoscaling baselines (DynamoLLM, AIBrix, Production Stack) ported onto the same engine.

Significance. If the central claims hold, this is a substantial contribution: it reframes a basic design question in LLM serving (what should scale), provides a working end-to-end system, and quantifies the benefit at cluster scale. The evaluation has genuine strengths: all baselines are run on the same nano-vLLM data plane, the traces are external production traces, the profiling model is validated on held-out interference measurements, and the offline-vs-online optimality gap is at least explicitly stated. The significance is conditional, however, on closing two technical gaps: the routing specification for KV-cache affinity, which is central to correctness of multi-token serving, and the handling of bursty arrivals by a Poisson-based provisioning model.

major comments (4)
  1. [§4.5, §5.2] The execution-plane description never specifies how a request's KV cache follows it across attention operator replicas. A request's KV cache is created at prefill on the replica that runs the attention operator for that request, and every decode iteration must read that same cache. The manuscript states that 'requests are dispatched to operator replicas via shortest-queue routing, weighted by replica capacity,' but it does not state whether routing is sticky per request, whether a shared or remote KV cache is accessible to all attention replicas, or whether KV entries are transferred between replicas. The ElasticBlockManager/kvcached mechanism handles memory allocation and reclamation, not per-request placement affinity. If routing is not sticky, the system as specified cannot serve multi-token requests; if it is sticky, the independent M/M/R model in Eq. (2) no longer describes the dispatch balance; and if KV is moved or read remotely, per-token latency and interconnect costs are omitted from the TBT model. This missing specification is load-bearing for the central SLO-attainment and sub-second-elasticity claims.
  2. [§3.1, Eq. (2); §5.7] The provisioning model treats each operator as an M/M/R queue with Poisson arrivals and exponential service times, and uses the Erlang-C formula to determine replica counts. However, the paper's own Figure 2 shows production LLM traffic with peak-to-min ratios of about 2x (Chat) and 5.8x (Code) within 10-second windows, which is far from Poisson. Section 5.7 reports an average queueing-model error of 0.8% 'across all evaluated scenarios and bursty trace replays,' but it does not state whether that validation set was held out from the data used to build the profile, nor how a Poisson assumption yields accurate waiting-time predictions under such burstiness. Since replica counts and SLO compliance in §4.3 and §5.2 are computed from this model, the paper needs to specify the validation protocol and, if the model is not Poisson, present the actual waiting-time distribution used.
  3. [§5.2–§5.3, Figs. 13 and 27] The paper formulates both TTFT and TBT as SLOs in §3.1 and Eq. (1), but the main autoscaling evaluation reports only P99 TTFT (Figures 13 and 27) and never presents TBT attainment for OpScale or the baselines. If the resource savings are achieved at the cost of degraded time-between-tokens, the headline 'meets latency SLOs' claim is incomplete. The authors should report TBT attainment (or state that TBT was not used as a scaling constraint), especially because decode-phase latency is the metric most sensitive to the KV-cache routing issue raised above.
  4. [§3, §4.3, §5.6] The manuscript repeatedly states that the online greedy provisioning heuristic has resource cost within 8% of the brute-force oracle, but no experiment or table in §5.6 (or elsewhere) measures this optimality gap; §5.6 reports only plan-generation latency. This is a quantitative claim that the reader cannot verify from the presented evidence. Either add an evaluation comparing the greedy plan's cost to the exhaustive oracle across workloads, or remove the 8% figure and state the heuristic's optimality as unquantified.
minor comments (4)
  1. [§5.1] The baselines are re-implementations on nano-vLLM, and the paper says they 'reproduce each baseline's autoscaling policy,' but it does not say how the ported policies were validated against the original systems; a sentence on fidelity would help readers interpret the magnitude of the reported savings.
  2. [§4.1] There is a spacing typo: 'would requireweeksof GPU time' should read 'would require weeks of GPU time.'
  3. [Table 1] The column headers 'P99 latency,' 'P90 latency,' 'Avg. latency' should be clarified as 'P99 scale-up latency' and similar, to avoid confusion with request-level latency.
  4. [§5.6] The claim that profiling a 57B model completes in under an hour on a single GB200 node is stated without a supporting measurement; a timing breakdown of the sparse-sampling profiling pipeline would strengthen this claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found; headline savings are measured against external baselines and the analytical model is a constrained relaxation, not a self-fulfilling definition.

full rationale

I walked the paper's claimed derivation chain: (1) operator profiling and sensitivity characterization (Section 2.1), (2) an analytical M/M/R queueing formulation for operator-level provisioning (Section 3.1), (3) a greedy provisioning and placement algorithm (Sections 4.3-4.4), and (4) end-to-end evaluation on real A100/GB200 clusters against DynamoLLM, AIBrix, and the vLLM Production Stack using production traces (Section 5). The headline results (up to 36.3% fewer GPUs, 28% less power, 44% higher throughput) are measured outcomes, not outputs of the paper's own model, so they do not reduce to the model's inputs by construction. The analytical benefit analysis in Section 3.2 is a legitimate optimization consequence: model-level provisioning is a restricted feasible case of operator-level provisioning, so the non-negativity of savings is structural, but the reported magnitudes depend on profiled operator sensitivities and the queueing model, and they are later corroborated by independent hardware measurements in Section 5.3. The queueing model validation in Section 5.7 reports prediction error against measured runtime behavior; even if the held-out status of the queueing validation is not fully explicit, the model's service rates come from offline microbenchmarks rather than being fitted to the evaluation's headline metrics. Self-citations (e.g., kvcached/Prism [66,67] for ElasticBlockManager, and µ-Serve [54] as a power baseline) are implementation components or comparison baselines, not load-bearing justifications of the central elasticity claim. No equation is used to define its own predicted quantity, and no fitted parameter is renamed as a prediction. The paper is self-contained against external benchmarks, so the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The main empirical inputs are the per-operator profiles (computation time, memory, communication) and the interference lookup table. The interference table is a fitted model of co-location slowdown. The queueing model and the multiplicative interference approximation are domain assumptions that may not hold under bursty, correlated traffic.

free parameters (2)
  • Interference factor lookup table Id,v(b,p) = Empirical table over batch size b and SM fraction p per operator pair
    Section 4.4 defines Id,v(b,p) as the measured ratio of co-execution to isolated latency, stored in a lookup table. It is a fitted empirical model of GPU contention used in placement.
  • SLO tolerance buffer epsilon = Not specified
    Algorithm 1 uses epsilon as a downscaling safety margin around the SLO. The value is not reported, so the tuning is not reproducible.
assumptions (5)
  • domain assumption Each operator is an independent M/M/R queue with Poisson arrivals and exponential service times.
    Section 3.1 Eq. (2) uses the Erlang-C formula. The paper's own Figure 2 shows bursty, non-Poisson production traffic.
  • domain assumption End-to-end iteration latency is the sum of per-operator computation, communication, and waiting times, with no queue interdependencies.
    Eq. (1) sums Tv + Cv + Wv over all operators, treating the DAG as a single critical path of independent queues.
  • domain assumption Three-way co-location interference is the product of pairwise interference factors.
    Section 4.4 approximates three-way co-location by multiplying pairwise factors; no three-way validation is shown.
  • domain assumption CUDA Green Contexts provide reliable SM partitioning for operators with run-to-complete kernels.
    Section 2.2 and 4.5 rely on spatial sharing via Green Contexts. If hardware or driver behavior differs, the interference model and SM allocation break.
  • domain assumption Operators are structurally equivalent across layers and model generations, so profiles transfer.
    Section 4.2 reuses profiles across layers and models, citing examples like SwiGLU MLP and RoPE attention. This requires unchanged kernel implementations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OpScale: Operator-level Provisioning and Autoscaling for LLM Serving." pith.science (2026). https://pith.science/paper/PSSB7EWO

@misc{pith2026260813499,
  author       = {Pith},
  title        = {Pith review of: OpScale: Operator-level Provisioning and Autoscaling for LLM Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PSSB7EWO}},
  note         = {Machine review of arXiv:2608.13499}
}
read the original abstract

Achieving cost efficiency while meeting strict user-facing SLOs (e.g., time-to-first-token) remains a fundamental challenge for cloud GPU clusters serving large language models (LLMs). Autoscaling is the key mechanism for cluster resource management, yet a basic system design question is open for serving LLMs: what should be the unit of scaling? Existing approaches primarily treat the entire model as a monolithic scaling unit--simple but unable to capture the fine-grained dynamics of inference workloads. As a result, such coarse-grained scaling often leads to either SLO violations under bursty demand or significant GPU under-utilization. Our characterization reveals substantial operator heterogeneity, exposing operator-level elasticity as a viable scaling primitive. We present OpScale, a practical operator-level orchestration framework of profiling, provisioning, placement, and runtime serving. OpScale is designed to tackle the high complexity and the space explosion problem, arising from operating at this finer granularity. Evaluated with production traces on up to 40 A100s and 24 GB200s, OpScale attains SLOs with up to 36.3% fewer GPUs and 28% less power, or achieves 44% higher throughput under fixed cost budgets.

Figures

Figures reproduced from arXiv: 2608.13499 by the authors.

Figure 1
Figure 1. LLM serving at model- vs. operator-level. First, model-level scaling is resource-inefficient. Genera￾tive models are data flows of heterogeneous operators (e.g., at￾tention, linear transformation, and normalization). Forcing all operators to scale uniformly causes over-provisioning, where non-bottleneck operators consume precious GPU memory and cycles. Second, model-level autoscaling is slow. Loading a full model on… view at source ↗
Figure 2
Figure 2. Burstiness in production LLM inference traffic [57] leads to large variability even within a 10-second timescale (for both Chat and Code services), making model-level au￾toscaling hard to adapt promptly. 1s 2s 5s 10s 20s 30s Model Scaling Latency 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Peak-to-Min Ratio Chat 1s 2s 5s 10s 20s 30s Model Scaling Latency 2.5 5.0 7.5 10.0 12.5 15.0 Coding [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Larger autoscaling latency misses more fine￾grained traffic variations across production workloads due to the increased load peak-to-min ratio. with distinct traffic patterns and strict SLOs on TTFT and TBT [57]. As a result, the central challenge arises from meet￾ing these performance targets under dynamic workloads while minimizing total resource costs. Today, the key solution is SLO-targeted autoscaling [19, 30, … view at source ↗
Figures from the paper (20 more)
Figure 4
Figure 4. Figure 4: Operator compute and memory sensitivities to sequence lengths across different model architectures. Each line is a different operator, with each data points measured at sequence lengths in [128, 256, ..., 64K] from left to right. The y = x marks the linear scaling with…
Figure 5
Figure 5. Figure 5: Operator compute sensitivity to batch sizes across different model architectures. Each line is a different operator. attention incurs quadratic complexity in sequence length (L), while remaining linear in batch size (b). This O(L 2b) com￾plexity is significantly higher…
Figure 6
Figure 6. Figure 6: Averaged per-operator compute- vs. memory￾sensitive in Qwen2-7B across different sequence lengths. ously heavy or lightweight (QKV Linear) in both dimensions. This two-dimensional heterogeneity highlights that workload￾induced pressure does not manifest uniformly acros…
Figure 7
Figure 7. Figure 7: Performance and SM utilization characteristics across operators in Qwen2-7B under varying SM allocations. Takeaway 4: Sensitivity to SM allocation varies widely across operators, sequence lengths, and prefill/decode phases. Particularly, spatial sharing is suitable for…
Figure 10
Figure 10. Figure 10: Saving comparisons across model sizes. Varying QPS [PITH_FULL_IMAGE:figures/full_fig_p006_10.png]
Figure 9
Figure 9. Figure 9: Benefits of op-level provisioning compared to model level under varying QPS (refer to [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]
Figure 12
Figure 12. Figure 12: OPSCALE control plane consists of operator pro￾visioning (§4.3) and placement (§4.4). profiling a 0.5B-parameter model (with 11 unique operators) would require weeks of GPU time. This is untenable for model iteration and deployment in production. Operator Configuratio…
Figure 13
Figure 13. Figure 13: Comparison of request performance, GPU usage, and cluster power usage during autoscaling for Qwen2-7B. the underlying inference runtime. 5.1 Experiment Setup Models and Traces. We evaluate OPSCALE using two repre￾sentative LLMs: Qwen2-7B [28] (a dense LLM) and Qwen2- …
Figure 14
Figure 14. Figure 14: Cluster-level power consumption comparison. 90 95 100 SLO Attainment (%) 8 10 12 14 Avg GPU Count OpScale AIBrix Prod. Stack DynamoLLM DynamoLLM+DVFS -Serve Better [PITH_FULL_IMAGE:figures/full_fig_p010_14.png]
Figure 17
Figure 17. Figure 17: Max throughput achieved at fixed provisioning. 5.4 Throughput Improvements Op-level provisioning also benefits static deployments, with a fixed number of GPUs. In static clusters up to 40 GPUs, we increase the request arrival load until each system reaches its SLO lim…
Figure 16
Figure 16. Figure 16: Comparison of the serving cost (in total GPUs and total power consumption) for varying QPS. an additional full-model replica. 5.3 Cost Savings when Meeting SLOs To understand the limits of cost savings, we evaluate the min￾imum capacity required to sustain varying wor…
Figure 18
Figure 18. Figure 18: Static provisioning comparison across serving granularities and hardware configurations. 1 2 5 10 15 20 Scaling Interval (seconds) 70 80 90 100 SLO Attainment (%) 97.4% 93.9% 91.4% 89.5% 89.4% 89.2% SLO = 1.0s DynamoLLM AIBrix (overprovisioning) Production Stack 0.5 1…
Figure 19
Figure 19. Figure 19: (Left) Impact of the scaling interval on SLO attain￾ment. (Right) Impact of SLO target. Both with Qwen2-7B. as the autoscaling interval decreases within the 1–20 sec￾onds range, reaching 97.4% when the interval is set to one second. This aligns with our observation fr…
Figure 20
Figure 20. Figure 20: Compute sensitivities to input data size, for various operators in different model architectures. 1K 5K 10K 20K 40K Sequence Length 10 0 10 1 10 2 Normalized Memory attn Qwen2-7B-Instruct Linear 1K 5K 10K 20K 40K Sequence Length 10 0 10 1 10 2 attn Llama3-8B-Instruct …
Figure 21
Figure 21. Figure 21: Memory sensitivities to input data size, for various operators in different model architectures [PITH_FULL_IMAGE:figures/full_fig_p019_21.png]
Figure 22
Figure 22. Figure 22: Benefits of operator-level resource management under varying sequence lengths. Left Y-axis denotes saving percentages. Right Y-axis denotes the absolute GPU usage, energy consumption, and memory footprint. computational and temporal characteristics drive different sca…
Figure 24
Figure 24. Figure 24: Queueing sensitivity to arrival rate. Insight 1: Prefill stages offer substantially higher opti￾mization potential than decode—up to 2–3× greater re￾source savings, as they are more compute-intensive and bursty, making them ideal targets for fine-grained op-level mode…
Figure 25
Figure 25. Figure 25: Operator input data volume for Qwen2-7B. 0 10 20 30 40 Average Savings (%) Chat Code Azure LLM Traces (2024) Prefill Decode GPU Energy Memory GPU Energy Memory 0 10 20 30 40 Average Savings (%) Chat Tool Mooncake LLM Traces (2025) [PITH_FULL_IMAGE:figures/full_fig_p0…
Figure 27
Figure 27. Figure 27: Comparison of GPU usage, request performance (P99 TTFT), and total power usage during autoscaling for Qwen2-57B-A14B (MoE). P50 P90 0 500 1K 1K 2K 2K 3K Cluster Power (W) OpScale AIBrix Prod. Stack DynamoLLM DynamoLLM (with DVFS) -Serve++ [PITH_FULL_IMAGE:figures/ful…
Figure 26
Figure 26. Figure 26: Benefits of operator-level resource management in prefill vs. decode stages for Qwen2-7B. F Additional Evaluation Results This section provides detailed evaluation results for the Mixture-of-Experts (MoE) model, Qwen2-57B-A14B. Due to the high operator heterogeneity i…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

71 extracted references · 52 canonical work pages

  1. [1]

    Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. InProceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024

    work page 2024

  2. [2]

    Medha: Efficiently serving multi-million context length LLM inference requests without approximations.arXiv preprint arXiv:2409.17264, 2024

    Amey Agrawal, Haoran Qiu, Junda Chen, Íñigo Goiri, Chaojie Zhang, Rayyan Shahid, Ramachandran Ramjee, Alexey Tumanov, and Esha Choukse. Medha: Efficiently serving multi-million context length LLM inference requests without approximations.arXiv preprint arXiv:2409.17264, 2024

    arXiv 2024

  3. [3]

    Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B

    Benjamin Spector, Jordan Juravsky, Stuart Sul, Owen Dugan, Dylan Lim, Dan Fu, Simran Arora, Chris Ré. Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B. Accessed from https://hazyresearch.stanford.edu/blog/ 2025-05-27-no-bubbles, 2025

    work page 2025

  4. [4]

    Internet and the Erlang formula.ACM SIGCOMM Computer Communication Review, 42(1):23–30, 2012

    Thomas Bonald and James W Roberts. Internet and the Erlang formula.ACM SIGCOMM Computer Communication Review, 42(1):23–30, 2012

    work page 2012

  5. [5]

    Stability, queue length, and delay of deterministic and stochastic queueing networks

    Cheng-Shang Chang. Stability, queue length, and delay of deterministic and stochastic queueing networks. IEEE Transactions on Automatic Control, 39(5):913–931, 1994

    work page 1994

  6. [6]

    TVM: An automated end-to-end optimizing compiler for deep learning

    Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et al. TVM: An automated end-to-end optimizing compiler for deep learning. InProceedings of The 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 578–594, 2018

    work page 2018

  7. [7]

    Towards high-goodput LLM serving with prefill-decode multiplexing

    Yukang Chen, Weihao Cui, Han Zhao, Ziyi Xu, Xiaoze Fan, Xusheng Chen, Yangjie Zhou, Shixuan Sun, Bingsheng He, and Quan Chen. Towards high-goodput LLM serving with prefill-decode multiplexing. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2026), pages 2030–2047, 2026

    work page 2026

  8. [8]

    Mirage Persistent Kernel: A Compiler and Runtime for Mega-Kernelizing Tensor Programs.arXiv preprint arXiv:2512.22219, 2025

    Xinhao Cheng, Zhihao Zhang, Yu Zhou, Jianan Ji, Jinchen Jiang, Zepeng Zhao, Ziruo Xiao, Zihao Ye, Yingyi Huang, Ruihang Lai, et al. Mirage Persistent Kernel: A Compiler and Runtime for Mega-Kernelizing Tensor Programs.arXiv preprint arXiv:2512.22219, 2025

    arXiv 2025

Show all 71 references
  1. [9]

    Serving heterogeneous machine learning models on multi-GPU servers with spatio-temporal sharing

    Seungbeom Choi, Sunho Lee, Yeonjae Kim, Jongse Park, Youngjin Kwon, and Jaehyuk Huh. Serving heterogeneous machine learning models on multi-GPU servers with spatio-temporal sharing. In2022 USENIX Annual Technical Conference (USENIX ATC 2022), pages 199–216, 2022

    2022

  2. [10]

    PaLM: Scaling language modeling with Pathways.Journal of Machine Learning Research, 24(240):1–113, 2023

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with Pathways.Journal of Machine Learning Research, 24(240):1–113, 2023

    2023

  3. [11]

    LithOS: An operating system for efficient machine learning on GPUs

    Patrick H Coppock, Brian Zhang, Eliot H Solomon, Vasilis Kypriotis, Leon Yang, Bikash Sharma, Dan Schatzberg, Todd C Mowry, and Dimitrios Skarlatos. LithOS: An operating system for efficient machine learning on GPUs. InProceedings of the ACM SIGOPS 31st Symposium on Operating ...

    2025

  4. [12]

    FlashAttention: Fast and memory-efficient exact attention with IO-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. Advances in Neural Information Processing Systems (NeurIPS 22), 35:16344–16359, 2022

    2022

  5. [13]

    GSLICE: controlled spatial sharing of GPUs for a scalable inference platform

    Aditya Dhakal, Sameer G Kulkarni, and KK Ramakrishnan. GSLICE: controlled spatial sharing of GPUs for a scalable inference platform. In Proceedings of the 11th ACM Symposium on Cloud Computing (SoCC 2020), pages 492–506, 2020

    2020

  6. [14]

    HydraInfer: Hybrid disaggregated scheduling for multimodal large language model serving.arXiv preprint arXiv:2505.12658, 2025

    Xianzhe Dong, Tongxuan Liu, Yuting Zeng, Liangyu Liu, Yang Liu, Siyu Wu, Yu Wu, Hailong Yang, Ke Zhang, and Jing Li. HydraInfer: Hybrid disaggregated scheduling for multimodal large language model serving.arXiv preprint arXiv:2505.12658, 2025. 14

    2025

  7. [15]

    MuxServe: Flexible spatial-temporal multiplexing for multiple LLM serving.arXiv preprint arXiv:2404.02015, 2024

    Jiangfei Duan, Runyu Lu, Haojie Duanmu, Xiuhong Li, Xingcheng Zhang, Dahua Lin, Ion Stoica, and Hao Zhang. MuxServe: Flexible spatial-temporal multiplexing for multiple LLM serving.arXiv preprint arXiv:2404.02015, 2024

    2024 arXiv

  8. [16]

    ServerlessLLM:low-latency serverless inference for large language models

    Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. ServerlessLLM:low-latency serverless inference for large language models. InProceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pag...

    2024

  9. [17]

    ATOM: Model-driven autoscaling for microservices

    Alim Ul Gias, Giuliano Casale, and Murray Woodside. ATOM: Model-driven autoscaling for microservices. In Proceedings of the IEEE 39th International Conference on Distributed Computing Systems (ICDCS 2019), pages 1994–2004. IEEE, 2019

    2019

  10. [18]

    Nano-vLLM.https: //github.com/GeeeekExplorer/nano-vllm, 2025

    GitHub. Nano-vLLM.https: //github.com/GeeeekExplorer/nano-vllm, 2025

    2025

  11. [19]

    NVIDIA Dynamo

    GitHub. NVIDIA Dynamo. https://github.com/ai-dynamo/dynamo, 2025

    2025

  12. [20]

    vLLM Production Stack.https: //github.com/vllm-project/production-stack, 2025

    GitHub. vLLM Production Stack.https: //github.com/vllm-project/production-stack, 2025

    2025

  13. [21]

    Semi-PD: Towards efficient LLM serving via phase-wise disaggregated computation and unified storage.arXiv preprint arXiv:2504.19867, 2025

    Ke Hong, Lufang Chen, Zhong Wang, Xiuhong Li, Qiuli Mao, Jianping Ma, Chao Xiong, Guanyu Wu, Buhe Han, Guohao Dai, et al. Semi-PD: Towards efficient LLM serving via phase-wise disaggregated computation and unified storage.arXiv preprint arXiv:2504.19867, 2025

    2025 arXiv

  14. [22]

    DEEPSERVE: Serverless large language model serving at scale

    Junhao Hu, Jiang Xu, Zhixia Liu, Yulong He, Yuetao Chen, Hao Xu, Jiang Liu, Jie Meng, Baoquan Zhang, Shining Wan, et al. DEEPSERVE: Serverless large language model serving at scale. InProceedings of the 2025 USENIX Annual Technical Conference (ATC 2025), 2025

    2025

  15. [23]

    DDiT: Dynamic resource allocation for diffusion transformer model serving.arXiv preprint arXiv:2506.13497, 2025

    Heyang Huang, Cunchen Hu, Jiaqi Zhu, Ziyuan Gao, Liangliang Xu, Yizhou Shan, Yungang Bao, Sun Ninghui, Tianwei Zhang, and Sa Wang. DDiT: Dynamic resource allocation for diffusion transformer model serving.arXiv preprint arXiv:2506.13497, 2025

    2025 arXiv

  16. [24]

    In 2026 IEEE International Symposium on High Performance Computer Architecture (HPCA 2026), pages 1–14

    Wenhao Huang, Zhaolin Duan, Laiping Zhao, Yuhao Zhang, Yanjie Wang, Yiming Li, Yihan Wang, Yichi Chen, Zhihang Tang, Kang Chen, et al.µShare: Non-intrusive kernel co-locating on NVIDIA GPUs. In 2026 IEEE International Symposium on High Performance Computer Architecture (HPCA 2...

    2026

  17. [25]

    Llama-3-8b.https://huggingface

    Hugging Face. Llama-3-8b.https://huggingface. co/meta-llama/Meta-Llama-3-8B, 2025

    2025

  18. [26]

    Mixtral-8x7B-v0.1.https:// huggingface.co/mistralai/Mixtral-8x7B-v0.1, 2025

    Hugging Face. Mixtral-8x7B-v0.1.https:// huggingface.co/mistralai/Mixtral-8x7B-v0.1, 2025

    2025

  19. [27]

    Qwen2-57B-A14B

    Hugging Face. Qwen2-57B-A14B. https://huggingface.co/Qwen/ Qwen2-57B-A14B-Instruct, 2025

    2025

  20. [28]

    QWen2-7B-Instruct.https: //huggingface.co/Qwen/Qwen2-7B-Instruct, 2025

    Hugging Face. QWen2-7B-Instruct.https: //huggingface.co/Qwen/Qwen2-7B-Instruct, 2025

    2025

  21. [29]

    Qwen2.5-VL-32B

    Hugging Face. Qwen2.5-VL-32B. https://huggingface.co/Qwen/Qwen2. 5-VL-32B-Instruct, 2025

    2025

  22. [30]

    Amant, Chetan Bansal, Victor Ruhle, Anoop Kulkarni, Steve Kofsky, and Saravan Rajmohan

    Shashwat Jaiswal, Kunal Jain, Yogesh Simmhan, Anjaly Parayil, Ankur Mallick, Rujia Wang, Renee St. Amant, Chetan Bansal, Victor Ruhle, Anoop Kulkarni, Steve Kofsky, and Saravan Rajmohan. SAGESERVE: Optimizing LLM serving on cloud data centers with forecast aware auto-scaling.P...

    2025

  23. [31]

    Pod-Attention: Unlocking full prefill-decode overlap for faster LLM inference

    Aditya K Kamath, Ramya Prabhu, Jayashree Mohan, Simon Peter, Ramachandran Ramjee, and Ashish Panwar. Pod-Attention: Unlocking full prefill-decode overlap for faster LLM inference. InProceedings of the 30th ACM International Conference on Architectural Support for Programming L...

    2025

  24. [32]

    A simulation analysis of sojourn times in a Jackson network

    Peter C Kiessler. A simulation analysis of sojourn times in a Jackson network. Technical report, 1980

    1980

  25. [33]

    Horizontal Pod Autoscaling.http: //kubernetes.io/docs/concepts/workloads/ autoscaling/horizontal-pod-autoscale, 2026

    Kubernetes. Horizontal Pod Autoscaling.http: //kubernetes.io/docs/concepts/workloads/ autoscaling/horizontal-pod-autoscale, 2026

    2026

  26. [34]

    AlpaServe: Statistical multiplexing with model parallelism for deep learning serving

    Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E Gonzalez, et al. AlpaServe: Statistical multiplexing with model parallelism for deep learning serving. InProceedings of the 17th USENIX Symposium on Oper...

    2023

  27. [35]

    Bullet: Boosting GPU utilization for LLM serving via dynamic spatial-temporal orchestration.arXiv preprint arXiv:2504.19516, 2025

    Zejia Lin, Hongxin Xu, Guanyi Chen, Xianwei Zhang, and Yutong Lu. Bullet: Boosting GPU utilization for LLM serving via dynamic spatial-temporal orchestration.arXiv preprint arXiv:2504.19516, 2025

    2025

  28. [36]

    Expert-as-a-service: Towards efficient, scalable, and robust large-scale MoE serving.arXiv preprint arXiv:2509.17863, 2025

    Ziming Liu, Boyu Tian, Guoteng Wang, Zhen Jiang, Peng Sun, Zhenhua Han, Tian Tang, Xiaohe Hu, 15 Yanmin Jia, Yan Zhang, et al. Expert-as-a-service: Towards efficient, scalable, and robust large-scale MoE serving.arXiv preprint arXiv:2509.17863, 2025

    2025

  29. [37]

    Azure VM NDm-A100-v4 sizes series.https://learn.microsoft.com/en-us/ azure/virtual-machines/sizes/ gpu-accelerated/ndma100v4-series, 2024

    Microsoft Azure. Azure VM NDm-A100-v4 sizes series.https://learn.microsoft.com/en-us/ azure/virtual-machines/sizes/ gpu-accelerated/ndma100v4-series, 2024

    2024

  30. [38]

    Azure VM ND GB200-v6 sizes series.https://learn.microsoft.com/en-us/ azure/virtual-machines/sizes/ gpu-accelerated/nd-gb200-v6-series, 2026

    Microsoft Azure. Azure VM ND GB200-v6 sizes series.https://learn.microsoft.com/en-us/ azure/virtual-machines/sizes/ gpu-accelerated/nd-gb200-v6-series, 2026

    2026

  31. [39]

    Documentation on NVIDIA Multi-Instance GPU (MIG).https://www.nvidia.com/en-us/ technologies/multi-instance-gpu/, 2025

    NVIDIA. Documentation on NVIDIA Multi-Instance GPU (MIG).https://www.nvidia.com/en-us/ technologies/multi-instance-gpu/, 2025

    2025

  32. [40]

    Documentation on NVIDIA Multi-Process Service (MPS).https: //docs.nvidia.com/deploy/mps/index.html, 2025

    NVIDIA. Documentation on NVIDIA Multi-Process Service (MPS).https: //docs.nvidia.com/deploy/mps/index.html, 2025

    2025

  33. [41]

    Nsight Systems.https: //developer.nvidia.com/nsight-systems, 2025

    NVIDIA. Nsight Systems.https: //developer.nvidia.com/nsight-systems, 2025

    2025

  34. [42]

    NVIDIA DCGM

    NVIDIA. NVIDIA DCGM. https://developer.nvidia.com/dcgm, 2025

    2025

  35. [43]

    NVIDIA Green Context Documentation

    NVIDIA. NVIDIA Green Context Documentation. https: //docs.nvidia.com/cuda/cuda-driver-api/ group_CUDA_GREEN_CONTEXTS.html, 2025

    2025

  36. [44]

    Introducing ChatGPT

    OpenAI. Introducing ChatGPT. https://openai.com/index/chatgpt/, 2022

    2022

  37. [45]

    ChatGPT Codex

    OpenAI. ChatGPT Codex. https://chatgpt.com/features/codex, 2025

    2025

  38. [46]

    Introducing Deep Research

    OpenAI. Introducing Deep Research. https://openai.com/index/ introducing-deep-research/, 2025

    2025

  39. [47]

    Measuring agents in production.arXiv preprint arXiv:2512.04123, 2025

    Melissa Z Pan, Negar Arabzadeh, Riccardo Cogo, Yuxuan Zhu, Alexander Xiong, Lakshya A Agrawal, Huanzhi Mao, Emma Shen, Sid Pallerla, Liana Patel, et al. Measuring agents in production.arXiv preprint arXiv:2512.04123, 2025

    2025 arXiv

  40. [48]

    Splitwise: Efficient generative LLM inference using phase splitting

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative LLM inference using phase splitting. InProceedings of the 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (IS...

    2024

  41. [49]

    Hierarchical autoscaling for large language model serving with Chiron.arXiv preprint arXiv:2501.08090, 2025

    Archit Patke, Dhemath Reddy, Saurabh Jha, Chandra Narayanaswami, Zbigniew Kalbarczyk, and Ravishankar Iyer. Hierarchical autoscaling for large language model serving with Chiron.arXiv preprint arXiv:2501.08090, 2025

    2025 arXiv

  42. [50]

    Gonzalez, Ion Stoica, and Harry Xu

    Yifan Qiao, Shu Anzai, Shan Yu, Haoran Ma, Shuo Yang, Yang Wang, Miryung Kim, Yongji Wu, Yang Zhou, Jiarong Xing, Joseph E. Gonzalez, Ion Stoica, and Harry Xu. ConServe: Fine-grained GPU harvesting for LLM online and offline co-serving, 2025

    2025

  43. [51]

    Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving.arXiv preprint arXiv:2407.00079, 2024

    Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving.arXiv preprint arXiv:2407.00079, 2024

    2024 arXiv

  44. [52]

    FIRM: An intelligent fine-grained resource management framework for SLO-oriented microservices

    Haoran Qiu, Subho S Banerjee, Saurabh Jha, Zbigniew T Kalbarczyk, and Ravishankar K Iyer. FIRM: An intelligent fine-grained resource management framework for SLO-oriented microservices. InProceedings of the 14th USENIX symposium on operating systems design and implementation (...

    2020

  45. [53]

    ModServe: Scalable and resource-efficient large multimodal model serving.arXiv preprint arXiv:2502.00937, 2025

    Haoran Qiu, Anish Biswas, Zihan Zhao, Jayashree Mohan, Alind Khare, Esha Choukse, Íñigo Goiri, Zeyu Zhang, Haiying Shen, Chetan Bansal, et al. ModServe: Scalable and resource-efficient large multimodal model serving.arXiv preprint arXiv:2502.00937, 2025

    2025

  46. [54]

    Power-aware deep learning model serving with µ-Serve

    Haoran Qiu, Weichao Mao, Archit Patke, Shengkun Cui, Saurabh Jha, Chen Wang, Hubertus Franke, Zbigniew Kalbarczyk, Tamer Ba¸ sar, and Ravishankar K Iyer. Power-aware deep learning model serving with µ-Serve. In2024 USENIX Annual Technical Conference (USENIX ATC 24), pages 75–93, 2024

    2024

  47. [55]

    USHER: Holistic interference avoidance for resource optimized ML inference

    Sudipta Saha Shubha, Haiying Shen, and Anand Iyer. USHER: Holistic interference avoidance for resource optimized ML inference. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 947–964, 2024

    2024

  48. [56]

    Efficiently serving large multimedia models using EPD disaggregation.arXiv preprint arXiv:2501.05460, 2024

    Gursimran Singh, Xinglu Wang, Ivan Hu, Timothy Yu, Linzi Xing, Wei Jiang, Zhefeng Wang, Xiaolong Bai, Yi Li, Ying Xiong, et al. Efficiently serving large multimedia models using EPD disaggregation.arXiv preprint arXiv:2501.05460, 2024

    2024 arXiv

  49. [57]

    DynamoLLM: Designing LLM inference clusters for performance and energy efficiency

    Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Josep Torrellas, and Esha Choukse. DynamoLLM: Designing LLM inference clusters for performance and energy efficiency. InProceedings of the 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA 25), pages ...

    2025

  50. [58]

    Orion: Interference-aware, fine-grained GPU sharing for ML applications

    Foteini Strati, Xianzhe Ma, and Ana Klimovic. Orion: Interference-aware, fine-grained GPU sharing for ML applications. InProceedings of the Nineteenth European Conference on Computer Systems (EuroSys 2024), pages 1075–1092, 2024

    2024

  51. [59]

    AIBrix: Towards scalable, cost-effective large language model inference infrastructure.arXiv preprint arXiv:2504.03648, 2025

    The AIBrix Team, Jiaxin Shan, Varun Gupta, Le Xu, Haiyang Shi, Jingyuan Zhang, Ning Wang, Linhui Xu, Rong Kang, Tongping Liu, et al. AIBrix: Towards scalable, cost-effective large language model inference infrastructure.arXiv preprint arXiv:2504.03648, 2025

    2025 arXiv

  52. [60]

    Distributed Inference and Serving

    vLLM. Distributed Inference and Serving. https://docs.vllm.ai/en/latest/serving/ distributed_serving.html, 2024

    2024

  53. [61]

    vLLM Profiler.https://docs.vllm.ai/en/ stable/contributing/profiling/, 2025

    vLLM. vLLM Profiler.https://docs.vllm.ai/en/ stable/contributing/profiling/, 2025

    2025

  54. [62]

    Step-3 is large yet affordable: Model-system co-design for cost-effective decoding.arXiv preprint arXiv:2507.19427, 2025

    Bin Wang, Bojun Wang, Changyi Wan, Guanzhe Huang, Hanpeng Hu, Haonan Jia, Hao Nie, Mingliang Li, Nuo Chen, Siyu Chen, et al. Step-3 is large yet affordable: Model-system co-design for cost-effective decoding.arXiv preprint arXiv:2507.19427, 2025

    2025

  55. [63]

    Autothrottle: A practical bi-level approach to resource management for SLO-targeted microservices

    Zibo Wang, Pinghe Li, Chieh-Jan Mike Liang, Feng Wu, and Francis Y Yan. Autothrottle: A practical bi-level approach to resource management for SLO-targeted microservices. InProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 2024), pag...

    2024

  56. [64]

    DeepScaling: microservices autoscaling for stable cpu utilization in large scale cloud systems

    Ziliang Wang, Shiyi Zhu, Jianguo Li, Wei Jiang, Kadangode K Ramakrishnan, Yangfei Zheng, Meng Yan, Xiaohong Zhang, and Alex X Liu. DeepScaling: microservices autoscaling for stable cpu utilization in large scale cloud systems. InProceedings of the 13th Symposium on Cloud Compu...

    2022

  57. [65]

    Aegaeon: Effective GPU pooling for concurrent LLM serving on the market

    Yuxing Xiang, Xue Li, Kun Qian, Yufan Yang, Diwen Zhu, Wenyuan Yu, Ennan Zhai, Xuanzhe Liu, Xin Jin, and Jingren Zhou. Aegaeon: Effective GPU pooling for concurrent LLM serving on the market. InProceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, pages...

    2025

  58. [66]

    Towards efficient and practical GPU multitasking in the era of LLM.arXiv preprint arXiv:2508.08448, 2025

    Jiarong Xing, Yifan Qiao, Simon Mo, Xingqi Cui, Gur-Eyal Sela, Yang Zhou, Joseph Gonzalez, and Ion Stoica. Towards efficient and practical GPU multitasking in the era of LLM.arXiv preprint arXiv:2508.08448, 2025

    2025 arXiv

  59. [67]

    Prism: Unleashing GPU sharing for cost-efficient multi-LLM serving.arXiv preprint arXiv:2505.04021, 2025

    Shan Yu, Jiarong Xing, Yifan Qiao, Mingyuan Ma, Yangmin Li, Yang Wang, Shuo Yang, Zhiqiang Xie, Shiyi Cao, Ke Bao, et al. Prism: Unleashing GPU sharing for cost-efficient multi-LLM serving.arXiv preprint arXiv:2505.04021, 2025

    2025 arXiv

  60. [68]

    DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In Proceedings of the 18th USENIX Symposium on Operating Systems Design and Imple...

    2024

  61. [69]

    NanoFlow: Towards optimal large language model serving throughput

    Kan Zhu, Yufei Gao, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Zihao Ye, Keisuke Kamahori, Chien-Yu Lin, et al. NanoFlow: Towards optimal large language model serving throughput. In Proceedings of the 19th USENIX Symposium on Operating Systems Design and Implem...

    2025

  62. [70]

    MegaScale-Infer: Serving mixture-of-experts at scale with disaggregated expert parallelism.arXiv preprint arXiv:2504.02263, 2025

    Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A Stuardo, Dongyang Wang, Xinlei Zhang, Huaping Zhou, Haoran Wei, Yang Cheng, et al. MegaScale-Infer: Serving mixture-of-experts at scale with disaggregated expert parallelism.arXiv preprint arXiv:2504.02263, 2025

    2025 arXiv

  63. [71]

    Serving large language models on Huawei CloudMatrix384.arXiv preprint arXiv:2506.12708, 2025

    Pengfei Zuo, Huimin Lin, Junbo Deng, Nan Zou, Xingkun Yang, Yingyu Diao, Weifeng Gao, Ke Xu, Zhangyu Chen, Shirui Lu, et al. Serving large language models on Huawei CloudMatrix384.arXiv preprint arXiv:2506.12708, 2025. 17 Algorithm 1Greedy Operator Provisioning Algorithm Requi...

    2024 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.