Pith. sign in

REVIEW 5 major objections 5 minor 31 references

MaaSO: SLO-aware Orchestration of Heterogeneous Model Instances for MaaS

T0 review · 5 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MaaSO claims heterogeneous LLM instance configurations raise SLO satisfaction by 15–30% and cut response latency by 40–60% vs. homogeneous serving.

desk verdict Plausible and well-structured system idea for heterogeneous LLM serving, but the headline gains are currently an unvalidated simulation result: the same fitted throughput model powers both the optimizer and the evaluator and is transferred from 3090/H20 to V100 without calibration. read the letter →

arxiv 2509.06362 v1 pith:IWGOA56T submitted 2025-09-08 cs.DC

Mo Xuan , Zhang yue , Wu Weigang This is my paper
classification cs.DC
keywords MaaSSLO-awareorchestrationheterogeneousmodelinstancesLLMinferenceparallelismstrategybatchsizecontinuousbatchingthroughputdecay
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Model-as-a-service platforms receive requests with very different deadline profiles: some need a first token almost instantly, others need long outputs within minutes. The paper argues that no single instance configuration—one way of splitting the model across GPUs and one batch-size cap—can serve that spread well, and that deliberately heterogeneous instances can. It proposes MaaSO, an orchestrator whose profiler estimates instance throughput with a logarithmic decay function, whose placer searches configurations and GPU partitions, and whose distributor routes requests by SLO while blocking admissions that would trigger cascaded timeouts. In simulation, MaaSO claims 15–30% higher SLO satisfaction and 40–60% lower response latency than homogeneous baselines, with lower configuration overhead. If right, this gives MaaS operators a practical way to meet diverse LLM application deadlines without buying more GPUs.

What carries the argument

The load-bearing object is the throughput decay function F(M,P,B,W) = T0(M,P)·(1 − δP·log(ϵP + min(B,W))), which turns a parallelism strategy P, a batch size B, and a workload level W into a per-instance decoding-throughput estimate. It is fit by least squares from a small number of real profiling runs, and it feeds every downstream decision: the simulator's SLO check uses it to estimate decoding and queuing latency, the placer's serving score uses it to compare configurations, and the distributor's overflow protection uses a worst-case value of it to block admissions. The configuration tree with its two pruning rules—dropping parallelism strategies that cannot beat data parallelism and keep

What would settle it

Replay one of the paper's workload traces on a real cluster with two deployments—MaaSO's selected heterogeneous configuration and a homogeneous baseline—and compare measured SLO attainment and queueing delays. If real queueing latency does not follow the paper's formula (for example, if it scales with batch occupancy or scheduling order rather than with workload times expected decoding time divided by total batch capacity), the reported 15–30% advantage will not reproduce.

Watch

Extended reading notes

Core claim

The paper's central claim is that heterogeneous instance configuration is the missing lever for SLO-aware LLM serving. For the same model, instances configured with different parallelism strategies and different inference batch sizes produce complementary performance regions; a portfolio of such instances can cover a wider range of first-token and between-token latency SLOs than any homogeneous deployment. MaaSO operationalizes this in three modules: a profiler that fits a throughput decay function from sparse measurements; a placer that prunes a configuration tree, uses a simulator to score candidate deployments by SLO satisfaction, throughput, and latency, and uses dynamic programming to p

Load-bearing premise

The simulator decides whether a request meets its deadline using a queuing-time formula that is asserted as a proportionality rather than derived or validated on a real system; every headline SLO number passes through that formula.

Editorial extensions

If this is right

  • With the same GPU budget, a heterogeneous set of (parallelism, batch-size) instances covers a broader SLO spectrum than any single homogeneous configuration, so a MaaS provider can accept a wider mix of LLM applications without adding capacity.
  • Because the distributor checks a request's deadline before admission and uses worst-case throughput for the estimate, continuous-batching systems avoid cascaded timeouts where one late request delays a whole batch.
  • Configuration-tree pruning plus sub-cluster dynamic programming keeps placer overhead low as cluster size grows, unlike homogeneous planners whose solver time climbs steeply.
  • Under bursty arrivals and high request volumes, heterogeneous orchestration holds SLO attainment steadier than homogeneous baselines, which degrade quickly under the same pressure.
  • The paper's own boundary analysis implies that when request SLOs are similar or every instance is saturated, heterogeneous gains converge away and the scheme can fall back to a homogeneous deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: The profiler's logarithmic decay function is fit on one serving engine and two GPU types; if the same functional form holds on other engines and accelerators, the whole MaaSO pipeline transfers with only a few profiling runs, but that transfer is untested in the paper.
  • Editorial: Because profiles are indexed by model, parallelism, batch size, and workload, the same architecture should extend to mixed model families and mixed hardware inside one cluster; the paper only simulates a homogeneous GPU pool.
  • Editorial: The most direct decisive test is to run MaaSO on a real cluster and compare measured SLO attainment and queueing delays against the simulator's predictions; the paper's queuing formula is the largest source of uncertainty.
  • Editorial: The paper's own traces 5 and 6 show that low SLO diversity favors homogeneous strategies, so a production orchestrator would likely need to toggle between heterogeneous and homogeneous modes online rather than pick one partition per request set.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. MaaSO is an orchestrator for MaaS platforms that aims to improve SLO satisfaction and reduce response latency by deploying multiple instances of the same LLM with heterogeneous configurations—different parallelism strategies and inference batch sizes—and by routing requests to these instances based on their SLO requirements. The system is decomposed into a profiler that fits a throughput-decay function F(M,P,B,W), a placer that selects instance configurations via simulator-guided pruning and dynamic programming over sub-clusters, and a distributor that performs SLO-aware request assignment with overflow protection. The paper reports, based on a discrete-event simulator, that MaaSO improves SLO satisfaction by 15–30% and reduces response latency by 40–60% compared to AlpaServe and Selective Replication baselines, while reducing solver overhead.

Significance. If the reported gains are real, MaaSO addresses a genuinely important problem: modern LLM serving engines typically use homogeneous instance configurations, and exploiting heterogeneity to meet diverse SLOs is a promising direction. The paper includes real profiling measurements on two GPU types, a modular algorithm design with concrete pruning heuristics, a dynamic-programming resource partition, and a set of six traces that vary SLO strictness, burstiness, and load. The ablation against MaaSO* is also a useful check on the role of the SLO weight alpha. However, the central empirical claims currently rest on a simulator that shares its timing model with the optimizer being evaluated, and the simulator is not validated against the target hardware or against real end-to-end deployments. The core idea is plausible and the system design is coherent, but the evidence as presented is not yet sufficient to establish the headline improvements.

major comments (5)
  1. [Section V-A] The profiling is done on 2×RTX 3090 and 8×H20 GPUs, but the simulated cluster is composed of AWS p3.16xlarge instances with 8×Tesla V100 GPUs. The only bridge offered is the phrase 'aligned with the profile data published by AlpaServe,' which is not a quantitative transfer function, calibration, or scaling law. Since every SLO attainment and latency number in Fig. 4 is produced by a simulator that uses Eq. (1) fitted on 3090/H20 data, the absolute and relative improvements (15–30% SLO, 40–60% latency) are not established for the claimed target hardware. Please add V100-specific profiling, a validated hardware transfer model, or an end-to-end deployment on the target cluster.
  2. [Section IV-B, Eq. (1), Fig. 1] The throughput decay function assumes monotonic logarithmic decay in workload W, but the measured data visibly violate monotonicity. For example, the DeepSeek-32B tp-8 row in Fig. 1(b) shows 81.5 token/s at W=2, 87.3 at W=4, and 95.3 at W=8. Least-squares fitting such non-monotonic data to a monotonic log curve can systematically misrepresent the true throughput landscape, especially at the low-to-moderate workload levels where SLO decisions matter most. The paper should report fit quality per (M,P), show residuals, and validate the decay function on held-out W and B values.
  3. [Section IV-C, Eq. (2)] The queuing-latency formula L_q ∝ W_a · E[L_d] / Σ B_i is introduced as a proportionality with no derivation, no queueing-theoretic justification, and no empirical validation under continuous batching. This formula directly determines whether a request is judged to meet its deadline in Eq. (3) and therefore affects every SLO attainment result in the evaluation. The paper needs to either derive this formula from a concrete queueing model under vLLM-style continuous batching, or validate it against measurements on a real inference engine with controlled batch sizes and workload levels.
  4. [Sections IV-E and V-A] The evaluation is partly circular: the same fitted throughput function F(·) is used inside the placer to score configurations (Eq. (6), Alg. 1) and inside the evaluation simulator to compute decoding latency and SLO attainment. A simulator that reuses the optimizer's own timing model will tend to reward configurations that exploit F's specific approximations, even if those approximations are not accurate for real vLLM serving. Independent validation is needed: either run MaaSO on a real cluster with real vLLM instances, or build an evaluation harness whose timing model does not consume the same fitted F. Releasing code and traces would also make the results reproducible.
  5. [Section V-B] The Limitations Analysis only discusses two request-pattern cases (low SLO diversity and uniform high load). It does not acknowledge the more basic threats to validity: the unvalidated Eq. (2), the lack of calibration to the simulated V100 hardware, and the shared timing model between the optimizer and the simulator. These limitations should be stated explicitly and addressed or the claims should be scoped accordingly.
minor comments (5)
  1. [Section III-B] The text refers to 'Fig. 2-(d) and Fig. 2-(e)' when comparing DeepSeek-7B instances with B=4 and B=8, but Fig. 2 contains only parts (a) and (b). The referenced subfigures do not exist; please fix the cross-references.
  2. [Section IV-C, constraints] Constraint (b) appears to have the summation indices swapped: it should be Σ_{i∈I} x_{r,i} ≤ 1 for each request r, not Σ_{r∈R} x_{r,i} ≤ 1 for each instance i. As written, it would imply each instance can serve at most one request, which is inconsistent with the rest of the paper.
  3. [Fig. 4 and Table I] The legend in Fig. 4 uses inconsistent names ('MaaS MaaSX* AlphaServe SR') while the text uses 'MaaSO', 'MaaSO*', 'AlpaServe', and 'SR'. Please standardize the naming.
  4. [Section I] Minor typo: 'required handling' should be 'require handling'.
  5. [Section V-A] The reference to 'profile data published by AlpaServe' needs a specific citation or supplementary artifact; otherwise it is not verifiable.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MaaSO's fitted throughput model is used consistently in both planning and simulation, but the SLO/latency results are not equivalent to the fitted inputs by construction; the unvalidated hardware transfer is a validation risk, not a circular step.

full rationale

The paper's derivation chain is self-contained in the ordinary sense: the profiler fits the throughput decay function F(M,P,B,W) in Eq. (1) to measured throughput data, and the placer, distributor, and simulator all use this same F to estimate latencies (Eq. (2)) and SLO attainment (Eq. (3)). This shared use is not circular because F is not defined in terms of SLO attainment, nor are the SLO results fitted and then re-predicted. The reported 15-30% SLO improvement and 40-60% latency reduction are produced by a simulator whose timing model is the fitted F, so any error in F, in the unvalidated queueing proportionality in Eq. (2), or in the transfer from 3090/H20 profiling to a V100 cluster ('aligned with the profile data published by AlpaServe') would affect the absolute and relative gains. That is a real external-validity/correctness limitation, but it does not reduce the claimed derivation to its own inputs. The baselines are evaluated under the same simulator, and the configuration-search and request-distribution logic carry independent content. There are no load-bearing self-citations: the cited AlpaServe, Orca, and normalized-deadline works are external, and the paper explicitly includes a Limitations Analysis (Section V-B) identifying cases where heterogeneous deployment yields no benefit. Therefore, no significant circularity is present.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on an empirical throughput model with fitted coefficients, an unvalidated queuing formula, and assumptions about workload realism and cross-hardware transfer. The serving score has hand-set weights and thresholds, and the throughput curve has fitted per-strategy coefficients. None of these are supported by independent external benchmarks, so they constitute the main epistemic burden of the paper.

free parameters (6)
  • delta_P (workload sensitivity) = not reported
    Fitted by least squares in Eq. (1) to per-strategy throughput measurements; controls the logarithmic decay slope.
  • epsilon_P (decay offset) = not reported
    Fitted alongside delta_P in Eq. (1) to keep the log argument positive.
  • alpha (SLO weight in serving score) = 4 for MaaSO, 10 for MaaSO*
    Hand-set priority of SLO attainment in the serving score of Eq. (6).
  • beta (throughput/latency balance) = 0.3
    Hand-set trade-off between normalized throughput and normalized latency in Eq. (6).
  • gamma_T (throughput normalization threshold) = maximum achievable throughput
    Cap used in Eq. (7) to normalize observed throughput; chosen by the authors from instance maxima.
  • gamma_L (latency normalization threshold) = maximum acceptable latency
    Floor used in Eq. (8) to normalize observed latency; no precise value is reported.
assumptions (5)
  • domain assumption Each LLM instance is deployed on exclusive GPU sets; co-deployment of multiple instances is inefficient
    Sec IV-A states this to justify one instance per GPU set, but co-location effects are not modeled or validated.
  • ad hoc to paper Throughput follows F(M,P,B,W)=T0(M,P)*(1-delta_P*log(epsilon_P+min(B,W)))
    Eq. (1) is an assumed functional form fitted to data. It forces monotonic logarithmic decay even though measured throughput fluctuates and is non-monotonic.
  • ad hoc to paper Queuing latency is proportional to W_a * E[L_d] / sum(B_i)
    Eq. (2) is the core scheduling model used by the simulator to decide SLO satisfaction, and it is not derived or empirically validated.
  • domain assumption Gamma-process traces with randomly assigned decode lengths and SLO factors represent realistic MaaS workloads
    Sec V-A constructs workloads following AlpaServe and related references; their representativeness for production MaaS traffic is assumed.
  • domain assumption Profiles measured on RTX 3090 and H20 servers transfer to a simulated V100 cluster
    Sec V-A says the profiles are 'aligned with the profile data published by AlpaServe,' but no cross-hardware transfer validation is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MaaSO: SLO-aware Orchestration of Heterogeneous Model Instances for MaaS." pith.science (2026). https://pith.science/paper/IWGOA56T

@misc{pith2026250906362,
  author       = {Pith},
  title        = {Pith review of: MaaSO: SLO-aware Orchestration of Heterogeneous Model Instances for MaaS},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IWGOA56T}},
  note         = {Machine review of arXiv:2509.06362}
}
read the original abstract

Model-as-a-Service (MaaS) platforms face diverse Service Level Objective (SLO) requirements stemming from various large language model (LLM) applications, manifested in contextual complexity, first-token latency, and between-token latency. On the other hand, an LLM instance, when configured with different parallelism strategies and inference batch sizes, exhibits distinct performance characteristics and can thus be used to serve different SLO requirements. However, current LLM inference systems typically deploy instances of the same model with identical configurations, lacking mechanisms to leverage such heterogeneity. To fill this research gap, we propose MaaSO, the first MaaS Orchestrator, which comprises three modules: (1) a profiler characterizing instance performance under diverse parallelism strategies and inference batch sizes; (2) a placer optimizing heterogeneous instance configurations; (3) a distributor enabling SLO-aware request distribution and preventing cascaded timeouts in continuous batching. Experiments show that MaaSO improves the SLO satisfaction ratio by 15 to 30% and reduces response latency by 40 to 60% compared to existing approaches, and significantly lowers overall orchestration overhead.

Figures

Figures reproduced from arXiv: 2509.06362 by the authors.

Figure 1
Figure 1. The decoding throughput of a serving LLM instance is determined by its parallelism strategy and the workload level (denoted as [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Optimized search process for instance configuration [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Example workflow of MaaSO with two sub-clusters. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Experimental results in three different scenarios. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 23 canonical work pages

  1. [1]

    A survey of knowledge enhanced pre- trained language models,

    L. Hu, Z. Liu, and Z. Z. et al., “A survey of knowledge enhanced pre- trained language models,” IEEE Trans. Knowl. Data Eng.TKDE, vol. 36, no. 4, pp. 1413–1430, 2024

    work page 2024

  2. [2]

    Model-as-a-service (maas): A survey,

    W. Gan, S. Wan, and P. S. Yu, “Model-as-a-service (maas): A survey,” in IEEE International Conference on Big Data, BigData , 2023

    work page 2023

  3. [3]

    The rise and potential of large language model based agents: a survey,

    Z. Xi, W. Chen, and X. G. et al., “The rise and potential of large language model based agents: a survey,” Science China Information Sciences , vol. 68, no. 2, 2025

    work page 2025

  4. [4]

    Llms meet multimodal generation and editing: A survey,

    Y . He, Z. Liu, and J. C. et al., “Llms meet multimodal generation and editing: A survey,” CoRR, vol. abs/2405.19334, 2024

    arXiv 2024

  5. [5]

    Intelligent condition monitoring of industrial plants: An overview of methodologies and uncertainty management strategies,

    M. Ahang, T. Charter, and O. O. et al., “Intelligent condition monitoring of industrial plants: An overview of methodologies and uncertainty management strategies,” CoRR, vol. abs/2401.10266, 2024

    arXiv 2024

  6. [6]

    Slo-aware scheduling for large language model inferences,

    J. Huang, Y . Xiong, and X. Y . et al., “Slo-aware scheduling for large language model inferences,” CoRR, vol. abs/2504.14966, 2025

    arXiv 2025

  7. [7]

    Control industrial automation system with large language models,

    Y . Xia, N. Jazdi, and J. Z. et al., “Control industrial automation system with large language models,” CoRR, vol. abs/2409.18009, 2024

    arXiv 2024

  8. [8]

    Advanced intelligent manufactur- ing in process industry using industrial artificial intelligence,

    F. K. Adjogble and J. e. a. Warschat, “Advanced intelligent manufactur- ing in process industry using industrial artificial intelligence,” in 2023 Portland International Conference on Management of Engineering and Technology, PICMET, 2023

    work page 2023

Show all 31 references
  1. [9]

    Enhancing document AI data generation through graph-based synthetic layouts,

    A. Agarwal, H. L. Patel, and P. P. et al., “Enhancing document AI data generation through graph-based synthetic layouts,” CoRR, vol. abs/2412.03590, 2024

    2024 arXiv

  2. [10]

    Towards a framework for local interrogation of ai ethics: A case study on text generators, academic integrity, and composing with chatgpt,

    M. A. Vetter, B. Lucia, and J. J. et al., “Towards a framework for local interrogation of ai ethics: A case study on text generators, academic integrity, and composing with chatgpt,” Computers and Composition , vol. 71, p. 102831, 2024

    2024

  3. [11]

    Alpaserve: Statistical multiplexing with model parallelism for deep learning serving,

    Z. Li, L. Zheng, and Y . Z. et al., “Alpaserve: Statistical multiplexing with model parallelism for deep learning serving,” in 17th Symposium on Operating Systems Design and Implementation, OSDI , 2023

    2023

  4. [12]

    Alpa: Automating inter- and intra- operator parallelism for distributed deep learning,

    L. Zheng, Z. Li, and H. Z. et al., “Alpa: Automating inter- and intra- operator parallelism for distributed deep learning,” in 16th USENIX Symposium on Operating Systems Design and Implementation,OSDI , 2022

    2022

  5. [13]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters,

    J. Rasley, S. Rajbhandari, and O. R. et al., “Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters,” in 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD , 2020

    2020

  6. [14]

    Efficient large-scale language model training on GPU clusters using megatron-lm,

    D. Narayanan, M. Shoeybi, and J. C. et al., “Efficient large-scale language model training on GPU clusters using megatron-lm,” in In- ternational Conference for High Performance Computing, Networking, Storage and Analysis, SC , 2021

    2021

  7. [15]

    Gpipe: Efficient training of giant neural networks using pipeline parallelism,

    Y . Huang, Y . Cheng, and A. B. et al., “Gpipe: Efficient training of giant neural networks using pipeline parallelism,” in Annual Conference on Neural Information Processing Systems, NeurIPS , 2019

    2019

  8. [16]

    Pipedream: generalized pipeline parallelism for DNN training,

    D. Narayanan, A. Harlap, and A. P. et al., “Pipedream: generalized pipeline parallelism for DNN training,” in 27th ACM Symposium on Operating Systems Principles, SOSP , 2019

    2019

  9. [17]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, and S. Z. et al., “Efficient memory management for large language model serving with pagedattention,” in 29th Symposium on Operating Systems Principles, SOSP , 2023

    2023

  10. [18]

    Sglang: Efficient execution of structured language model programs,

    L. Zheng, L. Yin, and Z. X. et al., “Sglang: Efficient execution of structured language model programs,” in Neural Information Processing Systems, NeurIPS, 2024

    2024

  11. [19]

    Orca: A distributed serving system for transformer-based generative models,

    G. Yu, J. S. Jeong, and G. K. et al., “Orca: A distributed serving system for transformer-based generative models,” in 16th USENIX Symposium on Operating Systems Design and Implementation, OSDI , 2022

    2022

  12. [20]

    Dynaserve: Unified and elastic tandem-style execution for dynamic disaggregated LLM serving,

    C. Ruan, Y . Chen, and D. T. et al., “Dynaserve: Unified and elastic tandem-style execution for dynamic disaggregated LLM serving,”CoRR, vol. abs/2504.09285, 2025

    2025 arXiv

  13. [21]

    Borg, omega, and kubernetes,

    B. Burns, B. Grant, and D. O. et al., “Borg, omega, and kubernetes,” ACM Commun., vol. 59, no. 5, pp. 50–57, 2016

    2016

  14. [22]

    Nvidia dynamo

    “Nvidia dynamo.” [Online]. https://github.com/ai-dynamo/dynamo/

  15. [23]

    Using deepspeed and megatron to train megatron-turing NLG 530b, A large-scale generative language model,

    S. Smith, M. Patwary, and B. N. et al., “Using deepspeed and megatron to train megatron-turing NLG 530b, A large-scale generative language model,” CoRR, vol. abs/2201.11990, 2022

    2022 arXiv

  16. [24]

    Beyond data and model parallelism for deep neural networks,

    Z. Jia, M. Zaharia, and A. Aiken, “Beyond data and model parallelism for deep neural networks,” in Second Conference on Machine Learning and Systems, SysML , 2019

    2019

  17. [25]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    DeepSeek-AI, D. Guo, D. Yang, and H. Z. et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” CoRR, vol. abs/2501.12948, 2025

    2025 arXiv

  18. [26]

    Qwen technical report,

    J. Bai, S. Bai, and Y . C. et al., “Qwen technical report,” CoRR, vol. abs/2309.16609, 2023

    2023 arXiv

  19. [27]

    Amazon ec2

    “Amazon ec2.” [Online]. https://aws.amazon.com/cn/ec2/

  20. [28]

    Serving dnns like clockwork: Performance predictability from the bottom up,

    A. Gujarati, R. Karimi, and S. A. et al., “Serving dnns like clockwork: Performance predictability from the bottom up,” in 14th USENIX Sym- posium on Operating Systems Design and Implementation, OSDI , 2020

    2020

  21. [29]

    Inferline: latency-aware provisioning and scaling for prediction serving pipelines,

    D. Crankshaw, G. Sela, and X. M. et al., “Inferline: latency-aware provisioning and scaling for prediction serving pipelines,” in ACM Symposium on Cloud Computing, SoCC , 2020

    2020

  22. [30]

    Nexus: a GPU cluster engine for accelerating dnn-based video analysis,

    H. Shen, L. Chen, and Y . J. et al., “Nexus: a GPU cluster engine for accelerating dnn-based video analysis,” in 27th ACM Symposium on Operating Systems Principles, SOSP , 2019

    2019

  23. [31]

    Clipper: A low-latency online prediction serving system,

    D. Crankshaw, X. Wang, and G. Z. et al, “Clipper: A low-latency online prediction serving system,” in 14th USENIX Symposium on Networked Systems Design and Implementation, NSDI , 2017

    2017

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.