Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

BestServe: Serving Strategies with Optimal Goodput in Collocation and Disaggregation Architectures

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read BestServe claims that the optimal LLM serving strategy — architecture, instance ratio, and parallelism — can be determined in minutes on a standard CPU by simulating inference with an adapted roofline model, replacing costly…

desk verdict A useful dispatch-aware goodput simulator with an overstated headline and a key missing check: does its top-ranked strategy match ground truth? read the letter →

arxiv 2506.05871 v1 pith:3G4WEZZ3 submitted 2025-06-06 cs.LG cs.DCcs.PF

Xiannan Hu , Tianyou Zeng , Xiaoming Yuan , Liwei Song , Guangyuan Zhang , Bangzheng He This is my paper
classification cs.LGcs.DCcs.PF
keywords LLMservinggoodputprefill-decodedisaggregationrooflinemodelsimulatorcontinuousbatchingdispatch-bounddecodeSLO-awareoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BestServe claims that choosing how to serve a large language model — which architecture, how many prefill versus decode instances, and what tensor-parallel size — can be automated by simulation instead of trial-and-error benchmarking. The paper builds a three-level analyzer: an estimator that predicts operator-level latencies from an adapted roofline model, a simulator that tracks requests through prefill and decode stages under SLO constraints, and an optimizer that binary-searches the highest sustainable arrival rate. The central assertion is that the optimal serving strategy can be found in minutes on a single standard CPU, with measured average absolute errors of 11.2%, 12.1%, 8.6%, and 30.1% across four operating scenarios. The motivation is that neither architecture wins universally; the better choice depends on request patterns such as input length, generation length, and arrival rate.

What carries the argument

The load-bearing object is the hierarchical analyzer: an Estimator built on an adapted roofline model, $P = \min\{e_c S_c, I e_m S_m\}$, which says an operator runs at the smaller of its compute peak and its memory-bandwidth peak, each discounted by an efficiency factor; a Simulator that treats disaggregated serving as a tandem queue and reproduces prefill-prioritizing, non-mixing scheduling for collocated serving; and an Optimizer that bisects arrival rates to find goodput. The two mechanisms that make the whole thing fast are the pseudo batch size $b^\dagger = \max(\lfloor(b+1)/\tau\rfloor, 1)$ with $\tau=2.5$, which approximates continuous batching interference without token-level loops, and caching of the latency oracle by its functional arguments.

What would settle it

Take a long-generation, short-input workload like the paper's OP4, run the simulator and the real serving system over several prefill-to-decode ratios and collocated configurations, and compare which strategy achieves the highest measured goodput. If the simulated rank order disagrees with the measured rank order — not just the magnitude of the error — the pseudo batch heuristic is misleading for the very decisions the tool is built to make. A more direct test isolates the decode phase: measure per-request token-generation latency as the number of concurrently served requests grows, and check whether the ratios follow the $b^\dagger$ curve for generation lengths of 1024 or more.

Watch

Extended reading notes

Core claim

The paper's central claim is that the maximum goodput of an LLM serving strategy — the highest request arrival rate whose P90 time-to-first-token and time-per-output-token stay within SLO bounds — can be estimated by lightweight simulation instead of measured by repeated benchmark runs. BestServe decomposes inference into operator-level work and memory traffic, applies an adapted roofline model with hardware efficiency parameters (MFU and MBU), and explicitly charges the time for the CPU to dispatch operators to the GPU, which the paper argues makes the decode phase dispatch-bound rather than memory-bound. A pseudo batch size heuristic, $b^\dagger = \max(\lfloor(b+1)/\tau\rfloor, 1)$ with $\tau = 2.5$, stands in for the full dynamics of continuous batching so each request is simulated once rather than token by token. An optimizer then bisects on arrival rate to produce the goodput of every candidate strategy. Validation against manual benchmarking across four operating scenarios reports average absolute relative errors of 11.2%, 12.1%, 8.6%, and 30.1%, with the largest error in the long-generation scenario attributed to the decode heuristic.

Load-bearing premise

The framework stands on one assumption: that a single heuristic formula, the pseudo batch size $b^\dagger = \max(\lfloor(b+1)/\tau\rfloor, 1)$ with $\tau = 2.5$, correctly captures how much slower token generation gets when requests share a GPU, for every generation length and load level.

Editorial extensions

If this is right

  • Service providers can compare collocation versus disaggregation, and different prefill-to-decode instance ratios, for their own traffic mix without renting GPUs for trial-and-error benchmarking.
  • Strategy search becomes a minutes-scale CPU job, so the optimal configuration can be re-derived as request patterns, model sizes, or SLOs change.
  • The framework backs the finding that disaggregation is not uniformly better than collocation: the winner depends on input length, generation length, and SLO thresholds, and BestServe predicts which configuration wins in each scenario.
  • If the dispatch-bound characterization of decode is correct, decode-phase hardware and software tuning should target CPU-GPU dispatch overhead rather than raw memory bandwidth alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own data fixes the failure regime: expect errors near 30% whenever generation length is long and input short (its OP4), and much lower errors in the opposite corner where prefill dominates. The abstract's 20% margin is a claim about typical scenarios, not a uniform bound.
  • The single global value $\tau=2.5$ in the pseudo batch size is the most promising lever: a generation-length-dependent or occupancy-dependent $\tau$ could recover OP4-style accuracy while keeping one simulation loop per request.
  • Because BestServe ignores memory capacity entirely, its 'optimal' strategy should be read as optimal up to memory feasibility; strategies it ranks first could still fail to load on small-memory hardware, especially under PagedAttention-style dynamic caching.
  • The dispatch-bound account yields a concrete prediction the authors only gesture at: reducing CPU-to-GPU dispatch overhead (for example by fusing operators) should improve decode-phase goodput roughly in proportion to the dispatch share of estimated latency, and BestServe's estimator could quantify that gain before any code is written.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. BestServe proposes a three-level framework (Estimator, Simulator, Optimizer) for ranking LLM serving strategies in collocated and disaggregated architectures by predicted goodput. The Estimator uses an adapted roofline model with MFU/MBU efficiency parameters and explicitly models CPU-GPU dispatch latency; the Simulator approximates request-level dynamics of prefill and decode phases with a pseudo batch size heuristic; and the Optimizer enumerates configurations and uses bisection on the arrival rate to find the maximum goodput satisfying SLO constraints. The paper validates on vLLM-Ascend with the CodeLlama-34b-Instruct model on Ascend 910B3 NPUs across four fixed-length operating scenarios, reporting average absolute relative errors of 11.2%, 12.1%, 8.6%, and 30.1% for OP1-OP4. The stated goal is to eliminate costly case-by-case benchmarking by determining the optimal serving strategy in minutes on a standard CPU.

Significance. If the framework's ranking accuracy were established, BestServe would be a practically valuable tool for early-stage deployment planning, as it replaces expensive trial-and-error benchmarking with lightweight simulation. The paper's strengths include its modular hierarchical design, explicit modeling of dispatch-bound decode behavior, and a detailed, mostly self-contained presentation of the estimator's operator-level cost tables in the appendices. The paper is also candid in Section 5 about the decode-phase heuristic and parameter-tuning limitations. However, the central claim is currently under-supported: the validation measures per-strategy goodput error rather than strategy-ranking accuracy; the efficiency parameters are calibrated on the same manual benchmarking data used as ground truth; and the abstract's 20% error claim is contradicted by the OP4 result. These issues are load-bearing for the paper's main contribution and must be addressed before the framework's usefulness for strategy selection can be accepted.

major comments (5)
  1. [§4.3 and Figure 11] The paper's stated purpose is strategy selection, but the validation reports only the per-strategy average absolute relative error of predicted goodput. It never reports whether the strategy with maximal predicted goodput matches the strategy with maximal ground-truth goodput in each operating scenario, nor any rank correlation between predicted and ground-truth orderings. A per-strategy mean error of 11.2-30.1% does not by itself guarantee correct ranking: strategies with similar goodput can be swapped without a large mean error, and Section 5 states that the OP4 error is concentrated in collocated strategies. Please report, for each scenario, the ground-truth and predicted argmax strategies and the goodput gap between them, and provide a rank correlation (e.g., Spearman) over the enumerated strategies. Also state the uncertainty in the manually interpolated ground truth, which the paper's own Figure 10a suggests can fluctuate by about 5% in P90 TTFT.
  2. [Abstract and §4.3] The abstract's claim of 'achieving predictions within a 20% error margin' is not supported by the reported numbers: §4.3 gives average absolute relative errors of 11.2%, 12.1%, 8.6%, and 30.1% for OP1-OP4, so OP4 violates the stated margin by a wide margin. The paper itself attributes this to the pseudo batch size heuristic in §5. The abstract and the conclusion should be revised to a qualified claim (e.g., 'within 20% for scenarios with short-to-moderate generation lengths'), or the validation must be extended to show that decode-heavy scenarios are also within the claimed margin.
  3. [§4.1 vs §4.3] The efficiency parameters ec, em, and e+ are determined in §4.1 by aligning the simulator's intermediate results (Figures 6 and 8) with real inference data from manual benchmarking, and the same manual benchmarking data are then used as the ground truth in §4.3. The reported error rates are therefore a measure of in-sample fit rather than predictive accuracy. Please state this explicitly, quantify how sensitive the predicted goodput and the selected strategy are to perturbations of these parameters within plausible ranges, and, if feasible, hold out one or more operating scenarios or strategies during calibration and report errors on the held-out cases.
  4. [§3.5, Algorithm 8] The bisection loop condition in Algorithm 8, 'while λu − λℓ < ε do', is inverted; it should be 'while λu − λℓ > ε do' so that the loop terminates when the interval is sufficiently small. As written, the pseudocode enters the loop when the difference is already below the tolerance and would not terminate. Please correct the algorithm and confirm that the implemented Optimizer uses the correct stopping condition.
  5. [Eq. (9) and §5] The pseudo batch size heuristic with the single global scalar τ=2.5 is the current bottleneck for decode-heavy scenarios. Section 5 states that in OP4 this heuristic overestimates goodput 'especially for serving strategies using the collocation architecture,' which is precisely the regime where a collocation-versus-disaggregation recommendation could invert. Please add a sensitivity analysis over τ (for example, τ ∈ {1.5, 2.5, 4.0}) and over the feasibility relaxation factor in Algorithm 9, showing whether the top-ranked strategy changes, and report OP4 ranking quality separately for collocated and disaggregated strategies.
minor comments (5)
  1. [Throughout] The name 'BestServe' is frequently concatenated with surrounding words (e.g., 'WepresentBestServe', 'BestServeleverages', 'BestServedemonstrates'); please fix the LaTeX spacing.
  2. [§4.1] The sentence 'linear regressions are conducted exploit the linear relationship' should read 'conducted to exploit the linear relationship'.
  3. [§3.4.2 and §4.1] Section 3.4.2 states that τ=2.5 provides a reasonable approximation, but §4.1's hyperparameter list mentions only MFU, MBU, and communication efficiency; please clarify how τ was set in the validation and whether it was tuned per scenario.
  4. [Appendix B.2] The sentence 'We still use (12) to estimate the computation time of the prefill phase of Attention module when TP is considered' should refer to the decode phase, since Eq. (12) is the decode-phase formula.
  5. [Figure 11] The histograms are sorted by predicted goodput but do not visually distinguish collocated from disaggregated strategies; adding markers or labels for the two architecture types would make it easier to see where ranking errors occur.

Circularity Check

1 steps flagged · score 6.0 of 10

Efficiency parameters are fit to the same manual-benchmark data later used as ground truth, so the reported 20% error margin is in-sample; the optimal-strategy claim is additionally not directly validated.

  1. fitted input called prediction [Section 4.1 (Hyperparameters) and Section 4.3 (Comparison)]
    "To determine appropriate values for ec and em, we aligned the intermediate results of BestServe’s simulator, such as Figures 6 and 8, with real inference data obtained from manual benchmarking. ... These values are only roughly tuned to reflect the practical efficiency of the hardware and software environment used in our experiments without losing too much generality. ... Using the ground truth obtained from manual benchmarking, we query BestServe to approximate the maximum goodput for each serving strategy across all four operating scenarios."

    The efficiency parameters ec, em, and e+ are calibrated by aligning BestServe's simulator outputs with manual-benchmarking data from the testbed (Section 4.1). Section 4.3 then measures 'prediction' error against that same manual-benchmarking ground truth. The reported average absolute relative errors (11.2%, 12.1%, 8.6%, and 30.1%) are therefore in-sample residuals after fitting, not out-of-sample prediction errors. The roofline per-operator estimates and queueing-inspired simulation have independent structure, so the framework is not wholly circular, but the headline claim of 'predictions within a 20% error margin' is partially forced by the fitting step.

full rationale

BestServe's construction is largely self-contained: the Estimator uses explicit FLOP and memory-traffic tables with an adapted roofline model, the Simulator implements queueing-inspired temporal dynamics, and the Optimizer searches configurations by simulated goodput. There is no load-bearing self-citation chain, and the cited systems (vLLM, DistServe, Mooncake) are external baselines. The concrete circularity is in the validation protocol: Section 4.1 fits ec, em, and e+ by aligning simulator outputs with manual-benchmarking data, and Section 4.3 evaluates the simulator's predictions against the same manual-benchmarking ground truth. The headline 20% error margin is thus an in-sample fit residual, not an independent prediction test, and the discrepancy is especially visible in OP4 (30.1% error, which the authors attribute to the pseudo-batch-size decode heuristic). This does not make the whole derivation circular, because the per-operator accounting, roofline equations, and temporal simulation contribute independent content. But the central quantitative promise is supported only by same-data residuals, so a partial-circularity score of 6 is appropriate. The separate gap that the paper never reports whether the Optimizer's top-ranked strategy equals the ground-truth optimal strategy is a missing validation rather than a circularity, so it is not counted in this score.

Assumptions & free parameters 10 free parameters · 6 assumptions · 1 invented entities

The framework depends on several hand-tuned or benchmark-fitted parameters (MFU, MBU, communication efficiency, tau values, dispatch constants) and on the dispatch-bound interpretation of the decode phase. The pseudo batch size is an invented modeling construct without independent evidence. The queueing and roofline elements are standard but are invoked under domain assumptions about the hardware and workload.

free parameters (10)
  • MFU (e_c) = 0.65 for prefill and decode
    Set by aligning simulator intermediate results with real inference data from manual benchmarking (§4.1).
  • MBU prefill (e_m) = 0.6
    Tuned in §4.1 to match prefill-phase measurements.
  • MBU decode (e_m) = 0.3
    Tuned in §4.1 to match decode-phase measurements.
  • Communication efficiency prefill (e_+) = 0.6
    Obtained via linear regression on transmission time vs batch size, sequence length and hidden size (§4.1).
  • Communication efficiency decode (e_+) = 0.3
    Obtained via linear regression on decode-phase communication (§4.1).
  • Pseudo batch size balancing scalar tau = 2.5
    Introduced in Eq. (9) as a heuristic; 'we find that tau=2.5 provides a reasonable approximation in most scenarios' (§3.4.2), without further justification.
  • Feasibility relaxation factor tau = 0.1
    Used in Algorithm 9; §3.5 states 'we acknowledge that this choice is not well-educated'.
  • Goodput upper bound slack constant = 1.2
    lambda_u = 1.2/T_min in Algorithm 8, a heuristic multiple of the queueing-theoretic bound.
  • KV-cache update/repeat/upcast coefficients (kappa_update, kappa_kv, kappa_upcast) = not disclosed
    Introduced in Eq. (12), Appendix A, to represent rates of memory operations in decode attention; no values or fitting procedure given.
  • Per-module dispatch time constants = e.g., 0.024 ms for RMSNorm, 0.190 ms for Attention (Table 3)
    Obtained by profiling a smaller model and reused for larger models (§3.3.3); not provided as a dataset.
assumptions (6)
  • domain assumption Adapted roofline model with MFU/MBU efficiency caps predicts operator-level latency on the target hardware.
    Foundation of the Estimator (§2.5, Eq. 3); assumes efficiency parameters are constant per phase and hardware.
  • domain assumption Decode phase is dispatch-bound, dominated by CPU-to-GPU instruction dispatch rather than memory bandwidth.
    Core claim of §3.3.5; presented as a correction of common belief but supported only by informal profiling.
  • domain assumption Dispatch times per module are constant across models in the LLaMa family and can be profiled on a small model.
    Stated in §3.3.3: 'these figures are highly sensitive to the underlying hardware and may vary significantly across different environments'.
  • domain assumption Request arrivals follow a Poisson process.
    Used for dummy request generation (§4.1) and in the feasibility check of Algorithm 9; real traffic may be bursty.
  • domain assumption vLLM scheduling behavior (prefill priority, no prefill/decode batching) is an accurate model for the collocation baseline.
    Collocation simulator is designed to mimic vLLM (§3.4.4); the availability logic is derived from this behavior.
  • standard math Queueing-theoretic upper bound lambda_u = 1.2/T_min with slack factor 1.2 bounds achievable goodput.
    Based on textbook queueing theory [13]; the 1.2 multiplier is a chosen heuristic without derivation.
invented entities (1)
  • Pseudo batch size b†
    purpose: One-per-request proxy for the dynamic batch size under continuous batching in the decode simulator.
    Introduced in Eq. (9) with hand-tuned tau=2.5; no token-level validation is provided, and the paper attributes the OP4 error to this heuristic.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BestServe: Serving Strategies with Optimal Goodput in Collocation and Disaggregation Architectures." pith.science (2026). https://pith.science/paper/3G4WEZZ3

@misc{pith2026250605871,
  author       = {Pith},
  title        = {Pith review of: BestServe: Serving Strategies with Optimal Goodput in Collocation and Disaggregation Architectures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3G4WEZZ3}},
  note         = {Machine review of arXiv:2506.05871}
}
abstract

Serving large language models (LLMs) to millions of users requires efficient resource allocation and parallelism strategies. It is a labor intensive trial-and-error process to find such a strategy. We present BestServe, a novel framework for ranking serving strategies by estimating goodput under various operating scenarios. Supporting both collocated and disaggregated architectures, BestServe leverages an inference simulator built on an adapted roofline model and CPU-GPU dispatch dynamics. Our framework determines the optimal strategy in minutes on a single standard CPU, eliminating the need for costly benchmarking, while achieving predictions within a $20\%$ error margin. It appeals to be practical for rapid deployment planning because of its lightweight design and strong extensibility.

Figures

Figures reproduced from arXiv: 2506.05871 by the authors.

Figure 1
Figure 1. Illustration of the collocation and disaggregation architectures in LLM serving. Requests are represented by dots: [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the roofline model. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Structural Overview of BestServe. The framework consists of three hierarchical components: Estimator, which predicts operator-level latencies using an adapted roofline model; Simulator, which models the temporal dynamics of request processing and computes serving metrics such as TTFT and TPOT; and Optimizer, which systematically explores serving strategies to identify the configuration that maximizes goodput while s… view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: The two prominent patterns in LLM inference: compute-bound and dispatch-bound. For simplicity, it is assumed [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Visualization of service metrics from the simulation [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Performance of P90 TTFTs and TPOTs against re [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Distribution of service metrics from the simulation [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Performance of P90 TTFTs and TPOTs against re [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Impact of one-shot and repetitive testing on the [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Comparison of the maximum goodput estimated by [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Simple is Better: Multiplication May Be All You Need for LLM Request Scheduling

    cs.DC 2026-03 conditional novelty 6.0 of 10

    Multiplying new-prefill-token count by instance batch size yields a hyperparameter-free LLM scheduler that matches or beats tuned linear and simulation policies on real production traces.

Reference graph

Works this paper leans on

36 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023

    2023

  2. [2]

    How continuous batching enables 23x throughput in LLM inference while reducing p50 latency

    Anyscale. How continuous batching enables 23x throughput in LLM inference while reducing p50 latency. https://www.anyscale.com/blog/continuous- batching-llm-inference, 2023

    work page 2023

  3. [3]

    Microsoft Azure.https://azure.microsoft.com/

    work page

  4. [4]

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016

    2016

  5. [5]

    Throughput is not all you need: Maximizing goodput in llm serving using prefill-decode disaggregation.https://hao-ai- lab.github.io/blogs/distserve/, 2024

    Junda Chen, Yinmin Zhong, Shengyu Liu, Yibo Zhu, Xin Jin, and Hao Zhang. Throughput is not all you need: Maximizing goodput in llm serving using prefill-decode disaggregation.https://hao-ai- lab.github.io/blogs/distserve/, 2024

    work page 2024

  6. [6]

    Google Cloud.https://cloud.google.com/

    work page

  7. [7]

    Huawei Cloud.https://www.huaweicloud.com/

    work page

  8. [8]

    EleutherAI and the HuggingFace Inc. team. modelling_llama.py. https://github.com/huggingface/ transformers/blob/main/src/transformers/ models/llama/modeling_llama.py, 2022

    work page 2022

Show all 36 references
  1. [9]

    Sigmoid-weighted linear units for neural network function approximation in reinforcement learning, 2017

    Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning, 2017

    2017

  2. [10]

    Text generation inference

    Hugging Face. Text generation inference. https://huggingface.co/docs/text-generation- inference/en/index

  3. [11]

    Low latency rnn inference with cellular batching

    Pin Gao, Lingfan Yu, Yongwei Wu, and Jinyang Li. Low latency rnn inference with cellular batching. In Proceedings of the Thirteenth EuroSys Conference, EuroSys ’18, New York, NY , USA, 2018. Association for Computing Machinery

    2018

  4. [12]

    Getting started with CUDA graphs

    Alan Gray. Getting started with CUDA graphs. https: //developer.nvidia.com/blog/cuda-graphs/, 2019

    2019

  5. [13]

    Shortle, James M

    Donald Gross, John F. Shortle, James M. Thompson, and Carl M. Harris.Fundamentals of Queueing Theory. Wiley-Interscience, USA, 4th edition, 2008

    2008

  6. [14]

    Pipedream: Fast and efficient pipeline parallel dnn training, 2018

    Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training, 2018

    2018

  7. [15]

    Inference without interference: Disaggregate llm inference for mixed downstream workloads, 2024

    Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. Inference without interference: Disaggregate llm inference for mixed downstream workloads, 2024

    2024

  8. [16]

    Le, Yonghui Wu, and Zhifeng Chen.GPipe: efficient training of giant neural networks using pipeline parallelism

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V . Le, Yonghui Wu, and Zhifeng Chen.GPipe: efficient training of giant neural networks using pipeline parallelism. Curran Associates Inc., Red Hook, NY , USA, 2019

    2019

  9. [17]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024

  10. [18]

    Efficient memory management for large language model serving with PagedAttention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles, SO...

    2023

  11. [19]

    Transformers KV caching explained

    João Lages. Transformers KV caching explained. https://medium.com/@joaolages/kv-caching- explained-276520203249, 2023

    2023

  12. [20]

    Sequence parallelism: Long sequence training from system perspective, 2022

    Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. Sequence parallelism: Long sequence training from system perspective, 2022

    2022

  13. [21]

    Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

    Meta. Llama 3.2: Revolutionizing edge AI and vision with open, customizable models. https://ai.meta.com/blog/llama-3-2-connect- 2024-vision-edge-mobile-devices/, 2024

    2024

  14. [22]

    NVIDIA TensorRT-LLM.https: //docs.nvidia.com/tensorrt-llm/index.html

    NVIDIA. NVIDIA TensorRT-LLM.https: //docs.nvidia.com/tensorrt-llm/index.html

  15. [23]

    OpenAI o3-mini

    OpenAI. OpenAI o3-mini. https://openai.com/index/openai-o3-mini/, 2025. 23

    2025

  16. [24]

    Splitwise: Efficient generative llm inference using phase splitting, 2024

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting, 2024

    2024

  17. [25]

    Mooncake: A KVCache-centric disaggregated architecture for LLM serving, 2024

    Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: A KVCache-centric disaggregated architecture for LLM serving, 2024

    2024

  18. [26]

    Focus: For tech giants, AI like Bing and Bard poses billion-dollar search problem

    Reuters. Focus: For tech giants, AI like Bing and Bard poses billion-dollar search problem. https://www.reuters.com/technology/tech- giants-ai-like-bing-bard-poses-billion- dollar-search-problem-2023-02-22/, 2023

    2023

  19. [27]

    Amazon Web Services.https://aws.amazon.com/

  20. [28]

    Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020

    2020

  21. [29]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InProceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, pages 6000–6010, Red H...

    2017

  22. [30]

    https://docs.vllm.ai/en/v0.4.2/index.html

    vLLM. https://docs.vllm.ai/en/v0.4.2/index.html

  23. [31]

    https://github.com/vllm-project/vllm-ascend

    vLLM Ascend. https://github.com/vllm-project/vllm-ascend

  24. [32]

    Simai: Unifying architecture design and performance tuning for large-scale large language model training with scalability and precision

    Xizheng Wang, Qingxu Li, Yichi Xu, Gang Lu, Dan Li, Li Chen, Heyang Zhou, Linkang Zheng, Sen Zhang, Yikai Zhu, Yang Liu, Pengchang Zhang, Kun Qian, Kunling He, Jiaqi Gao, Ennan Zhai, Dennis Cai, and Binzhang Fu. Simai: Unifying architecture design and performance tuning for la...

    2025

  25. [33]

    Roofline: an insightful visual performance model for multicore architectures.Commun

    Samuel Williams, Andrew Waterman, and David Patterson. Roofline: an insightful visual performance model for multicore architectures.Commun. ACM, 52(4):65–76, April 2009

    2009

  26. [34]

    Orca: A distributed serving system for Transformer-Based generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for Transformer-Based generative models. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, Carlsbad, CA, July 2022. ...

    2022

  27. [35]

    Root mean square layer normalization, 2019

    Biao Zhang and Rico Sennrich. Root mean square layer normalization, 2019

    2019

  28. [36]

    DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In USENIX Symposium on Operating Systems Design and Implementation, 2024. 24

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.