Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

Hetis claims that module-level parallelism—dense layers on fast GPUs, attention heads dynamically spread across all GPUs—raises LLM serving throughput up to 2.25x and cuts latency up to 1.49x relative to existing heterogeneity-aware systems

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Hetis dynamically splits LLM attention across heterogeneous GPUs at the granularity of individual heads while keeping dense computation on a selected subset, and reports up to 2.25x throughput gains over Splitwise and Hexgen.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Hetis is a credible heterogeneous-LLM-serving system with genuinely new head-wise dynamic attention dispatch; the headline 1.49x latency gain is a module-level number, but the end-to-end 1.39x/2.25x results hold up under scrutiny. the 5 major comments →

arxiv 2509.08309 v1 pith:RXRJRN67 submitted 2025-09-10 cs.DC

Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism

Zizhao Mo , Jianxiong Liao , Huanle Xu , Zhi Zhou , Chengzhong Xu This is my paper
classification cs.DC
keywords heterogeneous GPU clusterLLM servingdynamic parallelismattention head partitioningKV cache managementonline load dispatchingmodule-level parallelismtransformer inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Hetis is a serving system for LLM inference on clusters where a few fast GPUs sit alongside much slower ones, and its proposal is to stop parallelizing the model uniformly. Dense, parameter-heavy modules—QKV projection and MLP—run only on a chosen subset of 'primary workers,' while Attention, which has no parameters, is split at the granularity of individual heads across every GPU, so low-end cards become useful KV-cache holders and attention compute. An online dispatcher solves a linear program at each arrival to decide how many heads each device should handle, using profiled linear models of attention time and network transfer cost, with a re-dispatching scheme that moves only the affected heads when long contexts create imbalance. The authors report up to 2.25x higher serving throughput and up to 1.49x lower latency than the phase-splitting and asymmetric-partitioning baselines on a mixed A100/3090/P100 cluster. If the result holds, operators with a heterogeneous fleet can put slow GPUs to work instead of leaving them idle or letting them bottleneck dense math.

Core claim

The paper's claim is that the right unit of parallelism for heterogeneous LLM serving is the module, not the layer or the phase. Hetis separates the Transformer into two kinds of work: dense, parameter-heavy computation (QKV projection, MLP) and parameter-free Attention. For dense work it searches over data-, tensor-, and pipeline-parallel configurations on a subset of GPUs—the 'primary workers'—and deliberately excludes low-end devices whose contribution to dense computation is outweighed by communication cost. For Attention it treats each request's heads as independently schedulable: primary and attention workers each receive some number of heads plus the corresponding KV-cache slices, and

What carries the argument

The load-bearing mechanism is head-granularity dynamic attention dispatch. Each request's attention heads are partitioned across primary and attention workers, KV-cache blocks are indexed per head, and an online dispatcher solves a linear program that minimizes the maximum attention-plus-transfer time across devices, with per-device attention time modeled linearly in the number of heads and cache bytes and point-to-point transfer time modeled by a linear communication-cost model. The supporting mechanism is primary-worker selection: a hierarchical search removes low-end GPUs from dense computation whenever their contribution to reducing dense-module cost is below a small threshold, so slow G

Load-bearing premise

The dispatcher's head allocations are only as good as the profiled assumption that a GPU's attention time grows linearly with the number of heads it processes and the size of the cache it reads; if that curve bends under memory-bandwidth saturation or at very small head counts, the allocations will be wrong.

What would settle it

Profile attention kernel time on each GPU type at a fixed cache size while sweeping the number of heads from 1 up to the model's full head count, and repeat at several cache sizes; if the measured time is not an affine function of heads and cache size—for example, a sharp rise below a few heads due to launch overhead or a flattening as HBM bandwidth saturates—then the linear model behind the dispatcher is falsified and its head allocations are not optimal.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Low-end GPUs that are uncompetitive for MLP still contribute to serving capacity by hosting KV cache and computing attention heads, turning previously stranded memory into usable throughput.
  • Because rebalancing happens at head granularity, moving a long request to relieve a bottleneck costs only partial KV-cache transfers rather than whole-sequence migration, so adaptation can happen online.
  • The dispatch formulation supports both MHA and GQA by folding the query-to-KV head ratio into the communication cost, so the scheme is not tied to one attention architecture.
  • Heterogeneous serving no longer has to choose between phase splitting, which duplicates model parameters, and asymmetric parameter splitting, which strands cache space; the module-level role split avoids both.
  • Concretely, on the paper's A100/3090/P100 testbed Hetis sustains up to 2.25x the request rate of the phase-splitting baseline and up to 1.33x that of the asymmetric-partitioning baseline, with P95 time-per-output-token up to 1.39x lower.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same module-level split should generalize to other parameter-free or low-intensity components—normalization, MoE routing, or even attention on CPU memory—wherever a device's dense-compute throughput is much worse than its memory bandwidth.
  • A testable prediction: Hetis's throughput advantage over asymmetric partitioning should widen as the slowest GPU's MLP gap grows, until the network becomes the limiting factor; clusters with more low-end GPUs would expose that crossover.
  • The linear attention-time model is the assumption worth stress-testing: a convex piecewise extension inside the same LP framework would cover launch overhead and bandwidth saturation while keeping dispatch real-time.
  • Because newly arriving requests are dispatched without re-partitioning existing ones, workloads dominated by many short requests may leave some head-level slack; a more anticipatory rebalancing policy could close that gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. Hetis is an LLM serving system for heterogeneous GPU clusters that combines two forms of parallelism: (i) "primary worker parallelism," which selects a subset of GPUs for dense modules such as MLP, and (ii) dynamic head-wise parallelism for attention, in which attention heads and the associated KV caches are distributed across primary and attention workers. The dispatcher solves a linear program (Eq. 7) to minimize the maximum per-device attention time, with re-dispatching for long contexts and head-wise KV cache migration. The system is evaluated on a cluster with A100, 3090, and P100 GPUs for LLaMA-13B, OPT-30B, and LLaMA-70B on ShareGPT, HumanEval, and LongBench, against Splitwise and Hexgen. The paper claims up to 2.25× throughput improvement and 1.49× latency reduction over these baselines.

Significance. The head-granularity dynamic attention offloading idea is a genuine and useful departure from the coarse-grained, static partitioning used by Splitwise and Hexgen, and the module-level ablation in Fig. 13 is a good-faith attempt to attribute the gains to that mechanism. If the linear attention model and LP dispatch are robust, the system would be a solid contribution to heterogeneous LLM serving. The paper also ships a real prototype on vLLM, reports profiling accuracy, and includes sensitivity analyses for the re-dispatch threshold and profiling errors, which are strengths. However, the headline latency claim is not end-to-end, and several load-bearing algorithmic details (integrality of head allocation, completeness of the re-dispatch LP, validity of the linear model outside the profiled regime) need to be tightened before the central claims can be fully accepted.

major comments (5)
  1. [Abstract and §7.3] The headline claim 'reduce latency by 1.49×' is taken from the module-level attention latency in Fig. 13, not from the end-to-end results. The end-to-end P95 TPOT improvement is up to 1.39× and TTFT up to 1.47× (Fig. 12). The abstract and §1 should either report the end-to-end numbers or explicitly state that 1.49× is the attention-module latency reduction, otherwise the central advertised result is overstated.
  2. [§5.2, Eq. (7)] The paper states in §5.2.1 that x_i^j(t)/r must be an integer (head groups are indivisible), but §5.2.2 solves the problem as a continuous LP and does not describe rounding or an integrality argument. If fractional allocations are allowed, constraints (5) and (7c) are not truly enforced; if fractional solutions are rounded, the claimed optimality of the dispatch is not established. The authors should either solve an integer program (with a justification of tractability at serving timescales), prove total unimodularity of the constraint matrix, or report the effect of rounding on the dispatch quality.
  3. [§5.1, Eq. (3) and Fig. 7/16(b)] The dispatch objective rests on the linear attention-time model tau_i = a_i h_i + b_i g_i + c_i. The supporting evidence in Fig. 7 is limited to single-setting curves with no error bars and no coverage of the extremes (very small head counts, very large caches, or HBM-saturation regimes). The robustness test in Fig. 16(b) perturbs the coefficients by ±20%, but a structural nonlinearity (e.g., quadratic cache-size behavior or launch-overhead floor at small h_i) is not a coefficient perturbation. Since the LP in Eq. (7) will systematically misallocate heads if the model is misspecified, the authors should validate the linear model across the full operating range used by the dispatcher, or add a structural robustness test that compares LP decisions against a nonlinear ground-truth model.
  4. [§5.3.1] The re-dispatching optimization displayed in §5.3.1 is incomplete. As written, the constraint is only sum_i sum_j x_j^i(t) l_j(t) <= sum_i r M_i/2, with no per-request head-integrity constraint sum_i x_j^i(t) = H and no non-negativity/integrality constraints. Without those constraints, the reported 'ideal attention time' f* is not well defined (a trivial solution could assign zero heads to all requests). The formulation should be stated in full, matching Eq. (7).
  5. [§7.2, Figs. 8-10] The paper claims up to 2.25× throughput improvement, but throughput is never explicitly defined or measured. Figures 8-10 plot normalized latency against request rate; the throughput gain appears to be inferred from the highest request rate before latency rises steeply. The authors should define the throughput metric (e.g., achieved request rate under a latency SLO), state the SLO if one is used, and report per-dataset saturation throughput for Hetis, Splitwise, and Hexgen.
minor comments (6)
  1. [Figs. 8-13] The reported latency curves and bars do not include error bars or the number of runs. For a systems paper with P95 metrics, at least a few repetitions and confidence intervals should be reported, especially because some differences between systems are small.
  2. [§3.2 and §6] The text says head-wise KV cache migration has 'zero overhead,' but §6 describes low-priority CUDA streams to reduce the adverse impact, and Fig. 15(a) shows that re-dispatching still has a measurable cost. The 'zero-overhead' wording should be removed or qualified.
  3. [§7.4] The modeling-accuracy numbers ('up to 93.8%', '92.4% to 96.1%') are reported without defining the accuracy metric. Please state whether this is MAPE, R², or another measure.
  4. [§7.4] The large-scale search evaluation ('five GPU types with 32 GPUs each') is described only as a simulation, with no setup details. Please clarify whether this is an actual run of the search code and give the simulated device and model parameters.
  5. [§5.2.1, Eq. (5)] The notation sum_i x_j^i(t) ∈ {0,H} is unusual; since Eq. (7c) already enforces the H constraint for incoming requests, consider making explicit that the {0,H} form covers the no-allocation case and is not a typo.
  6. [§4.1] The text refers to a 'rigorous optimization problem formulation' but the GPU-exclusion criterion with Delta=0.05 and the one-by-one removal from lowest to highest end is a heuristic. The heuristic nature should be acknowledged in the main text, not only implicitly in the search description.

Circularity Check

0 steps flagged

No significant circularity: the reported 2.25x throughput and latency improvements are measured end-to-end against external baselines; the fitted attention-time model steers dispatch decisions but does not generate the headline numbers.

full rationale

The central claims of Hetis are empirical: throughput and P95 TTFT/TPOT gains in Figs. 8-13 are measured on a real A100/3090/P100 cluster against Splitwise and Hexgen, not computed from the paper's own models. The fitted linear Attention-time model (Eq. 3) and communication model (Eq. 4) are used only to choose head allocations in the dispatcher; they are not used to predict the reported improvements. The module-level breakdown in Fig. 13 is also measured execution time, so the claim that head-wise dynamic parallelism causes the gains is supported by experiment, not by construction. The only self-citations of the authors ([31,32,33]) appear in background discussion and in a modeling assumption ("perfect latency scaling [33]") used during the configuration search in Sec. 4.1. That assumption is a heuristic in the search, and the resulting configuration is validated empirically; it is not a uniqueness theorem and does not force the end-to-end result. The linearity of Eq. (3), supported only by the narrow plots in Figs. 7(b)-(c), is a legitimate correctness/robustness concern (and the paper's own ±20% parameter-perturbation test in Fig. 16(b) does not address structural nonlinearity), but that is a modeling-risk issue rather than circularity: nothing in the paper's derivation chain reduces a reported prediction to a fitted value or to a self-citation.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claims rest on per-device linear cost models for attention and transfer (Eqs. 3-4) whose coefficients are fitted from profiling on the very cluster used in evaluation, plus hand-chosen thresholds (Delta=0.05, Theta=50%) that control the primary-worker search and re-dispatching. No new physical entities are introduced; the novel structures are software roles (Primary worker, Attention worker, Hauler, Dispatcher) and head-wise cache management.

free parameters (5)
  • a_i, b_i, c_i = fitted from profiling, per GPU type (exact values not reported)
    Per-device coefficients of the linear attention-time model (Eq. 3). Fitted from measured attention times at eight h_i and eight g_i configurations on the evaluation cluster.
  • gamma_i, beta_i = fitted from profiling, per GPU type (exact values not reported)
    Per-device coefficients of the linear transfer-time model (Eq. 4), used to model network overhead for Attention workers.
  • Delta = 0.05
    Hand-chosen threshold in Section 4.1 that decides when a low-end GPU can be removed from the dense-compute Primary set.
  • Theta = 50%
    Hand-chosen re-dispatching trigger in Section 5.3.1; if the deviation between ideal and current Attention time exceeds this, one request is re-dispatched.
  • Profiling grid size = eight h_i and eight g_i configurations
    Section 7.4 states Hetis fits the model using eight h_i and eight g_i points; the choice of these points affects model accuracy.
axioms (5)
  • domain assumption Attention computation in the decode phase is well modeled as linear in the number of query heads and in the KV cache size (Eq. 3).
    Core of the dispatch and re-dispatch optimization; if false, the LP solutions do not minimize actual attention time.
  • domain assumption Point-to-point transfer overhead follows the Alpha-Beta linear model (Eq. 4).
    Used to compute communication cost for Attention workers; assumes no contention and linear scaling in data volume.
  • domain assumption All layers in a model have identical compute time, so profiling one layer suffices (Section 7.4: 'Thanks to the layer identity in LLM').
    Profiling is done once per configuration for the Attention module; layer-to-layer variation (e.g., different cache sizes per layer) is ignored.
  • domain assumption Perfect latency scaling of dense computation when constructing pipeline stages (Section 4.1, 'under the assumption of perfect latency scaling [33]').
    Used for the initial layer-to-stage mapping before searching TP/PP combinations; if perfect scaling fails, the excluded-GPU criterion is miscalibrated.
  • ad hoc to paper The GPU-exclusion criterion C_p(sigma-kappa,M,R) <= (1+Delta)*C_p(sigma,M,R) identifies low-end GPUs that can be safely removed from dense compute.
    Heuristic with hand-chosen Delta=0.05; no optimality guarantee, and the choice affects the Primary/Attention worker split.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism." pith.science (2026). https://pith.science/paper/RXRJRN67

@misc{pith2026250908309,
  author       = {Pith},
  title        = {Pith review of: Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RXRJRN67}},
  note         = {Machine review of arXiv:2509.08309}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The significant resource demands in LLM serving prompts production clusters to fully utilize heterogeneous hardware by partitioning LLM models across a mix of high-end and low-end GPUs. However, existing parallelization approaches often struggle to scale efficiently in heterogeneous environments due to their coarse-grained and static parallelization strategies. In this paper, we introduce Hetis, a new LLM system tailored for heterogeneous GPU clusters. Hetis addresses two critical challenges: (1) memory inefficiency caused by the mismatch between memory capacity and computational power in heterogeneous devices, and (2) computational inefficiency arising from performance gaps across different LLM modules. To tackle these issues, Hetis employs a fine-grained and dynamic parallelism design. Specifically, it selectively parallelizes compute-intensive operations to reduce latency and dynamically distributes Attention computations to low-end GPUs at a head granularity, leveraging the distinct characteristics of each module. Additionally, Hetis features an online load dispatching policy that continuously optimizes serving performance by carefully balancing network latency, computational load, and memory intensity. Evaluation results demonstrate that Hetis can improve serving throughput by up to $2.25\times$ and reduce latency by $1.49\times$ compared to existing systems.

Figures

Figures reproduced from arXiv: 2509.08309 by Chengzhong Xu, Huanle Xu, Jianxiong Liao, Zhi Zhou, Zizhao Mo.

Figure 1
Figure 1. Figure 1: Memory inefficiency in existing systems. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The execution time of the decoding MLP and At [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: The hierarchical searching process for configuring [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Illustration of dynamic Attention parallelism: each [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: The normalized end-to-end latency across different [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: The normalized end-to-end latency across different [PITH_FULL_IMAGE:figures/full_fig_p009_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: The normalized end-to-end latency across different [PITH_FULL_IMAGE:figures/full_fig_p009_10.png] view at source ↗
Figure 13
Figure 13. Figure 13: The P95 execution latency on Attention and MLP [PITH_FULL_IMAGE:figures/full_fig_p010_13.png] view at source ↗
Figure 12
Figure 12. Figure 12: The P95 TTFT and TPOT latency across datasets [PITH_FULL_IMAGE:figures/full_fig_p010_12.png] view at source ↗
Figure 15
Figure 15. Figure 15: The benefit and management overhead of dynamic [PITH_FULL_IMAGE:figures/full_fig_p011_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Robustness of Hetis system. simulation featuring five GPU types with 32 GPUs each, where the search process completes in just 15 seconds. Since this process is only executed once prior to the system’s initial deployment, its time cost is effectively negligible. Head-wise cache management overhead. We explored the over￾head associated with changing the token-wise cache management in vLLM to a head-wise app… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Lodestar: An Online-Learning LLM Inference Router

    cs.DC 2026-05 unverdicted novelty 6.0

    Lodestar deploys continuous online learning to route LLM inference requests across GPU clusters, reporting 1.41x lower average TTFT versus heuristics.

Reference graph

Works this paper leans on

53 extracted references · 31 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    NCCL: accelerated multi-GPU collective communication

    2023. NCCL: accelerated multi-GPU collective communication. https://images. nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf

    work page 2023

  2. [2]

    ShareGPT

    2023. ShareGPT. https://sharegpt.com/

    work page 2023

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report.arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  4. [4]

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming throughput-latency tradeoff in llm inference with sarathi-serve.arXiv preprint arXiv:2403.02310(2024)

    Pith/arXiv arXiv 2024

  5. [5]

    Ahsan Ali, Riccardo Pinciroli, Feng Yan, and Evgenia Smirni. 2020. Batch: Machine learning inference serving on serverless platforms with adaptive batching. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–15

    work page 2020

  6. [6]

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme 11 SC ’25, November 16–21, 2025, St Louis, MO, USA Zizhao Mo, Jianxiong Liao, Huanle Xu, Zhi Zhou, and Chengzhong Xu Penedo. 20...

    work page 2025

  7. [7]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al . 2023. Longbench: A bilingual, multitask benchmark for long context understanding.arXiv preprint arXiv:2308.14508(2023)

    Pith/arXiv arXiv 2023

  8. [8]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.Advances in neural information processing systems33 (2020), 1877–1901

    2020

  9. [9]

    Shubham Chaudhary, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, and Srinidhi Viswanatha. 2020. Balancing efficiency and fairness in heteroge- neous GPU clusters for deep learning. InProceedings of the Fifteenth European Conference on Computer Systems. 1–16

    work page 2020

  10. [10]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374(2021)

    Pith/arXiv arXiv 2021

  11. [11]

    Yujeong Choi, Yunseong Kim, and Minsoo Rhu. 2021. Lazy batching: An SLA- aware batching system for cloud machine learning inference. In2021 IEEE Inter- national Symposium on High-Performance Computer Architecture (HPCA). IEEE, 493–506

    work page 2021

  12. [12]

    Michael B Cohen, Yin Tat Lee, and Zhao Song. 2021. Solving linear programs in the current matrix multiplication time.Journal of the ACM (JACM)68, 1 (2021), 1–39

    work page 2021

  13. [13]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. InAdvances in Neural Information Processing Systems (NeurIPS)

    2022

  14. [14]

    Steven Diamond and Stephen Boyd. 2016. CVXPY: A Python-embedded modeling language for convex optimization.Journal of Machine Learning Research17, 83 (2016), 1–5

    work page 2016

  15. [15]

    Jiangfei Duan, Runyu Lu, Haojie Duanmu, Xiuhong Li, Xingcheng Zhang, Dahua Lin, Ion Stoica, and Hao Zhang. [n. d.]. MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving. InForty-first International Conference on Machine Learning

    work page

  16. [16]

    Michael R Garey, David S Johnson, and Ravi Sethi. 1976. The complexity of flowshop and jobshop scheduling.Mathematics of operations research1, 2 (1976), 117–129

    work page 1976

  17. [17]

    Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, and Ion Stoica. 2024. M\’elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity.arXiv preprint arXiv:2404.14527(2024)

    Pith/arXiv arXiv 2024

  18. [18]

    Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace. 2020. Serving{DNNs} like clockwork: Perfor- mance predictability from the bottom up. In14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). 443–462

    work page 2020

  19. [19]

    Jiaao He and Jidong Zhai. 2024. FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines.arXiv preprint arXiv:2403.11421 (2024)

    Pith/arXiv arXiv 2024

  20. [20]

    Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, et al. 2024. Inference with- out Interference: Disaggregate LLM Inference for Mixed Downstream Workloads. arXiv preprint arXiv:2401.11181(2024)

    Pith/arXiv arXiv 2024

  21. [21]

    Suhas Jayaram Subramanya, Daiyaan Arfeen, Shouxu Lin, Aurick Qiao, Zhihao Jia, and Gregory R Ganger. 2023. Sia: Heterogeneity-aware, goodput-optimized ML-cluster scheduling. InProceedings of the 29th Symposium on Operating Systems Principles. 642–657

    work page 2023

  22. [22]

    Xianyan Jia, Le Jiang, Ang Wang, Wencong Xiao, Ziji Shi, Jie Zhang, Xinyuan Li, Langshi Chen, Yong Li, Zhen Zheng, et al . 2022. Whale: Efficient giant model training over heterogeneous{GPUs}. In2022 USENIX Annual Technical Conference (USENIX ATC 22). 673–688

    work page 2022

  23. [23]

    Youhe Jiang, Ran Yan, Xiaozhe Yao, Yang Zhou, Beidi Chen, and Binhang Yuan

    work page

  24. [24]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361(2020)

    Pith/arXiv arXiv 2020

  25. [25]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles. 611–626

    2023

  26. [26]

    Tan N Le, Xiao Sun, Mosharaf Chowdhury, and Zhenhua Liu. 2020. Allox: compute allocation in hybrid clusters. InProceedings of the Fifteenth European Conference on Computer Systems. 1–16

    work page 2020

  27. [27]

    Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yan- ping Huang, Zhifeng Chen, Hao Zhang, Joseph E Gonzalez, et al. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23)

    work page 2023

  28. [28]

    Bin Lin, Tao Peng, Chen Zhang, Minmin Sun, Lanbo Li, Hanyu Zhao, Wencong Xiao, Qi Xu, Xiafei Qiu, Shen Li, et al. 2024. Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache.arXiv preprint arXiv:2401.02669(2024)

    Pith/arXiv arXiv 2024

  29. [29]

    Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshu- mali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. 2023. Deja vu: Contextual sparsity for efficient llms at inference time. InInternational Conference on Machine Learning. PMLR, 22137–22176

    work page 2023

  30. [30]

    Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia. 2023. Spotserve: Serving generative large language models on pre- emptible instances.arXiv preprint arXiv:2311.15566(2023)

    Pith/arXiv arXiv 2023

  31. [31]

    Zizhao Mo, Huanle Xu, and Wing Cheong Lau. 2024. Optimal Resource Effi- ciency with Fairness in Heterogeneous GPU Clusters. InProceedings of the 25th International Middleware Conference. 36–48

    work page 2024

  32. [32]

    Zizhao Mo, Huanle Xu, and Wing Cheong Lau. 2025. Fast and Fair Training for Deep Learning in Heterogeneous GPU Clusters. InProceedings of the 39th ACM International Conference on Supercomputing

    work page 2025

  33. [33]

    Zizhao Mo, Huanle Xu, and Chengzhong Xu. 2024. Heet: Accelerating Elas- tic Training in Heterogeneous Deep Learning Clusters. InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 499–513

    work page 2024

  34. [34]

    Deepak Narayanan, Keshav Santhanam, Fiodar Kazhamiaka, Amar Phanishayee, and Matei Zaharia. 2020. Heterogeneity-aware cluster scheduling policies for deep learning workloads. InProceedings of the 14th USENIX Conference on Oper- ating Systems Design and Implementation. 481–498

    work page 2020

  35. [35]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient generative llm inference using phase splitting. In2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 118–132

    2024

  36. [36]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer.The Journal of Machine Learning Research21, 1 (2020), 5485–5551

    2020

  37. [37]

    Timo Schneider. 2014. Communication Models. https://spcl.inf.ethz.ch/Teaching/ 2019-dphpc/lectures/lecture12-comm-models.pdf

    work page 2014

  38. [38]

    Haichen Shen, Lequn Chen, Yuchen Jin, Liangyu Zhao, Bingyu Kong, Matthai Philipose, Arvind Krishnamurthy, and Ravi Sundaram. 2019. Nexus: A GPU cluster engine for accelerating DNN-based video analysis. InProceedings of the 27th ACM Symposium on Operating Systems Principles. 322–337

    work page 2019

  39. [39]

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. Flexgen: High-throughput generative inference of large language models with a single gpu. InInternational Conference on Machine Learning. PMLR, 31094–31116

    2023

  40. [40]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971(2023)

    Pith/arXiv arXiv 2023

  41. [41]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288(2023)

    Pith/arXiv arXiv 2023

  42. [42]

    Taegeon Um, Byungsoo Oh, Minyoung Kang, Woo-Yeon Lee, Goeun Kim, Dongseob Kim, Youngtaek Kim, Mohd Muzzammil, and Myeongjae Jeon. 2024. Metis: Fast Automatic Distributed Training on Heterogeneous{GPUs}. In2024 USENIX Annual Technical Conference (USENIX ATC 24). 563–578

    work page 2024

  43. [43]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.Advances in neural information processing systems30 (2017)

    2017

  44. [44]

    Yuxin Wang, Yuhan Chen, Zeyu Li, Zhenheng Tang, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, and Xiaowen Chu. 2024. Towards Efficient and Reliable LLM Serving: A Real-World Workload Study.arXiv preprint arXiv:2401.17644 (2024)

    Pith/arXiv arXiv 2024

  45. [45]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung- Gon Chun. 2022. Orca: A distributed serving system for{Transformer-Based} generative models. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 521–538

    work page 2022

  46. [46]

    Chengliang Zhang, Minchen Yu, Wei Wang, and Feng Yan. 2019. {MArk}: Exploiting cloud services for{Cost-Effective},{SLO-Aware} machine learning inference serving. In2019 USENIX Annual Technical Conference (USENIX ATC 19). 1049–1062

    work page 2019

  47. [47]

    Hong Zhang, Yupeng Tang, Anurag Khandelwal, and Ion Stoica. 2023. {SHEPHERD}: Serving{DNNs} in the Wild. In20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). 787–808

    work page 2023

  48. [48]

    Shiwei Zhang, Lansong Diao, Chuan Wu, Zongyan Cao, Siyu Wang, and Wei Lin

    work page

  49. [49]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068 (2022)

    Pith/arXiv arXiv 2022

  50. [50]

    InProceedings of the Nineteenth European Conference on Computer Systems

    HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Auto- mated Program Synthesis. InProceedings of the Nineteenth European Conference on Computer Systems. 524–541. 12 Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism SC ’25, November 16–21, 2025, St Louis, MO, USA

    work page 2025

  51. [51]

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decod- ing for Goodput-optimized Large Language Model Serving.arXiv preprint arXiv:2401.09670(2024). 13

    Pith/arXiv arXiv 2024

  52. [52]

    Juntao Zhao, Borui Wan, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. LLM- PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization.arXiv preprint arXiv:2403.01136(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  53. [2024]

    InForty-first International Conference on Machine Learning

    HexGen: Generative Inference of Large Language Model over Heteroge- neous Environment. InForty-first International Conference on Machine Learning

    work page

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.