Pith. sign in

REVIEW 4 major objections 6 minor 14 references

HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that splitting prefill and decoding across heterogeneous GPUs, with a graph-partition and max-flow scheduler, delivers up to 2.0x serving throughput and 1.5x lower latency at the same price, and comparable performance at…

desk verdict A plausible new scheduler for disaggregated LLM serving on heterogeneous GPUs, but the headline numbers rest on unreported variance and an unquantified cost-model validation. read the letter →

arxiv 2502.07903 v1 pith:MRK7GAQ7 submitted 2025-02-11 cs.DC

Youhe Jiang , Ran Yan , Binhang Yuan This is my paper · ORCID
classification cs.DC
keywords LLMinferencedisaggregatedheterogeneousGPUsgraphpartitioningmax-flowschedulingKVcachetransferservingthroughputcostefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the disaggregated inference paradigm — running the compute-heavy prefill phase and the memory-bandwidth-heavy decoding phase on separate GPU groups — can be made to work well on a pool of heterogeneous GPUs and network links, not just on a uniform high-end cluster. Because heterogeneous GPU fleets are cheaper to assemble, an effective scheduler for them could lower the cost of LLM serving. The proposed system treats placement as a constrained optimization: partition GPUs into serving groups, decide which groups prefill and which decode, pick tensor and pipeline parallel strategies per phase, and route key-value cache transfers. On OPT-30B and Llama-2-70B workloads the authors report up to 2.0x and on average 1.3x higher throughput than state-of-the-art baselines at equal budget, 1.5x lower latency, and comparable performance at a 30 percent smaller budget.

What carries the argument

The load-bearing object is the directed max-flow graph. Prefill and decoding replicas are compute nodes whose internal edge capacity is the number of requests the replica can process in a fixed time window, estimated from the paper's inference cost model; valid edges run from the coordinator to a prefill node, from a prefill node to a decoding node (capacity derived from KV-cache transfer cost over the underlying GPU-to-GPU links), and from a decoding node back to the coordinator. The max flow through this network is the request-servicing plan, and its edge-by-edge utilization guides the graph-partition swaps in the outer loop. A coarsening step merges each GPU group into a supernode, and a secondary partition assigns supernodes to prefill or decoding types so that inter-type edge weight is maximized, favoring fast KV-cache links. The initial partition is formed by spectral graph partitioning followed by a pairwise node-swap refinement that balances memory across groups.

What would settle it

Run the paper's five heterogeneous settings with per-GPU measured bandwidth and the paper's own cost model, but replace the graph-partition and max-flow search with a simple bandwidth-first round-robin placement under the same budget; if the simple placement matches or beats the proposed scheduler on real throughput in any workload class, the central claim about the scheduler's contribution is refuted. Equivalently, a direct calibration experiment comparing the cost model's predicted per-replica capacity with measured steady-state throughput would settle whether the estimates are trustworthy.

Watch

Extended reading notes

Core claim

The central discovery is a scheduling algorithm that co-optimizes four decisions other systems make separately or not at all: which GPUs form a model replica, whether the replica prefills or decodes, what parallel strategy each replica uses, and how KV caches travel from prefill to decoding replicas. The algorithm builds a graph whose nodes are GPUs weighted by memory and whose edges are communication links weighted by measured bandwidth, partitions it into replica groups, then converts the groups into a directed flow network in which each replica's capacity is an edge estimated from an inference cost model. Running preflow-push max-flow assigns request traffic to replicas and KV-cache communication to links, and the flow values expose bottleneck and underutilized edges that drive an iterative edge-swap refinement. The paper claims this two-phase search finds placements that consistently outperform both a heterogeneous colocated-serving baseline and a homogeneous disaggregated-serving baseline at equal price, and that it scales polynomially with cluster size.

Load-bearing premise

The whole scheduling pipeline inherits its throughput estimates from a cost model originally built for colocated heterogeneous serving, and the paper asserts in Section 5.3 that estimated throughput closely tracks actual throughput without giving a quantitative comparison; if those estimates are off on real heterogeneous hardware, the optimality of the chosen placements, and therefore the claimed gains, does not follow.

Editorial extensions

If this is right

  • On a mixed cluster costing about the same as an 8-GPU homogeneous high-end instance, the scheduler should deliver materially higher token throughput for heavy-prefill and mixed workloads, not only for latency-tolerant batch jobs.
  • A 30 percent cheaper heterogeneous cluster should sustain roughly the same service quality, making mixed-GPU fleets a direct cost lever for LLM serving.
  • The chosen placements consistently put tensor parallelism on prefill replicas and pipeline or hybrid parallelism on decoding replicas, so future heterogeneous serving systems should treat the two phases as independently tunable.
  • KV-cache transfers should be scheduled over the highest-bandwidth links within a server to avoid making inter-phase communication the bottleneck.
  • The scheduling algorithm's convergence time grows polynomially with cluster size, so the approach is claimed to remain practical for clusters of hundreds of GPUs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not pursue is online recalibration: feeding measured per-replica throughput back into the cost model would let the same graph-partition and max-flow loop track workload drift and hardware aging.
  • The same flow formulation could schedule other disaggregated, transfer-heavy components, such as prompt caches or speculative-decoding draft models, by replacing the KV-cache edge cost with the relevant transfer volume.
  • If the capacity estimates were extended to quantized or partially offloaded weights, the scheduler would likely find even cheaper heterogeneous placements, since the cost model currently assumes fixed-precision parameters.
  • A testable prediction of the paper's logic is that the throughput gap over homogeneous serving should grow with the heterogeneity of the GPU pool, because the scheduler has more slack to exploit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. HexGen-2 extends the HexGen heterogeneous-LLM-serving framework to the disaggregated prefill/decode paradigm. The paper formulates the placement problem — which GPUs serve prefill versus decoding replicas, which parallel strategies (TP/PP) each replica uses, and how KV caches are routed between phases — as a constraint optimization problem, solved by a two-phase iterative algorithm that combines spectral/Kernighan-Lin graph partitioning (to form model-serving groups) with a preflow-push max-flow computation over a directed graph whose node and edge capacities are derived from the HexGen cost model, followed by max-flow-guided edge swaps that refine the partition. The system, built on libP2P and NCCL with continuous batching, FlashAttention, and PagedAttention, is evaluated on OPT-30B and Llama-2-70B in one homogeneous 8xH100 setting and five heterogeneous RunPod settings against DistServe, HexGen, and vLLM. The headline claims are up to 2.0x and on average 1.3x throughput gains at a comparable budget, an average 1.5x latency reduction, and comparable service at 70% of the budget; a scheduling ablation run 15 times shows that the max-flow-guided swap converges faster than a genetic-algorithm variant and a truncated variant without edge swaps.

Significance. Disaggregated inference over heterogeneous GPU pools is a timely and economically motivated problem, and the paper's formulation — graph partitioning coupled with a max-flow model whose capacities come from an analytic cost model — is a clean and reusable design. The strengths are concrete: the system is fully implemented and measured on rented cloud GPUs across two models, six hardware settings, four workload types, and three strong baselines; the ablation of the max-flow-guided edge swap is executed with 15 repetitions and reported as convergence curves; Appendix B's placement tables make the deployed configurations auditable; and the headline throughput numbers come from real runs rather than from fitting the scheduler's own cost model, so the comparison with baselines is not circular. The main risk is fidelity of the inherited HexGen cost model, which sets every capacity in the optimization but is validated only verbally in Section 5.3; a quantitative calibration study is needed to make the claimed gains load-bearing, and the abstract's latency claim is not backed by directly measured latency numbers.

major comments (4)
  1. [§5.3, Appendix A] The load-bearing validation of the cost model is missing. Every node capacity and KV-transfer edge capacity in the max-flow graph (§3.3) is an estimate from the HexGen cost model in Table 1 and Appendix A, yet §5.3 supports the model only by asserting that "in all cases, the estimated serving throughput closely aligns with the actual throughput," with no numbers, plot, per-component breakdown, or error bars. This matters for two reasons: (i) if the model is optimistic about decoding capacity or KV edge capacity, the max-flow solver can select placements whose estimated flow is high while the measured throughput is lower, which would inflate the reported 1.3x/2.0x gains; and (ii) the online loads in §5.1 are set to 75% of each cluster's estimated peak throughput, so a biased estimate changes the absolute SLO-attainment levels in Figure 8 and the fairness of the comparison. Please add a quantitative calibration study (estimated versus measured throughput for each setting, model, and workload, ideally per replica type), and state explicitly whether the online arrival rates were derived from the estimated or the measured peak.
  2. [Table 1 (Appendix A)] The printed memory-limit formula is not auditable as written. The row reads ((12H^2 B_type / |d_i,j|) + (2 bt (s_in+s_out) H B_type / |d_i,j|)) x l_i,j + 4 bt (s_in+s_out) H B_type, meaning the per-layer KV term inside the bracket uses a factor of 2 bt and is divided by the TP degree, while the trailing term uses 4 bt and is neither divided by |d_i,j| nor multiplied by l_i,j. Because the scheduler uses this memory constraint to set group sizes and avoid OOM, the feasibility of the placements in Table 2 cannot be checked from the printed equations. Please provide the corrected formula (e.g., the standard 4 bt (s_in+s_out) H B_type l_i,j / |d_i,j| form for a KV cache split across a tensor-parallel group) and confirm that it matches the model used in the implementation and in the original HexGen paper.
  3. [Abstract, §5.2, Figure 8] The abstract's claim that HexGen-2 reduces "the average inference latency by 1.5 times" is not supported by the reported evidence. The only latency-related result, Figure 8, shows SLO attainment as a function of SLO scale; no average, median, or percentile latency numbers are reported anywhere in the evaluation. SLO attainment and mean latency are different quantities, and the phrase "on average a 1.5x lower latency deadlines" in §5.2 does not resolve the mismatch. Either add measured end-to-end latency statistics (mean, p50, p99) for the online experiments and derive the 1.5x factor from them, or revise the abstract and §5.2 to claim a 1.5x improvement in SLO attainment rather than in average latency.
  4. [§5.2, Figures 6-9] The end-to-end results appear to be single runs with no variance information. While the scheduling-algorithm comparison in §5.3 was repeated 15 times ("all run 15 times," Figure 10), Figures 6-9 and Tables 3-4 report only point values, without the number of repetitions, error bars, or the observed run-to-run spread. On rented cloud hardware (RunPod), measurement noise is typically non-trivial, and several of the reported gaps are small (e.g., the 1.0x and 1.1x ratios in Figure 9). Please report means with error bars (or at least min-max ranges) over repeated runs for the headline throughput and SLO numbers, and state the repetition count used for each figure.
minor comments (6)
  1. [Footnote 1] The footnote states that Tesla K80 GPUs were released in 2006; the K80 was released in 2014, even though the broader point that old GPU generations remain in service is valid.
  2. [§5.1, Figure 4] The heterogeneous settings 1-4 have budgets of $28.8, $26.9, $27.1, and $26.3 per hour, which are not literally the "same price budget" as the $29.52/h homogeneous setting; please say "comparable budget" or match the budgets exactly.
  3. [Table 3 vs. Figures 6-7] For overlapping configurations (e.g., Heterogeneous Setting 1, HPLD and HPHD), the throughput values in Table 3 do not appear to match the corresponding bars in Figures 6-7; please clarify whether the table and figures report the same runs and reconcile any differences.
  4. [Appendix H] The claim that the scheduling algorithm "scales polynomially" is asserted without evidence; Table 5 reports convergence times at five cluster sizes, but no fitted scaling law or complexity analysis is given, so please either add the fit or soften the claim to "grows moderately with cluster size."
  5. [§3.1, §3.3, Appendix E.2] Several presentation issues should be fixed: "pradiagm" in §3.3, "detialed" in §3.2, "aims to fine the parallel strategy" in Appendix E.2, and the awkward phrase "1.5x lower latency deadlines" in §5.2.
  6. [§3.1, §3.3] The statement that finding the optimal placement is NP-hard is asserted without proof or citation, and the claim that the algorithm "iterates through all possible model parallelism combinations" would benefit from a statement of the search-space size and any pruning criteria; please add a short justification or reference for both.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported gains are measured against external baselines, and the HexGen cost-model self-citation is a modeling input, not a fitted prediction.

full rationale

The paper's central claims—2.0x/1.3x throughput improvement and 1.5x latency reduction—are supported by end-to-end experiments comparing HexGen-2 against HexGen, DistServe, and vLLM on rented GPUs. These are measured results, not quantities derived from the scheduler's own cost model. The scheduling algorithm does adopt the HexGen inference cost model (Section 3.3, Appendix A), which is a self-citation from overlapping authors, but that model is a parameterized input to the optimization, not a quantity fitted to the paper's own throughput results. The phrase 'the estimated serving throughput closely aligns with the actual throughput' in Section 5.3 is an empirical validation claim, albeit qualitative, and the max-flow objective is the estimated throughput while the reported end-to-end numbers are actual measured throughput. No equation in the paper defines a predicted result in terms of the same quantity it claims to predict. The unquantified cost-model validation and the apparent under-specification of the Table 1 memory-limit row are correctness/robustness concerns, not circularity, because the final performance claims do not reduce to the model's own estimates.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim does not introduce new physical entities. The main assumptions are the accuracy of the cost model and the representativeness of the experimental setup.

free parameters (2)
  • bt (batch size for memory estimation) = 32
    Used in Appendix A to estimate the total memory requirement for a single model replica (model parameter size plus 32 times KV cache size per request). This affects which groups can fit a replica and thus the partition.
  • T (time period for capacity estimation) = 10 minutes
    Used in Section 3.3 to compute the maximum number of requests a compute node can process by dividing the time period by the latency. This scales all node and edge capacities in the max-flow graph.
assumptions (5)
  • domain assumption The HexGen cost model (Table 1) accurately predicts inference latency and capacity on heterogeneous GPUs and networks.
    The schedule and max-flow capacities are computed from this cost model; if inaccurate, the scheduler's decisions may not optimize actual throughput. The paper asserts in Section 5.3 that estimated and actual throughput 'closely align' but provides no quantitative evidence.
  • domain assumption The measured NCCL bandwidths between each GPU pair (Figure 4) are stable during the evaluation.
    Cloud network bandwidth can vary; the scheduler depends on these edge weights to guide partition and KV cache placement.
  • domain assumption The Azure Conversation dataset samples and the heavy/light workload classification are representative of real LLM serving workloads.
    The claimed improvements are averaged over these workloads; other workloads might show different results.
  • domain assumption The RunPod pricing used for budget comparisons is constant and fair across the heterogeneous and homogeneous clusters.
    The 'same price budget' comparison depends on these prices; if they change, the cost benefit could differ.
  • domain assumption The max-flow formulation correctly represents the system's ability to process requests and transfer KV caches.
    The algorithm optimizes for max flow, but actual scheduling may have stragglers or interference not captured by the model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment." pith.science (2026). https://pith.science/paper/MRK7GAQ7

@misc{pith2026250207903,
  author       = {Pith},
  title        = {Pith review of: HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MRK7GAQ7}},
  note         = {Machine review of arXiv:2502.07903}
}
read the original abstract

Disaggregating the prefill and decoding phases represents an effective new paradigm for generative inference of large language models (LLM), which eliminates prefill-decoding interference and optimizes resource allocation. However, it is still an open problem about how to deploy the disaggregated inference paradigm across a group of heterogeneous GPUs, which can be an economical alternative to deployment over homogeneous high-performance GPUs. Towards this end, we introduce HexGen-2, a distributed system for efficient and economical LLM serving on heterogeneous GPUs following the disaggregated paradigm. Built on top of HexGen, the core component of HexGen-2 is a scheduling algorithm that formalizes the allocation of disaggregated LLM inference computations and communications over heterogeneous GPUs and network connections as a constraint optimization problem. We leverage the graph partitioning and max-flow algorithms to co-optimize resource allocation, parallel strategies for distinct inference phases, and the efficiency of inter-phase key-value (KV) cache communications. We conduct extensive experiments to evaluate HexGen-2, i.e., on OPT (30B) and Llama-2 (70B) models in various real-world settings, the results reveal that HexGen-2 delivers up to a 2.0 times and on average a 1.3 times improvement in serving throughput, reduces the average inference latency by 1.5 times compared with state-of-the-art systems given the same price budget, and achieves comparable inference performance with a 30% lower price budget.

Figures

Figures reproduced from arXiv: 2502.07903 by the authors.

Figure 1
Figure 1. Effects of batching on different phases (LLAMA-2 (7B) inference with an input length of 512 on a single A100 GPU). Parallel Runtime KV Caches First Tokens Prefill Model Replica Decode Model Replica KV Caches First Tokens Parallel Runtime Requests Transfer [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Illustration of each scheduling step. Step (ii) - Coarsen & secondary partition: We then determine the group type, where the graph is coarsened and partitioned again to determine the model replica type for each group. Note that coarsen is a common operation that merges nodes and edges to simplify graph partition (Hendrick￾son et al., 1995). Here, the coarsening operation merges graph nodes (GPUs) within the same gro… view at source ↗
Figure 4
Figure 4. Communication bandwidth (Gbps) matrix for different settings. Homogeneous setting [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (7 more)
Figure 6
Figure 6. Figure 6: Throughput results to evaluate HEXGEN-2 on LLAMA-2 (70B). Each row corresponds to a particular heterogeneous setting. The first four columns demonstrates the offline inference results on different LLM workloads. The last column represents the online inference results. …
Figure 5
Figure 5. Figure 5: Request traces for online testing. Online and offline testing. We test two different ar￾rival rates: In the online setting, we scale the average arrival rate to 75% of the cluster’s peak throughput to prevent request bursts that could cause system out￾ages due to out-o…
Figure 7
Figure 7. Figure 7: Throughput results to evaluate HEXGEN-2 on OPT (30B). 1 2 3 4 5 SLO Scale 50 62 75 87 100 SLO attainments (%) Hetero-1 1 2 3 4 5 SLO Scale 40 55 70 85 100 Hetero-2 1 2 3 4 5 SLO Scale 50 62 75 87 100 Hetero-3 1 2 3 4 5 SLO Scale 40 55 70 85 100 Hetero-4 HexGen-2 HexGen…
Figure 8
Figure 8. Figure 8: Latency results in online experiments. and DISTSERVE in the homogeneous setting. Given the same price budget, HEXGEN-2 outper￾forms its counterparts in almost all cases. In fact, compared with HEXGEN, HEXGEN-2 achieves up to a 1.5× and, on average, a 1.4× increase in s…
Figure 9
Figure 9. Figure 9: Throughput results with 70% price budget. [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Convergence comparison of our proposed search strategy, our strategy without edge [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: Throughput comparison in heterogeneous setting 1 among H [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 2 canonical work pages

  1. [1]

    Taming throughput-latency tradeoff in llm inference with sarathi-serve

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gula- vani, Alexey Tumanov, and Ramachandran Ramjee. Taming throughput-latency tradeoff in llm inference with sarathi-serve. arXiv preprint arXiv:2403.02310,

    arXiv

  2. [8]

    Mooncake: Kimi’s kvcache-centric architecture for llm serving

    Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xin- ran Xu. Mooncake: Kimi’s kvcache-centric architecture for llm serving. arXiv preprint arXiv:2407.00079,

    arXiv

  3. [9]

    Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530,

    arXiv

  4. [10]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al

    URL https://docs.runpod.io/. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288,

    arXiv

  5. [12]

    Jintao Zhang, Jia Wei, Pengle Zhang, Jun Zhu, and Jianfei Chen

    URL https://arxiv.org/abs/2411.10958. Jintao Zhang, Jia Wei, Pengle Zhang, Jun Zhu, and Jianfei Chen. Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration. In International Conference on Learning Rep- resentations (ICLR), 2025a. Jintao Zhang, Chendong Xiang, Haofeng Huang, Haocheng Xi, Jia Wei, Jun Zhu, and Jianfei Chen. Sparge...

    arXiv

  6. [13]

    Llm-pq: Serving llm on heterogeneous clusters with phase-aware partition and adaptive quantization

    Juntao Zhao, Borui Wan, Yanghua Peng, Haibin Lin, and Chuan Wu. Llm-pq: Serving llm on heterogeneous clusters with phase-aware partition and adaptive quantization. arXiv preprint arXiv:2403.01136,

    arXiv

  7. [14]

    {PetS}: A unified framework for {Parameter-Efficient} transformers serving

    Zhe Zhou, Xuechao Wei, Jiejing Zhang, and Guangyu Sun. {PetS}: A unified framework for {Parameter-Efficient} transformers serving. In 2022 USENIX Annual Technical Conference (USENIX ATC 22), pp. 489–504,

    work page 2022

  8. [1994]

    Fast distributed inference serving for large language models

    Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast distributed inference serving for large language models. arXiv preprint arXiv:2305.05920,

    arXiv

Show all 14 references
  1. [1995]

    Inference without interference: Disaggregate llm inference for mixed downstream workloads

    Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, et al. Inference without interference: Disaggregate llm inference for mixed downstream workloads. arXiv preprint arXiv:2401.11181,

    arXiv

  2. [2019]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024a. Youhe Jiang, Ran Yan, Xiaozhe ...

    arXiv

  3. [2022]

    M \’elange: Cost efficient large language model serving by exploiting gpu heterogeneity

    Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, and Ion Stoica. M \’elange: Cost efficient large language model serving by exploiting gpu heterogeneity. arXiv preprint arXiv:2404.14527,

    arXiv

  4. [2023]

    Helix: Distributed serving of large language models via max-flow on heterogeneous gpus

    Yixuan Mei, Yonghao Zhuang, Xupeng Miao, Juncheng Yang, Zhihao Jia, and Rashmi Vinayak. Helix: Distributed serving of large language models via max-flow on heterogeneous gpus. arXiv preprint arXiv:2406.01566,

    arXiv

  5. [2024]

    12 Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, ´I˜nigo Goiri, Saeed Maleki, and Ri- cardo Bianchini

    URL https://platform.openai.com/docs/models/ gpt-4o. 12 Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, ´I˜nigo Goiri, Saeed Maleki, and Ri- cardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual Internatio...

    2024

  6. [2025]

    P/d-serve: Serving disaggregated large language model at scale

    Yibo Jin, Tao Wang, Huimin Lin, Mingyang Song, Peiyang Li, Yipeng Ma, Yicheng Shan, Zhengfan Yuan, Cailong Li, Yajing Sun, et al. P/d-serve: Serving disaggregated large language model at scale. arXiv preprint arXiv:2408.08147,

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.