{"id":"936a7695-f6b0-4c4f-b01c-fcfa2192df77","arxiv_id":"2502.07903","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A scheduling algorithm for disaggregated LLM inference on heterogeneous GPUs that improves throughput by up to 2x and cuts cost by 30% relative to state-of-the-art serving systems.","lead":"HexGen-2 is a system that schedules large language model inference across a mix of cheaper GPUs by running the prefill and decoding phases on separate groups of hardware, reporting throughput up to 2x higher than prior systems at the same budget. The work suggests that using heterogeneous GPU pools could make LLM serving more economical without sacrificing performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed gains rest on an unquantified cost-model validation: §5.3 asserts estimated and actual throughput 'closely align' without numbers, and Table 1's memory formula as printed omits the layer count in KV-cache memory, so the max-flow capacities could be systematically optimistic.","rationale":"The paper is a serious systems contribution with a plausible design, extensive evaluation across two models, five heterogeneous settings, and multiple workload types; those are real strengths. The reader's conditional verdict is appropriate, and the weakest assumption identified by the reader is also the one I consider most load-bearing: the entire scheduling pipeline depends on the HexGen cost model, yet the paper provides no quantitative validation of it. Section 5.3's unquantified claim that estimated throughput 'closely aligns' with actual throughput is insufficient, especially because the same section is the only place where scheduling effectiveness is tied to actual measured throughput. My additional observation about Table 1's memory-limit formula is a concrete, checkable instance of why the cost model cannot be taken on faith: as printed, the KV-cache term is not multiplied by the number of layers and is not divided by the tensor-parallel size, so for a pipeline stage it could understate memory by a large factor. If the actual implementation uses a different formula, that discrepancy should be stated; if it uses the printed formula, the scheduler's memory-based grouping decisions are suspect. The latency claim in the abstract is also not directly supported by the reported SLO-attainment curves, but that is secondary. A focused reproducibility check comparing predicted and measured capacities would settle whether the central performance claim is robust or partly an artifact of optimistic cost estimates.","tokens_in":18988,"tokens_out":6446,"duration_ms":61264,"concrete_test":"Reproduce the Hetero-1 / Llama-2-70B / HPLD run from §5.2. Before deployment, log the scheduler's predicted prefill and decode node capacities, predicted KV-transfer edge capacities, and max-flow value. After deployment, measure per-replica achieved throughput, actual KV-transfer volume, and peak per-GPU memory over the same time window T. Compare predicted versus measured values per component, and also evaluate Table 1's memory formula for the chosen placement against nvidia-smi peak usage. If predicted capacities deviate by more than 15% on any component, or if peak memory exceeds the formula's estimate on the selected GPUs, the claimed gains are not reliably attributable to the scheduler.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that HexGen-2's max-flow scheduler produces the reported throughput and cost improvements, but every node and edge capacity in that max-flow graph comes from the HexGen cost model in Table 1 and Appendix A, not from direct measurement. Section 5.3 says only that 'the estimated serving throughput closely aligns with the actual throughput,' with no quantitative comparison, no per-component breakdown, and no error bars. If the cost model overestimates decoding capacity or KV-transfer edge capacity, the max-flow solution can choose placements whose estimated throughput is high while the actual throughput is lower; the reported 1.3x average gain over DistServe and HexGen could then be an artifact of optimistic capacity estimates rather than of the disaggregated placement itself. The printed model also raises a concrete red flag: the Memory limit row in Table 1 writes KV-cache memory as 4bt(s_in+s_out)HBtype with no multiplication by l_i,j and no division by |d_i,j|, which would mis-estimate memory by roughly the layer count for pipeline stages. Because the scheduler uses this memory estimate to set group sizes and avoid OOM, this is not a cosmetic issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"HexGen-2 extends the HexGen heterogeneous-LLM-serving framework to the disaggregated prefill/decode paradigm. The paper formulates the placement problem — which GPUs serve prefill versus decoding replicas, which parallel strategies (TP/PP) each replica uses, and how KV caches are routed between phases — as a constraint optimization problem, solved by a two-phase iterative algorithm that combines spectral/Kernighan-Lin graph partitioning (to form model-serving groups) with a preflow-push max-flow computation over a directed graph whose node and edge capacities are derived from the HexGen cost model, followed by max-flow-guided edge swaps that refine the partition. The system, built on libP2P and NCCL with continuous batching, FlashAttention, and PagedAttention, is evaluated on OPT-30B and Llama-2-70B in one homogeneous 8xH100 setting and five heterogeneous RunPod settings against DistServe, HexGen, and vLLM. The headline claims are up to 2.0x and on average 1.3x throughput gains at a comparable budget, an average 1.5x latency reduction, and comparable service at 70% of the budget; a scheduling ablation run 15 times shows that the max-flow-guided swap converges faster than a genetic-algorithm variant and a truncated variant without edge swaps.","tokens_in":19234,"tokens_out":12855,"duration_ms":107383,"significance":"Disaggregated inference over heterogeneous GPU pools is a timely and economically motivated problem, and the paper's formulation — graph partitioning coupled with a max-flow model whose capacities come from an analytic cost model — is a clean and reusable design. The strengths are concrete: the system is fully implemented and measured on rented cloud GPUs across two models, six hardware settings, four workload types, and three strong baselines; the ablation of the max-flow-guided edge swap is executed with 15 repetitions and reported as convergence curves; Appendix B's placement tables make the deployed configurations auditable; and the headline throughput numbers come from real runs rather than from fitting the scheduler's own cost model, so the comparison with baselines is not circular. The main risk is fidelity of the inherited HexGen cost model, which sets every capacity in the optimization but is validated only verbally in Section 5.3; a quantitative calibration study is needed to make the claimed gains load-bearing, and the abstract's latency claim is not backed by directly measured latency numbers.","major_comments":[{"comment":"The load-bearing validation of the cost model is missing. Every node capacity and KV-transfer edge capacity in the max-flow graph (§3.3) is an estimate from the HexGen cost model in Table 1 and Appendix A, yet §5.3 supports the model only by asserting that \"in all cases, the estimated serving throughput closely aligns with the actual throughput,\" with no numbers, plot, per-component breakdown, or error bars. This matters for two reasons: (i) if the model is optimistic about decoding capacity or KV edge capacity, the max-flow solver can select placements whose estimated flow is high while the measured throughput is lower, which would inflate the reported 1.3x/2.0x gains; and (ii) the online loads in §5.1 are set to 75% of each cluster's estimated peak throughput, so a biased estimate changes the absolute SLO-attainment levels in Figure 8 and the fairness of the comparison. Please add a quantitative calibration study (estimated versus measured throughput for each setting, model, and workload, ideally per replica type), and state explicitly whether the online arrival rates were derived from the estimated or the measured peak.","section":"§5.3, Appendix A"},{"comment":"The printed memory-limit formula is not auditable as written. The row reads ((12H^2 B_type / |d_i,j|) + (2 bt (s_in+s_out) H B_type / |d_i,j|)) x l_i,j + 4 bt (s_in+s_out) H B_type, meaning the per-layer KV term inside the bracket uses a factor of 2 bt and is divided by the TP degree, while the trailing term uses 4 bt and is neither divided by |d_i,j| nor multiplied by l_i,j. Because the scheduler uses this memory constraint to set group sizes and avoid OOM, the feasibility of the placements in Table 2 cannot be checked from the printed equations. Please provide the corrected formula (e.g., the standard 4 bt (s_in+s_out) H B_type l_i,j / |d_i,j| form for a KV cache split across a tensor-parallel group) and confirm that it matches the model used in the implementation and in the original HexGen paper.","section":"Table 1 (Appendix A)"},{"comment":"The abstract's claim that HexGen-2 reduces \"the average inference latency by 1.5 times\" is not supported by the reported evidence. The only latency-related result, Figure 8, shows SLO attainment as a function of SLO scale; no average, median, or percentile latency numbers are reported anywhere in the evaluation. SLO attainment and mean latency are different quantities, and the phrase \"on average a 1.5x lower latency deadlines\" in §5.2 does not resolve the mismatch. Either add measured end-to-end latency statistics (mean, p50, p99) for the online experiments and derive the 1.5x factor from them, or revise the abstract and §5.2 to claim a 1.5x improvement in SLO attainment rather than in average latency.","section":"Abstract, §5.2, Figure 8"},{"comment":"The end-to-end results appear to be single runs with no variance information. While the scheduling-algorithm comparison in §5.3 was repeated 15 times (\"all run 15 times,\" Figure 10), Figures 6-9 and Tables 3-4 report only point values, without the number of repetitions, error bars, or the observed run-to-run spread. On rented cloud hardware (RunPod), measurement noise is typically non-trivial, and several of the reported gaps are small (e.g., the 1.0x and 1.1x ratios in Figure 9). Please report means with error bars (or at least min-max ranges) over repeated runs for the headline throughput and SLO numbers, and state the repetition count used for each figure.","section":"§5.2, Figures 6-9"}],"minor_comments":[{"comment":"The footnote states that Tesla K80 GPUs were released in 2006; the K80 was released in 2014, even though the broader point that old GPU generations remain in service is valid.","section":"Footnote 1"},{"comment":"The heterogeneous settings 1-4 have budgets of $28.8, $26.9, $27.1, and $26.3 per hour, which are not literally the \"same price budget\" as the $29.52/h homogeneous setting; please say \"comparable budget\" or match the budgets exactly.","section":"§5.1, Figure 4"},{"comment":"For overlapping configurations (e.g., Heterogeneous Setting 1, HPLD and HPHD), the throughput values in Table 3 do not appear to match the corresponding bars in Figures 6-7; please clarify whether the table and figures report the same runs and reconcile any differences.","section":"Table 3 vs. Figures 6-7"},{"comment":"The claim that the scheduling algorithm \"scales polynomially\" is asserted without evidence; Table 5 reports convergence times at five cluster sizes, but no fitted scaling law or complexity analysis is given, so please either add the fit or soften the claim to \"grows moderately with cluster size.\"","section":"Appendix H"},{"comment":"Several presentation issues should be fixed: \"pradiagm\" in §3.3, \"detialed\" in §3.2, \"aims to fine the parallel strategy\" in Appendix E.2, and the awkward phrase \"1.5x lower latency deadlines\" in §5.2.","section":"§3.1, §3.3, Appendix E.2"},{"comment":"The statement that finding the optimal placement is NP-hard is asserted without proof or citation, and the claim that the algorithm \"iterates through all possible model parallelism combinations\" would benefit from a statement of the search-space size and any pruning criteria; please add a short justification or reference for both.","section":"§3.1, §3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper fits a systems venue, and the evaluation breadth (two models, six clusters, four workloads, three baselines, plus a vLLM comparison) is strong for a conference paper; for a journal version, the cost-model calibration gap and the single-run nature of the headline numbers are the main risks to the quantitative claims. The authors' novelty statement, \"first to adapt the disaggregated inference architecture for heterogeneous environments,\" should be sharpened: Helix (cited) already uses max-flow over heterogeneous GPUs, and concurrent work on heterogeneous disaggregation exists, so an explicit comparison of formulations would help. I would also flag that the abstract's most eye-catching number (1.5x average latency reduction) is the claim least supported by the evidence; it should be corrected early in the revision process."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read on HexGen-2. The paper is a legitimate systems contribution: it's the first work I know of that takes the disaggregated prefill/decode paradigm and explicitly optimizes it for heterogeneous GPU pools. The scheduling algorithm — graph partitioning to form model replicas, max-flow to assign parallel strategies and KV-cache routes, and iterative max-flow-guided edge swaps to refine — is a sensible integration of known pieces. The evaluation is also better than most arXiv systems papers: real RunPod rentals, four GPU types, five heterogeneous settings, OPT-30B and Llama-2-70B, and comparisons against DistServe, HexGen, and vLLM. The result that a 70% budget can match the homogeneous baseline is the most interesting thing in the paper.\n\nNow the soft spots, in decreasing order of severity. First, the headline latency claim doesn't match the reported metric: the abstract promises a 1.5x reduction in average latency, but Figure 8 reports SLO attainment. That's a real discrepancy and needs fixing. Second, the main throughput numbers in Figures 6–8 look like single runs — no error bars, no variance. With a 1.3x average improvement, reviewers will want to know whether that's stable across runs. Third, and relatedly, the scheduler's max-flow capacities come entirely from the inherited HexGen cost model, and Section 5.3's \"closely aligns\" is not supported by any numbers. The cost model is the load-bearing assumption; if it's off, the scheduler could be picking suboptimal placements without us knowing. There's also a possible typo in Table 1's memory formula, where one KV-cache term is printed without multiplication by the layer count or division by the group size. If that's not a formatting artifact, it would bias the memory estimates and the max-flow capacities. Finally, no code or data is released, which makes independent verification harder.\n\nNone of this makes me doubt the core system exists and works. The measured gains are from real runs, not fitted parameters, and the scheduler convergence analysis (vs. genetic algorithm, with 15 runs) is a good sign. But the paper overclaims in the abstract and under-reports the uncertainty in the experiments.\n\nVerdict: send it to peer review. A systems venue should give it a serious referee cycle; it needs a major revision on the evaluation rigor, but the contribution is timely and worth engaging with.\n\nBest,","headline":"A plausible new scheduler for disaggregated LLM serving on heterogeneous GPUs, but the headline numbers rest on unreported variance and an unquantified cost-model validation.","tokens_in":19721,"tokens_out":5698,"would_cite":true,"duration_ms":49977,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that splitting prefill and decoding across heterogeneous GPUs, with a graph-partition and max-flow scheduler, delivers up to 2.0x serving throughput and 1.5x lower latency at the same price, and comparable performance at…","keywords":["LLM inference","disaggregated inference","heterogeneous GPUs","graph partitioning","max-flow scheduling","KV cache transfer","serving throughput","cost efficiency"],"falsifier":"Run the paper's five heterogeneous settings with per-GPU measured bandwidth and the paper's own cost model, but replace the graph-partition and max-flow search with a simple bandwidth-first round-robin placement under the same budget; if the simple placement matches or beats the proposed scheduler on real throughput in any workload class, the central claim about the scheduler's contribution is refuted. Equivalently, a direct calibration experiment comparing the cost model's predicted per-replica capacity with measured steady-state throughput would settle whether the estimates are trustworthy.","tokens_in":18792,"feed_emoji":"⚙️","tokens_out":6856,"duration_ms":58033,"temperature":0.7,"pith_summary":"The paper sets out to show that the disaggregated inference paradigm — running the compute-heavy prefill phase and the memory-bandwidth-heavy decoding phase on separate GPU groups — can be made to work well on a pool of heterogeneous GPUs and network links, not just on a uniform high-end cluster. Because heterogeneous GPU fleets are cheaper to assemble, an effective scheduler for them could lower the cost of LLM serving. The proposed system treats placement as a constrained optimization: partition GPUs into serving groups, decide which groups prefill and which decode, pick tensor and pipeline parallel strategies per phase, and route key-value cache transfers. On OPT-30B and Llama-2-70B workloads the authors report up to 2.0x and on average 1.3x higher throughput than state-of-the-art baselines at equal budget, 1.5x lower latency, and comparable performance at a 30 percent smaller budget.","feed_headline":"Heterogeneous GPUs can serve LLMs twice as fast for the same price","feed_subtitle":"Splitting prefill and decoding across mixed GPUs yields 1.3x average throughput or 30 percent lower cost.","key_machinery":"The load-bearing object is the directed max-flow graph. Prefill and decoding replicas are compute nodes whose internal edge capacity is the number of requests the replica can process in a fixed time window, estimated from the paper's inference cost model; valid edges run from the coordinator to a prefill node, from a prefill node to a decoding node (capacity derived from KV-cache transfer cost over the underlying GPU-to-GPU links), and from a decoding node back to the coordinator. The max flow through this network is the request-servicing plan, and its edge-by-edge utilization guides the graph-partition swaps in the outer loop. A coarsening step merges each GPU group into a supernode, and a secondary partition assigns supernodes to prefill or decoding types so that inter-type edge weight is maximized, favoring fast KV-cache links. The initial partition is formed by spectral graph partitioning followed by a pairwise node-swap refinement that balances memory across groups.","core_discovery":"The central discovery is a scheduling algorithm that co-optimizes four decisions other systems make separately or not at all: which GPUs form a model replica, whether the replica prefills or decodes, what parallel strategy each replica uses, and how KV caches travel from prefill to decoding replicas. The algorithm builds a graph whose nodes are GPUs weighted by memory and whose edges are communication links weighted by measured bandwidth, partitions it into replica groups, then converts the groups into a directed flow network in which each replica's capacity is an edge estimated from an inference cost model. Running preflow-push max-flow assigns request traffic to replicas and KV-cache communication to links, and the flow values expose bottleneck and underutilized edges that drive an iterative edge-swap refinement. The paper claims this two-phase search finds placements that consistently outperform both a heterogeneous colocated-serving baseline and a homogeneous disaggregated-serving baseline at equal price, and that it scales polynomially with cluster size.","pith_inferences":["A natural extension the paper does not pursue is online recalibration: feeding measured per-replica throughput back into the cost model would let the same graph-partition and max-flow loop track workload drift and hardware aging.","The same flow formulation could schedule other disaggregated, transfer-heavy components, such as prompt caches or speculative-decoding draft models, by replacing the KV-cache edge cost with the relevant transfer volume.","If the capacity estimates were extended to quantized or partially offloaded weights, the scheduler would likely find even cheaper heterogeneous placements, since the cost model currently assumes fixed-precision parameters.","A testable prediction of the paper's logic is that the throughput gap over homogeneous serving should grow with the heterogeneity of the GPU pool, because the scheduler has more slack to exploit."],"forward_implications":["On a mixed cluster costing about the same as an 8-GPU homogeneous high-end instance, the scheduler should deliver materially higher token throughput for heavy-prefill and mixed workloads, not only for latency-tolerant batch jobs.","A 30 percent cheaper heterogeneous cluster should sustain roughly the same service quality, making mixed-GPU fleets a direct cost lever for LLM serving.","The chosen placements consistently put tensor parallelism on prefill replicas and pipeline or hybrid parallelism on decoding replicas, so future heterogeneous serving systems should treat the two phases as independently tunable.","KV-cache transfers should be scheduled over the highest-bandwidth links within a server to avoid making inter-phase communication the bottleneck.","The scheduling algorithm's convergence time grows polynomially with cluster size, so the approach is claimed to remain practical for clusters of hundreds of GPUs."],"supporting_citations":[{"why":"supplies the inference cost model used for node and edge capacity estimation and is the heterogeneous colocated baseline.","marker":"[Jiang et al., 2024b]"},{"why":"is the homogeneous disaggregated baseline and the source of the disaggregated prefill and decode design.","marker":"[Zhong et al., 2024]"},{"why":"motivates phase splitting and provides the real conversation traces used in the experiments.","marker":"[Patel et al., 2024]"},{"why":"provides the spectral partitioning method used for the initial GPU group partition.","marker":"[Alpert & Yao, 1995]"},{"why":"provides the pairwise node-swap refinement used to balance memory and reduce inter-group bandwidth.","marker":"[Kernighan & Lin, 1970]"},{"why":"provides the preflow-push algorithm used to compute max flow and derive edge utilization.","marker":"[Cheriyan & Maheshwari, 1989]"},{"why":"supplies the continuous batching optimization integrated into the serving runtime.","marker":"[Yu et al., 2022]"}],"fun_headline_variants":["HexGen-2: 2x LLM throughput on mixed GPUs","Disaggregated LLM serving: 2x throughput or 30% less cost","Heterogeneous GPUs: LLM inference cost cut by 30%","Max-flow scheduling doubles LLM serving speed on mixed GPUs","Graph partition plus max-flow: cheaper, faster LLM inference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole scheduling pipeline inherits its throughput estimates from a cost model originally built for colocated heterogeneous serving, and the paper asserts in Section 5.3 that estimated throughput closely tracks actual throughput without giving a quantitative comparison; if those estimates are off on real heterogeneous hardware, the optimality of the chosen placements, and therefore the claimed gains, does not follow.","fun_headline_variants_meta":{"raw":{"variants":["HexGen-2: 2x LLM throughput on mixed GPUs","Disaggregated LLM serving: 2x throughput or 30% less cost","Heterogeneous GPUs: LLM inference cost cut by 30%","Max-flow scheduling doubles LLM serving speed on mixed GPUs","Graph partition plus max-flow: cheaper, faster LLM inference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000381,"raw_usage":{"total_tokens":2062,"prompt_tokens":1023,"completion_tokens":1039,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":639,"completion_tokens_details":{"reasoning_tokens":942}},"tokens_in":639,"tokens_out":1039,"duration_ms":8452,"temperature":1.0,"reasoning_tokens":942,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T11:27:02.092060+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's five heterogeneous settings with per-GPU measured bandwidth and the paper's own cost model, but replace the graph-partition and max-flow search with a simple bandwidth-first round-robin placement under the same budget; if the simple placement matches or beats the proposed scheduler on real throughput in any workload class, the central claim about the scheduler's contribution is refuted. Equivalently, a direct calibration experiment comparing the cost model's predicted per-replica capacity with measured steady-state throughput would settle whether the estimates are trustworthy.","supporting_citations":[],"review_version":1}