{"id":"5b735131-d670-4c9d-8e8b-0c729ca7b02d","arxiv_id":"2509.08309","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Hetis dynamically splits LLM attention across heterogeneous GPUs at the granularity of individual heads while keeping dense computation on a selected subset, and reports up to 2.25x throughput gains over Splitwise and Hexgen.","lead":"Hetis is a new system for running large language model inference across a mix of powerful and weak GPUs, splitting the work so that fast GPUs handle dense compute and all GPUs, including slow ones, share the attention computation one head at a time. It reports up to 2.25x higher serving throughput and lower latency than two recent heterogeneous-serving baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Linear attention-time model (Eq. 3) is validated only on narrow single-setting plots; if it breaks in high-cache or low-head regimes, the LP dispatch misallocates heads and the measured gains may not generalize.","rationale":"The reader correctly identifies the linear attention-time model (Eq. 3) as the weakest link in the dispatch chain. The paper's own sensitivity analysis does not defend against nonlinearity, and the empirical validation is narrow. I considered other candidates — e.g., the LP's integrality gap (x_i^j must be integer but Eq. 7 is stated as an LP), baseline fairness, and the misleading abstract latency claim (1.49× is module-level, not end-to-end). While these are real weaknesses, the linear model is the one whose failure would directly undermine the causal mechanism claimed in the paper (head-wise dynamic dispatch). The module-level breakdown (Fig. 13) and ablation (Fig. 14) support the mechanism on the tested configuration, so the central claim is plausible but conditional on the cost model. My recommended verdict remains CONDITIONAL, matching the reader's assessment.","tokens_in":20122,"tokens_out":10292,"duration_ms":118782,"concrete_test":"Run a factorial measurement of decode-phase attention kernel time on the actual A100, 3090, and P100 devices, varying assigned heads h ∈ {1,2,4,8,16,32} and total cache size g (context lengths from 0.5K to 8K) with at least 10 repeats per point. Fit Eq. (3) and test for significant quadratic or interaction terms. Then, on the same testbed, run the Hetis dispatcher for the ShareGPT/HumanEval/LongBench workloads with the true measured attention time (lookup-table oracle) in place of Eq. (3). If the oracle-driven dispatch changes P95 TPOT or throughput by more than 10% relative to the linear-model dispatch, the linearity assumption is load-bearing; if within 5%, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central dispatch policy (Eq. 7, §5.2) minimizes the max per-device attention time by solving an LP whose objective is f_i(x) = a_i·h + b_i·g + c_i (Eq. 3). This linear model is the backbone of both initial dispatch and re-dispatching (§5.3). If the true attention time is nonlinear — e.g., HBM-bandwidth saturation makes time sublinear in cache size at large g, or kernel-launch overhead makes time superlinear at very small h — the LP will systematically misallocate heads, and the claimed 2.25×/1.39× improvements (Figs. 8–12) could shrink or invert on other workloads or cluster sizes. The evidence for linearity is Fig. 7(b)–(c): single-device, single-workload curves with no error bars and no coverage of the extremes (e.g., h < 4 or very large cache sizes). The robustness test in Fig. 16(b) perturbs a_i, b_i, c_i by ±20%, but a structural nonlinearity is not a parameter perturbation; a quadratic or interaction term would not be captured by that analysis. Because the whole system-level claim rests on optimal head allocation, this is the most load-bearing assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Hetis is an LLM serving system for heterogeneous GPU clusters that combines two forms of parallelism: (i) \"primary worker parallelism,\" which selects a subset of GPUs for dense modules such as MLP, and (ii) dynamic head-wise parallelism for attention, in which attention heads and the associated KV caches are distributed across primary and attention workers. The dispatcher solves a linear program (Eq. 7) to minimize the maximum per-device attention time, with re-dispatching for long contexts and head-wise KV cache migration. The system is evaluated on a cluster with A100, 3090, and P100 GPUs for LLaMA-13B, OPT-30B, and LLaMA-70B on ShareGPT, HumanEval, and LongBench, against Splitwise and Hexgen. The paper claims up to 2.25× throughput improvement and 1.49× latency reduction over these baselines.","tokens_in":20458,"tokens_out":5139,"duration_ms":58808,"significance":"The head-granularity dynamic attention offloading idea is a genuine and useful departure from the coarse-grained, static partitioning used by Splitwise and Hexgen, and the module-level ablation in Fig. 13 is a good-faith attempt to attribute the gains to that mechanism. If the linear attention model and LP dispatch are robust, the system would be a solid contribution to heterogeneous LLM serving. The paper also ships a real prototype on vLLM, reports profiling accuracy, and includes sensitivity analyses for the re-dispatch threshold and profiling errors, which are strengths. However, the headline latency claim is not end-to-end, and several load-bearing algorithmic details (integrality of head allocation, completeness of the re-dispatch LP, validity of the linear model outside the profiled regime) need to be tightened before the central claims can be fully accepted.","major_comments":[{"comment":"The headline claim 'reduce latency by 1.49×' is taken from the module-level attention latency in Fig. 13, not from the end-to-end results. The end-to-end P95 TPOT improvement is up to 1.39× and TTFT up to 1.47× (Fig. 12). The abstract and §1 should either report the end-to-end numbers or explicitly state that 1.49× is the attention-module latency reduction, otherwise the central advertised result is overstated.","section":"Abstract and §7.3"},{"comment":"The paper states in §5.2.1 that x_i^j(t)/r must be an integer (head groups are indivisible), but §5.2.2 solves the problem as a continuous LP and does not describe rounding or an integrality argument. If fractional allocations are allowed, constraints (5) and (7c) are not truly enforced; if fractional solutions are rounded, the claimed optimality of the dispatch is not established. The authors should either solve an integer program (with a justification of tractability at serving timescales), prove total unimodularity of the constraint matrix, or report the effect of rounding on the dispatch quality.","section":"§5.2, Eq. (7)"},{"comment":"The dispatch objective rests on the linear attention-time model tau_i = a_i h_i + b_i g_i + c_i. The supporting evidence in Fig. 7 is limited to single-setting curves with no error bars and no coverage of the extremes (very small head counts, very large caches, or HBM-saturation regimes). The robustness test in Fig. 16(b) perturbs the coefficients by ±20%, but a structural nonlinearity (e.g., quadratic cache-size behavior or launch-overhead floor at small h_i) is not a coefficient perturbation. Since the LP in Eq. (7) will systematically misallocate heads if the model is misspecified, the authors should validate the linear model across the full operating range used by the dispatcher, or add a structural robustness test that compares LP decisions against a nonlinear ground-truth model.","section":"§5.1, Eq. (3) and Fig. 7/16(b)"},{"comment":"The re-dispatching optimization displayed in §5.3.1 is incomplete. As written, the constraint is only sum_i sum_j x_j^i(t) l_j(t) <= sum_i r M_i/2, with no per-request head-integrity constraint sum_i x_j^i(t) = H and no non-negativity/integrality constraints. Without those constraints, the reported 'ideal attention time' f* is not well defined (a trivial solution could assign zero heads to all requests). The formulation should be stated in full, matching Eq. (7).","section":"§5.3.1"},{"comment":"The paper claims up to 2.25× throughput improvement, but throughput is never explicitly defined or measured. Figures 8-10 plot normalized latency against request rate; the throughput gain appears to be inferred from the highest request rate before latency rises steeply. The authors should define the throughput metric (e.g., achieved request rate under a latency SLO), state the SLO if one is used, and report per-dataset saturation throughput for Hetis, Splitwise, and Hexgen.","section":"§7.2, Figs. 8-10"}],"minor_comments":[{"comment":"The reported latency curves and bars do not include error bars or the number of runs. For a systems paper with P95 metrics, at least a few repetitions and confidence intervals should be reported, especially because some differences between systems are small.","section":"Figs. 8-13"},{"comment":"The text says head-wise KV cache migration has 'zero overhead,' but §6 describes low-priority CUDA streams to reduce the adverse impact, and Fig. 15(a) shows that re-dispatching still has a measurable cost. The 'zero-overhead' wording should be removed or qualified.","section":"§3.2 and §6"},{"comment":"The modeling-accuracy numbers ('up to 93.8%', '92.4% to 96.1%') are reported without defining the accuracy metric. Please state whether this is MAPE, R², or another measure.","section":"§7.4"},{"comment":"The large-scale search evaluation ('five GPU types with 32 GPUs each') is described only as a simulation, with no setup details. Please clarify whether this is an actual run of the search code and give the simulated device and model parameters.","section":"§7.4"},{"comment":"The notation sum_i x_j^i(t) ∈ {0,H} is unusual; since Eq. (7c) already enforces the H constraint for incoming requests, consider making explicit that the {0,H} form covers the no-allocation case and is not a typo.","section":"§5.2.1, Eq. (5)"},{"comment":"The text refers to a 'rigorous optimization problem formulation' but the GPU-exclusion criterion with Delta=0.05 and the one-by-one removal from lowest to highest end is a heuristic. The heuristic nature should be acknowledged in the main text, not only implicitly in the search description.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a systems venue and the core idea is sound enough to warrant a major revision. I would ask the authors to fix the headline claim, repair the LP formulations, and add structural robustness evidence for the linear attention model. There is no obvious concern about citation practices or novelty disclosure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The punchline: Hetis is a real systems contribution. It's not a paradigm shift, but it solves a practical problem—serving LLMs on clusters with mixed GPU generations—using an idea I hadn't seen in the baselines: keep dense compute (MLP/prefill attention) on a selected subset of high-end GPUs, and dynamically dispatch decode-phase attention at head granularity across all GPUs, including low-end ones, with a linear cost model driving an LP solver. The module-level ablation in Fig. 13 isolates the source of gains, and the reported 1.39x TPOT and 2.25x throughput improvements over Splitwise and Hexgen are consistent across three models and three workload types.\n\nThe paper is refreshingly honest about overheads: head-wise KV cache management costs 13% extra storage but saves 26% fetch time, and re-dispatching helps 1.06x/1.14x. The implementation is built on vLLM and the LP formulation is clean.\n\nThe main caveat is the linear Attention-time model (Eq. 3). The stress-test concern is fair: Fig. 7 shows linearity only in a narrow operating range, and the ±20% profiling-error sensitivity test (Fig. 16b) only perturbs coefficients; it wouldn't catch a structural nonlinearity like HBM saturation or kernel-launch overhead at very few heads. That said, the model drives dispatch decisions, not the reported improvements themselves, so even a misspecified model would degrade gracefully rather than invalidate the results. Still, the lack of released artifacts and the single 10-GPU testbed leave real uncertainty about how well it generalizes to larger, more mixed fleets.\n\nMinor issues: the abstract's \"reduce latency by 1.49x\" is the module-level Attention latency, not end-to-end (which is up to 1.39x). Also, the evaluation compares only against Splitwise and Hexgen; recent systems like LLM-PQ are mentioned in related work but not benchmarked. Error bars are absent throughout, which is common for systems papers but worth noting.\n\nOverall, I'd send this to peer review. The core idea is sound, the evaluation is honest, and the problem is real. A serious referee would ask for the abstract fix, error bars where feasible, and a deeper look at the linearity assumption, but I don't see a load-bearing flaw.","headline":"Hetis is a credible heterogeneous-LLM-serving system with genuinely new head-wise dynamic attention dispatch; the headline 1.49x latency gain is a module-level number, but the end-to-end 1.39x/2.25x results hold up under scrutiny.","tokens_in":20948,"tokens_out":3224,"would_cite":true,"duration_ms":31253,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hetis claims that module-level parallelism—dense layers on fast GPUs, attention heads dynamically spread across all GPUs—raises LLM serving throughput up to 2.25x and cuts latency up to 1.49x relative to existing heterogeneity-aware systems","keywords":["heterogeneous GPU cluster","LLM serving","dynamic parallelism","attention head partitioning","KV cache management","online load dispatching","module-level parallelism","transformer inference"],"falsifier":"Profile attention kernel time on each GPU type at a fixed cache size while sweeping the number of heads from 1 up to the model's full head count, and repeat at several cache sizes; if the measured time is not an affine function of heads and cache size—for example, a sharp rise below a few heads due to launch overhead or a flattening as HBM bandwidth saturates—then the linear model behind the dispatcher is falsified and its head allocations are not optimal.","tokens_in":19999,"feed_emoji":"⚡","tokens_out":11885,"duration_ms":116517,"temperature":0.7,"pith_summary":"Hetis is a serving system for LLM inference on clusters where a few fast GPUs sit alongside much slower ones, and its proposal is to stop parallelizing the model uniformly. Dense, parameter-heavy modules—QKV projection and MLP—run only on a chosen subset of 'primary workers,' while Attention, which has no parameters, is split at the granularity of individual heads across every GPU, so low-end cards become useful KV-cache holders and attention compute. An online dispatcher solves a linear program at each arrival to decide how many heads each device should handle, using profiled linear models of attention time and network transfer cost, with a re-dispatching scheme that moves only the affected heads when long contexts create imbalance. The authors report up to 2.25x higher serving throughput and up to 1.49x lower latency than the phase-splitting and asymmetric-partitioning baselines on a mixed A100/3090/P100 cluster. If the result holds, operators with a heterogeneous fleet can put slow GPUs to work instead of leaving them idle or letting them bottleneck dense math.","feed_headline":"Head-wise attention split lifts LLM serving 2.25x on mixed GPUs","feed_subtitle":"Dense layers stay on fast GPUs; attention heads and KV caches spread across all of them.","key_machinery":"The load-bearing mechanism is head-granularity dynamic attention dispatch. Each request's attention heads are partitioned across primary and attention workers, KV-cache blocks are indexed per head, and an online dispatcher solves a linear program that minimizes the maximum attention-plus-transfer time across devices, with per-device attention time modeled linearly in the number of heads and cache bytes and point-to-point transfer time modeled by a linear communication-cost model. The supporting mechanism is primary-worker selection: a hierarchical search removes low-end GPUs from dense computation whenever their contribution to reducing dense-module cost is below a small threshold, so slow G","core_discovery":"The paper's claim is that the right unit of parallelism for heterogeneous LLM serving is the module, not the layer or the phase. Hetis separates the Transformer into two kinds of work: dense, parameter-heavy computation (QKV projection, MLP) and parameter-free Attention. For dense work it searches over data-, tensor-, and pipeline-parallel configurations on a subset of GPUs—the 'primary workers'—and deliberately excludes low-end devices whose contribution to dense computation is outweighed by communication cost. For Attention it treats each request's heads as independently schedulable: primary and attention workers each receive some number of heads plus the corresponding KV-cache slices, and","pith_inferences":["The same module-level split should generalize to other parameter-free or low-intensity components—normalization, MoE routing, or even attention on CPU memory—wherever a device's dense-compute throughput is much worse than its memory bandwidth.","A testable prediction: Hetis's throughput advantage over asymmetric partitioning should widen as the slowest GPU's MLP gap grows, until the network becomes the limiting factor; clusters with more low-end GPUs would expose that crossover.","The linear attention-time model is the assumption worth stress-testing: a convex piecewise extension inside the same LP framework would cover launch overhead and bandwidth saturation while keeping dispatch real-time.","Because newly arriving requests are dispatched without re-partitioning existing ones, workloads dominated by many short requests may leave some head-level slack; a more anticipatory rebalancing policy could close that gap."],"forward_implications":["Low-end GPUs that are uncompetitive for MLP still contribute to serving capacity by hosting KV cache and computing attention heads, turning previously stranded memory into usable throughput.","Because rebalancing happens at head granularity, moving a long request to relieve a bottleneck costs only partial KV-cache transfers rather than whole-sequence migration, so adaptation can happen online.","The dispatch formulation supports both MHA and GQA by folding the query-to-KV head ratio into the communication cost, so the scheme is not tied to one attention architecture.","Heterogeneous serving no longer has to choose between phase splitting, which duplicates model parameters, and asymmetric parameter splitting, which strands cache space; the module-level role split avoids both.","Concretely, on the paper's A100/3090/P100 testbed Hetis sustains up to 2.25x the request rate of the phase-splitting baseline and up to 1.33x that of the asymmetric-partitioning baseline, with P95 time-per-output-token up to 1.39x lower."],"supporting_citations":[{"why":"Supplies the heterogeneous dense-computation cost model (communication plus compute) that the primary-worker search adapts, and serves as the asymmetric parameter-partitioning baseline.","marker":"[23]"},{"why":"The phase-splitting baseline; its duplicated model parameters motivate Hetis's memory-efficiency argument.","marker":"[35]"},{"why":"The open-source serving engine the prototype extends; contributes the block-based KV cache management and attention kernel that Hetis reworks to head granularity.","marker":"[25]"},{"why":"Cited for HBM-to-SRAM transfer overhead, supporting the linear dependence of attention time on cache size.","marker":"[13]"},{"why":"The point-to-point communication model used to quantify network transfer overhead in the dispatch objective.","marker":"[37]"},{"why":"Provides the polynomial-time linear-programming algorithm that makes real-time head dispatch tractable.","marker":"[12]"},{"why":"Defines grouped-query attention; the query-to-KV head ratio r enters the communication-volume formula and the GQA evaluation.","marker":"[40]"}],"fun_headline_variants":["Split attention heads, boost LLM throughput 2.25x on mixed GPUs","Hetis: head-granularity parallelism for heterogenous LLM serving","Dynamic head scheduling lifts LLM serving on uneven GPU clusters","Module-level parallelism gives 2.25x LLM throughput on mixed hardware"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The dispatcher's head allocations are only as good as the profiled assumption that a GPU's attention time grows linearly with the number of heads it processes and the size of the cache it reads; if that curve bends under memory-bandwidth saturation or at very small head counts, the allocations will be wrong.","fun_headline_variants_meta":{"raw":{"variants":["Split attention heads, boost LLM throughput 2.25x on mixed GPUs","Hetis: head-granularity parallelism for heterogenous LLM serving","Dynamic head scheduling lifts LLM serving on uneven GPU clusters","Module-level parallelism gives 2.25x LLM throughput on mixed hardware"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000175,"raw_usage":{"total_tokens":1123,"prompt_tokens":742,"completion_tokens":381,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":300}},"tokens_in":486,"tokens_out":381,"duration_ms":4368,"temperature":1.0,"reasoning_tokens":300,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T20:47:27.757660+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Profile attention kernel time on each GPU type at a fixed cache size while sweeping the number of heads from 1 up to the model's full head count, and repeat at several cache sizes; if the measured time is not an affine function of heads and cache size—for example, a sharp rise below a few heads due to launch overhead or a flattening as HBM bandwidth saturates—then the linear model behind the dispatcher is falsified and its head allocations are not optimal.","supporting_citations":[],"review_version":1}