{"id":"ee19ad86-ad62-4e73-9dd6-67c8cac9c86c","arxiv_id":"2508.19559","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"HeteroScale coordinates scaling of prefill and decode pools using decode TPS as a single robust signal, reporting a 26.6 percentage point GPU utilization gain in production.","lead":"A ByteDance team built HeteroScale, an autoscaling system for AI inference clusters that separates prefill and decode computation. It uses a single workload signal, decode tokens per second, to grow or shrink both pools together, and reports a 26.6 percentage point rise in average GPU utilization across tens of thousands of GPUs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Production gain of 26.6 percentage points is based on uncontrolled cross-service and before/after comparisons; no evidence rules out workload or hardware confounds, so the headline utilization claim is not yet established.","rationale":"The central claim of the paper is the production effectiveness number. The mechanism (coordinated scaling on decode TPS) is interesting, but the evidence for the mechanism's benefit rests entirely on the production comparisons. Those comparisons are uncontrolled: different services, different time periods, no error bars. A confounded comparison cannot tell us whether HeteroScale, the policy, the scheduler, or unrelated changes caused the utilization difference. The reader's weakest_assumption (uniform decode TPS) is a plausible internal-mechanism risk, but even if decode TPS is non-uniform, the negative-feedback latency controller (Section 3.3.2) provides a safety net that may preserve SLOs at some efficiency cost. The uncontrolled comparison has no such safety net; if it is confounded, the headline number and the 'hundreds of thousands of GPU-hours' claim are unsupported. This does not mean the paper is wrong; it means the core quantitative evidence is unverified. A controlled experiment or at least a difference-in-differences re-analysis would settle it. Given the system design is coherent and the empirical metric study is reasonable, a CONDITIONAL verdict is appropriate pending such evidence.","tokens_in":18153,"tokens_out":9614,"duration_ms":104411,"concrete_test":"Conduct a matched A/B deployment: select at least 10 production services with comparable traffic profiles and hardware types; randomly assign each to HeteroScale or the existing baseline (e.g., Kubernetes HPA or no autoscaling) for two weeks; measure GPU utilization and SM activity; report the mean difference with 95% confidence intervals and a permutation test. If the HeteroScale-minus-baseline utilization gain is not statistically significant or is materially below 26.6 pp, the headline claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 reports a 26.6 pp GPU-utilization increase by comparing 'services with HeteroScale enabled' to 'services without autoscaling.' This is a cross-sectional comparison of different services; it does not control for traffic volume, model mix, prompt/generation length distributions, or hardware pools. The before/after comparison ('a recent date with a date prior to the scaled deployment') is similarly confounded by any platform-level change over time (new models, traffic growth, scheduling changes). No confidence intervals, no matched cohorts, no difference-in-differences analysis are provided. The reported per-service numbers (prefill GPU utilization 46.8%→76.2%, decode GPU utilization 86.0%→82.2%) show the aggregate gain could be dominated by a shift in the service mix toward more prefill-heavy workloads rather than by autoscaling behavior. Therefore the central quantitative claim that HeteroScale 'increases average GPU utilization by 26.6 percentage points' is not supported by the evidence as presented. A secondary but related risk is that the coordinated policy assumes decode TPS is uniformly distributed across decode instances (Section 4.2.2) without presenting per-instance distribution data; if this assumption fails, the single-signal scaling rule could mis-provision, but this affects mechanism efficiency rather than the headline number directly.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents HeteroScale, a production autoscaling framework for prefill-decode (P/D) disaggregated LLM serving at ByteDance. The system combines a topology-aware scheduler that uses Deployment Groups and RDMA Subgroups to enforce network affinity and hardware-priority constraints, with a metrics-driven scaling policy. Based on an empirical comparison of eight autoscaling signals, the authors select decode TPS as the primary signal and scale prefill and decode pools jointly under a fixed P/D ratio, supplemented by a latency-based negative-feedback mechanism. The claimed contributions are a coordinated scaling policy that preserves architectural balance, a set of scheduling abstractions for heterogeneous hardware and network topology, and a large-scale production evaluation reporting a 26.6 percentage-point increase in average GPU utilization, a 9.2 percentage-point increase in SM activity, and hundreds of thousands of GPU-hours saved daily while preserving SLOs.","tokens_in":18513,"tokens_out":3523,"duration_ms":38806,"significance":"If the production-effectiveness claims were convincingly supported, this would be a significant systems contribution: it addresses real operational challenges of P/D-disaggregated LLM serving at a scale rarely reported in the literature. The paper's strengths include the use of production workload traces, the controlled P/D-ratio experiments on two services (Section 4.1), the replay comparison of eight scaling metrics (Section 4.2), and the concrete scheduling abstractions and heuristics in Section 3.4. The P/D coordination idea—using one robust signal to scale both pools with a fixed ratio—is plausible and well motivated. However, the headline quantitative claims in Section 4.3 rest on uncontrolled comparisons, and the metric-selection evidence in Section 4.2 is presented without quantitative scoring. The paper does not release code, data, or configurations, so the reported operational gains are not independently reproducible. The central contribution is credible but currently under-supported by the evidence as presented.","major_comments":[{"comment":"The headline claim of a 26.6 percentage-point increase in GPU utilization rests on comparing services with HeteroScale enabled to services without autoscaling on 'a representative day,' plus a before/after comparison. No controls are reported for service mix, traffic volume, prompt/generation length distribution, hardware pool, or concurrent platform changes. The per-service numbers in the same section (prefill GPU utilization 46.8%→76.2%, decode GPU utilization 86.0%→82.2%) show that the aggregate gain could be dominated by a shift in service mix toward prefill-heavy workloads. Please report matched cohorts or difference-in-differences estimates, with confidence intervals, and clarify which confounds were controlled.","section":"Section 4.3 (Production Deployment Analysis)"},{"comment":"The metric-replay evaluation is currently qualitative: eight replays are 'recorded and visualized' in Figure 6, and the text asserts that TPS-based autoscaling responds effectively while decode utilization is ineffective. No quantitative scoring is provided—no SLO violation counts, overshoot/oscillation metrics, time-to-scale, resource waste, or comparison against a no-scaling baseline. Since the selection of decode TPS as the primary signal is load-bearing for the coordinated policy, the replay experiments should be scored on objective criteria and a table of these metrics provided.","section":"Section 4.2.2 (Experiment Results)"},{"comment":"The coordinated policy assumes decode TPS is 'uniformly distributed across instances,' so that a pool-level average can serve as the single scaling signal for both prefill and decode pools. The paper provides no per-instance decode TPS distribution or load-imbalance measurements. If decode TPS is heterogeneous across instances, the single-signal proportional rule could scale the pool based on an average while some instances saturate and others idle, undermining the claimed balance. Please report per-instance decode TPS variance or provide evidence that instance-level imbalance is controlled by the scheduler.","section":"Section 4.2.2 and Section 3.3.2 (Metrics-driven Scaling Policy)"},{"comment":"Algorithm 1 obtains the optimal P/D ratio and target per-instance metric from a pressure test on service S and workload W, then selects the policy by simulating on the same W. This configuration appears to be the same one used in the production evaluation of Section 4.3. There is no held-out workload or time period, so the reported gains may partly reflect tuning to the evaluation conditions. Please add a validation split (e.g., a different time window or service) to demonstrate that the chosen ratio and thresholds generalize rather than overfit the calibration data.","section":"Section 3.3.3 (Workload-centric Policy Curation, Algorithm 1)"}],"minor_comments":[{"comment":"There are several typos and formatting issues: 'T aming' in the title, 'Geployment Groups' in Section 3.1, 'motivat' in Section 2.2, 'simutaneously' and 'address this this issue' in Section 3.4, and inconsistent variable names such as 'P odDelta'/'P odsDelta' and 'If inal' in the algorithms.","section":"Throughout"},{"comment":"The replay figures show scaling events but do not explain the axis units, the threshold calibration, or how the overlaid instance-count curves are computed. A legend and a description of the replay harness would make the comparison interpretable.","section":"Figure 6 and Appendix C"},{"comment":"The statement that the TPS-based policy manages '64% of the total GPU fleet under HeteroScale’s control' needs a denominator and a time window. Also, 'a representative day' and 'a recent date with a date prior to the scaled deployment' should be specified (dates, duration, and what changed between them).","section":"Section 4.3"},{"comment":"The text says the large latency fluctuations 'are an artifact of normalization' but does not explain the normalization or show the unnormalized curves. Please clarify or replot.","section":"Figure 7d"}],"recommendation":"major_revision","confidential_remarks":"The production-effectiveness claim is the main selling point of the paper, but as written it is not supported by a controlled comparison. I would encourage the authors to provide matched cohorts, confidence intervals, or a clearly stated quasi-experimental design, and to add quantitative scoring to the metric-replay study. The rest of the paper contains plausible and useful engineering contributions, so I see this as fixable within the scope of a revision rather than grounds for rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is worth reading as a production-systems design and as an empirical probe of autoscaling signals for P/D disaggregated serving. The system—coordinated proportional control using decode TPS as the single signal, with P/D ratio enforced during scaling—is coherent and directly motivated by the metric analysis. The Deployment Group and RDMA Subgroup abstractions are sensible tools for network-aware placement. The paper honestly engages with prior work, and the metric study uses real traces rather than synthetic loads. That is real substance.\n\nWhat is genuinely new: the first large-scale comparison of TPS, GPU utilization, SM activity, TTFT, and TBT as autoscaling signals in a P/D disaggregated production setting, and the specific finding that decode TPS is the most robust single signal. The coordinated policy that scales prefill and decode together from that one signal, keeping a fixed P/D ratio, is a clean answer to the architectural-imbalance problem the paper identifies.\n\nThe soft spots are in the evidence for the headline numbers. Section 4.3 compares services with HeteroScale enabled to services without autoscaling. That is a cross-sectional comparison of different services, with no covariate control, no matched cohorts, no confidence intervals. The before/after comparison is similarly confounded by any platform change over time. The 26.6 percentage point utilization gain and the hundreds of thousands of saved GPU-hours should be treated as upper-bound anecdotes, not measured effects. The metric replay (Section 4.2) is presented as visual inspection of eight replays, with no quantitative scoring and no comparison against a baseline like HPA. The paper also assumes decode TPS is 'uniformly distributed across instances' without showing per-instance distribution data; that assumption is load-bearing for the single-signal policy. And no code or data are released. None of this invalidates the design, but it does mean the central quantitative claim is not yet established.\n\nWho is this for? Systems researchers and operators building P/D disaggregated serving. A serious referee should look at it—the combination of a real deployment, a plausible design, and an interesting empirical signal analysis deserves referee time, even if the evaluation needs much stronger methodology. I would not cite the utilization numbers in my own work, but I would cite the metric analysis as a production observation if I needed that context.","headline":"A plausible production autoscaling design with a genuinely useful metric comparison, but the headline utilization gain rests on uncontrolled comparisons and should be read as anecdote until better evidence appears.","tokens_in":19030,"tokens_out":2398,"would_cite":true,"duration_ms":25575,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Decode tokens per second, not GPU utilization, is the robust autoscaling signal for disaggregated LLM serving; the production system built on it raises average GPU utilization by 26.6 percentage points.","keywords":["LLM inference serving","autoscaling","prefill-decode disaggregation","decode TPS","GPU utilization","network-aware scheduling","heterogeneous GPUs","production deployment"],"falsifier":"Collect per-instance decode TPS across a production decode pool during a traffic peak and compare the highest and lowest instances. If the spread is wide—say several instances at more than 1.5 times the pool mean while others sit near idle—and the pool-level decode TPS still drives scale-in/out decisions that let SLOs degrade, the uniform-distribution premise is falsified.","tokens_in":1527,"feed_emoji":"⚖️","tokens_out":1544,"duration_ms":73228,"temperature":0.7,"pith_summary":"This paper claims that autoscaling for prefill-decode disaggregated LLM serving fails when each phase is scaled independently on hardware metrics, because decode GPU utilization stays high even when idle due to KV-cache memory pressure. It presents HeteroScale, a production autoscaling system that uses decode tokens-per-second as a single scaling signal to resize both prefill and decode pools together while enforcing a fixed prefill-to-decode ratio, and places instances with network topology in mind. Deployed on tens of thousands of GPUs, the system reports a 26.6-percentage-point gain in average GPU utilization and daily savings of hundreds of thousands of GPU-hours while meeting latency SLOs. The significance is a template: one workload-level throughput signal plus coordinated scaling can keep heterogeneous, disaggregated inference balanced and efficient.","feed_headline":"Decode TPS alone lifts LLM GPU use by 26.6 points","feed_subtitle":"A single metric scales both pools; saves hundreds of thousands of GPU-hours per day.","key_machinery":"The load-bearing mechanism is the coordinated scaling policy built on decode TPS (tokens generated per second across the decode pool), used as a single proportional-control signal: it computes the needed total capacity and splits it by a fixed prefill-to-decode ratio, so the two pools are always scaled together. Its supporting mechanical counterpart is the Deployment Group abstraction—a logical container that pins a service's prefill and decode replicas to a shared network domain—plus RDMA Subgroups that rank those domains by hardware priority. Together they translate one aggregate workload signal into balanced, network-aware placement decisions.","core_discovery":"The central claim is that decode TPS is the robust autoscaling signal for prefill-decode disaggregated serving. The paper argues that decode-phase hardware metrics are misleading—KV-cache memory pressure keeps decode GPU utilization and SM activity high even under low load—while latency metrics react nonlinearly with a cliff-like threshold, and prefill TPS is confounded by KV-cache hit rates. Decode TPS, by contrast, tracks load with high signal-to-noise ratio and, the paper states, distributes uniformly across decode instances. Using decode TPS, HeteroScale's proportional-control policy computes the desired total capacity from a target per-instance throughput and then applies the fixed P/D","pith_inferences":["The single-signal design presumes decode TPS is balanced across instances; a testable extension is to monitor the per-instance distribution of decode TPS and trigger additional rebalancing if some decode replicas saturate while others idle.","The fixed P/D ratio is an operator-set constant derived from pressure tests; the paper's own future-work section implies it can be adapted online, and one concrete extension would be to adjust the ratio from queueing or latency signals during workload drift.","The same coordinated single-metric pattern may generalize to other memory-bound disaggregated pipelines, such as mixture-of-experts serving, where one stage's hardware utilization is decoupled from its true load.","The reported 26.6-point utilization gain is a fleet-level average across heterogeneous services; a sharper evaluation would separate services by workload type and hardware tier to see where the coordinated policy helps most."],"forward_implications":["Autoscalers for P/D-disaggregated serving should not scale decode pools on GPU utilization or SM activity; those metrics report high values regardless of actual decode load.","Decode TPS-based coordinated scaling maintains the prefill-to-decode ratio under diurnal traffic, avoiding the architectural imbalance that occurs when pools are scaled independently.","Network affinity-aware placement of prefill and decode instances is needed to preserve the bandwidth for KV-cache transfer that disaggregated serving depends on.","The TPS-driven policy outperforms a time-based periodic policy in production, delivering 10.0 percentage points higher GPU utilization and 11.1 points higher SM activity.","The fixed P/D ratio can protect both TTFT and TBT SLOs even during rapid scaling events, as long as prefill and decode instances register in a coordinated way."],"supporting_citations":[{"why":"Supplies the generic Kubernetes HPA baseline whose hardware-metric approach the paper argues is misleading for decode phases.","marker":"[29]"},{"why":"Establishes prefill/decode disaggregation and goodput-optimized serving that HeteroScale targets.","marker":"[57]"},{"why":"Supplies the KV-cache-centric architecture and motivates the network-affinity constraints for KV cache transfer.","marker":"[41]"},{"why":"Documents phase-splitting benefits and the compute-bound versus memory-bound distinction the metric analysis relies on.","marker":"[40]"},{"why":"Provides cost and efficiency evidence for phase-aware heterogeneous provisioning and the disaggregated MoE setting the system extends to.","marker":"[58]"},{"why":"Shows distributed disaggregated inference requires high-bandwidth KV cache transfer, grounding the Deployment Group affinity abstraction.","marker":"[7]"},{"why":"Describes serving disaggregated LLMs at scale, the operational context for coordinated prefill-decode scaling policies.","marker":"[25]"}],"fun_headline_variants":["Decode TPS metric lifts LLM GPU use 26.6 points","Single metric scales P/D pools, cuts GPU waste daily","HeteroScale: decode TPS autoscaling gains 26.6 pp","Decode TPS as load signal: 26.6-point GPU use boost","Coordinated autoscaling with decode TPS saves GPU-hours"],"cache_read_input_tokens":20736,"weakest_assumption_plain":"The whole coordinated policy assumes decode TPS is spread evenly across decode instances, so a pool-wide average tells the autoscaler what every instance is feeling; if real traffic concentrates tokens on some instances, the average hides imbalance and the single signal can leave some GPUs saturated while others idle.","fun_headline_variants_meta":{"raw":{"variants":["Decode TPS metric lifts LLM GPU use 26.6 points","Single metric scales P/D pools, cuts GPU waste daily","HeteroScale: decode TPS autoscaling gains 26.6 pp","Decode TPS as load signal: 26.6-point GPU use boost","Coordinated autoscaling with decode TPS saves GPU-hours"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000621,"raw_usage":{"total_tokens":2705,"prompt_tokens":725,"completion_tokens":1980,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":1897}},"tokens_in":469,"tokens_out":1980,"duration_ms":16992,"temperature":1.0,"reasoning_tokens":1897,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:40:32.231895+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect per-instance decode TPS across a production decode pool during a traffic peak and compare the highest and lowest instances. If the spread is wide—say several instances at more than 1.5 times the pool mean while others sit near idle—and the pool-level decode TPS still drives scale-in/out decisions that let SLOs degrade, the uniform-distribution premise is falsified.","supporting_citations":[{"cited_title":"Kubernetes horizontal pod au- toscaler","cited_arxiv_id":null,"evidence_quote":"Supplies the generic Kubernetes HPA baseline whose hardware-metric approach the paper argues is misleading for decode phases."},{"cited_title":"Day zero benchmarks for qwen 3 with sglang on baseten","cited_arxiv_id":null,"evidence_quote":"Establishes prefill/decode disaggregation and goodput-optimized serving that HeteroScale targets."},{"cited_title":"Splitwise: Efficient generative llm inference using phase splitting","cited_arxiv_id":null,"evidence_quote":"Supplies the KV-cache-centric architecture and motivates the network-affinity constraints for KV cache transfer."},{"cited_title":"P/d- serve: Serving disaggregated large language model at scale, 2024","cited_arxiv_id":null,"evidence_quote":"Describes serving disaggregated LLMs at scale, the operational context for coordinated prefill-decode scaling policies."}],"review_version":1}