{"id":"544c2b50-bcfa-4706-92e7-682ee44c1580","arxiv_id":"2608.13499","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Operator-level autoscaling meets LLM latency targets with 20-36% fewer GPUs and 14-28% less power than model-level replica scaling.","lead":"OPSCALE proposes scaling individual operators of a large language model, rather than whole model replicas, so GPU serving can react faster and waste less. On production traces it reports meeting latency targets with up to 36.3% fewer GPUs and 28% less power, or 44% more throughput at a fixed GPU budget.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Replica-level routing may break KV-cache affinity for replicated attention; §4.5 never specifies how a request's decode iterations find the attention replica holding its KV cache.","rationale":"The reader's Poisson/burstiness concern is real: Section 3.1's M/M/R assumption is in tension with the bursty traces in Figure 2, and Section 5.7's 0.8% validation does not describe holdout or calibration. I treat that as a secondary risk. The more load-bearing gap is in the execution plane: operator-level scaling only becomes a viable serving primitive if per-request state (especially the KV cache) survives dynamic replication and arbitrary dispatch. The paper describes shortest-queue routing and dynamic replica registration but never explains how a decode iteration is guaranteed to reach the attention replica that holds the request's KV cache. This is not a question of tuning or a calibration detail; it is a prerequisite for the system to function at all, and the possible solutions each contradict or complicate the queueing/routing abstractions used to compute the headline savings. Because the concern is addressable by code inspection or a focused experiment, and because the paper's controlled common-engine comparison has real value, I would keep the reader's CONDITIONAL verdict rather than escalate to REJECT or UNVERDICTED. My agreement is partial: I share the reader's unease about the provisioning model, but I locate the least secure condition in the underspecified per-request routing and KV-cache affinity, not primarily in the burstiness assumption.","tokens_in":27366,"tokens_out":11378,"duration_ms":128538,"concrete_test":"Inspect the Replica Manager dispatch path in the OPSCALE implementation (or, if no artifact is released, run a two-GPU experiment with attention replicated): serve one multi-token request and log which attention replica ID executes layer l on each decode iteration. If the ID is constant, verify that routing is sticky and reconcile this with shortest-queue dispatch; if the ID changes, locate the KV cache and measure the per-token remote-read or transfer overhead against a single-replica control. If TBT or per-token transfer time changes materially, the reported efficiency gains are partly an artifact of unaccounted KV movement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires operator-level elasticity to serve real multi-token requests. In the execution plane (§4.5), requests are dispatched to operator replicas via a weighted shortest-queue policy. For any attention operator, a request's KV cache is created at prefill on one replica and must be read on every decode iteration. If dispatch can send a later decode iteration to a different attention replica, that replica does not hold the needed cache. The paper does not state whether routing is sticky per request, whether a shared/remote KV cache is accessible to all replicas, or whether KV entries are transferred between replicas. The only KV-related mechanism described, the ElasticBlockManager based on kvcached, handles memory allocation and reclamation, not per-request placement affinity. Each of the missing mechanisms would materially alter the story: sticky routing invalidates the independent shortest-queue balance assumed by the M/M/R model in Eq. (2); remote KV access adds per-token latency that must appear in TBT; and KV transfer would consume NVLink bandwidth that the paper otherwise counts as available. Without this specification, the reported SLO attainment and sub-second scaling results are not reproducible from the text, and the measured gains could include hidden KV movement costs.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OpScale, a cluster-scale orchestration framework that changes the unit of autoscaling for LLM serving from whole model replicas to individual operators. It motivates this with an operator characterization study showing heterogeneity in compute, memory, and SM-allocation sensitivity, and then presents a control plane that profiles operators, provisions replicas via a queueing-theoretic model (M/M/R with Erlang-C), and places replicas with a contention-aware best-fit heuristic. The execution plane extends nano-vLLM with dynamic operator replicas, multi-stream management, and request routing. On up to 40 A100s and 24 GB200s with production traces, the paper reports up to 36.3% fewer GPUs, 28% lower power, and up to 44% higher throughput compared with model-level autoscaling baselines (DynamoLLM, AIBrix, Production Stack) ported onto the same engine.","tokens_in":27723,"tokens_out":6346,"duration_ms":61122,"significance":"If the central claims hold, this is a substantial contribution: it reframes a basic design question in LLM serving (what should scale), provides a working end-to-end system, and quantifies the benefit at cluster scale. The evaluation has genuine strengths: all baselines are run on the same nano-vLLM data plane, the traces are external production traces, the profiling model is validated on held-out interference measurements, and the offline-vs-online optimality gap is at least explicitly stated. The significance is conditional, however, on closing two technical gaps: the routing specification for KV-cache affinity, which is central to correctness of multi-token serving, and the handling of bursty arrivals by a Poisson-based provisioning model.","major_comments":[{"comment":"The execution-plane description never specifies how a request's KV cache follows it across attention operator replicas. A request's KV cache is created at prefill on the replica that runs the attention operator for that request, and every decode iteration must read that same cache. The manuscript states that 'requests are dispatched to operator replicas via shortest-queue routing, weighted by replica capacity,' but it does not state whether routing is sticky per request, whether a shared or remote KV cache is accessible to all attention replicas, or whether KV entries are transferred between replicas. The ElasticBlockManager/kvcached mechanism handles memory allocation and reclamation, not per-request placement affinity. If routing is not sticky, the system as specified cannot serve multi-token requests; if it is sticky, the independent M/M/R model in Eq. (2) no longer describes the dispatch balance; and if KV is moved or read remotely, per-token latency and interconnect costs are omitted from the TBT model. This missing specification is load-bearing for the central SLO-attainment and sub-second-elasticity claims.","section":"§4.5, §5.2"},{"comment":"The provisioning model treats each operator as an M/M/R queue with Poisson arrivals and exponential service times, and uses the Erlang-C formula to determine replica counts. However, the paper's own Figure 2 shows production LLM traffic with peak-to-min ratios of about 2x (Chat) and 5.8x (Code) within 10-second windows, which is far from Poisson. Section 5.7 reports an average queueing-model error of 0.8% 'across all evaluated scenarios and bursty trace replays,' but it does not state whether that validation set was held out from the data used to build the profile, nor how a Poisson assumption yields accurate waiting-time predictions under such burstiness. Since replica counts and SLO compliance in §4.3 and §5.2 are computed from this model, the paper needs to specify the validation protocol and, if the model is not Poisson, present the actual waiting-time distribution used.","section":"§3.1, Eq. (2); §5.7"},{"comment":"The paper formulates both TTFT and TBT as SLOs in §3.1 and Eq. (1), but the main autoscaling evaluation reports only P99 TTFT (Figures 13 and 27) and never presents TBT attainment for OpScale or the baselines. If the resource savings are achieved at the cost of degraded time-between-tokens, the headline 'meets latency SLOs' claim is incomplete. The authors should report TBT attainment (or state that TBT was not used as a scaling constraint), especially because decode-phase latency is the metric most sensitive to the KV-cache routing issue raised above.","section":"§5.2–§5.3, Figs. 13 and 27"},{"comment":"The manuscript repeatedly states that the online greedy provisioning heuristic has resource cost within 8% of the brute-force oracle, but no experiment or table in §5.6 (or elsewhere) measures this optimality gap; §5.6 reports only plan-generation latency. This is a quantitative claim that the reader cannot verify from the presented evidence. Either add an evaluation comparing the greedy plan's cost to the exhaustive oracle across workloads, or remove the 8% figure and state the heuristic's optimality as unquantified.","section":"§3, §4.3, §5.6"}],"minor_comments":[{"comment":"The baselines are re-implementations on nano-vLLM, and the paper says they 'reproduce each baseline's autoscaling policy,' but it does not say how the ported policies were validated against the original systems; a sentence on fidelity would help readers interpret the magnitude of the reported savings.","section":"§5.1"},{"comment":"There is a spacing typo: 'would requireweeksof GPU time' should read 'would require weeks of GPU time.'","section":"§4.1"},{"comment":"The column headers 'P99 latency,' 'P90 latency,' 'Avg. latency' should be clarified as 'P99 scale-up latency' and similar, to avoid confusion with request-level latency.","section":"Table 1"},{"comment":"The claim that profiling a 57B model completes in under an hour on a single GB200 node is stated without a supporting measurement; a timing breakdown of the sparse-sampling profiling pipeline would strengthen this claim.","section":"§5.6"}],"recommendation":"major_revision","confidential_remarks":"The KV-cache routing question is the main technical risk; I would ask the authors to specify the routing semantics for attention replicas before any accept decision. The paper is a good fit for a systems venue and I see no citation-pattern concerns. The TBT gap and the unmeasured 8% optimality claim should also be addressed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"OpScale is worth your time. It moves the scaling unit inside the model, to individual operators, and makes a credible case that this cuts GPU footprints materially. The characterization study is thorough, the evaluation is controlled (all baselines on the same nano-vLLM engine), and the production traces are external. The sub-second scale-up latency numbers are plausible and well measured. If the savings hold up, this is a significant engineering contribution.\n\nThe soft spots are real but not all equal. The biggest one is the execution plane. Requests are dispatched to operator replicas via weighted shortest-queue routing, but the paper never says how a decode iteration finds the attention replica that holds the KV cache created at prefill. Sticky routing would make the M/M/R queue independence assumption in Eq. (2) shaky; remote KV access would add per-token latency; KV transfer would consume bandwidth. None of these appear in the model or the evaluation. Without this specification, the headline SLO attainment is not reproducible from the text. This is load-bearing, because scaling attention is one of the paper's core moves.\n\nThe queueing model's Poisson assumption is a second concern, but the paper's own validation (0.8% average error in Table 2) partially mitigates it. Still, the paper doesn't say whether that validation set was held out from the calibration traces, and the burstiness in Figure 2 is exactly where an M/M/R model would be most fragile. I'd call this moderate.\n\nSmaller issues: the 'within 8% of oracle' claim appears without supporting measurement, and no artifact is released. Both are addressable.\n\nThis is a serious thinker's paper: the authors know the space, cite the right related work, and the mechanism is non-trivial. The central idea deserves a careful referee, but the paper needs to close the routing/KV affinity gap before the results can be trusted. I'd send it to review and ask for that explicitly.","headline":"A promising operator-level autoscaling system with a strong controlled evaluation, but the routing/KV-cache affinity gap is a load-bearing unspecified mechanism.","tokens_in":28131,"tokens_out":3164,"would_cite":true,"duration_ms":34068,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Operator-level scaling, not whole-model replicas, meets LLM latency SLOs with up to 36.3% fewer GPUs.","keywords":["operator-level autoscaling","LLM inference serving","GPU resource provisioning","SLO-aware scaling","queueing model","interference-aware placement","operator heterogeneity"],"falsifier":"Replay a held-out production trace through OPSCALE's provisioning and placement path, comparing the queueing model's predicted per-operator waiting times and P99 TTFT against measured values; if the P90 relative error exceeds the reported 1.9% on a trace not used for calibration, or if SLO attainment drops materially below the reported 97–98% at the same GPU count, the Poisson/Erlang-C foundation is the weak link.","tokens_in":27179,"feed_emoji":"⚙️","tokens_out":7698,"duration_ms":61319,"temperature":0.7,"pith_summary":"This paper tries to establish that the right unit of scaling for LLM serving is the individual operator, not the whole model replica. Model-level autoscaling is slow because loading a multi-billion-parameter model takes around ten seconds, and coarse because it duplicates every operator even when only one is the bottleneck. The paper builds OPSCALE, a cluster orchestration system that profiles each operator's sensitivity, computes per-operator replica counts with a queueing model, and packs replicas onto GPUs while accounting for interference. On production traces replayed across up to 40 A100s and 24 GB200s, the claimed payoff is meeting time-to-first-token SLOs with up to 36.3% fewer GPUs and 28% less power, or up to 44% higher token throughput under a fixed GPU budget. The central claim is that fine-grained elasticity is a practical, cluster-level efficiency win, not just a theoretical possibility.","feed_headline":"Scaling LLM parts, not whole models, saves up to 36% of GPUs","feed_subtitle":"Operator-level autoscaling meets latency SLOs while cutting GPU and power use in production traces.","key_machinery":"The load-bearing object is the operator-level deployment plan: for each operator $v$ in the model DAG, OPSCALE chooses batch size $B_v$, replica count $R_v$, tensor-parallel shards $P_v$, device assignment $A_v$, and SM allocation $S_v$. The core identity is the SLO decomposition $T^{itr}=\\sum_{v\\in V}(T_v+C_v+W_v)$, where each operator is modeled as an $M/M/R$ queue so $W_v$ follows the Erlang-C formula. A greedy critical-path algorithm adjusts $R_v$ and $B_v$ to push total latency under the SLO, and placement is done by a best-fit-decreasing bin-packing heuristic using a measured interference factor $I_{d,v}(b,p)=\\tilde{T}_v(b,p)/T_v(b,p)\\ge 1$ to reject placements that would inflate latency beyond the SLO. The machinery is what converts the heterogeneity observation into a concrete provisioning plan at millisecond timescales.","core_discovery":"The central discovery the paper argues for is operator heterogeneity: operators within one model differ widely in how their compute time, memory footprint, and SM usage respond to sequence length, batch size, and arrival rate, so scaling them uniformly wastes resources. OPSCALE treats the model as an operator DAG and scales only the operators that sit on the current critical path, using an SLO constraint of the form $T^{itr}=\\sum_{v}(T_v+C_v+W_v)$, with waiting $W_v$ from an Erlang-C queueing model. The paper reports that this yields SLO attainment of 98.4% for a dense model and 98.1% for a mixture-of-experts model, while model-level baselines achieve 84–97%, and that the savings grow with model size and operator heterogeneity. It further claims that op-level elasticity helps even in static deployments, extracting up to 44% more throughput from the same GPUs.","pith_inferences":["Editorial inference: prefill-decode disaggregation should multiply OPSCALE's gains, since the paper's own appendix shows prefill stages benefit two to three times more than decode stages; a split system could apply operator-level elasticity mainly to prefill.","Editorial inference: the savings should track operator heterogeneity across model families, so a testable prediction is that multimodal and mixture-of-experts models at the same parameter count will show larger GPU savings than dense text models; the paper's MoE results are consistent with this.","Editorial inference: the Poisson/$M/M/R$ assumption is the likeliest point of failure in a new deployment; swapping the Erlang-C waiting formula for a batch-aware or state-dependent arrival model would preserve the architecture while potentially tightening the savings estimate.","Editorial inference: if the mechanism generalizes, the same operator-DAG provisioning logic could apply to other generative model families whose operators also have heterogeneous sensitivity profiles."],"forward_implications":["Sub-second elasticity becomes the default: scaling only a bottleneck operator takes under half a second instead of the ten-plus seconds needed to load a full model replica.","GPU and power budgets for LLM serving can shrink by roughly a third at the same SLO attainment, with the largest gains on mixture-of-experts models and on clusters with fast interconnects.","Static GPU clusters can serve more tokens under the same latency SLO by directing the fixed budget to bottleneck operators; the paper reports up to 44% higher throughput at 40 GPUs.","Operator-level scaling is complementary to existing model-level autoscaling policies and to finer kernel-level spatial multiplexing, so it can be adopted as a mechanism beneath those policies."],"supporting_citations":[{"why":"supplies the production traffic traces, the burstiness measurements that motivate sub-second scaling, and the main model-level autoscaling baseline OPSCALE is compared against.","marker":"[57]"},{"why":"a utilization-based model-level autoscaling baseline used for cost and SLO comparisons.","marker":"[59]"},{"why":"a queue-driven model-level autoscaling baseline used for cost and SLO comparisons.","marker":"[20]"},{"why":"the GPU spatial-sharing primitive (SM allocation) that makes operator-level colocation and per-shard SM limits configurable.","marker":"[43]"},{"why":"multi-stream pipelined execution that OPSCALE adapts for operator-level pipelining and dispatch.","marker":"[69]"},{"why":"the attention kernel used in profiling, whose near-linear memory scaling shapes the memory-sensitivity characterization.","marker":"[12]"},{"why":"the Erlang-C formula that defines the per-operator waiting time in the queueing model.","marker":"[4]"},{"why":"documents the roughly ten-second model loading latency that motivates why operator-level scaling must be sub-second.","marker":"[16]"},{"why":"defines the prefill/decode iteration structure of LLM inference that the operator DAG model builds on.","marker":"[48]"}],"fun_headline_variants":["Scale only the ops you need: 36% fewer GPUs for LLM serving","Operator-level autoscaling cuts LLM GPU use by 36%","Per-operator scaling saves 36% GPUs and 28% power","Fine-grained LLM autoscaling: 44% more throughput or 36% less GPU","Scale LLM operators, not models, for 36% GPU savings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that each operator behaves as an independent $M/M/R$ queue with Poisson arrivals and exponential service times, even though the paper's own Figure 2 shows production traffic is bursty within ten-second windows.","fun_headline_variants_meta":{"raw":{"variants":["Scale only the ops you need: 36% fewer GPUs for LLM serving","Operator-level autoscaling cuts LLM GPU use by 36%","Per-operator scaling saves 36% GPUs and 28% power","Fine-grained LLM autoscaling: 44% more throughput or 36% less GPU","Scale LLM operators, not models, for 36% GPU savings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000262,"raw_usage":{"total_tokens":1605,"prompt_tokens":959,"completion_tokens":646,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":541}},"tokens_in":575,"tokens_out":646,"duration_ms":278865,"temperature":1.0,"reasoning_tokens":541,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:37:30.982521+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replay a held-out production trace through OPSCALE's provisioning and placement path, comparing the queueing model's predicted per-operator waiting times and P99 TTFT against measured values; if the P90 relative error exceeds the reported 1.9% on a trace not used for calibration, or if SLO attainment drops materially below the reported 97–98% at the same GPU count, the Poisson/Erlang-C foundation is the weak link.","supporting_citations":[{"cited_title":"DynamoLLM: Designing LLM inference clusters for performance and energy efficiency","cited_arxiv_id":null,"evidence_quote":"supplies the production traffic traces, the burstiness measurements that motivate sub-second scaling, and the main model-level autoscaling baseline OPSCALE is compared against."},{"cited_title":"vLLM Production Stack.https: //github.com/vllm-project/production-stack, 2025","cited_arxiv_id":null,"evidence_quote":"a queue-driven model-level autoscaling baseline used for cost and SLO comparisons."},{"cited_title":"NVIDIA Green Context Documentation","cited_arxiv_id":null,"evidence_quote":"the GPU spatial-sharing primitive (SM allocation) that makes operator-level colocation and per-shard SM limits configurable."},{"cited_title":"NanoFlow: Towards optimal large language model serving throughput","cited_arxiv_id":null,"evidence_quote":"multi-stream pipelined execution that OPSCALE adapts for operator-level pipelining and dispatch."},{"cited_title":"FlashAttention: Fast and memory-efficient exact attention with IO-awareness","cited_arxiv_id":null,"evidence_quote":"the attention kernel used in profiling, whose near-linear memory scaling shapes the memory-sensitivity characterization."},{"cited_title":"Internet and the Erlang formula.ACM SIGCOMM Computer Communication Review, 42(1):23–30, 2012","cited_arxiv_id":null,"evidence_quote":"the Erlang-C formula that defines the per-operator waiting time in the queueing model."},{"cited_title":"ServerlessLLM:low-latency serverless inference for large language models","cited_arxiv_id":null,"evidence_quote":"documents the roughly ten-second model loading latency that motivates why operator-level scaling must be sub-second."},{"cited_title":"Splitwise: Efficient generative LLM inference using phase splitting","cited_arxiv_id":null,"evidence_quote":"defines the prefill/decode iteration structure of LLM inference that the operator DAG model builds on."}],"review_version":1}