{"id":"92b023c5-51ec-4656-b8c3-2d53f60c944c","arxiv_id":"2507.06608","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Nexus performs proactive intra-GPU disaggregation of prefill and decode, using an analytical cost model and greedy search to dynamically partition SMs, achieving up to 2.2x throughput gains over vLLM.","lead":"A new LLM serving system called Nexus splits one GPU's compute between the prefill and decode stages, dynamically reallocating resources and using a cost model to plan ahead. In benchmarks on commodity GPUs it reports up to 2.2x higher throughput and much lower first-token and per-token latencies than vLLM.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cost model calibration is described as per-operator but the paper states parameters are extracted from full-stage end-to-end measurements; that procedure cannot yield per-operator Rsat/λ without hidden assumptions, and no validation of predicted vs measured concurrent latency is given.","rationale":"The reader identified the cost model and its calibration as the weakest assumption; I agree that this is the load-bearing point. My reading sharpens the concern in two ways that are directly testable. First, there is an internal inconsistency: the model is presented as operator-level (Eq. 5–7), but the calibration procedure described in §4.1.1 and §5 uses full-stage end-to-end latency measurements. Extracting per-operator saturation parameters from a single summed curve is underdetermined unless additional assumptions or kernel-level profiling are used but not reported. If the model is actually stage-level, the claimed ability to capture shifting bottlenecks (e.g., decode attention flipping between compute- and memory-bound) is not justified. Second, calibrating 'for each model and workload configuration' and then evaluating on those same configurations means the adaptive controller is tested in-sample; this conflates model fitting with proactive generalization. A disjoint calibration/evaluation split would settle whether the SM partition choices, rather than the SPF scheduler or other engineering details, are responsible for the reported gains. I do not think this requires rejection: the system is implemented and ablated, and the concerns are addressable with artifact release, a transparent calibration procedure, and a hold-out workload test. The reader's CONDITIONAL verdict remains appropriate, so the verdict is unchanged.","tokens_in":23898,"tokens_out":4255,"duration_ms":51774,"concrete_test":"Re-run the single-GPU end-to-end evaluations (§6.2) with cost-model parameters calibrated on a disjoint workload (e.g., calibrate only on Long Data Collections, then evaluate on Mixed workload) and compare dynamic SM partitioning against a static equal split on the same hardware. Additionally, attempt to recover per-operator Rsat/λ values from the described full-stage measurements alone; if the dynamic controller no longer beats the static split under disjoint calibration, or the per-operator parameters cannot be reproduced without kernel-level profiling, the proactive-adaptation claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the greedy SM search selecting near-optimal partitions via the cost model (Eq. 7–9). Section 4.1.1 says 'We extract Rsat and λ per operator from end-to-end measurements of the full stage (prefill or decode) under varying SM allocations.' A single stage-level latency curve cannot identify distinct per-operator parameters: many combinations of Rsat_i and λ_i yield the same summed latency. Either the profiler uses unstated per-kernel instrumentation, or the 'operator-level' model is actually a stage-level fit with per-operator claims that are not substantiated. Moreover, the memory contention model (Eq. 8–9) assumes full bandwidth saturation and proportional bandwidth sharing; the paper never reports predicted-vs-measured latency under concurrent prefill and decode execution. Section 5 states calibration is done 'for each model and workload configuration,' meaning the same workload distributions used in evaluation also set the model parameters. The reported 2.2x/20x/2.5x gains may therefore reflect fitting the controller to the exact evaluation traces rather than proactive adaptation to unseen conditions. Without a validated cost model, the causal attribution of improvements to dynamic SM repartitioning is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Nexus, a vLLM-based LLM serving engine that splits a single GPU's SMs between prefill and decode phases, executes the two phases concurrently on separate CUDA streams, and repartitions SMs dynamically using a two-regime saturation cost model (Eq. 7) with a memory-bandwidth contention term (Eqs. 8–9), a greedy partition search (Algorithm 1), and phase-specific SPF/FCFS schedulers. The paper claims up to 2.2x higher throughput, 20x lower TTFT, and 2.5x lower TBT than vLLM, up to 2x improvements over SGLang, and competitive or better performance than a two-GPU disaggregated vLLM. Evaluation covers three workloads, Qwen2.5-3B and LLaMA3.1-8B on one GPU and Qwen2.5-14B on two GPUs, plus ablations, offline makespan, and a latency breakdown.","tokens_in":24251,"tokens_out":7802,"duration_ms":71138,"significance":"If the cost-model validation gap is closed, this is a meaningful contribution: it offers a plausible path to prefill-decode isolation without extra GPUs, which is relevant to cost-sensitive LLM serving. The implementation is substantial (roughly 6K lines plus a CUDA extension), and the evaluation is broad in terms of workloads, model sizes, and single/dual-GPU configurations. The ablation study is well designed and supports the qualitative claim that both dynamic SM switching and SPF scheduling contribute to the end-to-end results. The paper also reports useful secondary evidence in the form of offline makespan and overhead breakdown. However, the central 'proactive' claim rests on a cost model whose calibration procedure is under-specified and not validated against concurrent execution, which is a load-bearing weakness.","major_comments":[{"comment":"The paper states that R_sat and λ are extracted 'per operator' from 'end-to-end measurements of the full stage (prefill or decode) under varying SM allocations,' but a single stage-level latency curve T_stage(r) = Σ_i T_i(r) cannot uniquely determine the per-operator parameters (R_sat,i, λ_i). Figure 5 shows that per-kernel breakdowns were measured, so a per-operator fit is plausible, but the text never describes the fitting procedure, including whether per-kernel timings were collected and what objective/constraints were used. Since Algorithm 1 and the 'proactive' adaptation claims depend on this model, the paper should either specify the per-operator fitting method or explicitly downgrade the model to stage-level, and should include a predicted-versus-measured validation for concurrent prefill/decode execution.","section":"§4.1.1, Eq. (7), §5"},{"comment":"Section 5 says λ is profiled 'for each model and workload configuration,' while the Related Work section asserts that Nexus 'uses a one-time profiling pass... avoiding offline workload tracing.' These statements contradict each other. If parameters are refit per workload distribution and the same three workload distributions from Table 1 are then used in Section 6, the reported gains may reflect calibration on the evaluation traces rather than proactive adaptation to unseen conditions. Please clarify precisely which quantities are workload-dependent in the profile and demonstrate generalization, for example by profiling on one workload and evaluating on a held-out workload, or by showing that the fitted parameters are stable across workload distributions.","section":"§5 vs. §7"},{"comment":"The memory-contention model assumes full bandwidth saturation during each overlap window and that bandwidth splits between prefill and decode in proportion to their memory traffic. Section 3.3 demonstrates qualitatively that prefill KV length affects decode latency, but no experiment validates the quantitative predictions of Eq. (9) under concurrent execution. Because the greedy controller adjusts SM ratios based on this model, the absence of a predicted-versus-measured comparison makes the causal attribution of the end-to-end gains to the cost model unsupported. Please add a validation experiment that varies prefill load, decode batch size, and SM split, and report the model's prediction error.","section":"§4.1.1, Eqs. (8)–(9)"},{"comment":"The evaluation reports no error bars, run counts, or confidence intervals. The headline numbers (up to 20x lower TTFT, 2.2x throughput, 2.5x lower TBT) are point estimates; without knowing the number of runs and the run-to-run variance, it is difficult to judge whether the improvements are robust. Please report the number of independent runs and either standard deviations or confidence intervals for the key metrics, at least for the largest claimed gains.","section":"§6.1, Figures 9–10"}],"minor_comments":[{"comment":"FastServe is a reimplementation; please report how parity with the original system was validated (for example, matching published throughput or latency numbers) or soften the conclusions drawn from this baseline.","section":"§6.1"},{"comment":"The third row of Figure 9 says 'Llama-3-1.8B', but Section 6.1 and the text use 'LLaMA3.1-8B'; the figure should be corrected.","section":"Figure 9 caption"},{"comment":"The phrase 'Green Contexts provide logical SM isolation' should refer to 'CUDA Green Context' for consistency with Section 5; the plural form is confusing.","section":"§4.2"},{"comment":"There are several typos that should be fixed, including 'bottneleck' (§6.5), 'Particularily' and 'employee' (§4.3), 'FIgure' (§6.2.2), 'repsectively' (§6.3), and 'acheives' (§6.3).","section":"§6.5 and elsewhere"},{"comment":"The sentence 'The maximum batch size and chunk size for prefill of Nexus are same as those of vLLM..' contains a doubled period and should be rephrased.","section":"§5"},{"comment":"The motivation experiments use NVIDIA MPS to control SM partitioning, while the implemented system uses CUDA Green Context; the paper should state whether the two mechanisms have equivalent partitioning granularity and runtime overhead, since the diminishing-returns evidence is collected with MPS.","section":"§3.2–3.3"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the cost-model calibration. If the authors can describe the per-operator fitting procedure, resolve the contradiction between per-workload profiling and 'one-time profiling', and add a held-out or predicted-versus-measured validation, I would support acceptance. Without those changes, the central proactive-adaptation claim is not fully established. The paper fits the journal's scope and is otherwise well within the normal revision effort for a systems paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Nexus is worth a serious referee. The reported 1.4–2.2x throughput gains over vLLM, SGLang, and disaggregated vLLM on the same GPU are plausible, and the design is coherent. But the central cost model has a calibration identification problem that the authors should fix, and the evaluation would be much stronger with code, data, and predicted-vs-measured latency checks.\n\nWhat is actually new: intra-GPU disaggregation itself is not new — Drift, Bullet, and semi-PD are cited and discussed. Nexus adds a contention-aware analytical cost model that explicitly captures memory bandwidth interactions between prefill and decode, a dual-objective optimization solved via a fast greedy SM search, and phase-specific schedulers. The ablation cleanly separates the effect of dynamic SM switching from SPF scheduling, which is good discipline. The comparison set is appropriate: vLLM, SGLang, disaggregated vLLM, and a reimplemented FastServe, across three workloads, two model families, and single- and dual-GPU setups.\n\nThe soft spots are real. Section 4.1.1 says Rsat and lambda are extracted per operator from end-to-end measurements of the full stage. A single stage-level latency curve cannot identify distinct per-operator parameters — many combinations produce the same summed latency. Either there is unstated per-kernel instrumentation, or the operator-level claim is not substantiated. Section 5 also says calibration is done for each model and workload configuration, and the evaluation uses those same configurations, so the gains may partly reflect fitting to the test domain rather than proactive adaptation. There is no predicted-vs-measured latency validation under concurrent prefill and decode execution, no error bars or run counts, no code or data release, and FastServe is a reimplementation with no public code to check.\n\nThese issues are not fatal to the design — the qualitative diminishing-returns and memory-contention observations are well motivated — but they mean the headline numbers should not be taken at face value. A reviewer should push for a disjoint profiling/evaluation split, per-kernel profiling details, and an artifact.\n\nThis is a paper for systems folks working on LLM serving. A reader wanting to know whether proactive SM repartitioning can beat monolithic batching will find a thoughtful design and a plausible yes. It deserves peer review, but acceptance should hinge on the authors addressing the calibration identification problem and providing independent validation.","headline":"A well-scoped systems paper with plausible large gains, but the cost model's calibration procedure is under-specified and the evaluation lacks independent validation, so the results need verification.","tokens_in":24717,"tokens_out":1871,"would_cite":true,"duration_ms":20480,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One GPU can give LLM serving the latency isolation of disaggregation and the utilization of a monolithic engine — by splitting its compute cores between prefill and decode on the fly.","keywords":["LLM serving","prefill-decode disaggregation","intra-GPU resource partitioning","dynamic SM allocation","analytical cost model","time-to-first-token","time-between-tokens","chunked prefill interference"],"falsifier":"Run the Long Data Collections workload on one L20 GPU with Qwen2.5-3B, log the SM ratio the controller chooses each iteration, and compare the cost model's predicted prefill and decode latencies with the measured latencies under concurrent execution; a systematic gap — especially decode TBT rising faster than predicted as prefill KV length grows — would show the offline-calibrated model fails to transfer to runtime conditions.","tokens_in":23717,"feed_emoji":"⚡","tokens_out":10048,"duration_ms":88886,"temperature":0.7,"pith_summary":"The paper claims that the two phases of LLM inference — compute-bound prefill and memory-bound decode — can be disaggregated inside a single GPU by dynamically partitioning the GPU's streaming multiprocessors (its kernel-executing compute units), without the hardware cost of engine-level disaggregation. It reports that both stages show diminishing returns beyond a saturation point, so each phase needs only the compute it actually uses, and that memory-bandwidth contention, not compute, is the main residual interference. Nexus predicts per-operator latency with an analytic cost model, picks SM partitions with a fast greedy search, and runs phase-specific schedulers. On Qwen and LLaMA models, the paper reports up to 2.2x higher throughput, 20x lower time-to-first-token, and 2.5x lower time-between-tokens than vLLM, while matching or exceeding two-GPU disaggregated serving.","feed_headline":"One GPU matches two-GPU LLM serving by splitting prefill from decode","feed_subtitle":"Nexus dynamically splits one GPU between LLM prefill and decode, cutting TTFT by 20x and boosting throughput 2.2x.","key_machinery":"The load-bearing mechanism is the runtime SM partition chosen by a greedy search over an analytic cost model. The model sums per-operator latencies, each the max of a compute term and a memory term; compute latency follows a two-regime saturation-decay curve, $T^{\\mathrm{compute}}_o(c_o,r) = c_o/(rC)$ below a saturation ratio $R_{\\mathrm{sat}}$ and $c_o/(R_{\\mathrm{sat}}C)\\cdot(1+\\lambda(r-R_{\\mathrm{sat}}))$ above it, with $R_{\\mathrm{sat}}$ and $\\lambda$ calibrated by offline profiling per model and workload. Decode memory latency is modeled by estimating the probability that decode overlaps prefill attention and then dividing decode's memory bytes by the bandwidth share it gets under assumed full saturation. The controller switches between decode-prioritized and prefill-prioritized objectives based on live KV-cache usage, the greedy search converges in 2–4 cost-model evaluations, and a hysteresis buffer filters out SM-ratio changes smaller than $\\delta$.","core_discovery":"Nexus's central claim is that intra-engine prefill-decode disaggregation, backed by a proactive cost model, achieves what monolithic and disaggregated designs each achieve only partially: monolithic chunked prefilling mixes compute-heavy prefill and memory-bound decode in one batch, causing fine-grained phase interference that inflates time-between-tokens, while engine-level disaggregation removes interference but wastes hardware and adds coordination overhead. Nexus instead runs prefill and decode as separate concurrent streams on one GPU, with a runtime controller that splits streaming multiprocessors between them. The split is chosen by a lightweight cost model that encodes two measured effects — per-operator latency saturating as SM ratio grows, and decode latency rising with prefill memory traffic — and by a greedy search that solves a dual-objective problem: prioritize prefill when KV-cache usage is low and decode when it is high, keeping the other phase within a slowdown budget. With phase-specific scheduling (shortest-prompt-first for prefill, FCFS for decode) and a hysteresis buffer that suppresses flapping, the paper claims Nexus matches the latency isolation of disaggregated systems and the utilization of monolithic systems on a single commodity GPU.","pith_inferences":["The offline-calibrated cost model suggests a natural extension: online calibration, in which measured iteration latencies continuously update $R_{\\mathrm{sat}}$ and $\\lambda$, would make the proactive controller adaptive without per-workload profiling.","The diminishing-returns result is more general than prefill/decode: any GPU co-tenant pair where one phase is compute-bound and the other memory-bound could use the same saturation-aware SM split.","Because the contention model assumes full bandwidth saturation, partition choices under low memory pressure may be conservative; a traffic-based contention model could shift more SMs to decode when actual bandwidth use is low.","The paper evaluates on three workloads; a stress test with rapidly switching prompt-length distributions would reveal whether the controller's KV-cache-triggered mode switching responds quickly enough under adversarial mixes."],"forward_implications":["If the cost model holds under concurrent execution, one GPU can deliver disaggregation-level time-to-first-token and time-between-tokens without a second engine, cutting hardware cost for latency-sensitive LLM serving.","Chunked prefill's interference is not inherent: separating phases into concurrent streams and giving decode enough SMs removes the fine-grained TBT inflation measured in mixed batches.","Phase-specific scheduling becomes practical once phases are decoupled: shortest-prompt-first prefill cuts TTFT substantially, while the decode scheduler can remain simple FCFS.","Dynamic SM reallocation can respond to workload shifts on sub-second timescales, since the greedy search needs only a few closed-form cost-model evaluations.","The same mechanism works across multiple GPUs: on two GPUs, the paper reports 2.2x throughput over vLLM and 2x over SGLang with better TTFT and TBT for a 14B model."],"supporting_citations":[{"why":"vLLM is the code base Nexus extends and the primary throughput/latency baseline.","marker":"[25]"},{"why":"Sarathi-Serve's chunked prefill is the monolithic technique whose batch-level interference motivates intra-engine separation.","marker":"[1]"},{"why":"DistServe provides the engine-level disaggregated baseline whose TTFT/TBT isolation Nexus aims to match on one GPU.","marker":"[67]"},{"why":"Splitwise supplies the core observation that prefill is compute-bound and decode is memory-bound.","marker":"[42]"},{"why":"CUDA Green Contexts is the GPU primitive Nexus uses to change SM partitions at runtime.","marker":"[35]"},{"why":"SGLang is the latency-optimized serving baseline Nexus compares against.","marker":"[66]"},{"why":"Bullet is the prior intra-GPU disaggregation system whose reactive tuning Nexus contrasts with proactive cost-model control.","marker":"[26]"},{"why":"Drift is the closest prior intra-GPU disaggregation work, which Nexus distinguishes by its contention-aware model and greedy search.","marker":"[9]"}],"fun_headline_variants":["One GPU matches two-GPU LLM serving by splitting prefill from decode","Proactive intra-GPU split of prefill/decode boosts LLM throughput 2.2x","Nexus splits one GPU between prefill and decode, cutting TTFT 20x","Intra-GPU disaggregation: prefill and decode on one GPU, no interference","One GPU, two streams: proactive PD disaggregation cuts TTFT 20x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The cost model's saturation and decay parameters are extracted from offline profiling for each model and workload configuration, and the memory-contention equations assume full bandwidth saturation; if those calibrated numbers do not match the latencies actually observed under concurrent prefill-decode execution, the chosen SM partitions will be off and the claimed gains will shrink.","fun_headline_variants_meta":{"raw":{"variants":["One GPU matches two-GPU LLM serving by splitting prefill from decode","Proactive intra-GPU split of prefill/decode boosts LLM throughput 2.2x","Nexus splits one GPU between prefill and decode, cutting TTFT 20x","Intra-GPU disaggregation: prefill and decode on one GPU, no interference","One GPU, two streams: proactive PD disaggregation cuts TTFT 20x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00023,"raw_usage":{"total_tokens":1537,"prompt_tokens":1057,"completion_tokens":480,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":673,"completion_tokens_details":{"reasoning_tokens":366}},"tokens_in":673,"tokens_out":480,"duration_ms":5063,"temperature":1.0,"reasoning_tokens":366,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:58:43.458035+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Long Data Collections workload on one L20 GPU with Qwen2.5-3B, log the SM ratio the controller chooses each iteration, and compare the cost model's predicted prefill and decode latencies with the measured latencies under concurrent execution; a systematic gap — especially decode TBT rising faster than predicted as prefill KV length grows — would show the offline-calibrated model fails to transfer to runtime conditions.","supporting_citations":[{"cited_title":"Gulavani, Alexey Tumanov, and Ramachandran Ramjee","cited_arxiv_id":null,"evidence_quote":"Sarathi-Serve's chunked prefill is the monolithic technique whose batch-level interference motivates intra-engine separation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DistServe provides the engine-level disaggregated baseline whose TTFT/TBT isolation Nexus aims to match on one GPU."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Bullet is the prior intra-GPU disaggregation system whose reactive tuning Nexus contrasts with proactive cost-model control."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Drift is the closest prior intra-GPU disaggregation work, which Nexus distinguishes by its contention-aware model and greedy search."}],"review_version":1}