{"id":"486c7dbf-a264-4baa-b1fc-722d68feb12a","arxiv_id":"2504.18154","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A time-sliced, cyclically staggered scheduling strategy, partial disaggregation, is shown to improve SLO-aware goodput for 30B to 72B LLMs on commodity-interconnect clusters relative to co-located and fully disaggregated baselines.","lead":"EcoServe describes a way to run large language models on ordinary data-center networks: each GPU alternates between reading prompts and generating tokens, while a group of GPUs staggers those turns so at least one is always ready for new prompts. The authors report large throughput gains at controlled latency for 30B to 70B models on Ethernet clusters, but the headline numbers do not match the evaluation section.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SLO attainment that defines goodput is produced by a controller whose admission check is not validated for prediction error; this is the load-bearing unverified step.","rationale":"I read the paper as a systems design and empirical evaluation: the central claim is that PaDG with rolling activation raises the SLO-attaining goodput frontier on commodity clusters. The architecture (temporal disaggregation, rolling activation, macro instance, mitosis scaling) is coherent, the comparison against NoDG baselines is meaningful and not regime-stacked, and the reported P90 improvements over vLLM and Sarathi are substantial. The FuDG comparisons are less probative because DistServe is run in a configuration noted as unable to meet SLOs and MoonCake is run on 10Gbps Ethernet; the paper itself flags DistServe's limitation, which I credit. Two independent issues support CONDITIONAL rather than UNCHANGED: (a) the abstract's headline percentages (82.49, 86.17, 122.76, 126.96) do not match the body's P90 figures (83.76, 71.97, 192.41, 218.22), which needs reconciliation; (b) no code, data, or detailed configuration is released, and no error bars are reported, so the empirical claims cannot currently be checked. The single most load-bearing concern, however, is the admission controller in Algorithm 2: it relies on profiled prefill-duration predictions and a mean saved-TPOT safeguard, and the SLO attainment levels that define goodput are exactly what this controller produces. These assumptions are not sensitivity-tested. The reader's weakest_assumption identifies the same concern, and I concur. My recommended verdict is CONDITIONAL: the central idea is plausible and positively evidenced, but a full acceptance of the empirical claims requires a reproduction with prediction-error sensitivity analysis, per-request TPOT safeguards, released configurations, reconciled numbers, and a discussion of the uncited MuxServe overlap.","tokens_in":22471,"tokens_out":2176,"duration_ms":20229,"concrete_test":"Instrument Algorithm 2 in a reproduction of the L20 cluster experiments and inject controlled prefill-duration prediction errors (multiply profiled prefill times by 0.8, 0.9, 1.1, 1.2) on ShareGPT and LongBench workloads with the stated SLOs, then measure the resulting P90 and P99 TTFT and TPOT attainment curves. Additionally, replace the mean(saved_tpots) check with min(saved_tpots) and recompute the Figure 8 goodput numbers; if P99 attainment collapses or the headline improvements shrink substantially under +/-10 percent prediction error, the controller is not robust and the claimed goodput advantage is conditional on the accuracy of the profiling tables.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's headline claim is an SLO-attaining goodput improvement, so the goodput numbers are only meaningful if the admission controller actually enforces the stated TTFT and TPOT SLOs. Algorithm 2 assumes (1) that prefill durations for pending requests can be predicted from sequence-length profiling (line 5: prefill_times is 'predict pending_prefills durations'), (2) that the sum t_total over pending prefills is a valid upper bound on the time until a newly admitted request receives its first token, and (3) that mean saved TPOT over currently decoding requests is a sufficient safeguard for every admitted request's TPOT SLO, including at the P99 attainment levels reported. These are load-bearing and unvalidated. Prefill time is highly sensitive to sequence length, batch composition, and memory bandwidth; a 5-10 percent profiling error on a single long prompt can invalidate the TTFT check. Mean saved TPOT can be positive while many individual decodes have negative saved TPOT, so the TPOT SLO is not per-request guaranteed. The paper reports results on LongBench (average input 2686.89 tokens), the dataset where this risk is most acute. No sensitivity analysis, no profiling-error measurement, no percentile breakdown of saved TPOT, and no release of code or configuration are provided, so the measured P50/P90/P99 goodput comparisons rest entirely on this controller's predictions being accurate. The reader identified exactly this as the weakest assumption, and I agree that it is the most load-bearing unverified step in the argument.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"EcoServe proposes a partially disaggregated (PaDG) strategy for LLM serving on clusters with commodity interconnects. The key idea is temporal disaggregation, in which each instance alternates between prefill and decode phases, combined with rolling activation, in which instances are cyclically staggered so that a prefill-capable instance is usually available. The paper also contributes an adaptive scheduling algorithm for request admission and routing, and a mitosis scaling approach for elastic instance management. The evaluation compares EcoServe against vLLM, Sarathi, DistServe, and MoonCake on two clusters, three models, and three datasets, reporting large goodput improvements, with the caveat that NoDG baselines remain competitive on the Alpaca workload. The central claim is that PaDG raises the TTFT/TPOT/throughput trade-off frontier without requiring high-performance interconnects.","tokens_in":22590,"tokens_out":3159,"duration_ms":33842,"significance":"If the reported results hold, EcoServe addresses a real and timely problem: the cost and complexity of fully disaggregated serving on commodity hardware. The evaluation is broad and uses externally released baselines, and the paper is honest about cases where baselines remain competitive, such as Alpaca. The PaDG concept of phase-slicing within instances plus cyclic staggering is a plausible and potentially practical design point. However, the headline numbers are inconsistent between the abstract and the body, and the SLO attainment that defines goodput is produced by an admission controller whose prediction assumptions are not validated. The paper does not ship code or configuration details, which limits reproducibility of the quantitative claims.","major_comments":[{"comment":"The abstract reports average goodput improvements of 82.49%, 86.17%, 122.76%, and 126.96% over the four baselines, while Section 4.2 reports P90 improvements of 83.76%, 71.97%, 192.41%, and 218.22%, and P50 improvements of 36.49%, 19.82%, 180.73%, and 194.62%. The abstract numbers match none of these lists, and the body does not state which percentile, if any, the abstract uses. Because the abstract is the headline claim, this discrepancy must be resolved.","section":"Abstract vs. §4.2"},{"comment":"The admission controller is load-bearing for the goodput comparison, but its prediction assumptions are unvalidated. Algorithm 2 admits a request only if the sum of predicted prefill durations t_total does not exceed SLO_TTFT (lines 5–7) and if mean saved TPOT exceeds t_total (lines 13–17). Prefill durations are obtained by profiling with no reported prediction-error measurement, and t_total is treated as a bound on time-to-first-token without accounting for batching effects, phase-switch overhead, or decode preemption. More importantly, using mean saved TPOT does not enforce a per-request TPOT SLO: the mean can be positive while individual decodes have negative saved TPOT, so the reported P99 attainment is not guaranteed. No sensitivity analysis, profiling-error characterization, or percentile breakdown of saved TPOT is provided, and the risk is highest on LongBench, which has long prompts.","section":"§3.4, Algorithm 2"},{"comment":"The end-to-end comparison reports no variance or repetition information. Figure 8 shows single throughput values at each SLO attainment level without error bars, and the text does not state how many trials were run or how request-rate sweeps were repeated. Given that the central claim is a quantitative goodput improvement, the absence of variance information makes it hard to judge whether the reported differences are within run-to-run noise.","section":"§4.2, Figure 8"},{"comment":"The LongBench comparison against FuDG systems excludes Llama-30B results because of execution failures, and the text says the improvement 'would be higher' as a result. This is disclosed, but the exclusion should be quantified or the reported average should be recomputed with a clearly stated policy. As written, the LongBench FuDG improvement of 164.42% is not directly comparable to the other application-level numbers.","section":"§4.2, Comparison Across Applications"}],"minor_comments":[{"comment":"The body reports P50 and P90 improvement percentages but never gives the corresponding P99 aggregate numbers, even though P99 attainment is discussed qualitatively. Please add the P99 numbers or state why they are omitted.","section":"§4.2"},{"comment":"The hyperparameters N_l and N_u are only specified for the dynamic-scaling experiment (N_l=4, N_u=16). The main evaluation in Section 4.2 does not state N_l, N_u, the prefill/decode phase window duration, or how these are chosen. These are free parameters listed in the design and should be reported for reproducibility.","section":"§4.3.2"},{"comment":"The paper states that no code or configuration is released. Given the strong quantitative claims and the custom admission controller, a public artifact would substantially strengthen the paper.","section":"§4.1"},{"comment":"There are several typos and grammar issues, e.g., 'requring' in §2.4.2, 'approximatebly' in §4.1, and 'processed intermediately' in §3.2.2. A careful proofread is needed.","section":"Throughout"},{"comment":"Reference [37] is listed as '1911' in the bibliography; the year should be corrected to 2019.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper has a genuinely interesting core idea and a broad evaluation, but the two issues I view as load-bearing are the unresolved mismatch between the abstract and body numbers and the unvalidated admission-control predictions that define SLO attainment. Both are fixable in revision. I would not reject the paper, but I would require the authors to reconcile the headline percentages and provide sensitivity analysis for Algorithm 2 before considering it for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"EcoServe has a real idea: time-slice prefill and decode inside each instance, then stagger instances cyclically so prefills are always available. That combination—PaDG—avoids KV-cache transfer, works on commodity Ethernet, and turns a group of instances into a single schedulable macro unit. The evaluation is broad and mostly honest: three models, three datasets, two clusters, four baselines, and they admit Alpaca cases where NoDG matches. The phase-switching component of TTFT is a genuinely useful observation.\n\nThe soft spots are where the quantitative claims rest. First, the abstract's four improvement figures (82.49%, 86.17%, 122.76%, 126.96%) do not match the P90 numbers in Section 4.2 (83.76%, 71.97%, 192.41%, 218.22%). That is a discrepancy that needs an explanation. Second, the SLO attainment that defines goodput is produced by EcoServe's own admission controller (Algorithm 2). It predicts prefill durations from profiling and uses mean saved TPOT across decoding requests to decide if a new prefill can be admitted. Prefill time is sensitive to sequence length and batch composition; mean saved TPOT can be positive while individual requests miss their TPOT SLO. No sensitivity analysis, no profiling-error measurement, and no percentile distribution of saved TPOT are given. LongBench, with its long prompts, is exactly where this risk is highest. Third, no code, data, or configurations are released, and there are no error bars. The NoDG comparisons (72–84% at P90) are the strongest evidence because they are not regime-stacked; the FuDG comparisons are less convincing since DistServe cannot meet SLOs on the 10Gbps Ethernet EcoServe was designed for.\n\nThere is also a citation gap: MuxServe (ISCA 2024) already does spatial-temporal multiplexing of prefill and decode on shared GPUs, and the paper does not discuss it. That does not kill the contribution, but it needs to be positioned.\n\nWho this paper is for: systems people building LLM serving on commodity clusters, and anyone working on the TTFT/TPOT trade-off. The idea deserves a serious referee. The flaws are addressable, not fatal.\n\nRecommendation: send to peer review. Require the authors to reconcile the reported numbers, add sensitivity analysis for the admission controller (or at least measure profiling error), release artifacts or detailed configs, and discuss MuxServe. If they do that, the empirical claim will be substantially stronger.","headline":"EcoServe's temporal disaggregation plus rolling activation is a plausible and useful scheduling idea, but the headline numbers do not match the body and the SLO-enforcing admission controller is unvalidated.","tokens_in":23354,"tokens_out":3116,"would_cite":true,"duration_ms":29716,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Time-slicing prefill and decode lifts LLM goodput on ordinary clusters.","keywords":["LLM serving","prefill-decode disaggregation","temporal disaggregation","rolling activation","macro instance","goodput","SLO attainment","commodity interconnects"],"falsifier":"Run EcoServe on a 50/50 mix of short (~100-token) and long (~4000-token) prompts at the same mean arrival rate, with the profiled prompt-duration table frozen, and count what fraction of long prompts exceed their time-to-first-token SLO; if that fraction rises faster than the saved per-token buffer can cover, the admission controller's core guarantee fails and the reported P90 goodput gap over DistServe and MoonCake should shrink or disappear.","tokens_in":22056,"feed_emoji":"⚡","tokens_out":10640,"duration_ms":95504,"temperature":0.7,"pith_summary":"This paper tries to establish that LLM serving does not have to choose between keeping the initial prompt-processing pass and the token-by-token generation pass in one instance (fast but interference-prone) or splitting them onto separate machines linked by fast interconnects (clean but expensive). It proposes a middle strategy, partial disaggregation (PaDG), in which each instance alternates between longer prompt-processing and generation time slices, and a group of instances is staggered so that some are always ready to accept a new prompt, with no KV cache—the cached key-value attention state—shipped across the network. The basic serving unit is a macro instance, and an adaptive scheduler admits requests only when the predicted prompt-processing time fits the time-to-first-token SLO and the spare generation time built up by existing requests can absorb the interruption. On a 32-GPU L20 cluster connected by ordinary 10Gbps Ethernet, the paper reports P90 goodput gains of 83.76% over vLLM, 71.97% over Sarathi, 192.41% over DistServe, and 218.22% over MoonCake (the abstract states overall averages of 82.49%, 86.17%, 122.76%, and 126.96%). If true, this would move the latency-throughput trade-off frontier for the many clusters that do not own InfiniBand or NVLink.","feed_headline":"Time-slicing prefill and decode lifts LLM goodput on ordinary clusters","feed_subtitle":"On 32 GPUs with plain Ethernet, EcoServe reports 72-218% goodput gains over four baselines.","key_machinery":"The load-bearing object is the macro instance: a set of GPU instances whose prompt-processing and generation phases are offset so that, at any moment, at least one member is accepting a new prompt while the others generate tokens. Inside an instance, temporal disaggregation fixes phase boundaries in time; across instances, rolling activation fixes the rotation; the constraint checker decides admission by comparing the predicted total prompt-processing time with the time-to-first-token SLO and with the mean saved per-token time accumulated by existing generations. The mitosis scaling layer then grows or shrinks a macro instance one GPU at a time and, at thresholds, splits or merges macro instances using a serializable proxy object, so capacity changes do not interrupt running generations.","core_discovery":"The central claim is that separating the prompt-processing and generation phases across time—rather than across hardware—captures most of the benefit of full disaggregation without its network dependency. In EcoServe, each instance runs only one phase at a time for an extended period, eliminating the fine-grained interference that hybrid batching creates; rolling activation then cycles a macro instance's members through their prompt-processing slots so that an arriving request is always routed to an instance that can start immediately. Because the phases still share the same GPU and model weights, no KV cache moves between instances, removing the cost and complexity that sinks fully disaggregated designs on commodity interconnects. The system's admission check is the mechanism that converts this schedule into SLO compliance: it estimates whether all pending prompt-processing work finishes within the time-to-first-token bound, and whether the mean spare time accumulated by token-generation batches that run faster than their per-token target covers that work, refusing admission otherwise. On the paper's testbed this yields a higher sustainable request rate at P50, P90, and P99 SLO attainment than four representative baselines, with the largest margins over the fully disaggregated systems on long-prompt workloads and on models with large KV caches.","pith_inferences":["A natural stress test the paper does not run is to freeze the profiled prompt-duration table and serve a heavy-tailed mix of short and long prompts; if time-to-first-token violations rise faster than the saved per-token buffer can absorb, the admission controller, not the phase schedule, is the real capacity ceiling.","The same time-slicing idea could be pushed one level deeper, for example alternating attention and feed-forward computation inside an instance for mixture-of-experts models, a direction the paper's discussion of module-level disaggregation already gestures toward.","If the reported gains replicate, cost comparisons of serving architectures should shift from interconnect choice to phase-scheduling policy, because a software-only schedule would capture much of what full disaggregation buys with hardware."],"forward_implications":["Clusters without high-performance interconnects can run tight-SLO LLM serving at throughputs the paper says previously required InfiniBand- or NVLink-class fabric.","The advantage grows as SLOs tighten: reported throughput gaps over the baselines widen from P50 to P90 and P99 SLO attainment.","Because instances never transfer KV cache, tensor and pipeline parallelism avoid PCIe contention and pipeline bubbles, so one system can serve 30B and 70B models on commodity nodes.","Fine-grained elastic scaling is possible by adding or removing single instances inside a macro instance, with instance migration overhead under 100 ms in the reported measurements.","PaDG is positioned as the middle ground for 30B-130B models; the paper says small models gain little from it and ultra-stringent SLOs may still require full disaggregation."],"supporting_citations":[{"why":"Serves as the NoDG baseline with prefill-priority separate batching and as EcoServe's single-device runtime.","marker":"[5]"},{"why":"Provides the Sarathi NoDG baseline using hybrid batching and chunked prefill, the main interference-mitigation technique EcoServe compares against.","marker":"[9]"},{"why":"Provides the MoonCake inter-node FuDG baseline whose KV-cache pool and InfiniBand dependence EcoServe aims to avoid.","marker":"[35]"},{"why":"Provides the DistServe intra-node FuDG baseline that transfers KV cache over NVLink, a cost EcoServe removes.","marker":"[50]"},{"why":"Introduces continuous batching, the execution model underlying the NoDG strategy that EcoServe extends with temporal phase separation.","marker":"[48]"},{"why":"Introduces phase splitting into prefill and decode instances, defining the FuDG strategy family EcoServe contrasts with.","marker":"[33]"},{"why":"Supplies PagedAttention, the memory management EcoServe's vLLM-based runtime relies on for KV cache.","marker":"[24]"}],"fun_headline_variants":["Temporal disaggregation boosts LLM serving on cheap Ethernet","Rolling activation for cost-effective LLM serving on plain Ethernet","Partial disaggregation: 82-127% goodput gains on commodity Ethernet","EcoServe: time-sliced prefill/decode lifts LLM goodput"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the system can predict from profiling how long a new prompt's first computation pass will take, accurately enough that the sum of pending predictions stays under the time-to-first-token limit, and that the average spare time built up by requests already generating tokens is enough to protect every request's per-token speed target; if those predictions are wrong—especially on very long or mixed-length prompts—accepted requests miss their latency targets and the measured goodput advantage changes.","fun_headline_variants_meta":{"raw":{"variants":["Temporal disaggregation boosts LLM serving on cheap Ethernet","Rolling activation for cost-effective LLM serving on plain Ethernet","Partial disaggregation: 82-127% goodput gains on commodity Ethernet","EcoServe: time-sliced prefill/decode lifts LLM goodput"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00098,"raw_usage":{"total_tokens":4240,"prompt_tokens":1104,"completion_tokens":3136,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":720,"completion_tokens_details":{"reasoning_tokens":3056}},"tokens_in":720,"tokens_out":3136,"duration_ms":21358,"temperature":1.0,"reasoning_tokens":3056,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:24:28.205289+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run EcoServe on a 50/50 mix of short (~100-token) and long (~4000-token) prompts at the same mean arrival rate, with the profiled prompt-duration table frozen, and count what fraction of long prompts exceed their time-to-first-token SLO; if that fraction rises faster than the saved per-token buffer can cover, the admission controller's core guarantee fails and the reported P90 goodput gap over DistServe and MoonCake should shrink or disappear.","supporting_citations":[{"cited_title":"vllm: Easy, fast, and cheap llm serving for everyone","cited_arxiv_id":null,"evidence_quote":"Serves as the NoDG baseline with prefill-priority separate batching and as EcoServe's single-device runtime."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Sarathi NoDG baseline using hybrid batching and chunked prefill, the main interference-mitigation technique EcoServe compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MoonCake inter-node FuDG baseline whose KV-cache pool and InfiniBand dependence EcoServe aims to avoid."},{"cited_title":"2024.{DistServe}: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving","cited_arxiv_id":null,"evidence_quote":"Provides the DistServe intra-node FuDG baseline that transfers KV cache over NVLink, a cost EcoServe removes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces continuous batching, the execution model underlying the NoDG strategy that EcoServe extends with temporal phase separation."}],"review_version":1}