{"id":"8bad0ba3-6956-470e-ba65-798d60ac14f9","arxiv_id":"2602.14516","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"AMPD adaptively routes incremental prefill tasks between prefill and decode workers and reorders queued prefill jobs to improve SLO attainment for multi-round LLM inference under prefill-decode disaggregation.","lead":"AMPD is a serving system that decides, at run time, where to run the prefill steps that reappear between rounds of a multi-round LLM conversation, and in what order, to hit latency goals more often. Read it if you run LLM inference infrastructure for agent-style workloads that alternate generation, tool calls, and retrieval.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ablation is confounded: deployment optimized for full AMPD may inflate AR+PR gains.","rationale":"The reader's weakest_assumption targets validation of the performance model/simulator. That is a legitimate reproducibility concern, but it is not the most load-bearing for the central design claim. Even if the model were perfectly accurate, the ablation would still be confounded by the deployment configuration: the planner optimizes the deployment for full AMPD, so disabling the online mechanisms leaves that variant with a suboptimal deployment. This concern is concrete, internal to the experimental design, and can be tested by re-planning deployments for the ablated variants. The end-to-end comparison to independently tuned baselines remains fair, so the CONDITIONAL verdict is still appropriate; however, the justification should shift toward the ablation confound and the need for a controlled ablation.","tokens_in":20106,"tokens_out":18929,"duration_ms":192584,"concrete_test":"Re-run the ablation of Figure 5 with two deployment configurations for each variant: (i) the current AMPD-optimal deployment from the planner, and (ii) a deployment re-optimized by the same ILP with adaptive routing and reordering disabled in the simulator (i.e., a remote-only policy). Compare full AMPD to the remote-only variant on its own optimal deployment. If the SLO improvement drops below the reported 44.47%–402%, the design claim that adaptive routing and reordering produce the gain is not established. Additionally, report the planner-chosen deployments for each variant to confirm whether they differ.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central design claim—that adaptive routing plus prefill reordering is what produces the SLO gains—rests on the ablation in §7.2 (Figure 5), which reports a 44.47%–402% improvement from these mechanisms. However, the ablation likely uses the single deployment configuration produced by the offline planner for full AMPD. The planner's ILP (§5) and simulator (§A.1) explicitly incorporate the adaptive routing policy from §4 when estimating P95 latency coefficients. The simulator 'implements the scheduling policy presented in §4.' Therefore, the chosen deployment is optimal for AMPD with adaptive routing and reordering enabled. When those mechanisms are disabled (AMPD w/o AR+PR, AMPD w/o PR), the same deployment is not necessarily optimal for a remote-only policy. For example, the planner may allocate fewer GPUs to prefill workers precisely because it expects some prefills to execute locally on decode workers; under remote-only, prefill workers become the bottleneck, artificially lowering the w/o AR+PR SLO. The reported ablation gain thus conflates the online scheduling mechanisms with a deployment choice that favors full AMPD. The paper does not state that deployments were re-optimized for the ablated variants. Because the headline empirical comparison (AMPD vs. tuned Dynamo/vLLM) is separate and fair, this concern does not necessarily invalidate the end-to-end result, but it directly undermines the attribution of the gain to adaptive routing and reordering, which the reader's strongest_claim includes.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AMPD, a disaggregated LLM serving framework targeting multi-round inference workloads. Its online stage consists of an adaptive routing mechanism that decides, per prefill task, whether to execute locally on the decode worker or remotely on a prefill worker, and a lookahead-based prefill reordering policy that reorders a small window of queued prefill tasks to maximize predicted TTFT SLO satisfaction. Its offline stage formulates deployment planning—choosing data-parallel and model-parallel configurations for prefill and decode workers under a GPU budget—as an ILP, with P95 latency coefficients obtained from a simulator. The system is implemented on NVIDIA Dynamo with Redis for shared metadata, NIXL for KV-cache transfer, and SCIP for ILP solving. Experiments compare AMPD against Dynamo, vLLM, and vLLM-Continuum over three models and four multi-round traces, reporting large SLO-attainment improvements; ablations and sensitivity studies support the individual online mechanisms.","tokens_in":20525,"tokens_out":5250,"duration_ms":57210,"significance":"If the reported results hold, AMPD makes a useful contribution: it identifies the interleaved prefill-decode pattern of multi-round inference as a first-class scheduling challenge for PD-disaggregated serving, and provides both an online adaptive routing/reordering mechanism and an offline deployment planner. The evaluation is broad—3 models, 4 traces, 5 arrival rates, 3 baselines—and includes hyperparameter sensitivity and a planner-ranking validation. The paper does not include code or machine-checked artifacts, so the reproducibility of the quantitative claims rests on the experimental description and on how completely the model and simulator are validated. The main risk is not the end-to-end comparison, which appears broadly sound, but the attribution of the gains to the specific online mechanisms and the indirect validation of the performance model that guides both routing and planning.","major_comments":[{"comment":"The ablation that attributes 44.47%–402% SLO improvement to adaptive routing plus prefill reordering is confounded with deployment optimization. The ILP in Eq. (5) and the simulator in App. A.1 explicitly incorporate the §4 adaptive routing and reordering policy when estimating τ_pre and τ_dec, so the single deployment used for full AMPD is chosen as optimal for the full system. When AR+PR or PR is disabled, that same deployment is not necessarily optimal. For example, the planner could allocate fewer GPUs to prefill workers precisely because local execution of incremental prefills is expected; disabling adaptive routing would then make prefill workers the bottleneck and inflate the apparent benefit of the mechanisms. The paper does not state whether deployments were re-optimized for the ablated variants. Please re-run the ablation with planner configurations that disable the component u","section":"§7.2 (Figure 5), §5, App. A.1"},{"comment":"Both online routing and offline planning rely on the profiled piecewise α-β model (T_pre, T_dec, T_kv) and on the simulator that produces P95 coefficients τ_pre(n) and τ_dec(n). Table 2 validates that the planner's top-3 configurations match the real system's ranking, but this does not validate that predicted latencies are accurate across the batching, concurrency, and parallelism regimes encountered under load. If the model systematically underestimates TTFT or ITL under load, both the routing decisions and the deployment plan could be wrong, while the evaluation could still pass because the same model guides the measured system. Please include a head-to-head comparison of predicted versus measured TTFT/ITL (or P95 latency) for representative configurations, covering varied concurrency and parallelism degrees.","section":"§3, §4.1 (Eq. 1–2), §5 (Eq. 5), App. A.1"},{"comment":"The pseudocode appears inconsistent with the surrounding text. Lines 2–5 place the local-execution check ('if dITL ≤ β·ITL_thres then return local') inside the 'for each prefill worker i' loop, so that the first prefill worker that is not TTFT-slack would trigger a local return before the remaining prefill workers are examined. The text says local execution is considered only after all prefill workers are under pressure. Since this is the core online algorithm, the exact control flow matters for reproducibility. Please fix the pseudocode indentation/control flow or clarify the intended semantics.","section":"§4.1, Algorithm 1"},{"comment":"The headline quantitative claims—'up to 967.54% and 3435.1%' and the average improvements—are reported as point estimates without error bars, repeated runs, or statistical significance. The request arrivals are Poisson and the scheduling decisions are online, so SLO attainment is a random quantity. A single run per configuration is insufficient to support precise percentage-level comparisons, especially at arrival rates where curves are close (e.g., Qwen3-32B GAIA at low load). Please report means/standard deviations over multiple seeds, or at least a stability check over repeated runs, for the main comparisons and for the ablation in Figure 5.","section":"§7.2 (Figures 4–6)"}],"minor_comments":[{"comment":"The arrows and percentage annotations on the bars are ambiguous: some entries read '↑543%' without stating the baseline to which the improvement refers. Please label each arrow with the baseline or move the baseline-specific claims to the text/table.","section":"Figure 4"},{"comment":"The label 'Llama3.1-70B · GAIA' appears twice in the Mixtral subplot row; the second occurrence should likely be 'Mixtral-8x7B · GAIA'.","section":"Figure 8"},{"comment":"Please state explicitly what 'real-system serving' ranking means in Table 2—is it measured by SLO attainment, P95 latency, or end-to-end latency? Also clarify how the top-3 configurations were selected from the larger configuration space.","section":"Table 2"},{"comment":"The protocol says baselines were 'tuned' and 'best results' reported, but the tuning grid (e.g., numbers of prefill/decode GPUs, tensor-parallel degrees) is not described. Please add this detail for reproducibility and to support the fairness claim.","section":"§7.1"},{"comment":"The abstract and introduction state 'brand new'; the paper would read more formally as 'new' or 'novel'. Also, reference [Li et al., 2025] is cited as 'vLLM-Continuum' in the experiments but as 'Continuum' in related work; please unify.","section":"§1/§7.1"}],"recommendation":"major_revision","confidential_remarks":"The end-to-end comparison against Dynamo and vLLM appears plausible and could be a solid empirical contribution. The main issue is the ablation attribution: the Figure 5 experiment needs deployments re-optimized for the ablated variants before the paper can claim that adaptive routing and prefill reordering are the cause of the gains. The performance-model validation gap (simulator coefficients used both in the planner and in the measured system) is also important. Neither issue invalidates the central claim, but both are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a serious systems paper, not a zero. The gap is real—multi-round, agent-style inference with interleaved incremental prefill under PD disaggregation hasn't been handled as a unit, and the paper's decomposition (adaptive local-vs-remote routing, prefill reordering, ILP deployment planner) is coherent and matches where the field is going. The evaluation is unusually broad: 3 models × 4 traces × 4 baselines, plus ablations and sensitivity. Credit where due: Table 2's planner-vs-real-system ranking check is a good idea, the implementation on Dynamo with RDMA KV transport is plausible, and the related-work survey is honest about the co-located vs. disaggregated split.\n\nNow the soft spots, in proportion. The real one is the ablation. The simulator explicitly implements the §4 scheduling policy when estimating P95 latency coefficients, so the ILP chooses a deployment that is optimal for full AMPD. When the authors disable AR+PR, they don't say they re-optimized the deployment for the ablated variant. If the planner allocated fewer GPUs to prefill workers because it expected some local execution, the \"w/o AR+PR\" variant is handicapped before the online policy even runs. That means the 44–402% ablation gains are likely inflated, and the causal story about adaptive routing being the key ingredient is weakened. The stress-test note is right; this is the single most important thing to fix.\n\nThat said, the end-to-end comparison (AMPD vs. tuned Dynamo/vLLM/vLLM-Continuum) is fair and separate, so the headline claim—better SLO attainment under multi-round workloads—survives. The weaker link underneath is that the whole online routing and offline planning depend on profiled performance models, and the paper only validates the planner ordinally (rankings), not the predicted latencies head-to-head against measured TTFT/ITL across batching and concurrency regimes. If the model misestimates under load, routing decisions go wrong without the experiments necessarily catching it. Casting the problem: that's a genuine but fixable validation gap, not a contradiction. Minor: no error bars, no repeat-run counts, no code or data release—for a systems paper with numbers like \"up to 3435%,\" that matters even though the averages (67–340%) are the fairer headline.\n\nOverall: the central design is sound, the paper is honest on its own terms, and the flaws are addressable. This deserves a serious referee. I'd want the ablation re-run with per-variant optimal deployments, a predicted-vs-measured latency plot, and code/data before taking the quantitative gains at face value, but the direction is right and the contribution is worth engaging with.","headline":"A solid, well-scoped systems paper that fills a real gap—PD-disaggregated multi-round inference—and the end-to-end gains are credible; the ablation attribution to adaptive routing+reordering is confounded because the deployment was optimized only for the full system.","tokens_in":20982,"tokens_out":1950,"would_cite":true,"duration_ms":24587,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AMPD shows that in disaggregated multi-round LLM serving, deciding per prefill task whether to run it locally on the decode worker or remotely on a prefill worker—guided by live TTFT/ITL slack and a profiled performance model—raises SLO att","keywords":["multi-round LLM inference","prefill-decode disaggregation","adaptive routing","prefill reordering","SLO attainment","ILP deployment planning","KV cache transmission","performance modeling"],"falsifier":"Measure predicted versus actual P95 time-to-first-token and inter-token latency on the target hardware across a sweep of batch sizes, concurrency levels, and parallelism degrees for the three test models; then perturb the model's prefill-time estimates by a known factor and check that SLO attainment degrades exactly as the model predicts. If predicted and measured latencies diverge, or the planner's top rankings change under a load sweep, the performance-model assumption is not load-independent.","tokens_in":20034,"feed_emoji":"⚡","tokens_out":9986,"duration_ms":87503,"temperature":0.7,"pith_summary":"Multi-round LLM workflows—agentic tool use, iterative retrieval—generate incremental prefill tasks that interrupt decoding. The paper argues that the conventional disaggregated paradigm, which always routes prefill to dedicated prefill workers, is suboptimal for this pattern, and that deployment planning that ignores round structure leaves performance on the table. Its system, AMPD, makes per-task routing decisions between local decode-worker execution and remote prefill-worker execution based on windowed TTFT/ITL slack, reorders queued prefills to maximize TTFT SLO hits within a small lookahead window, and solves an ILP to choose the data/model-parallel deployment. Across three models and four workloads, AMPD lifts SLO attainment by 67.29%–339.74% on average (up to 967.54% vs the disaggregated baseline and 3435.1% vs the co-located baseline), with ablation showing 44.47%–402% gains from the two online techniques. If correct, the result means serving systems can serve far more multi-round requests within latency targets at fixed GPU capacity simply by making routing and scheduling decisions that respect the interleaved pattern.","feed_headline":"Adaptive prefill placement lifts latency-goal attainment up to 34x","feed_subtitle":"Agent-style and retrieval-heavy workloads hit latency targets far more often — no extra GPUs.","key_machinery":"The central object is the adaptive routing decision between 'local execution' (on the request's bound decode worker) and 'remote execution' (on a prefill worker), driven by windowed TTFT/ITL statistics and a profiled piecewise α-β performance model (T_pre, T_dec, T_kv) that estimates prefill compute, KV transfer, and queuing costs. A lookahead reordering policy then reorders up to w queued prefill tasks to maximize the number meeting TTFT SLO, with a postponement cap to prevent starvation. An offline ILP planner, solving an unbounded-knapsack-style problem, chooses the data/model parallelism split that minimizes worst-case P95 latency across worker replicas under a GPU budget.","core_discovery":"The paper claims that the standard PD-disaggregated assumption—always send prefill to prefill workers, always decode on decode workers—fails for multi-round LLM workflows because each request's incremental prefills interleave with decoding. AMPD instead makes per-task routing decisions based on windowed TTFT/ITL slack: when prefill workers are congested, it executes some incremental prefills locally on the decode worker; when decode workers have slack, it routes them remotely. Together with a TTFT-aware prefill reordering policy and an ILP-based offline planner that minimizes P95 latency under GPU capacity, this lifts SLO attainment relative to both disaggregated and co-located baselines by","pith_inferences":["The adaptive-routing principle should generalize beyond the profiled trace set: if the decision surface (TTFT/ITL slack vs cost estimates) transfers, the same coordinator could handle mixes of single-round and multi-round traffic in one cluster, a scenario the paper does not test.","Because the routing cost estimates rely on queued-task counts, the system could be made even more proactive by predicting future incremental prefill arrivals (e.g., from the tool-use duration distribution) rather than reacting only to current queues.","The performance model's accuracy is the hidden hinge; a head-to-head validation against measured latency across a grid of batch sizes and parallelism degrees would test whether the planner's top-3 ranking matches real-system rankings not just on the tested traces but on a wider envelope.","The planning objective is P95 latency, not SLO attainment; a natural follow-up is to co-optimize for the SLO threshold directly, or to let the planner choose thresholds based on the workload's TTFT/ITL demands."],"forward_implications":["The adaptive routing decision itself (local vs remote) is the load-bearing mechanism: with the ablation showing 27.37%–350% SLO gain from routing alone, the interleaved pattern of multi-round inference is best served by a placement policy that reacts to real-time load.","A small lookahead window (w≈3) suffices to capture TTFT slack; larger windows give less than 3% SLO difference, suggesting the reordering policy scales cheaply.","The offline ILP planner, minimizing worst-case P95 latency under a GPU budget, produces deployment configurations that match real-system top rankings, so optimal resource split can be computed before serving.","The combined effect is capacity-free: SLO attainment improves without adding GPUs, meaning operators can either meet stricter SLOs on the same cluster or serve higher arrival rates."],"fun_headline_variants":["Prefill where it counts: adaptive routing boosts SLO 34x","Don't pigeonhole prefill: adapt placement for 34x SLO","Interleaved prefill-decode? AMPD adapts routing in real-time","Real-time prefill placement: boost SLO, no extra GPUs","Adaptive prefill scheduling: 34x SLO gain without extra GPUs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The routing, reordering, and deployment decisions all depend on the accuracy of a profiled piecewise performance model for prefill, decode, and KV-transfer times; if that model misestimates latencies under load, the system could pick the wrong workers or the wrong deployment even though the evaluation still passes because the same model guides the measured system.","fun_headline_variants_meta":{"raw":{"variants":["Prefill where it counts: adaptive routing boosts SLO 34x","Don't pigeonhole prefill: adapt placement for 34x SLO","Interleaved prefill-decode? AMPD adapts routing in real-time","Real-time prefill placement: boost SLO, no extra GPUs","Adaptive prefill scheduling: 34x SLO gain without extra GPUs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001731,"raw_usage":{"total_tokens":6671,"prompt_tokens":727,"completion_tokens":5944,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":5843}},"tokens_in":471,"tokens_out":5944,"duration_ms":44269,"temperature":1.0,"reasoning_tokens":5843,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T23:09:00.364354+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure predicted versus actual P95 time-to-first-token and inter-token latency on the target hardware across a sweep of batch sizes, concurrency levels, and parallelism degrees for the three test models; then perturb the model's prefill-time estimates by a known factor and check that SLO attainment degrades exactly as the model predicts. If predicted and measured latencies diverge, or the planner's top rankings change under a load sweep, the performance-model assumption is not load-independent.","supporting_citations":[],"review_version":1}