Pith. sign in

REVIEW 4 major objections 5 minor 6 cited by

AMPD shows that in disaggregated multi-round LLM serving, deciding per prefill task whether to run it locally on the decode worker or remotely on a prefill worker—guided by live TTFT/ITL slack and a profiled performance model—raises SLO att

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:09 UTC pith:2GKAZ4IY

load-bearing objection A solid, well-scoped systems paper that fills a real gap—PD-disaggregated multi-round inference—and the end-to-end gains are credible; the ablation attribution to adaptive routing+reordering is confounded because the deployment was optimized only for the full system. the 4 major comments →

arxiv 2602.14516 v2 pith:2GKAZ4IY submitted 2026-02-16 cs.DC

Efficient Multi-round LLM Inference over Disaggregated Serving

Wenhao He , Youhe Jiang , Penghao Zhao , Quanqing Xu , Eiko Yoneki , Bin Cui , Fangcheng Fu This is my paper
classification cs.DC
keywords multi-round LLM inferenceprefill-decode disaggregationadaptive routingprefill reorderingSLO attainmentILP deployment planningKV cache transmissionperformance modeling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Multi-round LLM workflows—agentic tool use, iterative retrieval—generate incremental prefill tasks that interrupt decoding. The paper argues that the conventional disaggregated paradigm, which always routes prefill to dedicated prefill workers, is suboptimal for this pattern, and that deployment planning that ignores round structure leaves performance on the table. Its system, AMPD, makes per-task routing decisions between local decode-worker execution and remote prefill-worker execution based on windowed TTFT/ITL slack, reorders queued prefills to maximize TTFT SLO hits within a small lookahead window, and solves an ILP to choose the data/model-parallel deployment. Across three models and four workloads, AMPD lifts SLO attainment by 67.29%–339.74% on average (up to 967.54% vs the disaggregated baseline and 3435.1% vs the co-located baseline), with ablation showing 44.47%–402% gains from the two online techniques. If correct, the result means serving systems can serve far more multi-round requests within latency targets at fixed GPU capacity simply by making routing and scheduling decisions that respect the interleaved pattern.

Core claim

The paper claims that the standard PD-disaggregated assumption—always send prefill to prefill workers, always decode on decode workers—fails for multi-round LLM workflows because each request's incremental prefills interleave with decoding. AMPD instead makes per-task routing decisions based on windowed TTFT/ITL slack: when prefill workers are congested, it executes some incremental prefills locally on the decode worker; when decode workers have slack, it routes them remotely. Together with a TTFT-aware prefill reordering policy and an ILP-based offline planner that minimizes P95 latency under GPU capacity, this lifts SLO attainment relative to both disaggregated and co-located baselines by

What carries the argument

The central object is the adaptive routing decision between 'local execution' (on the request's bound decode worker) and 'remote execution' (on a prefill worker), driven by windowed TTFT/ITL statistics and a profiled piecewise α-β performance model (T_pre, T_dec, T_kv) that estimates prefill compute, KV transfer, and queuing costs. A lookahead reordering policy then reorders up to w queued prefill tasks to maximize the number meeting TTFT SLO, with a postponement cap to prevent starvation. An offline ILP planner, solving an unbounded-knapsack-style problem, chooses the data/model parallelism split that minimizes worst-case P95 latency across worker replicas under a GPU budget.

Load-bearing premise

The routing, reordering, and deployment decisions all depend on the accuracy of a profiled piecewise performance model for prefill, decode, and KV-transfer times; if that model misestimates latencies under load, the system could pick the wrong workers or the wrong deployment even though the evaluation still passes because the same model guides the measured system.

What would settle it

Measure predicted versus actual P95 time-to-first-token and inter-token latency on the target hardware across a sweep of batch sizes, concurrency levels, and parallelism degrees for the three test models; then perturb the model's prefill-time estimates by a known factor and check that SLO attainment degrades exactly as the model predicts. If predicted and measured latencies diverge, or the planner's top rankings change under a load sweep, the performance-model assumption is not load-independent.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The adaptive routing decision itself (local vs remote) is the load-bearing mechanism: with the ablation showing 27.37%–350% SLO gain from routing alone, the interleaved pattern of multi-round inference is best served by a placement policy that reacts to real-time load.
  • A small lookahead window (w≈3) suffices to capture TTFT slack; larger windows give less than 3% SLO difference, suggesting the reordering policy scales cheaply.
  • The offline ILP planner, minimizing worst-case P95 latency under a GPU budget, produces deployment configurations that match real-system top rankings, so optimal resource split can be computed before serving.
  • The combined effect is capacity-free: SLO attainment improves without adding GPUs, meaning operators can either meet stricter SLOs on the same cluster or serve higher arrival rates.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The adaptive-routing principle should generalize beyond the profiled trace set: if the decision surface (TTFT/ITL slack vs cost estimates) transfers, the same coordinator could handle mixes of single-round and multi-round traffic in one cluster, a scenario the paper does not test.
  • Because the routing cost estimates rely on queued-task counts, the system could be made even more proactive by predicting future incremental prefill arrivals (e.g., from the tool-use duration distribution) rather than reacting only to current queues.
  • The performance model's accuracy is the hidden hinge; a head-to-head validation against measured latency across a grid of batch sizes and parallelism degrees would test whether the planner's top-3 ranking matches real-system rankings not just on the tested traces but on a wider envelope.
  • The planning objective is P95 latency, not SLO attainment; a natural follow-up is to co-optimize for the SLO threshold directly, or to let the planner choose thresholds based on the workload's TTFT/ITL demands.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes AMPD, a disaggregated LLM serving framework targeting multi-round inference workloads. Its online stage consists of an adaptive routing mechanism that decides, per prefill task, whether to execute locally on the decode worker or remotely on a prefill worker, and a lookahead-based prefill reordering policy that reorders a small window of queued prefill tasks to maximize predicted TTFT SLO satisfaction. Its offline stage formulates deployment planning—choosing data-parallel and model-parallel configurations for prefill and decode workers under a GPU budget—as an ILP, with P95 latency coefficients obtained from a simulator. The system is implemented on NVIDIA Dynamo with Redis for shared metadata, NIXL for KV-cache transfer, and SCIP for ILP solving. Experiments compare AMPD against Dynamo, vLLM, and vLLM-Continuum over three models and four multi-round traces, reporting large SLO-attainment improvements; ablations and sensitivity studies support the individual online mechanisms.

Significance. If the reported results hold, AMPD makes a useful contribution: it identifies the interleaved prefill-decode pattern of multi-round inference as a first-class scheduling challenge for PD-disaggregated serving, and provides both an online adaptive routing/reordering mechanism and an offline deployment planner. The evaluation is broad—3 models, 4 traces, 5 arrival rates, 3 baselines—and includes hyperparameter sensitivity and a planner-ranking validation. The paper does not include code or machine-checked artifacts, so the reproducibility of the quantitative claims rests on the experimental description and on how completely the model and simulator are validated. The main risk is not the end-to-end comparison, which appears broadly sound, but the attribution of the gains to the specific online mechanisms and the indirect validation of the performance model that guides both routing and planning.

major comments (4)
  1. [§7.2 (Figure 5), §5, App. A.1] The ablation that attributes 44.47%–402% SLO improvement to adaptive routing plus prefill reordering is confounded with deployment optimization. The ILP in Eq. (5) and the simulator in App. A.1 explicitly incorporate the §4 adaptive routing and reordering policy when estimating τ_pre and τ_dec, so the single deployment used for full AMPD is chosen as optimal for the full system. When AR+PR or PR is disabled, that same deployment is not necessarily optimal. For example, the planner could allocate fewer GPUs to prefill workers precisely because local execution of incremental prefills is expected; disabling adaptive routing would then make prefill workers the bottleneck and inflate the apparent benefit of the mechanisms. The paper does not state whether deployments were re-optimized for the ablated variants. Please re-run the ablation with planner configurations that disable the component u
  2. [§3, §4.1 (Eq. 1–2), §5 (Eq. 5), App. A.1] Both online routing and offline planning rely on the profiled piecewise α-β model (T_pre, T_dec, T_kv) and on the simulator that produces P95 coefficients τ_pre(n) and τ_dec(n). Table 2 validates that the planner's top-3 configurations match the real system's ranking, but this does not validate that predicted latencies are accurate across the batching, concurrency, and parallelism regimes encountered under load. If the model systematically underestimates TTFT or ITL under load, both the routing decisions and the deployment plan could be wrong, while the evaluation could still pass because the same model guides the measured system. Please include a head-to-head comparison of predicted versus measured TTFT/ITL (or P95 latency) for representative configurations, covering varied concurrency and parallelism degrees.
  3. [§4.1, Algorithm 1] The pseudocode appears inconsistent with the surrounding text. Lines 2–5 place the local-execution check ('if dITL ≤ β·ITL_thres then return local') inside the 'for each prefill worker i' loop, so that the first prefill worker that is not TTFT-slack would trigger a local return before the remaining prefill workers are examined. The text says local execution is considered only after all prefill workers are under pressure. Since this is the core online algorithm, the exact control flow matters for reproducibility. Please fix the pseudocode indentation/control flow or clarify the intended semantics.
  4. [§7.2 (Figures 4–6)] The headline quantitative claims—'up to 967.54% and 3435.1%' and the average improvements—are reported as point estimates without error bars, repeated runs, or statistical significance. The request arrivals are Poisson and the scheduling decisions are online, so SLO attainment is a random quantity. A single run per configuration is insufficient to support precise percentage-level comparisons, especially at arrival rates where curves are close (e.g., Qwen3-32B GAIA at low load). Please report means/standard deviations over multiple seeds, or at least a stability check over repeated runs, for the main comparisons and for the ablation in Figure 5.
minor comments (5)
  1. [Figure 4] The arrows and percentage annotations on the bars are ambiguous: some entries read '↑543%' without stating the baseline to which the improvement refers. Please label each arrow with the baseline or move the baseline-specific claims to the text/table.
  2. [Figure 8] The label 'Llama3.1-70B · GAIA' appears twice in the Mixtral subplot row; the second occurrence should likely be 'Mixtral-8x7B · GAIA'.
  3. [Table 2] Please state explicitly what 'real-system serving' ranking means in Table 2—is it measured by SLO attainment, P95 latency, or end-to-end latency? Also clarify how the top-3 configurations were selected from the larger configuration space.
  4. [§7.1] The protocol says baselines were 'tuned' and 'best results' reported, but the tuning grid (e.g., numbers of prefill/decode GPUs, tensor-parallel degrees) is not described. Please add this detail for reproducibility and to support the fairness claim.
  5. [§1/§7.1] The abstract and introduction state 'brand new'; the paper would read more formally as 'new' or 'novel'. Also, reference [Li et al., 2025] is cited as 'vLLM-Continuum' in the experiments but as 'Continuum' in related work; please unify.

Circularity Check

0 steps flagged

No significant circularity: the empirical SLO gains are measured against external baselines and the performance model is validated on real-system rankings rather than defining the outcome.

full rationale

The paper's central claims are empirical. AMPD's headline improvements (Fig. 4, §7.2) compare measured SLO attainment against Dynamo, vLLM, and vLLM-Continuum on real hardware, with baselines tuned for fairness; SLO attainment is computed from actual TTFT/ITL, not from the profiled model. The adaptive routing (Eq. 1-2) and reordering (Eq. 3-4) use profiled cost models as heuristics, but the evaluation does not substitute model predictions for measurements. The offline planner (§5, App. A.1) estimates P95 latencies via a simulator that incorporates operator-level profiling and the §4 policy; however, Table 2 validates the planner by comparing its top-3 configurations to real-system serving rankings, which is an independent internal check rather than a definitional reduction. The ablation (Fig. 5) compares full AMPD to variants without adaptive routing/reordering; any concern that the deployment was optimized for full AMPD is an experimental confound, not a circular derivation, and it does not affect the separate end-to-end comparison against baselines. No load-bearing self-citation chain is present.

Axiom & Free-Parameter Ledger

5 free parameters · 7 axioms · 0 invented entities

The paper introduces no new physical entities. Its load-bearing machinery is a fitted performance model and a simulator; the key free choices are routing thresholds α, β, window size w, and the profiled latency coefficients feeding the ILP.

free parameters (5)
  • α (TTFT slack threshold in adaptive routing) = 0.9 (default)
    Hand-chosen hyperparameter; sensitivity evaluated in Figure 6.
  • β (ITL slack threshold in adaptive routing) = 0.85 (default)
    Hand-chosen hyperparameter; sensitivity evaluated in Figure 6.
  • w (lookahead window size in prefill reordering) = 3 (default)
    Hand-chosen; sensitivity evaluated in Figure 6.
  • Performance-model coefficients for T_pre, T_dec, T_kv = not reported; profiled on H20 hardware
    The piecewise α-β model (§3, App. A.1) is fitted to operator-level profiling; specific coefficients are not given in the text.
  • P95 latency coefficients τ_pre(n), τ_dec(n) = from simulator; not listed
    Used as constant parameters in the ILP objective; estimated by the simulator rather than measured end-to-end.
axioms (7)
  • domain assumption Workload arrivals are Poisson processes with given rates (§7.1)
    Standard in serving papers, but real agent traffic may be burstier; the routing decisions rely on windowed statistics.
  • domain assumption KV-cache transmission time follows Hockney's α-β model (cited in App. A.1)
    Assumes latency is linear in message size with fixed overhead; not validated in this paper.
  • domain assumption Model parallelism degrees are drawn from a discrete set of powers of two (§5)
    Restricts the solution space of the ILP.
  • domain assumption Prefill tasks are prioritized over decode on the same worker (§3 fn. 3)
    Inherited from vLLM; the cost estimates in Eq. (1) do not account for priority preemption of decode.
  • ad hoc to paper The simulator's operator-level profiling and execution simulation accurately predict P95 latency for arbitrary configurations (App. A.1)
    The planner's coefficients and the routing costs come from this simulator, and it is only indirectly checked in Table 2.
  • domain assumption Windowed TTFT/ITL over the past 10 seconds reflects current load (§3)
    Routing decisions use this statistic; no analysis for highly bursty arrivals.
  • domain assumption The SLO thresholds (e.g., TTFT<=350ms, ITL<=40ms) are appropriate for the workloads
    SLO attainment is the primary metric; gains are sensitive to these thresholds.

pith-pipeline@v1.3.0-alltime-deepseek · 19751 in / 12066 out tokens · 111013 ms · 2026-08-02T23:09:00.364354+00:00 · methodology

0 comments
read the original abstract

With the rapid evolution of Large Language Models (LLMs), multi-round workflows, such as autonomous agents and iterative retrieval, have become increasingly prevalent. However, this raises hurdles for serving LLMs under prefill-decode (PD) disaggregation, a widely adopted paradigm that separates the compute-bound prefill phase and memory-bound decode phase onto individual resources. Specifically, existing systems overlook the interleaved prefill-decode workload pattern in multi-round inference, leading to sub-optimal handling of the incremental prefill workloads and model deployment for the two phases. In this work, we present AMPD, a brand new disaggregated serving framework for multi-round LLM inference. The core of AMPD is to coordinate the prefill workloads based on real-time workloads by adaptively determining where to carry out these workloads and how they are scheduled, in order to maximize service level objective (SLO) attainment. In addition, we tailor a planning algorithm for our scenario, facilitating the deduction of optimal resource allocation and parallel strategies for the two phases. Empirical results demonstrate that AMPD substantially improves SLO attainment compared to state-of-the-art baselines.

Figures

Figures reproduced from arXiv: 2602.14516 by Bin Cui, Eiko Yoneki, Fangcheng Fu, Penghao Zhao, Quanqing Xu, Wenhao He, Youhe Jiang.

Figure 1
Figure 1. Figure 1: Illustration of multi-round LLM inference. critical for both academia and industry (Kwon et al., 2023; Zheng et al., 2024; NVIDIA, 2026b; Zhang et al., 2025b). Generally, given a request, the LLM inference comprises two sequential phases: prefill and decode. The prefill phase processes the entire input prompt at once to compute the Key-Value (KV) cache, exhibiting compute-bound charac￾teristics. In contras… view at source ↗
Figure 2
Figure 2. Figure 2: System overview of AMPD. MARS (Shahout et al., 2025) and AugServe (Wang et al., 2025) predict the output lengths and memory consumption respectively in multi-round LLM inference, and leverage such information to guide the scheduling of requests. Although these works focus on how to optimize system performance for multi-round workflows, all of them are de￾signed under the co-located serving paradigm. To ach… view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of disaggregated LLM serving where prefill and decode workers are with different parallelism configurations. model parallelism degrees T (e.g., {1, 2, 4, 8}), the decision variables comprise two integer vectors, x, y ∈ Z |T | ≥0 (e.g., x = [x (1), x(2), x(4), x(8)], y = [y (1), y(2), y(4), y(8)]). For each model parallelism degree n ∈ T , the variable x (n) (y (n) ) denotes the number of prefi… view at source ↗
Figure 4
Figure 4. Figure 4: End-to-end comparison. The top three rows are the SLO attainment under different configurations (traces and request arrival rates) for the three models. The bottom row is a detailed breakdown of Llama-3.1-70B with a request arrival rate of 2 reqs/s, including the average TTFT for initial prefill, average TTFT for incremental prefill, and average ITL for decoding. which are Qwen3-32B (Yang et al., 2025), Ll… view at source ↗
Figure 5
Figure 5. Figure 5: Ablation studies (Llama3.1-70B, 2 reqs/s). Left: DuReader. Middle: GAIA. Right: Proportion of local and remote execution. w=2 w=3 w=4 w=5 0 15 30 45 60 75 SLO Attainment (%) 64.65 65.66 67.68 64.65 α=0.75 α=0.8 α=0.85 α=0.9 α=0.95 α=1 0 15 30 45 60 75 SLO Attainment (%) 45.46 55.56 61.62 65.66 62.63 58.59 β=0.75 β=0.8 β=0.85 β=0.9 β=0.95 0 15 30 45 60 75 SLO Attainment (%) 46.47 59.60 65.66 50.51 41.67 [P… view at source ↗
Figure 7
Figure 7. Figure 7: measures the time cost of our offline planning with different number of GPUs. Since the determination of deployment configuration is formulated as an Integer Linear Programming problem, it can be solved efficiently with existing solvers (Bolusani et al., 2024). Consequently, our planning finishes quickly, taking merely one minute over 256 GPUs. This represents a common cluster scale for LLM serving in many… view at source ↗
Figure 8
Figure 8. Figure 8: compares the average end-to-end latency of all counterparts. AMPD maintains low latencies that are comparable against Dynamo. Although Dynamo has lower latencies in some cases, the gap is small. More importantly, AMPD delivers substantial improvement in terms of SLO attainment (as evaluated in §7), which is essential for real-world serving. 0.8 1.0 1.2 1.4 1.6 1.8 2.0 2.2 2.4 load 3000 4000 5000 Average E2… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. TurboServe: Serving Streaming Video Generation Efficiently and Economically

    cs.DC 2026-06 unverdicted novelty 7.0

    TurboServe introduces the first serving system for streaming video generation workloads, using migration-aware placement and load-driven autoscaling to cut worst-case latency by 37.5% and GPU cost by 37.2%.

  2. Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Serving

    cs.DC 2026-06 unverdicted novelty 7.0

    Conversation-level scheduling in ConServe observes first-turn input length and KV occupancy to route prefill once and pin decoders, cutting p95 time-to-first-effective-token by 51% and improving energy efficiency by 7...

  3. HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Scheduling

    cs.DC 2026-05 unverdicted novelty 7.0

    HexAGenT reduces the SLO scale required for timely agentic LLM workflow completion by an average of 20.1% at 95% attainment and 33.0% at 99% attainment on heterogeneous A100/H100/H200 clusters.

  4. Autopoiesis: A Self-Evolving System Paradigm for LLM Serving Under Runtime Dynamics

    cs.DC 2026-04 unverdicted novelty 7.0

    Autopoiesis uses LLM-driven program synthesis to evolve serving policies online during deployment, delivering up to 53% and average 34% gains over prior LLM serving systems under runtime dynamics.

  5. DelAC: A Multi-agent Reinforcement Learning of Team-Symmetric Stochastic Games

    cs.MA 2026-05 unverdicted novelty 6.0

    Team-symmetric games always have team-symmetric Nash equilibria solvable via linear complementarity problems, and the DelAC actor-critic MARL algorithm outperforms existing methods in simulations.

  6. KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving

    cs.DC 2026-04 unverdicted novelty 6.0

    KAIROS reduces power by 27% on average (up to 39.8%) for agentic AI inference by using long-lived context to jointly manage GPU frequency, concurrency, and request routing across instances.

Reference graph

Works this paper leans on

29 extracted references · 15 linked inside Pith · cited by 6 Pith papers

  1. [2]

    Chen, S., Wang, Y ., Wu, Y .-F., Chen, Q., Xu, Z., Luo, W., Zhang, K., and Zhang, L. Advancing tool-augmented large language models: Integrating insights from errors in inference trees.Advances in Neural Information Process- ing Systems (NeurIPS 2024), 37:106555–106581,

    2024

  2. [3]

    Hydrainfer: Hybrid disaggregated scheduling for multimodal large language model serving.arXiv preprint arXiv:2505.12658,

    Dong, X., Liu, T., Zeng, Y ., Liu, L., Liu, Y ., Wu, S., Wu, Y ., Yang, H., Zhang, K., and Li, J. Hydrainfer: Hybrid disaggregated scheduling for multimodal large language model serving.arXiv preprint arXiv:2505.12658,

    arXiv

  3. [5]

    Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large lan- guage models.arXiv preprint arXiv:2403.07714,

    Guo, Z., Cheng, S., Wang, H., Liang, S., Qin, Y ., Li, P., Liu, Z., Sun, M., and Liu, Y . Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large lan- guage models.arXiv preprint arXiv:2403.07714,

    Pith/arXiv arXiv

  4. [8]

    Thunderserve: High-performance and cost-efficient llm serving in cloud environments

    JIANG, Y ., Fu, F., Yao, X., Wang, T., CUI, B., Klimovic, A., and Yoneki, E. Thunderserve: High-performance and cost-efficient llm serving in cloud environments. In Eighth Conference on Machine Learning and Systems (MLSys 2025). Jin, B., Zeng, H., Yue, Z., Yoon, J., Arik, S., Wang, D., Zamani, H., and Han, J. Search-r1: Training llms to reason and leverag...

    Pith/arXiv arXiv 2025

  5. [11]

    Continuum: Efficient and robust multi-turn llm agent scheduling with kv cache time-to-live.arXiv preprint arXiv:2511.02230,

    Li, H., Mang, Q., He, R., Zhang, Q., Mao, H., Chen, X., Cheung, A., Gonzalez, J., and Stoica, I. Continuum: Efficient and robust multi-turn llm agent scheduling with kv cache time-to-live.arXiv preprint arXiv:2511.02230,

    Pith/arXiv arXiv

  6. [12]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    Pith/arXiv arXiv

  7. [13]

    Augmented language mod- els: a survey.arXiv preprint arXiv:2302.07842,

    Mialon, G., Dess `ı, R., Lomeli, M., Nalmpantis, C., Pa- sunuru, R., Raileanu, R., Rozi`ere, B., Schick, T., Dwivedi- Yu, J., Celikyilmaz, A., et al. Augmented language mod- els: a survey.arXiv preprint arXiv:2302.07842,

    Pith/arXiv arXiv

  8. [14]

    Gaia: a benchmark for general ai assistants

    9 Efficient Multi-round LLM Inference over Disaggregated Serving Mialon, G., Fourrier, C., Wolf, T., LeCun, Y ., and Scialom, T. Gaia: a benchmark for general ai assistants. InThe Twelfth International Conference on Learning Represen- tations (ICLR 2024),

    2024

  9. [15]

    Nvidia dynamo documentation: Kv router

    NVIDIA. Nvidia dynamo documentation: Kv router. https://docs.nvidia.com/dynamo/ latest/router/README.html, 2026a. NVIDIA. Nvidia dynamo: A datacenter scale distributed in- ference serving framework. https://github.com/ ai-dynamo/dynamo, 2026b. NVIDIA. Nvidia dynamo: A datacenter scale distributed in- ference serving framework. https://github.com/ ai-dynam...

    2025

  10. [16]

    Splitwise: Efficient generative llm inference using phase splitting

    Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri,´I., Maleki, S., and Bianchini, R. Splitwise: Efficient generative llm inference using phase splitting. In2024 ACM/IEEE 51st Annual International Symposium on Computer Architec- ture (ISCA 2024), pp. 118–132. IEEE,

    2024

  11. [17]

    Fast inference for augmented large lan- guage models

    Shahout, R., Liang, C., Xin, S., Lao, Q., Cui, Y ., Yu, M., and Mitzenmacher, M. Fast inference for augmented large lan- guage models. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS 2025),

    2025

  12. [18]

    Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy.arXiv preprint arXiv:2305.15294,

    Shao, Z., Gong, Y ., Shen, Y ., Huang, M., Duan, N., and Chen, W. Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy.arXiv preprint arXiv:2305.15294,

    Pith/arXiv arXiv

  13. [19]

    Megatron-lm: Training multi- billion parameter language models using model paral- lelism.arXiv preprint arXiv:1909.08053,

    Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-lm: Training multi- billion parameter language models using model paral- lelism.arXiv preprint arXiv:1909.08053,

    Pith/arXiv arXiv 1909

  14. [21]

    N., Kaiser, Ł., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. At- tention is all you need.Advances in neural information processing systems (NeurIPS 2017), 30,

    2017

  15. [23]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  16. [24]

    Yang, Z., Qi, P., Zhang, S., Bengio, Y ., Cohen, W., Salakhut- dinov, R., and Manning, C. D. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pp. 2369–2380,

    2018

  17. [25]

    R., and Cao, Y

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y . React: Synergizing reasoning and act- ing in language models. InThe eleventh international con- ference on learning representations (ICLR 2023),

    2023

  18. [27]

    Blitzscale: Fast and live large model autoscaling with o(1) host caching

    Zhang, D., Wang, H., Liu, Y ., Wei, X., Shan, Y ., Chen, R., and Chen, H. Blitzscale: Fast and live large model autoscaling with o(1) host caching. In19th USENIX Sym- posium on Operating Systems Design and Implementation (OSDI 25), pp. 275–293, 2025a. Zhang, L., Jiang, Y ., He, G., Chen, X., Lv, H., Yao, Q., Fu, F., and Chen, K. Efficient mixed-precision ...

    Pith/arXiv arXiv 2024

  19. [28]

    More Details about Offline Planning A.1

    11 Efficient Multi-round LLM Inference over Disaggregated Serving A. More Details about Offline Planning A.1. Performance Simulation The simulator generates simulated performance metrics (i.e., P95 latency) for various model deployment configurations. Simulator inputs.In addition to the target model deployment configuration, the simulator requires three c...

    1991

  20. [29]

    More experimental results

    using Qwen3-32B, with each request invoking three retrieval calls. More experimental results. Figure 8 compares the average end-to-end latency of all counterparts. AMPD maintains low latencies that are comparable against Dynamo. Although Dynamo has lower latencies in some cases, the gap is small. More importantly, AMPD delivers substantial improvement in ...

    2000

  21. [1991]

    H., Gonzalez, J., Zhang, H., and Stoica, I

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y ., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th sympo- sium on operating systems principles (SOSP 2023), pp. 611–626,

    2023

  22. [1994]

    Inference without interference: Disaggregate llm inference for mixed down- stream workloads.arXiv preprint arXiv:2401.11181,

    Hu, C., Huang, H., Xu, L., Chen, X., Xu, J., Chen, S., Feng, H., Wang, C., Wang, S., Bao, Y ., et al. Inference without interference: Disaggregate llm inference for mixed down- stream workloads.arXiv preprint arXiv:2401.11181,

    Pith/arXiv arXiv

  23. [2015]

    Augserve: Adaptive request scheduling for augmented large language model inference serving.arXiv preprint arXiv:2512.04013,

    Wang, Y ., Jin, Z., Xu, J., Lin, W., Chen, Y ., and Chen, W. Augserve: Adaptive request scheduling for augmented large language model inference serving.arXiv preprint arXiv:2512.04013,

    Pith/arXiv arXiv

  24. [2018]

    Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts.arXiv preprint arXiv:2401.04088,

    Pith/arXiv arXiv

  25. [2019]

    Singh, G., Wang, X., Hu, Y ., Yu, T. T. L., Xing, L., Jiang, W., Wang, Z., Xiaolong, B., Li, Y ., Xiong, Y ., et al. Effi- ciently serving large multimodal models using epd disag- gregation. InForty-second International Conference on Machine Learning (ICML 2025),

    2025

  26. [2022]

    Inference scaling for long-context retrieval augmented generation.arXiv preprint arXiv:2410.04343,

    Yue, Z., Zhuang, H., Bai, A., Hui, K., Jagerman, R., Zeng, H., Qin, Z., Wang, D., Wang, X., and Bendersky, M. Inference scaling for long-context retrieval augmented generation.arXiv preprint arXiv:2410.04343,

    Pith/arXiv arXiv

  27. [2023]

    Tokenscale: Timely and accurate autoscaling for disaggregated llm serving with token velocity.arXiv preprint arXiv:2512.03416,

    Lai, R., Liu, H., Lu, C., Liu, Z., Cao, S., Shao, S., Zhang, Y ., Mai, L., and Ustiugov, D. Tokenscale: Timely and accurate autoscaling for disaggregated llm serving with token velocity.arXiv preprint arXiv:2512.03416,

    arXiv

  28. [2024]

    The scip optimiza- tion suite 9.0.arXiv preprint arXiv:2402.17702,

    Bolusani, S., Besanc ¸on, M., Bestuzheva, K., Chmiela, A., Dion´ısio, J., Donkiewicz, T., van Doornmalen, J., Eifler, L., Ghannam, M., Gleixner, A., et al. The scip optimiza- tion suite 9.0.arXiv preprint arXiv:2402.17702,

    Pith/arXiv arXiv

  29. [2025]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv