Pith. sign in

REVIEW 4 major objections 5 minor 55 references

EcoServe: Enabling Cost-effective LLM Serving with Proactive Intra- and Inter-Instance Orchestration

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Time-slicing prefill and decode lifts LLM goodput on ordinary clusters.

desk verdict EcoServe's temporal disaggregation plus rolling activation is a plausible and useful scheduling idea, but the headline numbers do not match the body and the SLO-enforcing admission controller is unvalidated. read the letter →

arxiv 2504.18154 v1 pith:LK57G3W6 submitted 2025-04-25 cs.DC

Jiangsu Du , Hongbin Zhang , Taosheng Wei , Zhenyi Zheng , Kaiyi Wu , Zhiguang Chen , Yutong Lu This is my paper · ORCID
classification cs.DC
keywords LLMservingprefill-decodedisaggregationtemporalrollingactivationmacroinstancegoodputSLOattainmentcommodityinterconnects
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that LLM serving does not have to choose between keeping the initial prompt-processing pass and the token-by-token generation pass in one instance (fast but interference-prone) or splitting them onto separate machines linked by fast interconnects (clean but expensive). It proposes a middle strategy, partial disaggregation (PaDG), in which each instance alternates between longer prompt-processing and generation time slices, and a group of instances is staggered so that some are always ready to accept a new prompt, with no KV cache—the cached key-value attention state—shipped across the network. The basic serving unit is a macro instance, and an adaptive scheduler admits requests only when the predicted prompt-processing time fits the time-to-first-token SLO and the spare generation time built up by existing requests can absorb the interruption. On a 32-GPU L20 cluster connected by ordinary 10Gbps Ethernet, the paper reports P90 goodput gains of 83.76% over vLLM, 71.97% over Sarathi, 192.41% over DistServe, and 218.22% over MoonCake (the abstract states overall averages of 82.49%, 86.17%, 122.76%, and 126.96%). If true, this would move the latency-throughput trade-off frontier for the many clusters that do not own InfiniBand or NVLink.

What carries the argument

The load-bearing object is the macro instance: a set of GPU instances whose prompt-processing and generation phases are offset so that, at any moment, at least one member is accepting a new prompt while the others generate tokens. Inside an instance, temporal disaggregation fixes phase boundaries in time; across instances, rolling activation fixes the rotation; the constraint checker decides admission by comparing the predicted total prompt-processing time with the time-to-first-token SLO and with the mean saved per-token time accumulated by existing generations. The mitosis scaling layer then grows or shrinks a macro instance one GPU at a time and, at thresholds, splits or merges macro instances using a serializable proxy object, so capacity changes do not interrupt running generations.

What would settle it

Run EcoServe on a 50/50 mix of short (~100-token) and long (~4000-token) prompts at the same mean arrival rate, with the profiled prompt-duration table frozen, and count what fraction of long prompts exceed their time-to-first-token SLO; if that fraction rises faster than the saved per-token buffer can cover, the admission controller's core guarantee fails and the reported P90 goodput gap over DistServe and MoonCake should shrink or disappear.

Watch

Extended reading notes

Core claim

The central claim is that separating the prompt-processing and generation phases across time—rather than across hardware—captures most of the benefit of full disaggregation without its network dependency. In EcoServe, each instance runs only one phase at a time for an extended period, eliminating the fine-grained interference that hybrid batching creates; rolling activation then cycles a macro instance's members through their prompt-processing slots so that an arriving request is always routed to an instance that can start immediately. Because the phases still share the same GPU and model weights, no KV cache moves between instances, removing the cost and complexity that sinks fully disaggregated designs on commodity interconnects. The system's admission check is the mechanism that converts this schedule into SLO compliance: it estimates whether all pending prompt-processing work finishes within the time-to-first-token bound, and whether the mean spare time accumulated by token-generation batches that run faster than their per-token target covers that work, refusing admission otherwise. On the paper's testbed this yields a higher sustainable request rate at P50, P90, and P99 SLO attainment than four representative baselines, with the largest margins over the fully disaggregated systems on long-prompt workloads and on models with large KV caches.

Load-bearing premise

The load-bearing premise is that the system can predict from profiling how long a new prompt's first computation pass will take, accurately enough that the sum of pending predictions stays under the time-to-first-token limit, and that the average spare time built up by requests already generating tokens is enough to protect every request's per-token speed target; if those predictions are wrong—especially on very long or mixed-length prompts—accepted requests miss their latency targets and the measured goodput advantage changes.

Editorial extensions

If this is right

  • Clusters without high-performance interconnects can run tight-SLO LLM serving at throughputs the paper says previously required InfiniBand- or NVLink-class fabric.
  • The advantage grows as SLOs tighten: reported throughput gaps over the baselines widen from P50 to P90 and P99 SLO attainment.
  • Because instances never transfer KV cache, tensor and pipeline parallelism avoid PCIe contention and pipeline bubbles, so one system can serve 30B and 70B models on commodity nodes.
  • Fine-grained elastic scaling is possible by adding or removing single instances inside a macro instance, with instance migration overhead under 100 ms in the reported measurements.
  • PaDG is positioned as the middle ground for 30B-130B models; the paper says small models gain little from it and ultra-stringent SLOs may still require full disaggregation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural stress test the paper does not run is to freeze the profiled prompt-duration table and serve a heavy-tailed mix of short and long prompts; if time-to-first-token violations rise faster than the saved per-token buffer can absorb, the admission controller, not the phase schedule, is the real capacity ceiling.
  • The same time-slicing idea could be pushed one level deeper, for example alternating attention and feed-forward computation inside an instance for mixture-of-experts models, a direction the paper's discussion of module-level disaggregation already gestures toward.
  • If the reported gains replicate, cost comparisons of serving architectures should shift from interconnect choice to phase-scheduling policy, because a software-only schedule would capture much of what full disaggregation buys with hardware.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. EcoServe proposes a partially disaggregated (PaDG) strategy for LLM serving on clusters with commodity interconnects. The key idea is temporal disaggregation, in which each instance alternates between prefill and decode phases, combined with rolling activation, in which instances are cyclically staggered so that a prefill-capable instance is usually available. The paper also contributes an adaptive scheduling algorithm for request admission and routing, and a mitosis scaling approach for elastic instance management. The evaluation compares EcoServe against vLLM, Sarathi, DistServe, and MoonCake on two clusters, three models, and three datasets, reporting large goodput improvements, with the caveat that NoDG baselines remain competitive on the Alpaca workload. The central claim is that PaDG raises the TTFT/TPOT/throughput trade-off frontier without requiring high-performance interconnects.

Significance. If the reported results hold, EcoServe addresses a real and timely problem: the cost and complexity of fully disaggregated serving on commodity hardware. The evaluation is broad and uses externally released baselines, and the paper is honest about cases where baselines remain competitive, such as Alpaca. The PaDG concept of phase-slicing within instances plus cyclic staggering is a plausible and potentially practical design point. However, the headline numbers are inconsistent between the abstract and the body, and the SLO attainment that defines goodput is produced by an admission controller whose prediction assumptions are not validated. The paper does not ship code or configuration details, which limits reproducibility of the quantitative claims.

major comments (4)
  1. [Abstract vs. §4.2] The abstract reports average goodput improvements of 82.49%, 86.17%, 122.76%, and 126.96% over the four baselines, while Section 4.2 reports P90 improvements of 83.76%, 71.97%, 192.41%, and 218.22%, and P50 improvements of 36.49%, 19.82%, 180.73%, and 194.62%. The abstract numbers match none of these lists, and the body does not state which percentile, if any, the abstract uses. Because the abstract is the headline claim, this discrepancy must be resolved.
  2. [§3.4, Algorithm 2] The admission controller is load-bearing for the goodput comparison, but its prediction assumptions are unvalidated. Algorithm 2 admits a request only if the sum of predicted prefill durations t_total does not exceed SLO_TTFT (lines 5–7) and if mean saved TPOT exceeds t_total (lines 13–17). Prefill durations are obtained by profiling with no reported prediction-error measurement, and t_total is treated as a bound on time-to-first-token without accounting for batching effects, phase-switch overhead, or decode preemption. More importantly, using mean saved TPOT does not enforce a per-request TPOT SLO: the mean can be positive while individual decodes have negative saved TPOT, so the reported P99 attainment is not guaranteed. No sensitivity analysis, profiling-error characterization, or percentile breakdown of saved TPOT is provided, and the risk is highest on LongBench, which has long prompts.
  3. [§4.2, Figure 8] The end-to-end comparison reports no variance or repetition information. Figure 8 shows single throughput values at each SLO attainment level without error bars, and the text does not state how many trials were run or how request-rate sweeps were repeated. Given that the central claim is a quantitative goodput improvement, the absence of variance information makes it hard to judge whether the reported differences are within run-to-run noise.
  4. [§4.2, Comparison Across Applications] The LongBench comparison against FuDG systems excludes Llama-30B results because of execution failures, and the text says the improvement 'would be higher' as a result. This is disclosed, but the exclusion should be quantified or the reported average should be recomputed with a clearly stated policy. As written, the LongBench FuDG improvement of 164.42% is not directly comparable to the other application-level numbers.
minor comments (5)
  1. [§4.2] The body reports P50 and P90 improvement percentages but never gives the corresponding P99 aggregate numbers, even though P99 attainment is discussed qualitatively. Please add the P99 numbers or state why they are omitted.
  2. [§4.3.2] The hyperparameters N_l and N_u are only specified for the dynamic-scaling experiment (N_l=4, N_u=16). The main evaluation in Section 4.2 does not state N_l, N_u, the prefill/decode phase window duration, or how these are chosen. These are free parameters listed in the design and should be reported for reproducibility.
  3. [§4.1] The paper states that no code or configuration is released. Given the strong quantitative claims and the custom admission controller, a public artifact would substantially strengthen the paper.
  4. [Throughout] There are several typos and grammar issues, e.g., 'requring' in §2.4.2, 'approximatebly' in §4.1, and 'processed intermediately' in §3.2.2. A careful proofread is needed.
  5. [References] Reference [37] is listed as '1911' in the bibliography; the year should be corrected to 2019.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central goodput claims are empirical comparisons against external baseline systems, and the admission-control heuristics are unvalidated robustness risks rather than reductions of the target result to its own inputs.

full rationale

EcoServe's central claim is that the PaDG strategy (temporal disaggregation plus rolling activation) improves goodput on commodity-interconnect clusters. The support for this claim is a measurement study against vLLM, Sarathi, DistServe, and MoonCake, which are externally released systems; none of the reported improvements are derived from an equation that contains the claim itself. The two self-citations by the authors, [13] (ATP) and [15] (Liger), appear in related-work enumerations of parallelism techniques and do not carry any load-bearing argument; in particular, no uniqueness theorem or ansatz from prior author work is invoked to force the PaDG design. Section 3.3's redefinition of TTFT to include phase-switching waiting time is transparent: it is a stricter metric applied to all systems, and the paper explicitly says the definition 'represents a stricter SLO'. This is a metric choice, not a self-definitional derivation. Algorithm 2's admission control uses profiled prefill durations and mean saved TPOT, and these are indeed unvalidated predictive heuristics; however, the paper reports SLO attainment as a measured quantity ('The throughput is collected by incrementally increasing the request rate until the system fails to reach the attainment', Section 4.1), so the reported goodput is not constructed from the controller's own estimates. The concern about prediction error or the mean-vs-percentile TPOT safeguard is a correctness and robustness risk, not a circularity: no quoted reduction shows that a reported result is equivalent by construction to an input fit or to a self-citation. The discussion section even acknowledges that FuDG may remain necessary for ultra-large models or stringent SLOs, further indicating that the paper's claims are bounded rather than tautological. Accordingly, no circular step can be exhibited under the required evidence standard.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The empirical claims rest on a small set of load-bearing choices: two hand-set scaling thresholds (N_l=4, N_u=16), an unreported prefill/decode window duration that controls the rolling-activation cadence, and a prefill-duration profiler that feeds the admission controller. The user-experience model assumes TPOT SLO is bankable as average slack and that TTFT including phase-switching wait is the right metric. No new physical or mathematical entities are introduced; the macro instance and the serializable proxy are internal software abstractions without independent falsifiable handles, and no code is released.

free parameters (4)
  • prefill/decode phase window duration = not reported
    Each instance alternates prefill and decode epochs; the epoch length sets switching overhead, TTFT wait, and rolling-activation cadence. The value is tuned implicitly but never stated.
  • N_l (minimum instances per macro instance) = 4
    Set by hand in Section 4.3.2: "we set the hyperparameters to N_l=4 and N_u=16." It drives split/merge triggers and scheduler load.
  • N_u (maximum instances per macro instance) = 16
    Upper bound on instances per macro instance, set by hand from weak-scaling experiments in Section 4.3.2.
  • prefill duration profile = profiling table (not included)
    Algorithm 2 Constraint 1 predicts prefill durations "by profiling sequences of various lengths"; this fitted profile gates every TTFT admission decision.
assumptions (4)
  • domain assumption Prefill is compute-bound and decode is memory-bound, so separating them in time or space avoids interference.
    Standard characterization used throughout Section 2.1 and the PaDG motivation.
  • domain assumption TPOT SLO can be banked: decodes that run faster than the SLO accumulate saved TPOT that can be spent on prefill interruptions.
    Underpins Algorithm 2 Constraint 2; assumes users experience average token rate, not per-token jitter. Paper states this in Section 3.2.1.
  • ad hoc to paper A fixed cyclic staggering of instances guarantees a prefill-active instance is always available for any workload within SLO.
    The paper asserts "EcoServe can theoretically satisfy SLOs" (Section 1) without a derivation or a condition on the instance count relative to the prefill:decode duration ratio.
  • domain assumption TTFT including phase-switching waiting time is the correct user-facing latency metric.
    Section 3.3 redefines TTFT to include phase-switching wait, calling it a stricter SLO; all SLO attainments and comparisons inherit this definition.
invented entities (2)
  • macro instance
    purpose: Group of cooperating instances used as the smallest scheduling unit; enables rolling activation and mitosis scaling.
    Internal software abstraction; no falsifiable handle outside the paper and no released code to reproduce the instance-to-macro mapping.
  • serializable proxy object (InstanceHandler)
    purpose: Logical migration of instances between macro instances without re-initialization, enabling split and merge.
    Implementation artifact; the claimed sub-100 ms migration overhead is not independently verifiable without code.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EcoServe: Enabling Cost-effective LLM Serving with Proactive Intra- and Inter-Instance Orchestration." pith.science (2026). https://pith.science/paper/LK57G3W6

@misc{pith2026250418154,
  author       = {Pith},
  title        = {Pith review of: EcoServe: Enabling Cost-effective LLM Serving with Proactive Intra- and Inter-Instance Orchestration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LK57G3W6}},
  note         = {Machine review of arXiv:2504.18154}
}
read the original abstract

Existing LLM serving strategies can be categorized based on whether prefill and decode phases are disaggregated: non-disaggregated (NoDG) or fully disaggregated (FuDG). However, the NoDG strategy leads to strong prefill-decode interference and the FuDG strategy highly relies on high-performance interconnects, making them less cost-effective. We introduce EcoServe, a system that enables cost-effective LLM serving on clusters with commodity interconnects. EcoServe is built on the partially disaggregated (PaDG) strategy, applying temporal disaggregation and rolling activation for proactive intra- and inter-instance scheduling. It first disaggregates the prefill and decode phases along the time dimension within a single instance to mitigate inter-phase interference and enhance throughput. Next, it coordinates multiple instances and cyclically activates them to ensure the continuous availability of prefill processing, thereby improving latency. Thus, EcoServe's basic serving unit is the macro instance, within which multiple instances collaborate. It further integrates an adaptive scheduling algorithm to route requests in a macro instance and a mitosis scaling approach to enable fine-grained capacity scaling. Beyond delivering high goodput, EcoServe excels in load balancing, hardware cost, parallelism compatibility, and even engineering simplicity compared to existing solutions. When serving 30B- and 70B-scale models on a production-level cluster with 32 NVIDIA L20 GPUs using commodity Ethernet, EcoServe averagely improves goodput by 82.49%, 86.17%, 122.76%, and 126.96% over four representative NoDG and FuDG systems.

Figures

Figures reproduced from arXiv: 2504.18154 by the authors.

Figure 1
Figure 1. The NoDG, FuDG, and PaDG strategies. response times meet service level objectives (SLOs) becomes a primary goal. LLM inference consists of two distinct phases, the prefill phase and the decode phase, each associated with a different SLO, time to first token (TTFT) for the prefill phase and time per output token (TPOT) for the decode phase. The interplay between TTFT, TPOT, and throughput forms an inherent performanc… view at source ↗
Figure 2
Figure 2. LLM autoregressive decoding process. assigned to an instance that has just switched to process the decode phase, it would suffer from an unacceptably high TTFT. Rolling activation proactively coordinates multiple instances in a cyclic pattern. At any given time, there are instances specifically activated for processing prefills, capa￾ble of delivering acceptable TTFT latency. A group of such cooperating instances is… view at source ↗
Figure 3
Figure 3. Tensor parallelism and pipeline parallelism. Inter-decode data dependency Decode Batch Prefill Batch Inter-batch Imbalance Inter-layer data dependency Time GPU0 GPU1 GPU2 GPU3 Prefill-Decode Imbalance Inter-batch Imbalance [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Pipeline bubbles. these two phases exhibit distinct performance characteristics, i.e. compute-bound and memory-bound, the prefill phase saturates GPU computation even at a batch size of just one, while the decode phase requires a batch size in hundreds. By contrast, hy…
Figure 5
Figure 5. Figure 5: EcoServe Architecture Overview. the smallest scheduling unit in EcoServe is the macro in￾stance, adjusting capacity at the granularity of an entire macro instance can be inflexible and often leads to resource under utilization or waste. Inspired by biological cell mi￾t…
Figure 6
Figure 6. Figure 6: Runtime and frontend Timing. thereby ensuring the continuous availability of prefill pro￾cessing. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: The illustration of the expansion and contraction processes. Here 𝑁𝑙 = 3 and 𝑁𝑢 = 6. a macro instance, and subsequently adds or removes entire macro instances through splitting or merging. 3.5.1 Expansion and Contraction. We initially set two hyperparameters, 𝑁𝑙 and 𝑁𝑢…
Figure 8
Figure 8. Figure 8: End-to-end performance comparison. MoonCake and DistServe cannot meet SLOs in Llama-30B with LongBench. Metrics. We also use SLO attainment as the evaluation metric following prior works. Specifically, we compare sys￾tem throughput under different levels of SLO attainm…
Figure 9
Figure 9. Figure 9: demonstrates that both models’ serving achieve superlinear improvement under P90 SLO attainment. For 1 2 4 8 16 Instance Number 2 2 2 3 2 4 2 5 2 6 Throughput (req/s) CodeLlama2-34B L20 ShareGPT EcoServe P90 Linear 1 2 4 8 Instance Number 2 1 2 2 2 3 2 4 Qwen2-72B L20 …
Figure 10
Figure 10. Figure 10: Dynamic fine-grained scaling. Individual instances are dynamically added to a macro instance as request rates increase. Here 𝑁𝑙 = 4 and 𝑁𝑢 = 16. 100 200 300 400 500 TPOT SLO (ms) 20 40 Throughput (req/s) CodeLlama2-34B ShareGPT L20 P90 100 200 300 400 500 TPOT SLO (ms…
Figure 11
Figure 11. Figure 11: Pipeline parallel compatibility. overhead. For instance, re-initializing CodeLlama2-34B from L20 node’s local storage takes about 3 minutes, and this can be further prolonged when weights are loaded from remote storage. In conclusion, the mitosis scaling approach can …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 23 canonical work pages

  1. [1]

    Github Copilot

    2023. Github Copilot. https://github.com/features/copilot

    work page 2023

  2. [2]

    2024. Chatgpt. https://chat.openai.com

    work page 2024

  3. [3]

    Faster Transformer

    2024. Faster Transformer. https://github.com/NVIDIA/ FasterTransformer

    work page 2024

  4. [4]

    2024. SGLang. https://github.com/sgl-project/sglang

    work page 2024

  5. [5]

    vllm: Easy, fast, and cheap llm serving for everyone

    2024. vllm: Easy, fast, and cheap llm serving for everyone. https: //github.com/vllm-project/vllm

    work page 2024

  6. [6]

    Character ai

    2025. Character ai. https://character.ai

    work page 2025

  7. [7]

    2025. Cursor. https://www.cursor.com

    work page 2025

  8. [8]

    Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant Nair, Ilya Soloveychik, and Purushotham Kamath. 2024. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems 6 (2024), 114–127

    work page 2024

Show all 55 references
  1. [9]

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ram- jee. 2024. Taming{Throughput-Latency} Tradeoff in{LLM} Infer- ence with{Sarathi-Serve}. In 18th USENIX Symposium on Operating Systems Design and Imp...

    2024

  2. [10]

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. GQA: Training General- ized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv:2305.13245 [cs.CL] https://arxiv.org/abs/2305.13245

    2023 arXiv

  3. [11]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learn- ers. Advances in neural information processing systems 33 (2020), 1877–1901

    2020

  4. [12]

    Shiyi Cao, Shu Liu, Tyler Griggs, Peter Schafhalter, Xiaoxuan Liu, Ying Sheng, Joseph E Gonzalez, Matei Zaharia, and Ion Stoica. 2025. Moe- lightning: High-throughput moe inference on memory-constrained gpus. In Proceedings of the 30th ACM International Conference on Architect...

    2025

  5. [13]

    Shenggan Cheng, Ziming Liu, Jiangsu Du, and Yang You. 2023. ATP: Adaptive Tensor Parallelism for Foundation Models. arXiv preprint arXiv:2301.08658 (2023)

    2023 arXiv

  6. [14]

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

  7. [15]

    Jiangsu Du, Jinhui Wei, Jiazhi Jiang, Shenggan Cheng, Dan Huang, Zhiguang Chen, and Yutong Lu. 2024. Liger: Interleaving Intra-and Inter-Operator Parallelism for Distributed Large Model Inference. In Proceedings of the 29th ACM SIGPLAN Annual Symposium on Principles and Practi...

    2024

  8. [16]

    Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. 2024. Attention- store: Cost-effective attention reuse across multi-turn conversations in large language model serving. arXiv e-prints (2024), arXiv–2403

    2024

  9. [17]

    In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandel- wal, and Lin Zhong. 2024. Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems 6 (2024), 325–338

    2024

  10. [18]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  11. [19]

    Jiaao He and Jidong Zhai. 2024. Fastdecode: High-throughput gpu- efficient llm serving using heterogeneous pipelines. arXiv preprint arXiv:2403.11421 (2024)

    2024 arXiv

  12. [20]

    Yanping Huang, Youlong Cheng, Ankur Bapna, et al . 2019. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, Decem...

    2019

  13. [21]

    Ranggi Hwang, Jianyu Wei, Shijie Cao, Changho Hwang, Xiaohu Tang, Ting Cao, and Mao Yang. 2024. Pre-gated moe: An algorithm- system co-design for fast and scalable mixture-of-expert inference. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA)...

    2024

  14. [22]

    Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. 2023. DeepSpeed Ulysses: System Optimizations for Enabling Training of Ex- treme Long Sequence Transformer Models. arXiv:2309.14509 [cs.LG] https://arxiv.org...

    2023 arXiv

  15. [23]

    Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, and Xin Jin. 2024. Ragcache: Efficient knowledge caching for retrieval- augmented generation. arXiv preprint arXiv:2404.12457 (2024)

    2024 arXiv

  16. [24]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica

  17. [25]

    Jiamin Li, Yimin Jiang, Yibo Zhu, Cong Wang, and Hong Xu. 2023. Accelerating distributed{MoE} training and inference with lina. In 2023 USENIX Annual Technical Conference (USENIX ATC 23). 945– 959

    2023

  18. [26]

    Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, and Ion Stoica. 2021. Terapipe: Token-level pipeline parallelism for training large-scale language models. In International Conference on Machine Learning. PMLR, 6543–6552

    2021

  19. [27]

    Yunkai Liang, Zhangyu Chen, Pengfei Zuo, Zhi Zhou, Xu Chen, and Zhou Yu. 2025. Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregation. arXiv preprint arXiv:2503.20552 (2025)

    2025 arXiv

  20. [28]

    Bin Lin, Chen Zhang, Tao Peng, Hanyu Zhao, Wencong Xiao, Minmin Sun, Anmin Liu, Zhipeng Zhang, Lanbo Li, Xiafei Qiu, et al . 2024. Infinite-llm: Efficient llm service for long context with distattention and distributed kvcache. arXiv preprint arXiv:2401.02669 (2024)

    2024 arXiv

  21. [29]

    Hao Liu, Matei Zaharia, and Pieter Abbeel. 2023. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:2310.01889 (2023)

    2023 arXiv

  22. [30]

    Shu Liu, Asim Biswal, Audrey Cheng, Xiangxi Mo, Shiyi Cao, Joseph E Gonzalez, Ion Stoica, and Matei Zaharia. 2024. Optimizing llm queries in relational workloads. arXiv preprint arXiv:2403.05821 (2024)

    2024 arXiv

  23. [31]

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. 2023. Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification. arX...

    2023 arXiv

  24. [32]

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. 2019. PipeDream: Generalized pipeline parallelism for DNN training. In Proceedings of the 27th ACM Symposium on Operating Systems Princip...

    2019

  25. [33]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient generative llm inference using phase splitting. In2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 118–132

    2024

  26. [34]

    Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ram- jee, and Ashish Panwar. 2024. vattention: Dynamic memory man- agement for serving llms without pagedattention. arXiv preprint arXiv:2405.04437 (2024)

    2024 arXiv

  27. [35]

    Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading More Storage for Less Computation — A KVCache-centric 14 EcoServe: Enabling Cost-effective LLM Serving with Proactive Intra- and Inter-Insta...

    2025

  28. [36]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wen- han Xio...

    2024 arXiv

  29. [37]

    Noam Shazeer. 1911. Fast transformer decoding: One write-head is all you need, 2019. URL https://arxiv. org/abs (1911)

    1911

  30. [38]

    Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. 2018. Mesh-tensorflow: Deep learn- ing for supercomputers. Advances in neural information processing systems 31 (2018)

    2018

  31. [39]

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang

  32. [40]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, et al. 2019. Megatron- LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. CoRR abs/1909.08053 (2019). arXiv:1909.08053 http: //arxiv.org/abs/1909.08053

    2019 arXiv

  33. [41]

    Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He, and Abhinav Bhatele. 2023. A hybrid tensor- expert-data parallelism approach to optimize mixture-of-experts training. In Proceedings of the 37th International Conference on Supercomputing. 203–214

    2023

  34. [42]

    In International Conference on Machine Learning

    Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning. PMLR, 31094–31116

  35. [43]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  36. [44]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. At- tention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  37. [45]

    Hanshi Sun, Li-Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen. 2024. Shadowkv: Kv cache in shadows for high-throughput long-context llm inference. arXiv preprint arXiv:2410.21465 (2024)

    2024 arXiv

  38. [46]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guant- ing Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jin...

    2024 arXiv

  39. [47]

    Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2025. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowl- edge Fusion. In Proceedings of the Twentieth European Conference on Computer System...

    2025

  40. [48]

    Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. Loongserve: Efficiently serving long-context large language models with elastic sequence parallelism. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles. 640–654

    2024

  41. [49]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing...

    2023

  42. [50]

    2024.{DistServe}: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xu- anzhe Liu, Xin Jin, and Hao Zhang. 2024.{DistServe}: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation ...

    2024

  43. [51]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for {Transformer-Based} generative models. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 521– 538

    2022

  44. [52]

    Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A. Stuardo, Dongyang Wang, Xinlei Zhang, Huaping Zhou, Haoran Wei, Yang Cheng, Jianzhe Xiao, Xinyi Zhang, Lingjun Liu, Haibin Lin, Li-Wen Chang, Jianxi Ye, Xiao Yu, Xuanzhe Liu, Xin Jin, and Xin Liu. 2025. MegaScale-Infer: Se...

    2025 arXiv

  45. [54]

    Kan Zhu, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Yufei Gao, Qinyu Xu, Tian Tang, Zihao Ye, et al. 2024. Nanoflow: Towards optimal large language model serving throughput. arXiv preprint arXiv:2408.12757 (2024)

    2024 arXiv

  46. [2022]

    Advances in neural information processing systems 35 (2022), 16344–16359

    Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems 35 (2022), 16344–16359

    2022

  47. [2023]

    In Proceedings of the 29th Symposium on Operating Systems Principles

    Efficient memory management for large language model serv- ing with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles. 611–626

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.