Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Beyond the Buzz: A Pragmatic Take on Inference Disaggregation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Disaggregated inference serving—splitting prefill and decode onto separate GPU pools—expands the throughput–interactivity frontier mainly for prefill-heavy traffic and models above about 10 billion parameters, provided the ratio between…

desk verdict A clearly written, ambitious simulation study from NVIDIA on when disaggregated LLM serving wins, but the load-bearing quantitative results rest on an unvalidated proprietary simulator and I found a concrete bug in the rate-matching algorithm in Appendix B. read the letter →

arxiv 2506.05508 v1 pith:EMBERR3Z submitted 2025-06-05 cs.DC cs.AI

Tiyasa Mitra , Ritika Borkar , Nidhi Bhatia , Ramon Matas , Shivam Raj , Dheevatsa Mudigere , Ritchie Zhao , Maximilian Golub
show 5 more authors
Arpan Dutta Sailaja Madduri Dharmesh Jani Brian Pharris Bita Darvish Rouhani
This is my paper
classification cs.DCcs.AI
keywords LLMinferenceservingprefill-decodedisaggregationthroughput-interactivityParetofrontierdynamicratematchingKVcachetransferchunkedpipelineparallelismGPUperformancesimulationBlackwelldatacenter
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Disaggregated inference—splitting an LLM's prefill phase and decode phase onto separate GPU pools—is often presented as a universal speedup, but this paper argues the benefit is conditional. Using a proprietary datacenter-scale GPU simulator, the authors sweep hundreds of thousands of design points across models, traffic patterns, latency targets, and hardware, and conclude that disaggregation clearly wins only under prefill-heavy traffic ($\mathrm{ISL} \gg \mathrm{OSL}$) and for larger models (above roughly 10 billion parameters). In those regimes it expands the throughput–interactivity frontier beyond what co-located serving can reach. The optimal context-to-generation GPU ratio shifts sharply with latency targets, so a fixed ratio leaves throughput on the table, and KV cache transfer between pools can be hidden with existing datacenter bandwidth. The paper's design rules are conditional on the simulator's fidelity, but if its trends hold they tell operators where disaggregation is worth the added complexity.

What carries the argument

The organizing object is the throughput–interactivity Pareto frontier, in which overall throughput in tokens per second per GPU trades against interactivity measured as tokens per second per user (the reciprocal of token-to-token latency). The argument is carried by a proprietary, high-fidelity GPU performance simulator that takes a model architecture, traffic pattern, and GPU configuration and outputs latency and throughput for many batch sizes and parallelism strategies, together with a rate-matching integer solver that, for a fixed prefill configuration, finds the minimal total GPU count and the balanced ratio of prefill to decode instances subject to the token-to-token latency constraint. Two named techniques do specific work: Chunked Pipeline Parallelism (CPP), which splits long input sequences into chunks and overlaps the later layers of earlier chunks with earlier layers of later ones, lets prefill pools meet strict first-token latency without wide tensor parallelism; and the layer-by-layer KV cache transfer model, captured in egress and ingress bandwidth equations, shows that transfer can overlap with computation rather than becoming a bottleneck.

What would settle it

Run the same prefill-heavy DeepSeek-R1 workload (input length 16K, output length 2K) on a real Blackwell cluster, measure the throughput–interactivity curves for co-located and disaggregated serving, and check whether disaggregation beats co-location in the medium-latency regime and whether a fixed context-to-generation ratio loses as much throughput as Figure 10 predicts.

Watch

Extended reading notes

Core claim

The paper's central claim is that disaggregation provides the greatest benefits in prefill-heavy traffic scenarios ($\mathrm{ISL} \gg \mathrm{OSL}$) and when serving larger models (e.g., more than 10 billion parameters). Larger models benefit because they are mapped across more GPUs, giving prefill and decode pools a richer space of distinct parallelism strategies to choose from; prefill-heavy traffic benefits because co-located mappings that favor decode speed end up compromising prefill throughput, while a separate decode pool can push tensor parallelism aggressively without hurting prefill. The paper also finds that dynamic rate matching is required to stay on the Pareto frontier, that chunked pipeline parallelism in the prefill pool is the best way to meet strict first-token latency targets, and that layer-by-layer KV cache transfer keeps bandwidth requirements within provisioned datacenter capacity. For co-located serving, context-chunked piggybacking helps most under relaxed latency targets and generation-heavy traffic, and its effectiveness is architecture-sensitive: multi-latent attention pays a redundant computation cost per prefill chunk that grouped-query attention does not.

Load-bearing premise

All conclusions rest on the proprietary GPU performance simulator reproducing real Blackwell datacenter behavior, but the paper reports no validation against measured cluster results, no error bars, and no calibration details.

Editorial extensions

If this is right

  • Operators should target disaggregation first at prefill-heavy workloads ($\mathrm{ISL} \gg \mathrm{OSL}$) and models larger than about 10 billion parameters, where it actually expands the throughput–interactivity frontier.
  • A static context-to-generation GPU ratio is a trap: a ratio that is best at relaxed latency targets degrades sharply as latency tightens, so production systems need dynamic rate matching.
  • For co-located serving, context-chunked piggybacking is worth adopting mainly under relaxed latency targets and generation-heavy traffic, and its benefit depends on the attention mechanism (for example, multi-latent attention versus grouped-query attention).
  • KV cache transfer need not block adoption: overlapping transfer with prefill computation keeps the required egress and ingress bandwidth within provisioned datacenter capacity across the studied range.
  • Larger NVLink domains improve disaggregated serving because they widen the choice of tensor and expert parallelism available to the decode pool.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the simulator's trends transfer to real clusters, the same phase-asymmetry argument should extend to vision-language models, and to future models with cheaper attention mechanisms, where the relative benefit of disaggregation could shift as KV cache size shrinks.
  • The model-size result is attributed to a richer parallelization search space, so disaggregation gains should continue to grow as models and NVLink domains scale; small models with narrow parallelism options should remain co-located even under prefill-heavy traffic.
  • The paper's demonstration that P50 sequence lengths approximate dynamic traffic suggests that production operators could decide between co-located and disaggregated serving from coarse traffic percentiles alone, without maintaining full workload traces.
  • A direct way to test the generality of the result would be to run the same design-space sweep on an open, auditable performance model or against measured cluster data, which would show whether the trends are an artifact of the proprietary simulator.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents a simulation-based study of disaggregated LLM inference, claiming to be the first systematic exploration of the throughput-interactivity Pareto frontier at datacenter scale. Using a proprietary GPU-performance simulator, it evaluates hundreds of thousands of design points across models, traffic patterns, parallelism strategies, and latency targets. The central findings are that disaggregation is most beneficial for prefill-heavy traffic and larger models, that dynamic rate matching between prefill and decode pools is critical, and that KV-cache transfer bandwidth is not a bottleneck under the modeled assumptions. The paper also contributes analytic bandwidth formulas (Eqs. 1 and 2) and a discussion of context chunking and NVLink domain size sensitivity.

Significance. If the simulator faithfully reproduces real Blackwell datacenter behavior, the paper's conclusions would be practically valuable: they would provide concrete design guidance for disaggregated serving, identify the regimes where disaggregation pays off, and emphasize the need for elastic, rate-matched resource allocation. The analytic bandwidth derivation in Section 5.1 is a useful, parameter-free first-order check that can be evaluated independently. The paper is also candid about the trend-based nature of its normalized results and acknowledges scenarios where disaggregation offers limited benefit. At present, however, the lack of any real-system validation, combined with an internal inconsistency in the rate-matching algorithm described in Appendix B, means that the quantitative Pareto frontiers and the headline comparisons rest on unverified and partly contradictory foundations.

major comments (3)
  1. [Section 3.1 and Figures 1, 6-11] All central claims, including the Section 1 statement that disaggregation provides the greatest benefits in prefill-heavy traffic and for larger models, are derived entirely from a 'proprietary, high-fidelity GPU performance simulator' with no validation against measurements from a real Blackwell cluster, no calibration details, no error bars, and no sensitivity analysis. Because every Pareto-frontier comparison inherits this unverified modeling, the paper's headline conclusions are not independently checkable and could reflect simulator artifacts rather than real system behavior.
  2. [Appendix B, Algorithm 2] The rate-matching formulas on lines 9-10 of Algorithm 2 do not implement the balance condition. Since alpha = round(P/D) is the ratio of per-GPU prefill request throughput to per-GPU decode request throughput, balanced total throughput requires N_d/N_p = alpha, i.e., N_p = d*K and N_d = n*K when alpha = n/d. The algorithm instead sets num_prefill_gpus = numerator(alpha) * G and num_decode_gpus = denominator(alpha) * best_prefill_config.num_gpus, giving N_d/N_p = (d * p)/(n * G), which equals alpha only when the per-replica prefill and decode GPU counts happen to be equal. For p=4 prefill GPUs, G=8 decode GPUs, and alpha=2, this returns N_p=16 and N_d=4, a 4:1 imbalance in the wrong direction rather than the intended decode-heavy ratio. Since every disaggregated Pareto point in Figures 1, 6-11 is produced by this rate-matching step, the quantitative frontiers are invalid as written. If the actual implementation uses the integer solver mentioned in Section 3.2 and differs from Algorithm 2, then the appendix is misleading and the real algorithm is not disclosed.
  3. [Appendix C, Figure 14] The claim that the P50 power-of-two ISL/OSL approximation 'provides a reliable representation of the Pareto frontier under dynamic real-world traffic conditions' is validated by comparing the approximation to a direct simulation of the dynamic distribution using the same unvalidated simulator. This does not establish that the P50 proxy is faithful to real dynamic workloads; it only shows that the two simulation modes agree within the simulator's own modeling assumptions. Since all main results use the P50 proxy, this validation gap is load-bearing for the traffic-sensitivity conclusions.
minor comments (5)
  1. [Section 3.1, paragraph on simulator] The phrase 'high-fidelity GPU performance simulator' should be accompanied by at least a high-level description of its calibration approach, input parameters, and any previous validation against real hardware, so readers can assess its applicability.
  2. [Figure 1 caption] The caption states results are normalized but does not define the normalization baseline or display axis units; adding a brief explanation would make the Pareto frontiers interpretable.
  3. [Section 5.1, Eq. (1)] The variable 'bytes_element' is described as 'KV cache bytes per token' but is multiplied by d_head and N_kvheads; the naming is confusing and should be clarified to indicate bytes per element or the formula should be rewritten with a per-token-per-layer KV size.
  4. [Algorithm 1] The returned 'best_prefill_throughput' is used later in Algorithm 2, but the variable 'G' inside the loop is the prefill config's GPU count; this is fine, yet the code block would benefit from explicit type annotations or comments distinguishing requests/second/GPU from tokens/second/GPU.
  5. [Appendix B, Algorithm 2, line 8] The function round(x, tolerance) is not standard and its rounding semantics are unspecified; if this is intended as a tolerance-aware rational approximation, it should be defined or replaced with a standard integer rounding that is consistent with the integer-solver description in Section 3.2.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: simulator-driven study with parameter-free formulas; minor non-load-bearing self-citation.

full rationale

The paper's derivation chain is not circular. The central load-bearing claims (disaggregation is most effective for prefill-heavy traffic and larger models; dynamic rate matching is important) are presented as observed trends from a proprietary high-fidelity GPU performance simulator (Section 3.1), not as conclusions forced by an equation that already contains the answer. The only closed-form derivations, Eq. (1) and Eq. (2) in Section 5.1, are dimensional first-principles calculations of KV-transfer bandwidth; none of their terms is fitted to the reported Pareto results, so the bandwidth conclusions do not reduce to their inputs. Appendix B's rate-matching routines are algorithmic and, as written, appear internally inconsistent for general GPU counts (a feasibility bug rather than a circularity); they do not define the dependent variable as the independent variable. Reference [13], a self-citation used for FP4 precision, is a hardware-precision background citation and is not load-bearing for the main claims. Appendix C's validation of the P50 approximation compares two outputs of the same unvalidated simulator, which is a weak internal-consistency check rather than external validation, but it is not circular in the sense of a fitted parameter renamed as a prediction or an ansatz smuggled in by citation. The absence of real-cluster validation is a correctness and evidentiary risk, not a logical circularity. The score reflects one minor self-citation that is not load-bearing; no circular step was identified.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

No new physical or architectural entities are postulated. The paper introduces no new forces, particles, or hardware components; it evaluates existing mechanisms (disaggregation, chunked pipeline parallelism) in a simulator.

free parameters (4)
  • FTL cutoff (10 s) = 10 seconds
    Chosen by hand (Section 3.2) to prune the design space. All Pareto frontiers drop configurations with FTL > 10 s, which shapes which designs appear and can favor disaggregated prefill pools that hit the cutoff.
  • Rate matching tolerance = 0.03
    Integer solver rounding tolerance in Algorithm 2 (Appendix B). The tolerance controls how finely the context-to-generation GPU ratio is rounded and therefore affects every Pareto point.
  • P50 ISL/OSL power-of-two proxy = power-of-two quantiles
    Dynamic traffic is approximated by constant sequence lengths equal to the closest power-of-two to the 50th percentile (Section 4.2, Appendix C). The proxy is validated only by the same simulator used for the main results.
  • Hidden simulator calibration parameters = unknown
    The 'proprietary, high-fidelity GPU performance simulator' (Section 3.1) must be calibrated to GPU performance, but the calibration is not released. Every trend in the paper depends on these hidden parameters.
assumptions (6)
  • ad hoc to paper The proprietary simulator is high-fidelity
    Invoked in Section 3.1 as the source of all latency and throughput numbers; no validation against a real cluster is shown.
  • domain assumption Sufficient GPUs and incoming requests exist to fully utilize rate-matched deployments
    Stated in Section 3.2. This removes queueing effects and under-utilization, which real deployments may experience.
  • domain assumption KV cache transfer can be overlapped with prefill compute
    Stated in Section 3.2 and analyzed in Section 5.1. If overlap is imperfect, the bandwidth requirements would be higher than computed.
  • domain assumption P50 ISL/OSL is a reliable proxy for dynamic traffic
    Used throughout Section 4.2; the supporting evidence in Appendix C is generated by the same simulation stack.
  • domain assumption The enumerated parallelism strategies (TP, EP, PP, CPP, TEP) cover the practically relevant design space
    Section 3.1 restricts the search to these strategies; other techniques (e.g., speculative decoding, KV cache reuse) are deferred to future work.
  • domain assumption FP4 precision on Blackwell is the representative target
    Section 3.1 states the analysis focuses on Blackwell systems using FP4 precision, which may not generalize to other precision or hardware generations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond the Buzz: A Pragmatic Take on Inference Disaggregation." pith.science (2026). https://pith.science/paper/EMBERR3Z

@misc{pith2026250605508,
  author       = {Pith},
  title        = {Pith review of: Beyond the Buzz: A Pragmatic Take on Inference Disaggregation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EMBERR3Z}},
  note         = {Machine review of arXiv:2506.05508}
}
read the original abstract

As inference scales to multi-node deployments, disaggregation - splitting inference into distinct phases - offers a promising path to improving the throughput-interactivity Pareto frontier. Despite growing enthusiasm and a surge of open-source efforts, practical deployment of disaggregated serving remains limited due to the complexity of the optimization search space and system-level coordination. In this paper, we present the first systematic study of disaggregated inference at scale, evaluating hundreds of thousands of design points across diverse workloads and hardware configurations. We find that disaggregation is most effective for prefill-heavy traffic patterns and larger models. Our results highlight the critical role of dynamic rate matching and elastic scaling in achieving Pareto-optimal performance. Our findings offer actionable insights for efficient disaggregated deployments to navigate the trade-off between system throughput and interactivity.

Figures

Figures reproduced from arXiv: 2506.05508 by the authors.

Figure 1
Figure 1. Throughput–interactivity Pareto frontier for DeepSeek-R1. The benefits of disaggregated [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Visualization of (left) co-located and (right) disaggregated inference serving, illustrating [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. High-level overview of rate matching for disaggregated serving. KV cache and weights are [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: High-level overview of chunked pipeline parallelism. It works by: (i) splitting the input [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Chunked pipeline parallelism during Prefill is an optimal strategy to maximize throughput [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Disaggregated vs. co-located serving. Co-located serving overall (red-solid) is the [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Larger models benefit more from disaggregated serving due to a richer search space. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Disaggregation helps most under prefill-heavy traffic. [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: The optimal ratio of ctx-to-gen GPUs varies across models and target latencies. [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: Optimal rate matching dynamically adapts Ctx:Gen ratio to deliver Pareto optimal [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]
Figure 11
Figure 11. Figure 11: Larger NVLink domain helps disaggregated serving performance. DeepSeek-R1 benefits [PITH_FULL_IMAGE:figures/full_fig_p007_11.png]
Figure 12
Figure 12. Figure 12: Bandwidth requirements for KV cache transfer: Maximum of egress and ingress bandwidth [PITH_FULL_IMAGE:figures/full_fig_p008_12.png]
Figure 13
Figure 13. Figure 13: Example distribution of ISL and OSL in dynamic traffic. [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: Comparison of Pareto frontiers using dynamic traffic simulation versus P50 approximation. [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sangam: Efficiently Serving Diffusion LLMs with the AR Stack

    cs.DC 2026-07 conditional novelty 6.5 of 10

    Deficit token-budget scheduling plus hybrid prefill overflow makes AR-style colocated and disaggregated serving work for cached diffusion LLMs with block-sized, recurring prefills.

  2. Efficient and Scalable Agentic AI with Heterogeneous Systems

    cs.LG 2025-07 conditional novelty 5.0 of 10

    A system design and preliminary simulation argue that pairing older H100 GPUs with Gaudi 3 accelerators can match or beat all-B200 clusters on total cost of ownership for AI agent workloads.

Reference graph

Works this paper leans on

28 extracted references · 12 canonical work pages · cited by 2 Pith papers

  1. [1]

    Gulavani, and Ramachandran Ramjee

    Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, and Ramachandran Ramjee. SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills. 2023. URLhttps://arxiv.org/abs/2308.16369

    arXiv 2023

  2. [2]

    Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. pages 117–134, 2024. URL https://www.usenix.org/conference/osdi24/ presentation/agrawal

    work page 2024

  3. [3]

    TensorRT-LLM: Disaggregated Serving Implementation

    NVIDIA. TensorRT-LLM: Disaggregated Serving Implementation. 2024. URL https://docs.nvidia. com/dynamo/latest/architecture/disagg_serving.html. NVIDIA Documentation

    work page 2024

  4. [4]

    vLLM: Disaggregated Prefilling (experimental)

    vLLM Team. vLLM: Disaggregated Prefilling (experimental). 2024. URL https://docs.vllm.ai/ en/stable/features/disagg_prefill.html. vLLM Documentation

    work page 2024

  5. [5]

    DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serv- ing

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serv- ing. pages 193–210, 2024. URL https://www.usenix.org/conference/osdi24/presentation/ zhong-yinmin. 9

    work page 2024

  6. [6]

    Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot

    Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot. pages 155–170, 2025. URL https://www.usenix.org/ conference/fast25/presentation/qin

    work page 2025

  7. [7]

    P/D-Serve: Serving Disaggregated Large Language Model at Scale

    Yibo Jin, Tao Wang, Huimin Lin, Mingyang Song, Peiyang Li, Yipeng Ma, Yicheng Shan, Zhengfan Yuan, Cailong Li, Yajing Sun, Tiandeng Wu, Xing Chu, Ruizhi Huan, Li Ma, Xiao You, Wenting Zhou, Yunpeng Ye, Wen Liu, Xiangkun Xu, Yongsheng Zhang, Tiantian Dong, Jiawei Zhu, Zhe Wang, Xijian Ju, Jianxun Song, Haoliang Cheng, Xiaojing Li, Jiandong Ding, Hefei Guo,...

    arXiv 2024

  8. [8]

    Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. 2020. URL https://arxiv.org/abs/1909.08053

    arXiv 2020

Show all 28 references
  1. [9]

    GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. 2021. URLhttps://openreview.net/forum?id=qrwe7XHTmYb

    2021

  2. [10]

    GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, and zhifeng Chen. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. 32, 2019. URL https://proceedings.neurips.cc/pape...

    2019

  3. [11]

    Devanur, Gregory R

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. PipeDream: generalized pipeline parallelism for DNN training. pages 1–15, 2019. URLhttps://doi.org/10.1145/3341301.3359646

    2019

  4. [12]

    NVIDIA Blackwell Architecture Technical Brief, 2024

    NVIDIA. NVIDIA Blackwell Architecture Technical Brief, 2024. URL https://cdn. prod.website-files.com/61dda201f29b7efc52c5fbaf/6602ea9d0ce8cb73fb6de87f_ nvidia-blackwell-architecture-technical-brief.pdf . NVIDIA Technical Documenta- tion

    2024

  5. [13]

    Microscaling Data Formats for Deep Learning

    Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, Stosic Dusan, Venmugil Elango, Maximilian Golub, Alexander Heinecke, Phil James-Roxby, Dharmesh Jani, Gaurav Kolhe, Ma...

    2023 arXiv

  6. [14]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025 arXiv

  7. [15]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Ar...

    2024 arXiv

  8. [16]

    DeepSpeed-Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, and Yuxiong He. DeepSpeed-Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale. pages 1–15, 202...

    2022 arXiv

  9. [17]

    Orca: A Distributed Serving System for Transformer-Based Generative Models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A Distributed Serving System for Transformer-Based Generative Models. pages 521–538, 2022. URL https://www.usenix.org/conference/osdi22/presentation/yu

    2022

  10. [18]

    Efficient Memory Management for Large Language Model Serving with PagedAttention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient Memory Management for Large Language Model Serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles, SO...

    2023

  11. [19]

    Efficiently Scaling Transformer Inference

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently Scaling Transformer Inference. pages 606–624, 2023. URL https://proceedings.mlsys.org/paper_files/paper/2023/file/ c4be71ab8d2...

    2023

  12. [20]

    Splitwise: Efficient Generative LLM Inference Using Phase Splitting

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. pages 118–132, 2024. URLhttps://doi.org/10.1109/ISCA59077.2024.00019

    2024

  13. [21]

    DéjàVu: KV- cache Streaming for Fast, Fault-tolerant Generative LLM Serving

    Foteini Strati, Sara Mcallister, Amar Phanishayee, Jakub Tarnawski, and Ana Klimovic. DéjàVu: KV- cache Streaming for Fast, Fault-tolerant Generative LLM Serving. pages 46745–46771, 2024. URL https://proceedings.mlr.press/v235/strati24a.html

    2024

  14. [22]

    DynaServe: Unified and Elastic Tandem-Style Execution for Dynamic Disaggregated LLM Serving

    Chaoyi Ruan, Yinhe Chen, Dongqi Tian, Yandong Shi, Yongji Wu, Jialin Li, and Cheng Li. DynaServe: Unified and Elastic Tandem-Style Execution for Dynamic Disaggregated LLM Serving. 2025. URL https://arxiv.org/abs/2504.09285

    2025 arXiv

  15. [23]

    S. Chen, R. Jiang, D. Yu, J. Xu, M. Chao, F. Meng, C. Jiang, W. Xu, and H. Liu. KVDirect: Distributed Disaggregated LLM Inference. 2024. URLhttps://arxiv.org/abs/2501.14743

    2024 arXiv

  16. [24]

    Inference without Inter- ference: Disaggregate LLM Inference for Mixed Downstream Workloads

    Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. Inference without Inter- ference: Disaggregate LLM Inference for Mixed Downstream Workloads. 2024. URL https: //arxiv.org/ab...

    2024 arXiv

  17. [25]

    FastDecode: High-Throughput GPU-Efficient LLM Serving using Heteroge- neous Pipelines

    Jiaao He and Jidong Zhai. FastDecode: High-Throughput GPU-Efficient LLM Serving using Heteroge- neous Pipelines. 2024. URLhttps://arxiv.org/abs/2403.11421

    2024 arXiv

  18. [26]

    HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment

    Youhe Jiang, Ran Yan, and Binhang Yuan. HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment. 2024. URLhttps://openreview.net/forum?id=Cs6MrbFuMq. 12

    2024

  19. [27]

    MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool

    Cunchen Hu, Heyang Huang, Junhao Hu, Jiang Xu, Xusheng Chen, Tao Xie, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool. 2024. URLhttps://arxiv.org/abs/2406.17565

    2024 arXiv

  20. [28]

    Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregation

    Shaoyuan Chen, Wencong Xiao, Yutong Lin, Mingxing Zhang, Yingdi Shan, Jinlei Jiang, Kang Chen, and Yongwei Wu. Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregation. 2025. URLhttps://arxiv.org/abs/2405.01814. 13 A Terminology This section de...

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.