Pith. sign in

REVIEW 2 major objections 1 minor 2 cited by

AFD sustains around 4k tokens/s throughput for DeepSeek-V3.2 under strict TTFT/TPOT SLOs where non-AFD is infeasible.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Operator-level attention-FFN disaggregation enables ~4k tokens/s throughput for DeepSeek-V3.2 under tight TTFT/TPOT SLOs where chunked-prefill and prefill-decode baselines cannot.

T0 review reviewed 2026-06-29 challenge →

load-bearing objection The paper maps operator-level AFD for MoE serving and claims 4k tokens/s under tight SLOs where baselines fail, but the number rests on unvalidated simulation. the 2 major comments →

arxiv 2605.28302 v1 pith:NJC3P2HV submitted 2026-05-27 cs.LG cs.AIcs.DC

How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving

Hanjiang Wu , Abhimanyu Rajeshkumar Bambhaniya , Sarbartha Banerjee , Tuhin Khare , Sudarshan Srinivasan , Suvinay Subramanian , Souvik Kundu , Madhu Kumar
show 4 more authors
Midhilesh Elavazhagan William Won Amir Yazdanbakhsh Tushar Krishna
This is my paper
classification cs.LG cs.AIcs.DC
keywords MoE LLM servingattention-FFN disaggregationprefill-decode disaggregationTTFT TPOT SLOssystem throughputGPU resource allocationDeepSeek-V3.2
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper investigates how far disaggregation can be pushed in serving large mixture-of-experts language models while meeting tight per-user latency requirements. It compares chunked-prefill and prefill-decode disaggregation against attention-FFN disaggregation, which moves attention and expert FFN computations onto separate GPU groups to match their differing resource profiles. A framework fusing on-device kernel data with network simulation evaluates the options across workloads that vary in sequence length, prefix reuse, and latency constraints. The analysis shows AFD enables viable high-throughput operation where earlier disaggregation levels cannot sustain the required performance.

Core claim

By placing attention and MoE-FFN execution on separate GPU groups, AFD manages the distinct demands of memory-bound attention, compute-intensive expert FFNs, and MoE dispatch/combine communication, sustaining around 4k tokens/s of system throughput on DeepSeek-V3.2 under strict TTFT/TPOT SLOs across chat, coding, and agentic-coding workloads, where non-AFD deployments are infeasible.

What carries the argument

Attention-FFN Disaggregation (AFD), which assigns attention computations and MoE expert FFN computations to distinct GPU groups.

Load-bearing premise

The fused on-device kernel measurements and high-fidelity network simulation accurately predict real rack- and cluster-scale behavior for AFD scheduling and communication patterns.

What would settle it

Deploying AFD at rack scale on real hardware for DeepSeek-V3.2 and checking whether measured system throughput under the same TTFT/TPOT SLOs reaches the simulated value of around 4k tokens/s.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Partitioning attention and FFN across GPUs must be chosen as a function of workload sequence lengths, prefix-KV reuse, and model architecture.
  • AFD becomes necessary to sustain both throughput and interactivity when strict per-user latency constraints are enforced on large MoE models.
  • Design principles for current rack- and cluster-scale deployments favor deeper disaggregation to handle MoE operator heterogeneity.
  • Joint optimization of throughput and interactivity requires evaluating all three disaggregation levels against the specific input/output characteristics of the target workload.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Extending the simulation framework to other MoE architectures could test whether the 4k tokens/s threshold generalizes beyond DeepSeek-V3.2.
  • Real interconnect topologies that deviate from the simulated network model might shift the workload regimes where AFD provides the largest gains.
  • The same measurement-plus-simulation approach could be applied to explore whether finer operator-level splits become worthwhile on next-generation hardware.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The manuscript explores the design space of operator-level Attention-FFN Disaggregation (AFD) for MoE LLM serving, comparing it to chunked-prefill and prefill-decode (P/D) disaggregation. Using a framework that fuses on-device kernel measurements with high-fidelity network simulation, it claims that under strict TTFT/TPOT SLOs, AFD sustains ~4k tokens/s system throughput on DeepSeek-V3.2 across chat, coding, and agentic-coding workloads (where non-AFD baselines are infeasible), and distills workload-dependent partitioning guidelines for attention and FFN across GPU groups.

Significance. If the simulation framework is shown to be accurate at rack/cluster scale, the work would supply concrete, workload-specific design principles for when AFD pays off versus prior disaggregation levels in MoE inference. The systematic coverage of input/output lengths, prefix-KV reuse, and latency constraints is a strength of the empirical characterization.

major comments (2)
  1. [Abstract and framework description] Abstract and framework description: The central quantitative claim (~4k tokens/s throughput and infeasibility of non-AFD baselines) is produced by fusing on-device kernel measurements with high-fidelity network simulation. The manuscript supplies no validation of this simulation against real hardware at rack or cluster scale for MoE dispatch/combine volumes, AFD scheduling overheads, prefix-KV reuse, or cross-GPU contention; any systematic bias in modeled interconnect or runtime costs would alter both the reported throughput and the feasibility conclusions.
  2. [Throughput results (workload tables/figures)] Throughput results (workload tables/figures): No error bars, sensitivity analysis to simulation parameters, or explicit workload exclusion criteria are reported for the 4k tokens/s figure or the baseline infeasibility statements. This makes it impossible to assess robustness of the cross-workload claim that only AFD meets the SLOs.
minor comments (1)
  1. [Notation and definitions] Notation for TTFT/TPOT SLO thresholds and the exact definition of 'system throughput' (tokens/s) should be stated once in a dedicated subsection rather than scattered across the abstract and results.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments highlighting the reliance on simulation and the need for robustness analysis. We address each point below and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: [Abstract and framework description] Abstract and framework description: The central quantitative claim (~4k tokens/s throughput and infeasibility of non-AFD baselines) is produced by fusing on-device kernel measurements with high-fidelity network simulation. The manuscript supplies no validation of this simulation against real hardware at rack or cluster scale for MoE dispatch/combine volumes, AFD scheduling overheads, prefix-KV reuse, or cross-GPU contention; any systematic bias in modeled interconnect or runtime costs would alter both the reported throughput and the feasibility conclusions.

    Authors: We agree that the framework has not been validated end-to-end against real hardware at rack or cluster scale for the listed MoE operations and overheads. The approach fuses real on-device kernel measurements with a high-fidelity network simulator, but this remains a limitation. In revision we will add an explicit Limitations section describing the kernel-level validation performed, the assumptions in the network model, and the potential effects of unmodeled contention or scheduling overheads, while qualifying the reported throughput numbers. revision: yes

  2. Referee: [Throughput results (workload tables/figures)] Throughput results (workload tables/figures): No error bars, sensitivity analysis to simulation parameters, or explicit workload exclusion criteria are reported for the 4k tokens/s figure or the baseline infeasibility statements. This makes it impossible to assess robustness of the cross-workload claim that only AFD meets the SLOs.

    Authors: We accept this observation. The revision will add error bars from repeated simulation runs, a sensitivity analysis on key parameters (interconnect bandwidth, latency, and dispatch volume), and explicit workload selection criteria. These additions will allow readers to evaluate the robustness of the 4k tokens/s claim and the baseline infeasibility statements. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical simulation-based characterization

full rationale

The paper is an empirical design-space exploration that fuses on-device kernel measurements with high-fidelity network simulation to evaluate attention-FFN disaggregation for MoE models. No derivation chain, equations, or predictions are presented that reduce by construction to fitted inputs or self-citations. The central throughput claims (e.g., ~4k tokens/s under SLOs) are direct outputs of the simulation framework rather than any self-referential definition or renaming of known results. The work is self-contained against external benchmarks via measurements and simulation, with no load-bearing self-citation or ansatz smuggling.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Only the abstract is available; no free parameters, axioms, or invented entities are identifiable.

reviewed 2026-06-29 · how reviews work

0 comments
Cite this review

Pith. "Pith review of How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving." pith.science (2026). https://pith.science/paper/NJC3P2HV

@misc{pith2026260528302,
  author       = {Pith},
  title        = {Pith review of: How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NJC3P2HV}},
  note         = {Machine review of arXiv:2605.28302}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Modern large language model (LLM) inference has progressively disaggregated to keep pace with growing model sizes and tight TTFT and TPOT service-level objectives: from chunked-prefill aggregation, to prefill-decode (P/D) disaggregation, and most recently to operator-level Attention-FFN Disaggregation (AFD). This trend is especially important for mixture-of-experts (MoE) models, where memory-bound attention, compute-intensive expert FFNs, and MoE dispatch/combine communication create distinct resource demands. AFD further exposes this heterogeneity by placing attention and MoE-FFN execution on separate GPU groups. Each level of disaggregation deepens the scheduling design space across workload characteristics, resource allocation, and interconnect topology, raising the central question: when does each level actually pay off? We systematically characterize this trade-off for MoE inference across realistic workloads spanning input/output sequence lengths, prefix-KV reuse, and per-user latency constraints. Using chunked-prefill and P/D disaggregation as baselines, we study the benefits and limits of AFD at scale through a framework that fuses on-device kernel measurements with high-fidelity network simulation. Under strict TTFT/TPOT SLOs, AFD sustains around 4k tokens/s of system throughput on DeepSeek-V3.2 across chat, coding, and agentic-coding workloads, where non-AFD deployments are infeasible. We distill concrete takeaways for jointly optimizing throughput and interactivity, including how to partition attention and FFN across GPUs as a function of workload and model architecture, providing design principles for current rack- and cluster-scale deployments as well as future disaggregated AI infrastructure.

Figures

Figures reproduced from arXiv: 2605.28302 by Abhimanyu Rajeshkumar Bambhaniya, Amir Yazdanbakhsh, Hanjiang Wu, Madhu Kumar, Midhilesh Elavazhagan, Sarbartha Banerjee, Souvik Kundu, Sudarshan Srinivasan, Suvinay Subramanian, Tuhin Khare, Tushar Krishna, William Won.

Figure 1
Figure 1. Figure 1: AIC++ Framework Overview. AIC++ takes model architecture, hardware configuration, and workload constraints as inputs and performs design-space exploration to identify optimal scheduling across token-level parallelism—data (DP), sequence (SP), tensor (TP), pipeline (PP), and expert parallelism (EP)—as well as phase-level prefill/decode (P/D) and operator-level attention–FFN disaggregation (AFD) [C1]. The fr… view at source ↗
Figure 2
Figure 2. Figure 2: System throughput at the best feasible deployment on [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: (a) Runtime breakdown and (b) memory component breakdown for different model [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Mapping AFD to 4 different stages in the model execution [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Evaluation of a Cluster with 128 B200 GPUs for the representative model architectures and [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Throughput frontier for the long-context Qwen3-235B workload on B200 GPUs. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: GPU Placement Worker Gattn Gffn Node (a) Segregated by phase — all prefill first, then decode Prefill0 {0, 1} {2, 3} 0 Prefill1 {4, 5} {6, 7} 0 Decode0 {8, 9} {10, 11} 1 Decode1 {12, 13} {14, 15} 1 (b) Paired P&D per node — co-locate each (𝑃𝑖 , 𝐷𝑖) Prefill0 {0, 1} {2, 3} 0 Decode0 {4, 5} {6, 7} 0 Prefill1 {8, 9} {10, 11} 1 Decode1 {12, 13} {14, 15} 1 [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: KV transfer latency under 2P2D: segregated by phase (dotted) vs. paired P&D per node [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Think Before You Grid-Search: Floor-First Triage for LLM Serving

    cs.PF 2026-07 conditional novelty 6.0

    A five-dimensional resource-vector floor model computes latency bounds and capacity walls for LLM serving, predicting when TP16 or EP16+DP attention layouts dominate based on operating point.

  2. Think Before You Grid-Search: Floor-First Triage for LLM Serving

    cs.PF 2026-07 conditional novelty 6.0

    LLM serving should triage by five-resource analytical floors and wall ordering, not grid search; on 16×H20, TP16 is capacity-capped at ~70 while EP+DP attention reaches ~644 concurrent 8K requests.

Reference graph

Works this paper leans on

13 extracted references · 7 canonical work pages · cited by 1 Pith paper · 3 internal anchors

  1. [1]

    SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills

    Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, and Ra- machandran Ramjee. Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills.arXiv preprint arXiv:2308.16369,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Vidur: A Large-Scale Simulation Framework For LLM Inference

    URLhttps://arxiv.org/abs/2405.05465. Vaswani Ashish. Attention is all you need.Advances in neural information processing systems, 30:I,

    work page Pith review arXiv

  3. [3]

    URLhttps://arxiv.org/abs/2504.09775. DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo G...

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

    URLhttps://arxiv.org/abs/2405.04434. Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    Efficient memory management for large language model serving with pagedattention,

    Association for Computing Machinery. ISBN 9798400702297. doi: 10.1145/3600006.3613165. URLhttps://doi.org/ 10.1145/3600006.3613165. Xupeng Miao, Yujie Wang, Youhe Jiang, Chunan Shi, Xiaonan Nie, Hailin Zhang, and Bin Cui. Galvatron: Efficient transformer training over multiple gpus using automatic parallelism.arXiv preprint arXiv:2211.13878,

    work page doi:10.1145/3600006.3613165

  6. [6]

    Accessed: 2025-04-10. Nvidia. Rubin (microarchitecture) - wikipedia, 6

    2025

  7. [7]

    [Online; accessed 2025-04-11]

    URLhttps://en.wikipedia.org/wiki/ Rubin_(microarchitecture). [Online; accessed 2025-04-11]. NVIDIA. Accelerating mixture of experts training with rail-optimized infiniband networking in crusoe cloud (presented by crusoe) s63014 | gtc san jose 2024 | nvidia on-demand.https: //www.nvidia.com/en-us/on-demand/session/gtc24-s63014/,

    2025

  8. [8]

    NVIDIA. InsideNVIDIAGroq3LPX:TheLow-LatencyInferenceAcceleratorfortheNVIDIAVera Rubin Platform.https://developer.nvidia.com/blog/inside-nvidia-groq-3-lpx-t he-low-latency-inference-accelerator-for-the-nvidia-vera-rubin-platform/ , 2026a. Accessed: 2026-05-07. NVIDIA. TensorRT-LLM. https://github.com/NVIDIA/TensorRT-LLM , 2026b. GitHub repository. Accessed...

    2026

  9. [9]

    StepMesh: A high-performance, low-latency communication library for attention-ffn disaggregation, 2025.https://github.com/stepfun-ai/StepMesh

    StepFun. StepMesh: A high-performance, low-latency communication library for attention-ffn disaggregation, 2025.https://github.com/stepfun-ai/StepMesh. vLLM Project. [Model] Attention FFN Disaggregation (AFD) for MoE Models.https://github .com/vllm-project/vllm/pull/29772,

    2025

  10. [10]

    Aiconfigurator: Lightning-fast configuration optimization for multi- framework llm serving.arXiv preprint arXiv:2601.06288, 2026

    URLhttps://arxiv.org/abs/2601.06288. Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributedservingsystemfor {Transformer-Based}generativemodels. In16thUSENIXSymposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538,

    work page arXiv

  11. [11]

    MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism

    URL https://arxiv.org/abs/2504.02263. 12 6 Appendix 6.1 vLLM Implementation for AFD We prototype AFD on top of vLLM v0.16.0rc2 Kwon et al

    work page Pith review arXiv

  12. [12]

    The prototype is used for correctness and implementation guidance, while the cluster-scale results use measured backend costs and AstraSim communication modeling

    to verify functional correctness of the full bipartite attention–FFN execution path and to align the implementation structure with the communication pattern modeled inAIC++. The prototype is used for correctness and implementation guidance, while the cluster-scale results use measured backend costs and AstraSim communication modeling. Our starting point i...

    2025

  13. [13]

    In (a), KV transfers cross the node boundary; in (b), each prefill–decode pair shares a node

    Attn0 Attn1 Node 0 FFN0 FFN1 Prefill 0Attn0 Attn1 FFN0 FFN1 Prefill 1 Attn0 Attn1 Node 0 FFN0 FFN1 Prefill 0Attn0 Attn1 FFN0 FFN1 Decode0 Attn0 Attn1 Node 1 FFN0 FFN1 Decode 0Attn0 Attn1 FFN0 FFN1 Attn0 Attn1 Node 1 FFN0 FFN1 Prefill 1Attn0 Attn1 FFN0 FFN1 Decode1 Decode 1 ✅ ✅ ❌ ❌Slower Inter-node BW Faster Intra-node BW Assignment of GPUs by Contiguous P...

    2024

This paper was first reviewed by grok-4.3 on June 29, 2026.