Pith. sign in

REVIEW 3 major objections 6 minor 46 references

The paper identifies X-Stage, a post-issue stage where accepted remote stores drain while the issuing GPU resumes computation, and shows finite drain capacity, not just schedule, determines sender backpressure in fused kernels.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-07-31 23:53 UTC pith:IH7VMAAA

load-bearing objection A credible new scheduling lever for GPU remote stores, well-validated at mechanism level, with a legitimate open question about whether its calibrated drain rate transfers to bidirectional workloads. the 3 major comments →

arxiv 2607.23264 v1 pith:IH7VMAAA submitted 2026-07-25 cs.DC cs.AI

X-Stage: An Overlooked Pipeline Stage for Communication-Computation Overlap in DiT Inference

Jianwen Xian , Zhiyuan Xu , Yuchen Li , Ziliang Lai , Kang He , Zhen Huang , Aichen Feng , Jinyan Chen
show 3 more authors
Yilin Zhang Qinqin Chen Chengru Song
This is my paper
classification cs.DC cs.AI
keywords remote storedevice-initiated communicationcommunication-computation fusionbackpressureburst-gap modelmixture of expertssequence parallelismperformance modeling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Persistent GPU kernels that communicate by issuing remote stores have long been scheduled by two events: when the store is issued and when the receiver may consume it. The paper argues there is a third, overlooked event: after a store is accepted, it continues to drain toward remote visibility while the issuing GPU returns to computation. This post-issue phase, called X-Stage, is finite — sustained bursts fill an effective outstanding capacity and backpressure later issues, eventually stalling the Tensor Core producer. The paper models X-Stage as a Burst-Gap process with three calibrated platform parameters, then uses the model to make two scheduling changes: interleaving independent computation between concentrated bursts in a fused mixture-of-experts kernel, and piggybacking the output all-to-all exchange onto the natural compute loop of sequence-parallel attention. If the model is right, communication overhead in fused kernels can be hidden by shaping when bursts enter the network, rather than by reserving dedicated communication warps or SMs.

Core claim

The paper's central claim: there is a software-visible phase, X-Stage, between remote-store issue and remote-visible completion. Short bursts drain while the issuer resumes work; sustained injection exhausts finite outstanding capacity and delays later issues. A Burst-Gap model parameterized by backpressure-free issue time, effective drain rate (~717 GB/s), and outstanding capacity (~4.25 MiB) predicts sender-visible issue time as max(T_iss^0, V/R − G), a max law rather than a completion-coupled sum. Using the model, the paper interleaves independent linear work between concentrated bursts in a fused MoE kernel (1.18x geo-mean speedup over 84 configurations) and piggybacks the output all-to-

What carries the argument

The central object is the Burst-Gap model, a fluid queue abstraction calibrated by three measurable platform parameters: backpressure-free issue time T_iss^0, effective aggregate drain rate R, and effective outstanding capacity Q. It asserts the sender-visible issue time for burst volume V followed by a producer-side gap G is T_iss = max(T_iss^0, V/R − G), which yields a max law for the steady-state period and a recovery gap G* = [V/R − T_iss^0]+. Combined with the single-burst capacity bound [V − R·T_iss^0]+ ≤ Q, it tells a scheduler whether the natural compute gap is sufficient or whether injection must be reshaped — the test that drives both kernel redesigns.

Load-bearing premise

The model assumes the downstream remote-store path behaves as a single work-conserving queue with a constant effective drain rate R over the measured regime, calibrated on one eight-GPU system; if drain rate varies with burst interleaving, receiver congestion, memory placement, or clock state, the max law and capacity bound can mispredict, and other communication operations are explicitly outside the validated scope.

What would settle it

On a different eight-GPU node, run the paper's zero-gap periodic-burst microbenchmark and an isolated-burst sweep. If the drain-limited period V/R does not scale linearly with V with the same R across producer counts, or if the isolated-burst knee moves so that Q differs by more than measurement noise, the calibrated fluid model does not transfer. A sharper test: keep V and G fixed but split the gap into two sub-intervals with different interleavings; the model predicts identical issue time, since only the total G matters.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Sender-visible issue time depends only on burst volume, gap, drain rate, and capacity: adding useful computation between bursts reduces issue stall one-for-one until the backpressure-free floor is reached.
  • A fused kernel is backpressure-free when both the long-term rate bound V/(T_iss^0 + G) ≤ R and the single-burst capacity bound [V − R·T_iss^0]+ ≤ Q hold; violating either predicts sender-side stall that can propagate to the Tensor Core critical path.
  • A natural compute loop that already exceeds the recovery gap — such as the full Q-loop of sequence-parallel attention at long sequence lengths — can absorb the post-issue drain, so no dedicated communication warp or SM is needed.
  • Cross-wave interleaving of independent linear-layer work achieves a 1.18x geometric-mean and 1.62x maximum speedup over the wave-ordered baseline across 84 MoE configurations, with skewed routing benefiting most.
  • The completion-coupled model (issuer blocked until remote visibility) would predict a residual that grows linearly with sequence length; the X-Stage model predicts a flat, near-zero residual, and the measured slope matches the flat prediction.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same audit could be applied to other fused kernels with remote-store epilogues: if the natural gap already clears G*, piggyback issue on the output-owning role; if not, reorder independent work or reduce burst volume. The paper demonstrates only the two cases, but the test is general.
  • The calibration parameters R and Q were measured on one eight-GPU system for one-sided remote stores; a portable implementation would need to recalibrate per system and per operation class (loads, atomics, collectives) and should check whether the drain rate is invariant to receiver-side load.
  • A testable extension is to treat burst shaping as a runtime decision: choose tile size, staging depth, and interleaving distance jointly against the rate and capacity bounds, while accounting for the cache-locality regression the paper observed in some balanced-routing MoE configurations.
  • The model's claim that adding a producer-side gap overlaps with draining rather than delaying the next burst suggests that latency models of fused kernels should replace completion-coupled sums with the max law; this is a direct, checkable prediction on other hardware.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper identifies X-Stage, a software-visible post-issue pipeline stage for device-initiated remote stores on NVIDIA GPUs, in which accepted stores continue progressing toward remote-visible completion after the issuing role resumes execution. A Burst–Gap model parameterized by backpressure-free issue time T_iss^0, effective drain rate R, and effective outstanding capacity Q predicts sender-visible issue overhead, recovery gaps, and capacity-induced backpressure. The model is used to guide two fused-kernel schedules: in DeepGEMM MegaMoE, cross-wave interleaving of Linear-1 and Linear-2 tiles is claimed to reshape concentrated Combine bursts, yielding a 1.18x geometric-mean / 1.62x maximum kernel speedup over 84 configurations; in Ulysses sequence-parallel attention, per-tile piggybacking of the post-attention All-to-All into the FlashAttention Q-loop is claimed to hide sender-side issue overhead without a dedicated communication warp or SM, with maximum sender-visible speedups of 1.43x (FA3) and 1.42x (FA4). The paper also reports mechanism-level validations: Tensor Core timelines, per-tile remote-store span distributions, local-store controls, a DSv4-Pro negative control, and a sequence-length residual-slope test that distinguishes X-Stage behavior from a completion-coupled counterfactual.

Significance. If the quantitative claims hold, the paper makes a useful systems contribution: it identifies a measurable, modelable scheduling dimension—post-issue remote-store progress—that existing tile-fusion and communication-overlap treatments omit. The Burst–Gap model is simple, and the three parameters are calibrated from microbenchmarks and then used without application-specific refitting; the paper also commits to a falsifiable prediction (the residual-slope test in Figure 11), which is a strength. The mechanism-level evidence is unusually thorough: the local-store control, the DSv4-Pro negative control, the per-tile instrumentation, and the comparison against a completion-coupled counterfactual give the X-Stage phenomenon independent support beyond aggregate kernel speedups. However, the application-regime transfer of the calibrated drain rate R is the main technical risk, as explained below. The paper is not a trivial incremental report; if the transfer concern is resolved, it should be of interest to the systems and GPU-kernel communities.

major comments (3)
  1. [§3.3, §4.3, §5.3, §6.3] R (717 GB/s) is calibrated from unidirectional peer-to-peer remote stores, but both target workloads are bidirectional/all-to-all: MegaMoE Combine is many-to-many and Ulysses A2A is all-to-all, so every GPU is simultaneously a sender and receiver. Section 6.3 explicitly excludes receiver congestion. This is load-bearing: Eq. (14) (MegaMoE sufficiency), Eq. (10) (rate and capacity bounds), and the FA design test depend directly on R. The flat E_res slope in Figure 11 is insensitive to R in the measured range—with M ≥ 8,192 and M*_ub ≈ 1.0K, even a 20–30% lower R would leave G_Qloop ≫ G*, so the slope test does not validate R transfer. I request a bidirectional/A2A calibration of R, or a sensitivity analysis for Eq. (14), Eq. (10), and Table 3, or an explicit restriction of the quantitative model claims to the unidirectional calibration domain.
  2. [§4.3, Eq. (10)] The capacity bound for the FlashAttention piggyback design has a very thin margin: V≈4.6MiB, (V−R·T_iss^0)^+≈4.1MiB, Q≈4.25MiB, leaving about 0.15MiB. R is reported without an uncertainty estimate. Under bidirectional traffic, R could plausibly drop enough to violate the capacity bound, which would undermine the stated design test. Please report uncertainty in R and Q, or provide a sensitivity bound showing over what R range the inequality holds.
  3. [§4.2, Eq. (14)] The aggregate burst volume in Eq. (14) is V = K_act·V_t, where K_act is defined as “the number of producers whose tile boundaries align in a burst.” The paper never specifies how K_act is measured, bounded, or estimated in the MegaMoE evaluation. Without this operational definition, Eq. (14)'s sufficiency test and the claim that interleaving moves each tile to the backpressure-free floor cannot be reproduced from the text. Please state the alignment assumption and report the K_act values used for the configurations in Figure 8 and Table 2.
minor comments (6)
  1. [§5.1] Please name the exact GPU model and NVLink generation. “A recent NVIDIA architecture” with 148 SMs is not reproducible, and the values of R, Q, and T_iss^0 are hardware-specific.
  2. [Table 3] Report confidence intervals, per-trial spread, or repeated-run variability for the reported medians. Several hiding ratios exceed 100% and are attributed to noise; without variance information it is hard to assess the short-sequence speedup claims.
  3. [Figure 11] The caption refers to “prior predictions” and a ±2% DVFS envelope, but the derivation of the dotted issue floor and the envelope is not fully specified in the caption or text. Please make the plotted curves directly traceable to Eqs. (18)–(20).
  4. [Algorithm 1] The notation G, stride G, lead D, P, P1, P2, and the roles are under-defined. Please add a table or a few explanatory sentences so the scheduler can be understood without reverse-engineering.
  5. [References] Reference [10] is marked “Working reference; add commit hash and access date before submission,” and [25], [27] have similar unresolved access-date placeholders. These should be completed; the DeepGEMM commit hash is needed to reproduce the MegaMoE baseline.
  6. [§5.3, Figure 11] The ±2% DVFS noise band is large relative to the claimed residual values (e.g., ±107 µs at M=65,536 for FA4). The statement that residuals are “consistent with zero” is therefore weak unless clock-stabilized reruns are described in more detail and the actual spread is reported.

Circularity Check

0 steps flagged

No significant circularity: platform parameters are calibrated independently and applied to applications without refitting; remaining concerns are external validity, not circularity.

full rationale

The central derivation chain is the Burst-Gap model (Eqs. 1–5) and its application in Eqs. (10), (14), (18), and (20). The parameters T_iss^0, R, and Q are obtained from dedicated remote-store microbenchmarks in Section 3 (zero-gap drain-limited steady state, recovered plateau, isolated-burst knee) and are explicitly held fixed when predicting MegaMoE and FlashAttention–A2A behavior: "Platform parameters (T_iss^0,R,Q) are calibrated once by the microbenchmarks in Section 3 and are not refitted to applications." The empirical checks are non-tautological: the max law in Eq. (3) is contrasted with the completion-coupled sum; the MegaMoE span shift is compared with local-store controls; DSv4-Pro provides a negative control; and the FA4 residual-vs-sequence-length slope is tested against the M*_ub bound rather than fitted to the application results. The limitations stated in Section 6.3 (one-sided remote stores, no receiver congestion, no synchronization overheads) are external-validity threats: if the unidirectional R does not transfer to bidirectional A2A/MoE regimes, the predictions could be wrong, but that is not a reduction of the conclusion to its inputs. No load-bearing self-citation was found: references to DeepGEMM, FlashAttention, and prior fusion systems are external implementations, and the Section 2.3 mismatch is explicitly called a motivation rather than a proof. The score reflects only the mild calibration/validation overlap inherent in using microbenchmarks both to discover and to parameterize the abstraction, not an actual circular step.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 1 invented entities

The model rests on three calibrated platform parameters (T_iss^0, R, Q) plus one measured compute slope (sigma), and on the domain assumption that remote stores behave as posted writes draining through a fluid, work-conserving path. X-Stage itself is the only genuinely invented conceptual entity, and it has independent predictive handles.

free parameters (4)
  • R (effective aggregate drain rate) = ~717 GB/s
    Fitted from zero-gap periodic-burst steady-state periods (Figure 3d) and used in Eqs. (3), (5), (6), (10), (14), and (20).
  • T_iss^0(K,V) (backpressure-free issue time) = e.g., ~0.76 us at K=148, B=32 KB
    Calibrated from the recovered large-gap plateau in microbenchmark sweeps; central parameter of the Burst-Gap model.
  • Q(K) (effective outstanding capacity) = ~4.25 MiB for K=148
    Derived from the isolated-burst knee using Eq. (6); used in the single-burst capacity bound Eq. (10) and M*_ub computation.
  • sigma (Q-loop gap slope) = ~0.79 us per KV tile
    Measured from FlashAttention Q-loop compute gap; used to derive M*_ub in Eq. (20).
axioms (5)
  • domain assumption Downstream remote-store path drains in a work-conserving manner at constant rate R.
    Invoked in Section 3.2, Eq. (1) and throughout the model; not proven independently on other topologies or under receiver congestion.
  • domain assumption Remote stores are posted writes accepted by the sender; issue completion can precede remote-visible completion.
    This is the existence condition for X-Stage, established by microbenchmarks but assumed as the hardware model for the applications.
  • domain assumption The producer-side gap G introduces no new remote stores.
    Used in Eqs. (1)-(3) and in the MegaMoE interleaving argument where Linear-1 work is treated as a clean drain window.
  • domain assumption Effective outstanding capacity Q is finite and depends only on the producer configuration K; application configurations stay within K<=148.
    Capacity is measured for one configuration and extrapolated to the FlashAttention producer count and MegaMoE tile alignment.
  • domain assumption Steady-state fluid abstraction ignores synchronization, receiver congestion, launch overhead, and DVFS variation.
    Acknowledged as a limitation in Section 6.3; the model does not account for all sender-side overheads.
invented entities (1)
  • X-Stage (software-visible post-issue pipeline stage) independent evidence
    purpose: Names the interval between remote-store issue and remote-visible completion, during which accepted stores drain while the issuer resumes work.
    The paper gives falsifiable predictions not used to fit the constants: residual-vs-sequence-length slope, M*_ub convergence bound, and the drain-limited vs backpressure-free span regimes; these are compared against instrumentation measurements.

pith-pipeline@v1.3.0-alltime-deepseek · 21965 in / 11649 out tokens · 111027 ms · 2026-07-31T23:53:38.219766+00:00 · methodology

0 comments
read the original abstract

Fine-grained, device-initiated communication lets persistent GPU kernels in distributed diffusion transformer (DiT) inference issue remote stores and overlap data movement with Tensor Core computation. Existing systems schedule when communication is issued and when received data becomes consumable, but omit post-issue progress before remote-visible completion, making sender backpressure hard to predict. We identify X-Stage, a software-visible post-issue pipeline stage. Measurements on an eight-GPU node with a recent NVIDIA architecture show that short remote-store bursts drain as the issuer resumes work, whereas sustained injection exhausts finite outstanding capacity and delays later issues. A lightweight Burst-Gap model parameterized by backpressure-free issue time, effective drain rate, and outstanding capacity predicts issue overhead, recovery between bursts, and the onset of backpressure. Guided by the model, we redesign two communication-computation fused kernels. For DeepGEMM MegaMoE, interleaving Linear-1 and Linear-2 work across expert waves places computation between concentrated remote-store bursts, yielding a 1.18x geometric-mean and 1.62x maximum kernel speedup over the Expert-Wave baseline across 84 configurations. For Ulysses sequence-parallel attention, tile-granular fusion of the post-attention All-to-All with FlashAttention lets an output-tile owner issue remote stores and resume computation without a dedicated communication warp or streaming multiprocessor. FlashAttention-3 and FlashAttention-4 reach maximum sender-visible speedups of 1.43x and 1.42x over serial execution, and at long sequences their steady-state times approach those of FlashAttention alone. These results establish post-issue progress as a measurable scheduling lever for shaping bursts, avoiding backpressure, and hiding sender-side overhead.

Figures

Figures reproduced from arXiv: 2607.23264 by Aichen Feng, Chengru Song, Jianwen Xian, Jinyan Chen, Kang He, Qinqin Chen, Yilin Zhang, Yuchen Li, Zhen Huang, Zhiyuan Xu, Ziliang Lai.

Figure 1
Figure 1. Figure 1: Remote stores issued by an epilogue continue to make progress in X-Stage while the next mainloop performs useful computation. The remote-store burst and the intervening mainloop form the recurring burst–gap pattern used throughout this paper. interface for fine-grained one-sided operations without re￾turning to the host or launching a separate collective kernel for every transfer [28]. Fine-grained communi… view at source ↗
Figure 2
Figure 2. Figure 2: Task-level and X-Stage-aware views of the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Remote-store microbenchmark characterization of X-Stage. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: X-Stage-aware MegaMoE scheduling with a separate epilogue role. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Ulysses sequence-parallel attention. QKV pro￾jection and the pre-attention All-to-All transform sequence￾partitioned tokens into head partitions (a–c). FlashAttention and the post-attention All-to-All produce the output and restore the original sequence partition (d–e); our FA+A2A design fuses these latter two operations [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: FlashAttention Q-loop and output tiling. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Piggybacked FlashAttention–A2A pipeline for FlashAttention-4. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: MegaMoE kernel speedup. Speedup of the X-Stage-aware interleaved scheduler over the expert-wave scheduler, across models, precisions, and routing configurations [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: MegaMoE Tensor Core timeline (Tensor-pipe [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 11
Figure 11. Figure 11: Relative sender-visible residual 𝐸res(𝑀)/𝑇FA for FA4+A2A. Prior predictions use the once-calibrated platform parameters and measured compute-side timing, without application refitting. The dotted reference represents an approximately 16 𝜇s issue floor, and the blue band is a ±2% DVFS envelope. The worst-case convergence threshold 𝑀∗ ub (K=148) ≈ 1.0K lies below all measured points. estimates range from −3… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 12 linked inside Pith

  1. [1]

    Ionescu, Klaus E

    Albert Alexandrov, Mihai F. Ionescu, Klaus E. Schauser, and Chris Scheiman. 1995. LogGP: Incorporating Long Messages into the LogP Model. InProceedings of the Seventh Annual ACM Symposium on Par- allel Algorithms and Architectures (SPAA). 95–105

    1995

  2. [2]

    Li-Wen Chang, Wenlei Bao, Qi Hou, Chengquan Jiang, Ningxin Zheng, Yinmin Zhong, Xuanrun Zhang, Zuquan Song, Chengji Yao, Ziheng Jiang, Haibin Lin, Xin Jin, and Xin Liu. 2024. FLUX: Fast Software- based Communication Overlap on GPUs Through Kernel Fusion.arXiv preprint arXiv:2406.06858(2024)

    Pith/arXiv arXiv 2024

  3. [3]

    Meghan Cowan, Saeed Maleki, Madanlal Musuvathi, Olli Saarikivi, and Yifan Xiong. 2023. MSCCLang: Microsoft Collective Communication Language. InProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS). doi:10.1145/3575693.3575724Originally released as GC3, arXiv:2201.11840

    arXiv 2023

  4. [4]

    Minyu Cui, Anna Wingkvist, and Morgan Ericsson. 2026. Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts.arXiv preprint arXiv:2607.19539 (2026)

    Pith/arXiv arXiv 2026

  5. [5]

    David Culler, Richard Karp, David Patterson, Abhijit Sahay, Klaus Erik Schauser, Eunice Santos, Ramesh Subramonian, and Thorsten von Eicken. 1993. LogP: Towards a Realistic Model of Parallel Computation. InProceedings of the Fourth ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP). 1–12

    1993

  6. [6]

    Tri Dao. 2024. FlashAttention-2: Faster Attention with Better Paral- lelism and Work Partitioning. InInternational Conference on Learning Representations (ICLR). 35549–35562

    2024

  7. [7]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

  8. [8]

    DeepSeek-AI. 2024. DeepSeek-V3 Technical Report.arXiv preprint arXiv:2412.19437(2024)

    Pith/arXiv arXiv 2024

  9. [9]

    DeepSeek-AI. 2025. EPLB: Expert Parallelism Load Balancer. GitHub repository.https://github.com/deepseek-ai/EPLB

    2025

  10. [10]

    DeepSeek-AI. 2026. DeepGEMM: High-performance Tensor Core Kernels and Mega MoE. GitHub repository. Working reference; add commit hash and access date before submission

    2026

  11. [11]

    DeepSeek-AI. 2026. DeepSeek-V4: Towards Highly Efficient Million- Token Context Intelligence.arXiv preprint arXiv:2606.19348(2026)

    arXiv 2026

  12. [12]

    Jiarui Fang and Shangchun Zhao. 2024. USP: A Unified Sequence Parallelism Approach for Long Context Generative AI.arXiv preprint arXiv:2405.07719(2024)

    Pith/arXiv arXiv 2024

  13. [13]

    Zhengcong Fei, Mingyuan Fan, Changqian Yu, Debang Li, and Junshi Huang. 2024. Scaling Diffusion Transformers to 16 Billion Parameters. arXiv preprint arXiv:2407.11633(2024)

    Pith/arXiv arXiv 2024

  14. [14]

    Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. 2023. MegaBlocks: Efficient Sparse Training with Mixture-of-Experts. In Proceedings of Machine Learning and Systems (MLSys), Vol. 5. 288–304

    2023

  15. [15]

    Jiaao He, Jidong Zhai, Tiago Antunes, Haojie Wang, Fuwen Luo, Shangfeng Shi, and Qin Li. 2022. FasterMoE: Modeling and Optimizing Training of Large-Scale Dynamic Pre-trained Models. InProceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP). 120–134. doi:10.1145/3503221.3508418

    arXiv 2022

  16. [17]

    Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. 2023. DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models.arXiv preprint arXiv:2309.14509(2023)

    Pith/arXiv arXiv 2023

  17. [18]

    Abhinav Jangda, Jun Huang, Guodong Liu, Amir Hossein Nodehi Sa- bet, Saeed Maleki, Youshan Miao, Madanlal Musuvathi, Todd Mytkow- icz, and Olli Saarikivi. 2022. Breaking the Computation and Commu- nication Abstraction Barrier in Distributed Machine Learning Work- loads. InProceedings of the 27th ACM International Conference on Ar- chitectural Support for ...

    arXiv 2022

  18. [19]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guil- laume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Tev...

    Pith/arXiv arXiv 2024

  19. [20]

    Chao Jin, Ziheng Jiang, Zhihao Bai, Zheng Zhong, Juncai Liu, Xiang Li, Ningxin Zheng, Xi Wang, Cong Xie, Qi Huang, Wen Heng, Yiyuan Ma, Wenlei Bao, Size Zheng, Xuegui Zheng, Yanghua Peng, Haibin Lin, Xuanzhe Liu, Xin Jin, and Xin Liu. 2026. MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production. InProceeding...

    arXiv 2026

  20. [21]

    John D. C. Little. 1961. A Proof for the Queuing Formula: 𝐿=𝜆𝑊 . Operations Research9, 3 (1961), 383–387

    1961

  21. [22]

    Hao Liu, Matei Zaharia, and Pieter Abbeel. 2024. RingAttention with Blockwise Transformers for Near-Infinite Context. InInternational Conference on Learning Representations (ICLR). 3992–4008

    2024

  22. [23]

    Yijun Ma, Siyuan Shen, Tiancheng Chen, Akhil Langer, Jiri Kraus, Ben- jamin Glick, Craig Belusar, Jeff Hammond, and Torsten Hoefler. 2026. Demystifying NVSHMEM: A System-Level Analysis on Symmetric Memory and Device-Initiated Operations in GPU Communication. arXiv preprint arXiv:2606.05951(2026)

    Pith/arXiv arXiv 2026

  23. [24]

    Xinxin Mei, Qiang Wang, and Xiaowen Chu. 2017. A Survey and Measurement Study of GPU DVFS on Energy Conservation.Digital X-Stage: An Overlooked Pipeline Stage for Communication–Computation Overlap in DiT Inference Communications and Networks3, 2 (2017), 89–100

    2017

  24. [25]

    NVIDIA. 2026. CUDA C++ Programming Guide: Peer Device Memory Access. NVIDIA Developer Documentation. Accessed 2026; add exact URL and access date before submission

    2026

  25. [26]

    NVIDIA. 2026. NVIDIA Collective Communications Library (NCCL) Documentation.https://docs.nvidia.com/deeplearning/nccl/user- guide/docs/. Version 2.30.7, accessed July 2026

    2026

  26. [27]

    NVIDIA. 2026. NVLink and NVSwitch System Overview. NVIDIA Data Center Documentation. Accessed 2026; add exact URL and access date before submission

    2026

  27. [28]

    NVIDIA. 2026. NVSHMEM: GPU-Initiated Communication Library. NVIDIA Developer Documentation. Accessed 2026; add exact URL and access date before submission

    2026

  28. [29]

    Xinglin Pan, Wenxiang Lin, Lin Zhang, Shaohuai Shi, Zhenheng Tang, Rui Wang, Bo Li, and Xiaowen Chu. 2025. FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 (ASPLOS). 524–539. doi:10....

    arXiv 2025

  29. [30]

    William Peebles and Saining Xie. 2023. Scalable Diffusion Models with Transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 4195–4205

    2023

  30. [31]

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. 2024. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision. InAdvances in Neural Information Processing Systems 37 (NeurIPS). 68658–68685

    2024

  31. [32]

    Noam Shazeer. 2020. GLU Variants Improve Transformer.arXiv preprint arXiv:2002.05202(2020)

    Pith/arXiv arXiv 2020

  32. [33]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv preprint arXiv:1909.08053(2019)

    Pith/arXiv arXiv 2019

  33. [34]

    Sul, Simran Arora, Benjamin F

    Stuart H. Sul, Simran Arora, Benjamin F. Spector, and Christopher Ré. 2026. ParallelKittens: Systematic and Practical Simplification of Multi-GPU AI Kernels. InProceedings of Machine Learning and Systems (MLSys)

    2026

  34. [35]

    Haotian Sun, Tao Lei, Bowen Zhang, Yanghao Li, Haoshuo Huang, Ruoming Pang, Bo Dai, and Nan Du. 2025. EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing. InInternational Conference on Learning Representations (ICLR). 72383–72401

    2025

  35. [36]

    Didem Unat, Ilyas Turimbetov, Mohammed Issa, Dogan Sagbili, Flavio Vella, Daniele De Sensi, and Ismayil Ismayilov. 2026. The Landscape of GPU-Centric Communication.Comput. Surveys58, 12 (2026). doi:10.1 145/3813799

    2026

  36. [37]

    Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai

  37. [38]

    Yingping Wang, Yi Wu, Xiangyu Wu, Junwei Cui, Weilin Cai, Zhijiang Guo, and Jiayi Huang. 2026. ReaLB: Real-Time Load Balancing for Multimodal MoE Inference.arXiv preprint arXiv:2604.19503(2026)

    Pith/arXiv arXiv 2026

  38. [39]

    Samuel Williams, Andrew Waterman, and David Patterson. 2009. Roofline: An Insightful Visual Performance Model for Multicore Ar- chitectures.Commun. ACM52, 4 (2009), 65–76

    2009

  39. [40]

    Yanpeng Yu, Haiyue Ma, Krish Agarwal, Nicolai Oswald, Qijing Huang, Hugo Linsenmaier, Chunhui Mei, Ritchie Zhao, Ritika Borkar, Bita Darvish Rouhani, David Nellans, Ronny Krashinsky, and Anurag Khan- delwal. 2025. Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokens.arXiv preprint arXiv:2512.09277 (2025)

    arXiv 2025

  40. [41]

    Yike Yuan, Ziyu Wang, Zihao Huang, Defa Zhu, Xun Zhou, Jingyi Yu, and Qiyang Min. 2025. Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Experts. InProceed- ings of the 42nd International Conference on Machine Learning (ICML). 73671–73682

    2025

  41. [42]

    Ted Zadouri, Markus Hoehnerbach, Jay Shah, Timmy Liu, Vijay Thakkar, and Tri Dao. 2026. FlashAttention-4: Algorithm and Ker- nel Pipelining Co-Design for Asymmetric Hardware Scaling.arXiv preprint arXiv:2603.05451(2026)

    arXiv 2026

  42. [43]

    Shulai Zhang, Ningxin Zheng, Haibin Lin, Ziheng Jiang, Wenlei Bao, Chengquan Jiang, Qi Hou, Weihao Cui, Size Zheng, Li-Wen Chang, Quan Chen, and Xin Liu. 2025. Comet: Fine-grained Computation- Communication Overlapping for Mixture-of-Experts. InProceedings of Machine Learning and Systems (MLSys)

    2025

  43. [44]

    Size Zheng, Jin Fang, Xuegui Zheng, Qi Hou, Wenlei Bao, Ningxin Zheng, Ziheng Jiang, Dongyang Wang, Jianxi Ye, Haibin Lin, Li-Wen Chang, and Xin Liu. 2025. TileLink: Generating Efficient Compute- Communication Overlapping Kernels Using Tile-Centric Primitives. InProceedings of Machine Learning and Systems (MLSys)

    2025

  44. [45]

    Size Zheng, Xuegui Zheng, Li-Wen Chang, and Jidong Zhai. 2026. UniEP: Unified Expert-Parallel MegaKernel MoE for LLM Training. In Proceedings of the 35th International Symposium on High-Performance Parallel and Distributed Computing (HPDC). 387–401. doi:10.1145/38 06645.3807818

    arXiv 2026

  45. [2022]

    InAdvances in Neural Information Processing Systems 35 (NeurIPS)

    FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. InAdvances in Neural Information Processing Systems 35 (NeurIPS)

  46. [2024]

    Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of- Experts.arXiv preprint arXiv:2408.15664(2024)

    Pith/arXiv arXiv 2024