Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Reordering MoE rows by destination rank lets the return all-to-all hide behind expert compute, giving up to 2.64x end-to-end and 2.74x layer speedups on four A100 GPUs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 12:26 UTC pith:W2ZARMHG

load-bearing objection Original and mostly plausible, but Section 4.5's padding bound is off by a factor of E and the correctness anomaly is unresolved; worth a serious referee but needs real revisions. the 3 major comments →

arxiv 2607.19539 v1 pith:W2ZARMHG submitted 2026-07-21 cs.DC cs.AI

Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts

Minyu Cui , Anna Wingkvist , Morgan Ericsson This is my paper
classification cs.DC cs.AI
keywords Mixture-of-Expertsdistributed LLM inferenceall-to-all communicationcomputation-communication overlappersistent GPU kernelstile-level signalingexpert parallelismremote-owner-aligned layout
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that the second all-to-all in distributed Mixture-of-Experts (MoE) inference—the transfer of expert outputs back to the ranks that originally sent each token—does not have to wait for expert computation to finish. A persistent GEMM kernel (the producer) runs all local experts in a single launch and publishes a readiness flag as each output tile completes; a persistent communication kernel (the consumer) on a small dedicated SM partition polls those flags and forwards completed row segments to their owner ranks while the GEMM continues. The enabling device is a remote-owner-aligned row layout: rows are grouped by destination rank and padded to tile boundaries so every output tile belongs to exactly one remote rank, reducing transfer bookkeeping to a few array lookups and letting remote-bound tiles be scheduled first. The paper reports up to 2.64x end-to-end and 2.74x MoE-layer speedups over four established MoE systems on a 4-A100 platform, with overlap ratios of 71.9–99.9%. If the claim holds, MoE inference can be accelerated without modifying the underlying GEMM kernels or communication primitives.

Core claim

The paper's central claim is that the second all-to-all of an MoE layer is not an indivisible tail but a stream of tile-sized dependencies that can be consumed as they are produced. A rank-wide persistent GEMM kernel (the producer) runs all local experts in one launch, schedules remote-bound tiles first, and publishes a readiness flag after each tile's epilogue store; a persistent communication kernel on a small SM partition (the consumer) polls those flags and issues one contiguous remote write per row-band segment. The remote-owner-aligned row layout makes every output tile map to a single destination rank, so the consumer needs no per-row routing. Across three MoE models, four baselines,

What carries the argument

The central mechanism is the remote-owner-aligned row layout, which groups each expert's input rows by destination rank and pads to tile-height multiples so every output tile carries a statically known owner. This is combined with tile-level signaling (a per-tile readiness flag published in the GEMM epilogue after a thread-fence), a remote-first tile schedule that emits the largest remote-bound transfers earliest, and a persistent consumer kernel that transfers ready row-band segments (contiguous strips of output rows spanning the full width) as single contiguous remote writes. The consumer runs on a small tunable SM partition, so the producer's GEMMs and the transfer pipeline genuinely exec

Load-bearing premise

The load-bearing premise is that a tile written by the producer is globally visible to the remote-write engine before the consumer observes the readiness flag; if that ordering is not guaranteed, the consumer can transfer stale or partially-written data.

What would settle it

Run the producer-consumer pair under a memory-ordering stress test with checksummed tiles while deliberately weakening or removing the thread-fence between tile store and flag publication: any reproducible corruption at the receiving rank would falsify the overlap's correctness claim. Alternatively, reproduce the single tolerance miss the paper observed under stress_skew: if that outlier becomes reproducible across seeds, the design is not reliably safe.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • The return all-to-all in MoE layers can be almost fully hidden, with measured overlap ratios from 71.9% to 99.9% as the number of experts per rank varies.
  • Operator-level speedups reach 2.97–3.01x over a sequential compute-then-communicate baseline, and MoE-layer speedups reach 1.61–1.78x across router modes.
  • The benefit grows with expert count and with return-communication volume, so larger expert-parallel configurations gain the most.
  • Resource balance is decisive: with too few consumer SMs the overlap collapses and can be slower than the baseline, while the best latency appears at 10–20 consumer SMs.
  • The final per-token output matches the sequential baseline within FP16 rounding tolerance, so the overlap does not change numerical results beyond expected noise.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's explicit scope, the same producer-consumer signaling pattern could be applied to the first all-to-all (dispatch) and to MoE training's backward pass, since both share the GEMM-then-transfer structure; the paper leaves those directions open.
  • The padding overhead scales with the number of ranks and tile height, so on larger clusters the remote-owner-aligned layout may need intra-node versus inter-node grouping to avoid excessive padding and bandwidth loss—this is an editorial extrapolation, not a paper claim.
  • The strong sensitivity to consumer SM share suggests that a runtime-adaptive partition selector, sizing the consumer by observed routing skew and tile production rate, could be tested as a drop-in improvement.
  • The bandwidth curves that motivate segment sizing were measured on one 4-A100 NVLink node; on systems with a different compute-to-interconnect ratio, the optimal segment size and SM partition would shift, so the speedup numbers should not be expected to transfer unchanged to other topologies.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a producer-consumer co-design for distributed MoE inference that overlaps expert GEMM with the second (return) all-to-all communication. A persistent rank-wide GEMM kernel (producer) uses tile-level signaling from the CUTLASS epilogue; a persistent communication kernel (consumer) runs on a dedicated SM partition and issues segment-granular NVSHMEM puts as tiles become ready. A remote-owner-aligned row layout and a remote-first tile schedule ensure that each output tile maps to a single destination rank and that remote-bound tiles are produced early. The evaluation on a 4-A100 node covers three MoE models, four baselines, shape/router/swEEP sweeps, SM-partition contention, and 1440 correctness checks, claiming up to 2.64x end-to-end and 2.74x MoE-layer speedups.

Significance. If the empirical claims hold, this is a practical and useful contribution: it achieves fine-grained overlap without intrusive kernel fusion, reusing standard CUTLASS and NVSHMEM primitives. The remote-owner-aligned row layout is an elegant way to make tile-level signaling applicable to the dynamic routing of MoE return traffic. The paper is also relatively honest: it reports the losing M-Trans-xl configuration and the c_CTA=2 failure, and it ships a broad sensitivity study rather than cherry-picked points. However, the overhead analysis contains a provable arithmetic error, and the memory-ordering argument for correctness is underspecified. These issues must be fixed before the central claims can be accepted.

major comments (3)
  1. [§4.5 / Algorithm 1] The padding-overhead bound is wrong by a factor of E, the number of experts per rank. §4.5 states the maximum padded rows per rank is (W−1)(tbM−1), giving 765 rows and calling it 'a few percent.' But Algorithm 1 pads each remote rank group independently for every local expert: lines 2–7 loop over r≠r_self inside the per-expert loop. The true worst case is sum_{e,r≠r_self} (tbM − (|G_{e,r}| mod tbM)), bounded by E(W−1)(tbM−1), not (W−1)(tbM−1). For the evaluated E=64, W=4, tbM=256 this bound is 48,960 rows, larger than the M=16,384 GEMM dimension in Table 2; even roughly balanced routing gives per-source-per-expert groups of M/(EW)=64 rows, requiring 192 padded rows each, i.e., ~36,864 padded rows per rank. This invalidates the 'a few percent' overhead claim and weakens the extrapolation in §4.3 that overlap gains grow with E, because the baseline does not pay this padding cost. The measu
  2. [§3.3.1 / Table 3] The correctness argument relies on a memory-ordering protocol that is under-specified. The paper says only that after writing a tile the producer performs a thread-fence and then publishes a readiness flag; the consumer polls the flag and issues an NVSHMEM put. CUDA's __threadfence() alone does not constitute a portable acquire/release protocol unless the flag is atomic/volatile with the correct memory scope, and the paper does not specify the types, scope, or ordering of the flag operations, nor how the NVSHMEM put observes the tile stores. Since a stale or partial tile would silently corrupt MoE outputs, this is load-bearing. Table 3 also contains an internal inconsistency: the text says one stress_skew iteration initially exceeded the tolerance, but the table reports 1440/1440 pass without qualification. Please specify the exact instruction-level protocol (e.g., atomicAcquire/Release
  3. [§4.2–§4.3] All latency and speedup numbers appear to be single measurements with no repetitions, error bars, or variance information. Given the sharp c_CTA=2 slowdown and the one unexplained correctness anomaly, a few repeated runs per configuration are needed to establish that the reported 1.3–5.3x speedups are not within run-to-run noise. Please report at least the min/median (or mean ± std) over five runs for the headline figures (Figures 4–8), or state explicitly how many repetitions were used.
minor comments (4)
  1. [Abstract / §4.2.1] The abstract says the approach 'consistently improves' performance across configurations, but §4.2.1 reports that M-Trans-xl trails Tutel, Megatron-CUTLASS, and Megatron-TE, and §4.3.4 shows c_CTA=2 is slower than the baseline. Please qualify the 'consistent' claim to reflect these disclosed exceptions.
  2. [Table 3] The entry '(480)/480' for stress_skew is confusing. If one initial run failed and then passed on rerun, state that explicitly (e.g., 479/480 initial passes, 480/480 after rerun) and remove the implication that all 1440 checks passed on the first attempt.
  3. [Algorithm 1 / §4.5] Algorithm 1 uses R for the rank count while §4.5 uses W. Use one symbol throughout. Also, the notation 'row band' is introduced in §3.2.1 but the exact relation between row bands, tiles, and the tunable mgb parameter would benefit from a single definition/equation.
  4. [§2.2] Typo: 'withou host-side synchronization' should be 'without host-side synchronization'.

Circularity Check

0 steps flagged

No significant circularity: the paper's claims are empirical speedups measured against external baselines, not derivations from its own inputs.

full rationale

The paper's central claims are measured improvements (up to 2.64x end-to-end, 2.74x MoE-layer speedup) over named external systems and a sequential baseline. The proposed mechanism—tile-level signaling, remote-owner-aligned row layout, persistent producer/consumer kernels—is a constructed engineering design rather than a mathematical derivation. No equation or fitted parameter is renamed as a prediction: the tunable consumer SM count (c_CTA) and interior segment size (mgb) are swept openly, and the selected default is presented as an empirical choice with full sensitivity data, so these do not force the headline result by construction. Correctness is checked against the baseline on identical inputs; the one unreproduced tolerance outlier is an unresolved empirical anomaly, not a circular step. The Section 4.5 padding-bound analysis appears arithmetically inconsistent with Algorithm 1 (padding is per expert and per remote rank, so the (W-1)(tbM-1) bound omits the per-expert loop), but that is an overhead-accounting/correctness-risk flaw, not a case of the result being equivalent to its inputs. The paper cites prior signaling work (EVT, T3) for inspiration, but no load-bearing step reduces to a self-citation, and none of the authors' own prior results are invoked as an external uniqueness theorem or ansatz. Overall, the derivation chain is not circular; the main vulnerability is empirical validation and the padding-overhead accounting, which are correctness concerns outside the circularity definition.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The design introduces no new physical or conceptual entities; the 'remote-owner-aligned row layout' is a data layout, not an invented entity in the axis's sense. The free parameters are tuning knobs of the system, not fitted constants in a derivation.

free parameters (3)
  • consumer SM budget c_CTA = 14 (default; swept 2-24)
    Number of SMs/CTAs dedicated to the persistent communication consumer; chosen empirically on the 4-A100 platform; performance is highly sensitive (Section 4.3.4).
  • middle segment row-band count mgb = 1 or 2 (swept)
    Number of consecutive row bands coalesced into interior transfer segments; the optimal value is workload-dependent (Section 3.3.2).
  • correctness tolerance tau = 8e-3
    Relative-distance tolerance for per-token output comparison; set several times above FP16 noise (Section 4.4).
axioms (3)
  • domain assumption A thread-fence in the GEMM epilogue followed by a device flag publication makes tile output visible to the consumer kernel polling the flag on the same GPU (and the consumer's NVSHMEM put reads correct data).
    Section 3.3.1: signaling is fused into the epilogue after a thread-fence; correctness of the whole overlap depends on this memory-ordering guarantee.
  • domain assumption NVSHMEM one-sided puts from a consumer kernel to peer GPU symmetric memory operate correctly under concurrent kernel execution and reach the measured bandwidth plateau (~87 GB/s at 8 SMs).
    Section 2.3 and Figure 3a; the overlap design relies on device-initiated communication with the stated bandwidth characteristics.
  • domain assumption Router output (token-to-expert assignment) is available before the return path, so per-tile destination metadata can be precomputed.
    Section 3.2.3: combine plan pre-resolves transfer metadata; if routing were dynamic/uncertain, the plan would need recomputation.

pith-pipeline@v1.3.0-alltime-deepseek · 16422 in / 12918 out tokens · 109411 ms · 2026-08-01T12:26:00.568793+00:00 · methodology

0 comments
read the original abstract

Mixture-of-Experts (MoE) architectures increase model capacity without proportionally increasing computation cost and have become a key building block for scaling large language models (LLMs) to trillion-parameter regimes. Efficient deployment of these MoE models relies on distributed execution across multiple GPUs, where each MoE layer involves two all-to-all communications: dispatching tokens to expert ranks and returning the expert outputs to their source ranks. Conventional MoE implementations launch this return all-to-all after expert compute completes, exposing communication latency on the critical path and reducing GPU utilization. We present a fine-grained approach that overlaps expert compute with the second all-to-all via tile-level signaling and scheduling. Our producer-consumer co-design combines: (1) a persistent per-rank computation kernel (producer) that covers all local experts on the rank to eliminate repeated kernel launch overhead and prioritizes remote-critical tiles, and (2) a persistent communication kernel (consumer) on a small dedicated partition of streaming multiprocessors (SMs) that issues segment-granular transfers as tiles become ready. Our co-design avoids intrusive changes to the underlying computation operators or communication primitives, making it practical for improving distributed MoE execution efficiency on multi-GPU systems. On a 4-A100 GPU platform, evaluated on three MoE models against four state-of-the-art MoE systems, our approach achieves up to 2.64x end-to-end speedup and 2.74x MoE-layer speedup. Compared with a conventional non-overlap baseline, our approach consistently improves both operator- and MoE-layer-level performance across varying GEMM shapes, router modes, and a broad range of producer/consumer SM partitions, while preserving correctness.

Figures

Figures reproduced from arXiv: 2607.19539 by Anna Wingkvist, Minyu Cui, Morgan Ericsson.

Figure 1
Figure 1. Figure 1: Overview of our design for the MoE layer. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Optimization strategies to hide communication. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Communication Granularity. (a) Measured band [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: End-to-End and MoE layer latency (↓ is better) in MoE models. 4 8 16 32 64 Experts per Rank 0 20 40 Time (ms) FasterMoE Tutel Megatron-CUTLASS Megatron-TE Ours (a) Latency with varying E 4 8 16 32 64 Experts per Rank 0 25 50 75 100 Overlap ratio (%) (b) Overlap ratio with varying E [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Single MoE-layer microbenchmark. 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Speedup 4 8 16 32 64 16384×8192 2048 E M×N K 2.97 4 8 16 32 64 32768×8192 2048 2.81 base ours(mgb=1) ours(mgb=2) (a) balanced 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Speedup 4 8 16 32 64 16384×8192 2048 E M×N K 2.89 4 8 16 32 64 32768×8192 2048 2.94 base ours(mgb=1) ours(mgb=2) (b) moderate_skew 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Speedup 4 8 16 32 64 16384×8192 204… view at source ↗
Figure 6
Figure 6. Figure 6: Speedup at operator-level. 4.2.1 End-to-End and MoE Layer Evaluation in MoE Models. Fig￾ure 4 depicts the end-to-end (E2E) and MoE-layer latency of Faster￾MoE, Tutel, Megatron-CUTLASS, Megatron-TE, and ours on the three MoE models, normalized to fasterMoE. For end-to-end la￾tency in the left sub-figure of [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Speedup of the MoE layer. hidden dimension (𝐷𝑖𝑚. = 1024) provides enough per-tile useful work to amortize the fixed overhead of fine-grained overlap. 4.3 Scalability Analysis We have demonstrated both the MoE layer and the end-to-end effectiveness of our approach in Section 4.2. This section further an￾alyzes its scalability under the configurations listed in [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Normalized throughput (MTokens/sec). 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00 2.25 Norm. Latency 4 8 16 32 64 16384×8192 2048 E M×N K 4 8 16 32 64 32768×8192 2048 4 8 16 32 64 16384×8192 2048 4 8 16 32 64 32768×8192 2048 4 8 16 32 64 16384×8192 2048 4 8 16 32 64 32768×8192 2048 (a) balanced (b) moderate_skew (c) stress_skew cCTA=24 cCTA=22 cCTA=20 cCTA=18 cCTA=16 cCTA=14 cCTA=12 cCTA=10 cCTA=8 cCTA=6 cCTA=4… view at source ↗
Figure 9
Figure 9. Figure 9: Normalized latency (↓ is better) of the MoE layer across all evaluated SM partitions. The dashed line denotes the baseline. baseline runs a per-expert GEMM followed by a bulk all-to-all. Any difference in the final per-token output comes solely from rounding noise inherent to using two different FP16 GEMM implementations on identical inputs. On every rank and every iteration, a check is considered as pass … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. X-Stage: An Overlooked Pipeline Stage for Communication-Computation Overlap in DiT Inference

    cs.DC 2026-07 conditional novelty 7.0

    X-Stage is a measurable post-issue window for GPU remote stores; a Burst-Gap model predicts backpressure and guides scheduling, yielding up to 1.62x MoE and 1.43x attention speedups.

Reference graph

Works this paper leans on

32 extracted references · 4 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Osayamen Jonathan Aimuyo, Byungsoo Oh, and Rachee Singh. 2025. FlashMoE: Fast Distributed MoE in a Single Kernel. InAdvances in Neural Information Processing Systems (NeurIPS ’25). arXiv:2506.04667

    arXiv 2025

  2. [2]

    Chang Chen, Xiuhong Li, Qianchao Zhu, Jiangfei Duan, Peng Sun, Xingcheng Zhang, and Chao Yang. 2024. Centauri: Enabling Efficient Scheduling for Communication-Computation Overlap in Large Model Training via Communi- cation Partitioning. InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating ...

    2024

  3. [3]

    Zhaodong Chen, Andrew Kerr, Richard Cai, Jack Kosaian, Haicheng Wu, Yufei Ding, and Yuan Xie. 2024. EVT: Accelerating Deep Learning Training with Epilogue Visitor Tree. InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3(La Jolla, CA, USA)(ASPLOS ’24). Association for Co...

    arXiv 2024

  4. [4]

    Jiang Chenyu, Tian Ye, Jia Zhen, Zheng Shuai, Wu Chuan, and Wang Yida. 2024. Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation- Communication Overlapping. InProceedings of Machine Learning and Systems, Vol. 6. 74–86

    2024

  5. [5]

    Le, and Ruslan Salakhutdinov

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Linguistics, 2978–2988

    2019

  6. [6]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.Journal of Machine Learning Research23, 120 (2022), 1–39

    2022

  7. [7]

    Jiaao He, Jidong Zhai, Tiago Antunes, Haojie Wang, Fuwen Luo, Shangfeng Shi, and Qin Li. 2022. FasterMoE: Modeling and Optimizing Training of Large-Scale Dynamic Pre-Trained Models. InProceedings of the ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP)

    2022

  8. [8]

    Ke Hong, Xiuhong Li, Minxu Liu, Qiuli Mao, Tianqi Wu, Zixiao Huang, Lufang Chen, Zhong Wang, Yichong Zhang, Zhenhua Zhu, Guohao Dai, and Yu Wang

  9. [9]

    Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, HoYuen Chau, Peng Cheng, Fan Yang, Mao Yang, and Yongqiang Xiong. 2023. Tutel: Adaptive Mixture-of-Experts at Scale. InProceedings of Machine Learning and Systems (MLSys), Vol. 5. 269–287

    2023

  10. [10]

    Abhinav Jangda, Jun Huang, Guodong Liu, Amir Hossein Nodehi Sabet, Saeed Maleki, Youshan Miao, Madanlal Musuvathi, Todd Mytkowicz, and Olli Saarikivi

  11. [11]

    Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, Yulu Jia, Sun He, Hongmin Chen, Zhihao Bai, Qi Hou, Shipeng Yan, Ding Zhou, Yiyao Sheng, Zhuo Jiang, Haohan Xu, Haoran Wei, Zhang Zhang, Pengfei Nie, Leqi Zou, Sida Zhao, Liang Xiang, Zherui Liu, Zhe Li, Xiaoying Jia, Jianxi Ye, Xin J...

    2024

  12. [12]

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2021. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. InInternational Conference on Learning Representations (ICLR)

    2021

  13. [13]

    2025.The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation.https://ai.meta.com/blog/llama-4-multimodal- intelligence/ Accessed: 2025-12-12

    AI @ Meta Llama Team. 2025.The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation.https://ai.meta.com/blog/llama-4-multimodal- intelligence/ Accessed: 2025-12-12

    2025

  14. [14]

    Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. Mixed Precision Training. InInternational Confer- ence on Learning Representations (ICLR). https://arxiv.org/abs/1710.03740

    Pith/arXiv arXiv 2018

  15. [15]

    2023.CUTLASS: CUDA Templates for Linear Algebra Subroutines

    NVIDIA. 2023.CUTLASS: CUDA Templates for Linear Algebra Subroutines. https: //github.com/NVIDIA/cutlass

    2023

  16. [16]

    NVIDIA. 2024. Grouped GEMM for MoE. https://github.com/fanshiqing/grouped_ gemm

    2024

  17. [17]

    NVIDIA. 2024. Transformer Engine. https://github.com/NVIDIA/ TransformerEngine. Fine-grained Computation-Communication Overlap for Mixture-of-Experts ICPP ’26, September 28-October 01, 2026, Singapore, Singapore

    2024

  18. [18]

    Sinclair

    Suchita Pati, Shaizeen Aga, Mahzabeen Islam, Nuwan Jayasena, and Matthew D. Sinclair. 2024. T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectives. InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), Volume 2. 1146–1164. doi:10.1145/36206...

    arXiv 2024

  19. [19]

    Beckmann

    Kishore Punniyamurthy, Khaled Hamidouche, and Bradford M. Beckmann. 2024. Optimizing Distributed ML Communication with Fused Computation-Collective Operations. InSC24: International Conference for High Performance Computing, Networking, Storage and Analysis. 1–17. doi:10.1109/SC41406.2024.00094

    Pith/arXiv arXiv 2024

  20. [20]

    PyTorch. 2024. Introducing Async Tensor Parallelism in PyTorch. https://discuss.pytorch.org/t/distributed-w-torchtitan-introducing-async- tensor-parallelism-in-pytorch/209487

    2024

  21. [21]

    Shaohuai Shi, Xinglin Pan, Xiaowen Chu, and Bo Li. 2023. PipeMoE: Accelerating Mixture-of-Experts through Adaptive Pipelining. InIEEE INFOCOM 2023 - IEEE Conference on Computer Communications. 1–10. doi:10.1109/INFOCOM53939. 2023.10228874

    arXiv 2023

  22. [22]

    Shaohuai Shi, Xinglin Pan, Qiang Wang, Chengjian Liu, Xiaozhe Ren, Zhongzhe Hu, Yu Yang, Bo Li, and Xiaowen Chu. 2024. ScheMoE: An Extensible Mixture- of-Experts Distributed Training System with Tasks Scheduling. InProceedings of the Nineteenth European Conference on Computer Systems (EuroSys ’24). 236–249. https://doi.org/10.1145/3627703.3650083

    arXiv 2024

  23. [23]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.arXiv:1909.08053(2019)

    Pith/arXiv arXiv 2019

  24. [24]

    Guanhua Wang, Chengming Zhang, Zheyu Shen, Ang Li, and Olatunji Ruwase

  25. [25]

    Hulin Wang, Yaqi Xia, Donglin Yang, Xiaobo Zhou, and Dazhao Cheng. 2025. Harnessing Inter-GPU Shared Memory for Seamless MoE Communication- Computation Fusion. InProceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming (PPoPP ’25). 170–182

    2025

  26. [26]

    Shibo Wang, Jinliang Wei, Amit Sabne, Andy Davis, Berkin Ilbeyi, Blake Hecht- man, Dehao Chen, Karthik Srinivasa Murthy, Marcello Maggioni, Qiao Zhang, Sameer Kumar, Tongfei Guo, Yuanzhong Xu, and Zongwei Zhou. 2022. Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models. InProceedings of the 28th ACM Internationa...

    2022

  27. [27]

    Shulai Zhang, Ningxin Zheng, Haibin Lin, Ziheng Jiang, Wenlei Bao, Chengquan Jiang, Qi Hou, Weihao Cui, Size Zheng, Li-Wen Chang, Quan Chen, and Xin Liu. 2025. COMET: Fine-grained Computation-communication Overlapping for Mixture-of-Experts. InEighth Conference on Machine Learning and Systems (MLSys)

    2025

  28. [28]

    Zheng Zhang, Donglin Yang, Yaqi Xia, Liang Ding, Dacheng Tao, Xiaobo Zhou, and Dazhao Cheng. 2023. MPipeMoE: Memory Efficient MoE for Pre-trained Models with Adaptive Pipeline Parallelism. In2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS). 167–177

    2023

  29. [29]

    Size Zheng, Jin Fang, Xuegui Zheng, Qi Hou, Wenlei Bao, Ningxin Zheng, Ziheng Jiang, Dongyang Wang, Jianxi Ye, Haibin Lin, Li-Wen Chang, and Xin Liu. 2025. TileLink: Generating Efficient Compute-Communication Overlapping Kernels using Tile-Centric Primitives. InEighth Conference on Machine Learning and Systems (MLSys)

    2025

  30. [2022]

    InProceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS)

    Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads. InProceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS)

  31. [2024]

    Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping. (2024). arXiv:arXiv:2409.15241

    Pith/arXiv arXiv 2024

  32. [2026]

    InEuropean Conference on Computer Systems (EuroSys ’26)

    Efficient and Adaptable Overlapping for Computation and Communica- tion via Signaling and Reordering. InEuropean Conference on Computer Systems (EuroSys ’26). arXiv:2504.19519

    arXiv