Pith. sign in

REVIEW 2 major objections 5 minor 89 references

StrataCL claims that on scale-up fabrics, registering buffers at allocation time removes both redundant copies and registration latency from communication, delivering up to 1.6x bandwidth and 1.9x end-to-end speedups.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 15:38 UTC pith:C4ZHAS2O

load-bearing objection StrataCL is a genuine new mechanism for supernode communication with strong results, but the 'off critical path' claim rests on gap stats, not direct measurements of the readiness barrier. the 2 major comments →

arxiv 2607.26444 v1 pith:C4ZHAS2O submitted 2026-07-29 cs.DC

StrataCL: Fabric-Native Communication Library for Production Supernodes

Tiancheng Hu , Jin Qin , Yuzheng Wang , Ke Liu , TangShengsheng Li , Sheng Wang , Zhongzhe Hu , Tianlun Hu
show 9 more authors
Wei Wang Lijun Li Jingbin Zhou Xiaoming Bao Hongwei Sun Jieru Zhao Huimin Cui Tao Xie Chenxi Wang
This is my paper
classification cs.DC
keywords collective communicationuser-buffer registrationregistration-on-allocationshadow virtual addressingscale-up fabricMoE dispatch/combineworkload-balanced partitioningSDMA offloading
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

StrataCL is a communication library for tightly coupled supernodes that aims to make direct communication over user buffers free of both redundant data copies and registration overhead. Its central idea is registration-on-allocation: register a buffer for remote access in the background the moment it is allocated, before the first communication use, and defer only a cheap local readiness check. Shadow virtual addressing gives every NPU the same virtual address for each buffer, so operators need no peer-to-peer address translation. On a 384-NPU supernode, the paper reports up to 1.6x collective bus bandwidth, up to 1.4x MoE dispatch/combine bandwidth, and, across three production workloads, 1.9x LLM inference throughput, 2.2x lower P99 time-to-first-token, and 1.4x and 1.3x faster LLM and recommendation-model training iterations.

Core claim

The paper claims that on a scale-up fabric exposing a global unified physical address space, a communication library can eliminate both redundant staging copies and registration latency by registering physical memory allocations eagerly and asynchronously rather than at first communication use. Registration-on-allocation exploits a long gap between allocation and first use, while shadow virtual addressing (each peer maps the same buffer at the same virtual address) removes address-translation metadata. The paper also redesigns operator execution around the fabric's non-uniform access: a full-mesh remote-slice abstraction minimizes synchronization, a workload-balanced NPU-core partitioner mod

What carries the argument

The central mechanism is registration-on-allocation with shadow virtual addressing. Registration-on-allocation intercepts physical memory allocation events and asynchronously broadcasts the buffer's physical-memory handle to peer NPUs, creating remote fabric mappings in the background; only a local readiness check is deferred to the first communication touch. Shadow virtual addressing assigns each NPU a disjoint virtual-address range and mirrors every allocated buffer at the same virtual address on all peers, so remote-slice transfers use identical source and destination addresses with no per-peer translation. The second pillar is the operator execution model: a full-mesh remote-slice abstra

Load-bearing premise

The load-bearing premise is that in production AI workloads a freshly allocated buffer sits unused long enough for background registration to finish before any communication touches it, and that on-demand page remapping is rare enough that the fallback barrier never lands on the critical path; the paper's support is empirical (minimum allocation-to-communication gaps of seconds in three workloads, remapping in under 4% of MoE batches).

What would settle it

Run a microbenchmark that allocates a fresh buffer and immediately launches an all-gather on it, repeated across many ranks, on the same supernode. If the measured allocation-to-first-use gap in that workload is comparable to or smaller than the fabric's remote-mapping latency, registration-on-allocation cannot hide registration, and the design reverts to the just-in-time barrier it claims to remove.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • User-buffer direct communication becomes transparent to application frameworks: no allocator changes, no dedicated communication pool, and no just-in-time registration barrier for common-case allocations.
  • Registration overhead no longer scales with rank count on the communication critical path; the per-rank serial registration cost seen in just-in-time schemes is hidden behind the allocation-to-communication gap.
  • On-demand allocators with runtime page remapping remain usable: remapping is handled as incremental registration and, in the paper's MoE-serving measurements, occurs in under 4% of request batches.
  • Communication occupies far fewer NPU cores during transfer: the SDMA-offloaded path cuts NPU-core occupation by over 95% at a 9% latency cost, which favors compute-communication overlap.
  • Workload-balanced partitioning reduces the fastest-to-slowest NPU-core completion gap from about 43% to within 5%, removing the long-tail effect of non-uniform fabric access.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If registration-on-allocation generalizes, the same eager-registration pattern could remove staging copies on other scale-up fabrics with a global address space, not just the specific fabric studied here; the paper includes a small prototype on an NVLink-domain system showing a similar trend.
  • The allocation-to-communication gap is the key resource: workloads that allocate a buffer and communicate on it almost immediately (or that rely on very short-lived tensors) would force the fallback to a just-in-time barrier, so the benefit is inherently workload-dependent.
  • The workload-balanced partitioning problem is shown to be NP-hard, and the paper uses an LPT-style approximation; a testable extension is comparing that policy against exact or metaheuristic schedules on real MoE routing skew to quantify remaining suboptimality.
  • The full-mesh choice trades peak large-payload bandwidth for latency at small and medium sizes (a roughly 6% average gap at large payloads); a workload-aware operator selector that switches to multi-step algorithms for large messages is an obvious next step, and the paper explicitly notes this possibility.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. StrataCL proposes a communication library for the Huawei CloudMatrix384 supernode that enables user-buffer direct communication without placing registration on the critical path. The key mechanism is registration-on-allocation: intercepted physical allocations are asynchronously registered in a global UB address space, with shadow virtual addressing so all peers access a buffer at the same virtual address and only a local readiness check (Phase 2) is deferred to first communication use. The communication layer adds a full-mesh abstraction, workload-balanced NPU-core partitioning, and NPU-driven SDMA offloading. Evaluation on CM384 reports up to 1.6x collective bus bandwidth, up to 1.4x MoE dispatch/combine bandwidth, 1.9x LLM inference throughput, 2.2x P99 TTFT reduction, and 1.4x/1.3x LLM/Recsys training iteration-time improvements. The paper includes an ablation study, a scalability study, an NVIDIA prototype, and appendices on complexity and workload-partitioning overhead.

Significance. If the reported results hold, StrataCL is a substantial contribution: it addresses a real bottleneck in modern AI systems by eliminating both staging copies and just-in-time registration overhead on scale-up fabrics with a global physical address space. The design is well motivated by measurements of UB registration latency, topology non-uniformity, and core contention. The paper is refreshingly honest about limitations: it acknowledges that HCCL-zerocopy is slightly better at large payloads (Section 8.1), that SDMA offloading has a 9% latency penalty (Section 9.5), that remapping occurs in less than 4% of MoE batches (Section 5.3), and that the LPT-style bound in Appendix B does not directly apply to the constrained problem. The cost model tau(s) = alpha_t + s/beta_t uses measured tier parameters rather than parameters fitted to the reported speedups, which strengthens the credibility of the partitioning evaluation. The main weakness is that the central 'off the critical path' property of registration-on-allocation is not directly instrumented: Phase 2 readiness-barrier hit rates and blocking times are never reported, leaving the P99 TTFT claim and the generality of the 1.9x/1.4x

major comments (2)
  1. [§5.1, §5.3, §9.4] The paper's central claim is that registration-on-allocation keeps user-buffer registration off the communication critical path, but the Phase 2 readiness barrier is never measured. Section 9.4 reports allocation-to-communication gaps of seconds, and Section 5.3 reports that runtime remapping occurs in under 4% of MoE batches with 'typically' tens of milliseconds of intervening computation. These observations do not establish that Phase 2 never blocks a communication operator: a freshly remapped buffer that is touched before background UB mapping completes will trigger exactly the just-in-time barrier the paper claims to eliminate. The paper provides no count of Phase 2 barrier hits, no distribution of blocking time, and no correlation with the P99 TTFT tail. Because P99 tail latency is particularly sensitive to rare events, the 4% remapped batches could be precisely the tail requests dr
  2. [§8.2, Fig. 11/12] The end-to-end HCCL-zerocopy baseline is not the production HCCL-zerocopy interface but a pool-based construction in which the whole HBM is pre-registered and user tensors are allocated from that pool. The paper labels this 'practical and optimistic,' and the ablation in Figure 13a helps separate JIT registration from registration-on-allocation. However, the end-to-end claims such as '1.6x over HCCL-zerocopy' conflate the registration strategy with the memory-fragmentation effects of disabling the expandable-segment allocator. The authors should explicitly state, for each end-to-end comparison, how much of the gap is attributable to avoiding pool-based fragmentation versus to the asynchronous registration mechanism itself. Otherwise a reader cannot tell whether the comparison is to a straw-man baseline or to a realistic deployment of the production HCCL-zerocopy path.
minor comments (5)
  1. [§6.2, Appendix B] The LPT approximation bound given in Appendix B applies only to the unconstrained variant; the authors acknowledge that the stripe-level fan-out cap invalidates the bound. This is acceptable for a heuristic, but the main text should explicitly say that no worst-case approximation guarantee is claimed for the actual constrained algorithm, rather than leaving the impression that the classical LPT bound covers StrataCL's partitioning.
  2. [§9.3] The NVIDIA portability result is demonstrated only on an 8-GPU DGX B200, not on a GB200 NVL72. The expectation that the benefit grows at larger NVLink-domain scales is plausible but speculative; please mark it clearly as an extrapolation.
  3. [§8.1/§9.2] The paper repeatedly mentions a 'workload-aware operator selection policy' as future work to address the large-payload regression. It would strengthen the paper to at least state where the crossover point occurs for the evaluated collectives, so readers can understand the current deployment boundary.
  4. [§7] The paper states the source code will be released upon publication, but no artifact or data is currently available. Given the production-scale claims, a reproducibility appendix with at least the microbenchmark harness and configuration details would be valuable.
  5. [Global] Several references are dated 2026 and some are to arXiv or vendor documentation. Please verify that all cited works exist and are accessible; in particular, the DeepSeek-V4 Flash citation should be checked.

Circularity Check

0 steps flagged

No significant circularity: StrataCL's central claims rest on direct measurements and independent baselines, not on equations that reduce to fitted targets or on load-bearing self-citations.

full rationale

I walked the claimed derivation chain. The central 'off critical path' claim for registration-on-allocation is supported by an independent empirical measurement (§9.4) of allocation-to-first-use gaps across three workloads; the mechanism is not defined in terms of the reported throughput gains. Section 5.3 explicitly concedes that when a freshly mapped region is accessed before asynchronous UB mapping completes, StrataCL still uses a synchronization barrier 'no worse than just-in-time registration' — a stated limitation, not a circular justification. The partitioning cost model τ(s)=α_t+s/β_t uses measured per-tier access latency and bandwidth (Table 1), not parameters fitted to the reported makespan reductions or end-to-end speedups; its approximation question is grounded in the external LPT bound from Graham 1969, and Appendix B openly notes that the constrained variant does not inherit the classical bound. Operator-level gains (Figure 10, Table 2) and ablations (§9.1, §9.5, §9.6) are direct before/after comparisons against baselines, and the paper reports cases where StrataCL is worse (large payloads vs HCCL-zerocopy in §8.1; slightly lower peak bandwidth at 256 ranks in §9.2), which is inconsistent with a forced or definitionally guaranteed result. No load-bearing self-citation appears; references such as [15], [42], [48], [58], and [79] are external or non-load-bearing. The absence of direct Phase-2 readiness-barrier hit-rate/latency instrumentation is a validation gap for the generality of the off-critical-path claim, but the paper's own §5.3 and §9.4 present the relevant empirical conditions and limitations. Under the stated rules, a missing measurement is a correctness/robustness concern, not a circularity. I find no equation equal to an input, no fitted parameter renamed as a prediction, and no self-citation chain that reduces to the paper's conclusions.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central result rests on empirically measured per-tier hardware constants, two hand-chosen scheduling knobs (unit size S_t and fan-out cap H_t), and hardware capabilities of CloudMatrix384. There are no new physical entities; the paper's novelty is in mechanisms such as registration-on-allocation and shadow addressing, which are design artifacts rather than postulated entities.

free parameters (4)
  • tier access latency alpha_t = die-to-die: 0.2 us; intra-node: 0.7 us; inter-node: 2.1 us (Table 1)
    Measured hardware latencies used in tau(s)=alpha_t+s/beta_t to estimate transfer-unit cost. Measured, not fitted to target throughput.
  • tier bandwidth beta_t = die-to-die: 210; intra-node: 170; inter-node: 150 GB/s (Table 1)
    Measured single-direction bandwidth used in the cycle-cost model for NPU-core partitioning.
  • tier-specific unit size S_t = not reported
    Partitioning granularity chosen by hand to balance policy-generation overhead against residual per-core imbalance (Section 6.2). A free design knob.
  • fan-out cap H_t = not reported
    Tier-specific cap on instantaneous number of cores targeting the same peer (Section 6.2). No value or sensitivity analysis is reported.
axioms (5)
  • domain assumption CM384 exposes a global unified physical address space in which all NPU HBM is addressable, and a local VA can be mapped to remote HBM physical pages.
    Section 2.3: 'CM384 maps the HBM physical address ranges of all NPUs into a unified UB address space.' The entire registration and shadow addressing design rests on this hardware property.
  • domain assumption A buffer's physical allocation is typically separated from its first communication use by seconds, and VMM remapping is rare enough to be hidden.
    Sections 5.1 and 9.4 (minimum allocation-to-communication gap about 2.6s) and Section 5.3 (VMM remapping in <4% of MoE batches). If this fails, registration-on-allocation degrades to just-in-time registration.
  • domain assumption UB address translation allows multiple virtual addresses to alias the same physical page, enabling asynchronous peer unbinding while the local VA is reclaimed.
    Section 5.4: 'UB address translation allows multiple virtual addresses to map to the same physical page.' This justifies non-blocking deregistration.
  • standard math The NPU-core partitioning problem is NP-hard via reduction from P||Cmax, and the LPT-style approximation is an acceptable heuristic for the constrained variant.
    Appendix B proves NP-hardness and gives the classical LPT bound; Appendix B also notes the bound does not directly apply under the fan-out cap, so the constrained case is heuristic.
  • domain assumption SGLang, TorchTitan, and TorchRec can be integrated with StrataCL by intercepting memory APIs without changing application semantics.
    Section 7 describes interposition on aclrtMalloc/aclrtMapMem. No public code is available to verify that interception is safe across all allocation paths.

pith-pipeline@v1.3.0-daily-deepseek · 23516 in / 12612 out tokens · 100090 ms · 2026-08-01T15:38:15.121841+00:00 · methodology

0 comments
read the original abstract

Modern distributed AI workloads run across hundreds of accelerators, making communication a major bottleneck. Existing communication libraries remain largely buffer-centric because user and communication buffers are managed separately, causing redundant data copies or costly user-buffer registration. This paper presents StrataCL, a zero-redundancy and fabric-native communication library for production supernodes. StrataCL introduces registration-on-allocation to realize user-buffer direct communication, and designs communication operators with workload-balanced NPU-core partitioning and NPU-driven SDMA offloading to exploit supernode architecture features. On the Huawei CloudMatrix384, StrataCL improves collective bus bandwidth by up to 1.6x and improves MoE dispatch/combine bus bandwidth by up to 1.4x. Across three production workloads, StrataCL improves LLM inference throughput by 1.9x, reduces P99 TTFT by 2.2x, and reduces LLM and Recsys training iteration time by 1.4x and 1.3x, respectively.

Figures

Figures reproduced from arXiv: 2607.26444 by Chenxi Wang, Hongwei Sun, Huimin Cui, Jieru Zhao, Jingbin Zhou, Jin Qin, Ke Liu, Lijun Li, Sheng Wang, TangShengsheng Li, Tao Xie, Tiancheng Hu, Tianlun Hu, Wei Wang, Xiaoming Bao, Yuzheng Wang, Zhongzhe Hu.

Figure 3
Figure 3. Figure 3: a reports the AllGather bus bandwidth [50] of HCCL and HCCL-zerocopy at 32 ranks. HCCL-zerocopy improves bus bandwidth by more than 30% for all payloads above 8 MiB by eliminating redundant data movement on the stag￾ing path. For an AllGather across 𝑁 ranks with 𝑀 bytes per rank, HCCL copies 𝑀 bytes from each user input buffer into 1 4 16 64 256 Per-rank payload (MiB) 0 50 100 150 200 Bus BW (GB/s) HCCL HC… view at source ↗
Figure 2
Figure 2. Figure 2: Simplified architecture of an Ascend 910C NPU. Each NPU contains two dies, each with 24 AI Cores; each AI Core integrates one Cube Core and two Vector Cores. Each Ascend 910C NPU is a dual-die package connected by a high-bandwidth SIO fabric with up to 540 GB/s aggre￾gate bandwidth. As shown in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Inter-NPU memory registration on supernode. (a) Registration latency comparison of RDMA and UB. (b) Time breakdown of UB registration, normalized to the total. 2 8 32 128 Rank count N 10 −1 10 0 10 1 10 2 10 3 Latency (ms) 2 MiB 128 MiB 1024 MiB DeepSeek V4 inference DeepSeek V3.2 training 0 1 2 3 4 5 Fragmentation (GiB) [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: (a) User-buffer registration overhead varying with the number of ranks. (b) HBM fragmentation on CM384 when PyTorch’s expandable-segment allocator is disabled. the latency breakdown of these steps. For smaller payloads, handle export and import can cause significant delays. As the payload grows, the mapping step becomes dominant because NPU page-table updates scale with the registered size. 3 Motivation 3.… view at source ↗
Figure 7
Figure 7. Figure 7: StrataCL design overview. Figure 6a, full-mesh outperforms ring across the small-to￾medium range, on average over 2× faster below 1 MiB and up to 4.5× at 64 KiB, and remains ahead through 8 MiB. The latency breakdown shows that synchronization accounts for more than 50% of the ring’s end-to-end time at small pay￾loads and reaches ∼77% at 64 KiB. Ring becomes preferable beyond 16 MiB, when lower cross-node … view at source ↗
Figure 8
Figure 8. Figure 8: Registration workflow in StrataCL. When a physi￾cal allocation is intercepted, an asynchronous registration starts (Phase 1). When a communication operator first ac￾cesses the region, StrataCL performs a readiness barrier to ensure that the corresponding UB mappings have completed (Phase 2). and requires each NPU to maintain address-translation meta￾data, which becomes increasingly expensive at scale. Stra… view at source ↗
Figure 9
Figure 9. Figure 9: (a) Device-side SDMA issue path. (b) Two comple￾tion mechanisms: (I) NPU cores poll a status flag for in-kernel synchronization, and (II) the SDMA engine raises a notify event that a downstream or AI CPU waits on. attribution. Hardware-metric profilers, such as Nsight Com￾pute [43], expose hardware information but do not directly at￾tribute latency to developer-defined communication stages. To address this… view at source ↗
Figure 10
Figure 10. Figure 10: Bus bandwidth of AllGather and AllReduce. Remote-DRAM UB addresses are then mapped into the ac￾cessing process’s CPU page table, allowing CPU cores to access peer host DRAM through ordinary virtual addresses. This direct path reduces metadata-broadcast latency from 60 µs to 7–8 µs compared with the NPU-relay path. Generality on NVIDIA GPUs. StrataCL relies on a scale-up fabric with a global address space … view at source ↗
Figure 11
Figure 11. Figure 11: LLM inference performance. (a) Normalized la￾tency vs. request rate. (b) P99 TTFT and P99 TPOT at 15 req/s. 200 2000 5000 Iteration 0 10 20 30 40 Iter. time (s) 5 15 25 Cache (GiB) 0 400 800 1200 Iter. time (ms) HCCL HCCL-zerocopy StrataCL [PITH_FULL_IMAGE:figures/full_fig_p010_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: (a) LLM training iteration time at different train￾ing iterations. (b) Recsys training iteration time under dif￾ferent embedding cache sizes. Cache size here refers to the memory allocated to the embedding table for each NPU. maintaining the same serving batch size as HCCL. As a re￾sult, StrataCL improves inference throughput by 1.9× over HCCL and 1.6× over HCCL-zerocopy. The gain comes from workload-bala… view at source ↗
Figure 13
Figure 13. Figure 13: (a) Performance breakdown of StrataCL’s LLM in￾ference throughput (JIT: just-in-time user-buffer registration, RoA: registration-on-allocation, Part.: workload-balanced NPU-core partitioning). (b) Peak AllGather bus bandwidth. HCCL and ∼16% over HCCL-zerocopy, demonstrating its ef￾fectiveness for sparse embedding communication. The com￾munication benefit becomes more pronounced when the embedding cache is… view at source ↗
Figure 15
Figure 15. Figure 15: (a) Allocation-to-communication gap. (b) NPU￾core occupation time under SDMA offloading. NPU cores 0.6 0.8 1.0 Completion time (normalized) ≈43% without balance NPU cores ≤5% makespan −19% with balance [PITH_FULL_IMAGE:figures/full_fig_p012_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Per-NPU-core completion time statistics. that registration-on-allocation can remove remote-access registration from the communication critical path. 9.5 NPU-driven SDMA Offloading This section evaluates whether SDMA offloading reduces NPU-core occupation (§6.3). Using the NPU-side profiler, a 32-rank 128 MiB AllGather is profiled under two paths. In the MTE path, NPU cores directly perform data movement. … view at source ↗
Figure 17
Figure 17. Figure 17: Extended collective microbenchmarks. APPENDICES A Extended Microbenchmark [PITH_FULL_IMAGE:figures/full_fig_p017_17.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

89 extracted references · 10 linked inside Pith

  1. [1]

    Anirudha Agrawal, Shaizeen Aga, Suchita Pati, and Mahzabeen Is- lam. 2025. ConCCL: Optimizing ML Concurrent Computation and Communication with GPU DMA Engines. In2025 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS 25). 1–11. doi:10.1109/ISPASS64960.2025.00018

    arXiv 2025

  2. [2]

    Osayamen Jonathan Aimuyo, Byungsoo Oh, and Rachee Singh. 2025. FlashMoE: Fast Distributed MoE in a Single Kernel. InAdvances in Neural Information Processing Systems (NeurIPS 25).https://arxiv.org/ abs/2506.04667

    arXiv 2025

  3. [3]

    AMD. 2024. AMD Instinct MI300X Accelerator Data Sheet. https://www.amd.com/content/dam/amd/en/documents/instinct- tech-docs/data-sheets/amd-instinct-mi300x-data-sheet.pdf

    2024

  4. [4]

    AMD. 2024. RCCL: ROCm Communication Collectives Library.https: //github.com/ROCm/rccl

    2024

  5. [5]

    AMD ROCm. 2025. rocSHMEM: GPU-Centric Intra-Kernel Networking through an OpenSHMEM-like Interface.https://github.com/ROCm/ rocSHMEM

    2025

  6. [6]

    Quentin Anthony, Benjamin Michalowicz, Jacob Hatef, Lang Xu, Mustafa Abdul Jabbar, Aamir Shafi, Hari Subramoni, and Dha- baleswar K. Panda. 2024. Demystifying the Communication Char- acteristics for Distributed Transformer Models. InProceedings of the 31st IEEE Symposium on High-Performance Interconnects (HOTI 24). https://doi.org/10.1109/HOTI63208.2024.00020

    arXiv 2024

  7. [7]

    Meghan Cowan, Saeed Maleki, Madanlal Musuvathi, Olli Saarikivi, and Yifan Xiong. 2023. MSCCLang: Microsoft Collective Communication Language. InProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 23).https://doi.org/10.1145/3575693.3575724

    arXiv 2023

  8. [8]

    Criteo AI Lab. 2015. Criteo 1TB Click Logs dataset.https://ailab.criteo. com/criteo-1tb-click-logs-dataset/

    2015

  9. [9]

    DeepSeek-AI. 2025. DeepEP: An Efficient Expert-Parallel Communica- tion Library.https://github.com/deepseek-ai/DeepEP

    2025

  10. [10]

    DeepSeek-AI. 2025. deepseek-ai/profile-data: Analyze computation- communication overlap in V3/R1.https://github.com/deepseek-ai/ profile-data. Accessed: 2026-05-28

    2025

  11. [11]

    DeepSeek-AI. 2025. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556 [cs.CL]https://arxiv.org/ abs/2512.02556

    Pith/arXiv arXiv 2025

  12. [12]

    DeepSeek-AI. 2026. DeepSeek-V4 Flash: Towards Highly Effi- cient Million-Token Context Intelligence.https://huggingface.co/ deepseek-ai/DeepSeek-V4-Flash.https://huggingface.co/deepseek- ai/DeepSeek-V4-Flash

    2026

  13. [13]

    Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. 2024. Better & Faster Large Language Models via Multi-token Prediction.arXiv preprint arXiv:2404.19737(2024). https://arxiv.org/abs/2404.19737

    Pith/arXiv arXiv 2024

  14. [14]

    Raja Gond, Nipun Kwatra, and Ramachandran Ramjee. 2025. Token- Weave: Efficient Compute-Communication Overlap for Distributed LLM Inference. arXiv preprint arXiv:2505.11329.https://arxiv.org/ abs/2505.11329

    Pith/arXiv arXiv 2025

  15. [15]

    R. L. Graham. 1969. Bounds on Multiprocessing Timing Anomalies. SIAM J. Appl. Math.(1969).https://doi.org/10.1137/0117039

    doi:10.1137/0117039 1969

  16. [16]

    Zhiyuan Guo, Yizhou Shan, Xuhao Luo, Yutong Huang, and Yiying Zhang. 2022. Clio: A Hardware-Software Co-Designed Disaggregated Memory System. InProceedings of the 27th ACM International Confer- ence on Architectural Support for Programming Languages and Operat- ing Systems (ASPLOS 22).https://doi.org/10.1145/3503222.3507762

    arXiv 2022

  17. [17]

    Songlin Huang and Chenshu Wu. 2025. Neutrino: Fine-grained GPU Kernel Profiling via Programmable Probing. In19th USENIX Sympo- sium on Operating Systems Design and Implementation (OSDI 25).https: //www.usenix.org/conference/osdi25/presentation/huang-songlin

    2025

  18. [18]

    Huawei. 2024. CANN: Compute Architecture for Neural Networks. https://www.hiascend.com/en/cann. Huawei Ascend heterogeneous computing architecture. Accessed 2026-06-02

    2024

  19. [19]

    Huawei. 2024. CANN-HCCL: Huawei Collective Communication Library (Open-Source).https://gitee.com/ascend/cann-hccl

    2024

  20. [20]

    Huawei. 2025. Ascend SHMEM: Shared-Memory Communication Library for Ascend NPUs.https://gitee.com/ascend/shmem

    2025

  21. [21]

    Huawei. 2025. MindStudio.https://www.hiascend.com/en/developer/ software/mindstudio. Huawei Ascend development toolchain. Ac- cessed 2026-06-02

    2025

  22. [22]

    Huawei Technologies. 2025. HCCL Zero-Copy User-Buffer Direct Communication API.https://www.hiascend.com/document/detail/ zh/canncommercial/81RC1/apiref/hcclapiref/hcclcpp_07_0053.html. CANN Commercial Edition 8.1.RC1 HCCL API (C)

    2025

  23. [23]

    Changho Hwang, Peng Cheng, Roshan Dathathri, Abhinav Jangda, Saeed Maleki, Madan Musuvathi, Olli Saarikivi, Aashaka Shah, Ziyue Yang, Binyang Li, et al. 2026. MSCCL++: Rethinking GPU Communi- cation Abstractions for AI Inference. InProceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems...

    arXiv 2026

  24. [24]

    Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, et al

  25. [25]

    Changho Hwang, KyoungSoo Park, Ran Shu, Xinyuan Qu, Peng Cheng, and Yongqiang Xiong. 2023. ARK: GPU-driven Code Execution for Distributed Deep Learning. InProceedings of the 20th USENIX Sym- posium on Networked Systems Design and Implementation (NSDI 23). https://www.usenix.org/conference/nsdi23/presentation/hwang 13

    2023

  26. [26]

    InfiniBand Trade Association. 2014. Supplement to InfiniBand Architecture Specification Volume 1 Release 1.2.1, Annex A17: RoCEv2.https://www.infinibandta.org/ibta-announces-new-roce- specification/

    2014

  27. [27]

    InfiniBand Trade Association. 2023. InfiniBand Architecture Spec- ification Volume 1, Release 1.7.https://www.infinibandta.org/ibta- specification/

    2023

  28. [28]

    Dmytro Ivchenko et al. 2022. TorchRec: a PyTorch Domain Library for Recommendation Systems. InProceedings of the 16th ACM Conference on Recommender Systems (RecSys 22).https://doi.org/10.1145/3523227. 3547387

    doi:10.1145/3523227 2022

  29. [29]

    Sylvain Jeaugey, Giuseppe Congiu, Thomas Gillis, Ben Williams, and Fred Oh. 2025. New Scaling Algorithm and Initialization with NVIDIA Collective Communications Library 2.23. NVIDIA Technical Blog.https://developer.nvidia.com/blog/new-scaling-algorithm-and- initialization-with-nvidia-collective-communications-library-2-23/

    2025

  30. [30]

    Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, et al

  31. [31]

    Chao Jin, Ziheng Jiang, Zhihao Bai, Zheng Zhong, Juncai Liu, Xiang Li, Ningxin Zheng, Xi Wang, Cong Xie, Qi Huang, et al. 2026. MegaScale- MoE: Large-Scale Communication-Efficient Training of Mixture-of- Experts Models in Production. InProceedings of the 21st European Conference on Computer Systems (EUROSYS 26).https://doi.org/10. 1145/3767295.3769325

    arXiv 2026

  32. [32]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. InProceedings of the 29th Sym- posium on Operating Systems Principles (SOSP 23). ACM, 611–626. doi:10.1145/3600006.3613165

    arXiv 2023

  33. [33]

    Xingyi Li, Yadong Liu, Xiaojie Huang, Yiran Zhang, Shuai Wang, Shangguang Wang, Zhehao Lin, Yinben Xia, Chang Yu, Qihang Liu, et al. 2026. SwiftEP: Accelerating MoE Inference with Buffer Fusion and TMA Offloading. InProceedings of the 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). 1073–1089. https://www.usenix.org/conferen...

    2026

  34. [34]

    Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, Andrew Gu, Chien-Chin Huang, Iris Zhang, Wei Feng, Howard Huang, Junjie Wang, et al. 2025. TorchTitan: One-stop PyTorch native solution for production ready LLM pretraining. InThe Thirteenth International Conference on Learning Representations (ICLR 25).https://arxiv.org/ abs/2410.06511

    Pith/arXiv arXiv 2025

  35. [35]

    Heng Liao, Bingyang Liu, Xianping Chen, Zhigang Guo, Chuanning Cheng, Jianbing Wang, Xiangyu Chen, Peng Dong, Rui Meng, Wenjie Liu, et al. 2025. UB-Mesh: A Hierarchically Localized nD-FullMesh Datacenter Network Architecture. arXiv preprint arXiv:2503.20377. https://arxiv.org/abs/2503.20377

    Pith/arXiv arXiv 2025

  36. [36]

    Heng Liao, Jiajin Tu, Jing Xia, Hu Liu, Xiping Zhou, Honghui Yuan, and Yuxing Hu. 2021. Ascend: a Scalable and Unified Architecture for Ubiquitous Deep Neural Network Computing: Industry Track Paper. InIEEE International Symposium on High-Performance Computer Ar- chitecture (HPCA 21).https://doi.org/10.1109/HPCA51647.2021.00071

    arXiv 2021

  37. [37]

    Jinkun Lin, Ziheng Jiang, Zuquan Song, Sida Zhao, Menghan Yu, Zhanghan Wang, Chenyuan Wang, Zuocheng Shi, Xiang Shi, Wei Jia, et al . 2025. Understanding Stragglers in Large Model Train- ing Using What-if Analysis. InProceedings of the 19th USENIX Sym- posium on Operating Systems Design and Implementation (OSDI 25). https://www.usenix.org/conference/osdi2...

    2025

  38. [38]

    Apoorve Mohan, Robert Walkup, Bengi Karacali, Ming-Hung Chen, Abdullah Kayi, Liran Schour, Shweta Salaria, Sophia Wen, I-Hsin Chung, Abdul Alim, et al . 2025. Vela: A Virtualized LLM Training System with GPU Direct RoCE. InProceedings of the 30th ACM Interna- tional Conference on Architectural Support for Programming Languages and Operating Systems (ASPLO...

    doi:10.1145/3676641 2025

  39. [39]

    Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, and others. 2019. Deep Learning Recommendation Model for Personalization and Recommen- dation Systems.https://arxiv.org/abs/1906.00091

    Pith/arXiv arXiv 2019

  40. [40]

    NVIDIA. 2022. NVIDIA H100 Tensor Core GPU Architecture Whitepa- per.https://resources.nvidia.com/en-us-hopper-architecture/nvidia- h100-tensor-c

    2022

  41. [41]

    NVIDIA. 2024. GB200 NVL72.https://www.nvidia.com/en-us/data- center/gb200-nvl72/

    2024

  42. [42]

    NVIDIA. 2024. NCCL: Optimized Primitives for Collective Multi-GPU Communication.https://github.com/NVIDIA/nccl

    2024

  43. [43]

    NVIDIA. 2025. Nsight Compute.https://developer.nvidia.com/nsight- compute. NVIDIA developer tools. Accessed 2026-06-02

    2025

  44. [44]

    NVIDIA. 2025. Nsight Systems.https://developer.nvidia.com/nsight- systems. NVIDIA developer tools. Accessed 2026-06-02

    2025

  45. [45]

    NVIDIA. 2025. NVLink & NVLink Switch: Fastest HPC Data Center Platform.https://www.nvidia.com/en-us/data-center/nvlink/

    2025

  46. [46]

    NVIDIA. 2025. NVSHMEM: A Parallel Programming Interface Based on OpenSHMEM for NVIDIA GPU Clusters.https://developer.nvidia. com/nvshmem

    2025

  47. [47]

    NVIDIA. 2025. Optimizing for Low-Latency Communication in Inference Workloads with JAX and XLA. NVIDIA Technical Blog.https://developer.nvidia.com/blog/optimizing-for-low-latency- communication-in-inference-workloads-with-jax-and-xla/

    2025

  48. [48]

    NVIDIA. 2025. User Buffer Registration — NCCL Documenta- tion.https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/ usage/bufferreg.html

    2025

  49. [49]

    NVIDIA. 2026. GPUDirect RDMA (CUDA Documentation).https: //docs.nvidia.com/cuda/gpudirect-rdma/

    2026

  50. [50]

    NVIDIA Corporation. 2026. Performance reported by NCCL tests.https://github.com/NVIDIA/nccl-tests/blob/master/doc/ PERFORMANCE.md. Accessed: 2026-05-22

    2026

  51. [51]

    Adam Paszke et al . 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. InAdvances in Neural Information Processing Systems 32 (NeurIPS 19). https://papers.neurips.cc/paper/9015-pytorch-an-imperative- style-high-performance-deep-learning-library

    2019

  52. [52]

    Pitch Patarasuk and Xin Yuan. 2009. Bandwidth optimal all-reduce algorithms for clusters of workstations.J. Parallel and Distrib. Comput. (2009).https://doi.org/10.1016/j.jpdc.2008.09.002

    doi:10.1016/j.jpdc.2008.09.002 2009

  53. [53]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. In51st ACM/IEEE Annual International Symposium on Computer Architecture (ISCA 24). https://doi.org/10.1109/ISCA59077.2024.00019

    arXiv 2024

  54. [54]

    Sinclair

    Suchita Pati, Shaizeen Aga, Mahzabeen Islam, Nuwan Jayasena, and Matthew D. Sinclair. 2023. Tale of Two Cs: Computation vs. Com- munication Scaling for Future Transformers on Future Hardware. In IEEE International Symposium on Workload Characterization (IISWC 23).https://doi.org/10.1109/IISWC59245.2023.00026

    arXiv 2023

  55. [55]

    Sinclair

    Suchita Pati, Shaizeen Aga, Mahzabeen Islam, Nuwan Jayasena, and Matthew D. Sinclair. 2024. T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectives. InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 24).https: //doi.org/10.1145/3620665.3640410

    arXiv 2024

  56. [56]

    Suchita Pati, Mahzabeen Islam, Shaizeen Aga, and Mohamed Assem Ibrahim. 2025. DMA Collectives for Efficient ML Communication Offloads.arXiv preprint arXiv:2511.06605(2025).https://arxiv.org/abs/ 14 2511.06605

    Pith/arXiv arXiv 2025

  57. [57]

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. Efficiently Scaling Transformer Inference. InProceedings of Machine Learning and Systems 5 (MLSys 23).https: //arxiv.org/abs/2211.05102

    Pith/arXiv arXiv 2023

  58. [58]

    PyTorch Contributors. 2024. CUDA Semantics — PyTorch CUDA Caching Memory Allocator (Memory Management).https://docs. pytorch.org/docs/stable/notes/cuda.html#memory-management

    2024

  59. [59]

    PyTorch Contributors. 2024. Understanding CUDA Memory Usage.https://docs.pytorch.org/docs/stable/torch_cuda_memory. html. PyTorch documentation; CUDA memory snapshots via torch.cuda.memory._record_memory_history and _snapshot. Ac- cessed 2026-06-01

    2024

  60. [60]

    PyTorch Team. 2024. CUDA Semantics: Expandable Segments.https: //docs.pytorch.org/docs/stable/notes/cuda.html

    2024

  61. [61]

    Kun Qian, Yongqing Xi, Jiamin Cao, Jiaqi Gao, Yichi Xu, Yu Guan, Binzhang Fu, Xuemei Shi, Fangbo Zhu, Rui Miao, et al. 2024. Alibaba HPN: A Data Center Network for Large Language Model Training. InProceedings of the ACM SIGCOMM 2024 Conference (SIGCOMM 24). https://doi.org/10.1145/3651890.3672265

    arXiv 2024

  62. [62]

    SGLang Team. 2025. DeepEP-Ascend: Ascend Implementation of DeepEP.https://github.com/sgl-project/sgl-kernel-npu/tree/main/ python/deep_ep

    2025

  63. [63]

    Aashaka Shah, Vijay Chidambaram, Meghan Cowan, Saeed Maleki, Madan Musuvathi, Todd Mytkowicz, Jacob Nelson, and Olli Saarikivi

  64. [64]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. In International Conference on Learning Representations (ICLR 17).https: //openreview.net/forum?id=B1ckMDqlg

    2017

  65. [65]

    Min Si, Pavan Balaji, Yongzhou Chen, et al. 2025. Collective Commu- nication for 100k+ GPUs. arXiv preprint arXiv:2510.20171.https: //arxiv.org/abs/2510.20171

    arXiv 2025

  66. [66]

    In20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23)

    TACCL: Guiding Collective Algorithm Synthesis using Commu- nication Sketches. In20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX Association.https: //www.usenix.org/conference/nsdi23/presentation/shah

  67. [67]

    Rajeev Thakur, Rolf Rabenseifner, and William Gropp. 2005. Opti- mization of Collective Communication Operations in MPICH.Inter- national Journal of High Performance Computing Applications(2005). https://dl.acm.org/doi/10.1177/1094342005051521

    doi:10.1177/1094342005051521 2005

  68. [68]

    vLLM-Ascend Team. 2025. ACL Graph. vLLM-Ascend documenta- tion, Developer Guide.https://docs.vllm.ai/projects/ascend/en/latest/ developer_guide/feature_guide/ACL_Graph.htmlAccessed 2026-06-

    2025

  69. [69]

    Athinagoras Skiadopoulos, Zhiqiang Xie, Mark Zhao, Qizhe Cai, Sak- sham Agarwal, Jacob Adelmann, David Ahern, Carlo Contavalli, Michael Goldflam, Vitaly Mayatskikh, et al. 2024. High-throughput and Flexible Host Networking for Accelerated Computing. InProceed- ings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24).https...

    2024

  70. [70]

    Weiyang Wang, Moein Khazraee, Zhizhen Zhong, Manya Ghobadi, Zhi- hao Jia, Dheevatsa Mudigere, Ying Zhang, and Anthony Kewitsch. 2023. TopoOpt: Co-optimizing Network Topology and Parallelization Strat- egy for Distributed Training Jobs. InProceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23).https://www.usenix.or...

    2023

  71. [71]

    Zilong Wang, Layong Luo, Qingsong Ning, Chaoliang Zeng, Wenxue Li, et al. 2023. SRNIC: A Scalable Architecture for RDMA NICs. InPro- ceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23).https://www.usenix.org/conference/ nsdi23/presentation/wang-zilong

    2023

  72. [72]

    Describes ACL Graph capture and replay for graph mode on As- cend

  73. [73]

    Mattias De Wael, Stefan Marr, Bruno De Fraine, Tom Van Cutsem, and Wolfgang De Meuter. 2015. Partitioned Global Address Space Languages.Comput. Surveys(2015).https://doi.org/10.1145/2716320

    doi:10.1145/2716320 2015

  74. [74]

    Ao Xiao, Bangzheng He, Baoquan Zhang, Baoxing Huai, Bingji Wang, et al. 2025. xDeepServe: Model-as-a-Service on Huawei CloudMa- trix384. arXiv preprint arXiv:2508.02520.https://arxiv.org/abs/2508. 02520

    arXiv 2025

  75. [75]

    Guanbin Xu, ZhenGuo Xu, Yuzhe Li, Youhui Bai, Ping Gong, Chaoyi Ruan, and Cheng Li. 2026. Lagom: Unleashing the Power of Commu- nication and Computation Overlapping for Distributed LLM Training. arXiv preprint arXiv:2602.20656.https://arxiv.org/abs/2602.20656

    arXiv 2026

  76. [76]

    Xingda Wei, Fangming Lu, Tianxia Wang, Jinyu Gu, Yuhan Yang, Rong Chen, and Haibo Chen. 2023. No Provisioned Concurrency: Fast RDMA-codesigned Remote Fork for Serverless Computing. In Proceedings of the 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23).https://www.usenix.org/conference/ osdi23/presentation/wei-rdma

    2023

  77. [77]

    Tianyuan Wu, Lunxi Cao, Hanfeng Lu, Xiaoxiao Jiang, Yinghao Yu, Siran Yang, Guodong Yang, Jiamang Wang, Lin Qu, Liping Zhang, et al

  78. [78]

    Xing, et al

    Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P. Xing, et al. 2022. Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning. InProceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22).https://www.usenix.org...

    2022

  79. [79]

    Gonzalez, et al

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, et al . 2024. SGLang: Efficient Execution of Structured Language Model Programs. InAd- vances in Neural Information Processing Systems (NeurIPS 24). https://proceedings.neurips.cc/paper_files/paper/2024/hash/ 724...

    2024

  80. [80]

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xu- anzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24).https://www.usenix.org/conference/osdi24/ presentation/zhong-yinmin

    2024

Showing first 80 references.