Pith. sign in

REVIEW 3 major objections 5 minor 70 references

Every Microsecond Matters: Achieving Near Speed-of-Light Latency in GPU Collectives

T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By eliminating global memory barriers, this paper claims, small-message GPU collectives can run within about 7% of the hardware speed-of-light lower bound, with measurable gains for LLM inference and HPC workloads.

desk verdict Solid latency work with a genuinely new kernel and API, but the 'absolute SoL bound' is a model estimate that flatters the headline number. read the letter →

arxiv 2607.16100 v1 pith:WT4ADOFA submitted 2026-07-17 cs.DC

Siyuan Shen , Anton Korzh , John Bachan , Tiancheng Chen , Arnav Goel , Ludwig Schneider , Pouya Kousha , Zhenhao He
show 5 more authors
Sylvain Jeaugey Kamil Iskra Nishank Chandawala Jeff R. Hammond Torsten Hoefler
This is my paper
classification cs.DC
keywords GPUcollectivesAllReducelatencyspeed-of-lightboundbarrier-freesynchronizationsymmetricmemoryLLprotocolsentinelpollingLL128atomic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Small-message collective communication—the frequent AllReduce calls that sit on the critical path of LLM decoding and HPC time-stepping—is latency-bound, not bandwidth-bound. This paper tries to show that such collectives can approach the hardware speed-of-light floor on a scale-up network (one NVLink domain) by removing global memory barriers altogether. The authors identify global barriers as the dominant overhead, often two barriers accounting for about 40% of a small AllReduce, and replace them with lower-cost synchronization: packing data and a flag into atomic stores, polling sentinel-initialized buffers, and using double buffering so each incoming chunk authorizes the next send. They implement these ideas as reusable device-side primitives in a widely used collective library and build new AllReduce kernels on top. If the claims hold, latency-critical workloads pay less per token: LLM inference shows 7–13% lower inter-token latency, a distributed dense linear algebra library shows up to 7% speedup, and small-message AllReduce overhead drops to about 7% over the estimated lower bound.

What carries the argument

The load-bearing piece is a set of synchronization mechanisms plus a cost model. The LL protocol makes a 16-byte atomic store carry both data and an incrementing epoch flag, so the receiver can poll one word and know when data is valid without a separate flag write or barrier. Sentinel polling achieves the same with full bandwidth by pre-filling buffers with a sentinel value (such as NaN) and watching for it to change. Double buffering extends either mechanism across iterations by having each rank alternate between two scratch buffers as it exchanges chunks with a peer, so a receive from a peer—rather than a global barrier—grants permission to overwrite that peer's earlier buffer. The LL128

What would settle it

Measure the latency of a single 128-byte two-GPU AllReduce on the same hardware. If any hand-tuned kernel can complete below 1.404 µs—the paper's computed speed-of-light floor—the bound is not absolute. Alternatively, measure whether the latency of issuing remote stores grows with the number of peers for a fixed message size: if sending to 64 peers costs more than sending to 1 peer, the model's rank-independence assumption is violated and the 'within 7%' claim is an underestimate of the gap.

Watch

Extended reading notes

Core claim

The paper's central claim is that the remaining gap between practical GPU AllReduce latency and the hardware floor is synchronization, not data movement. Global memory barriers that coordinate thread blocks across GPUs cost more than one microsecond each, and two such barriers can account for roughly 40% of a small AllReduce. The authors replace those barriers with three mechanisms: packing data with a flag in 16-byte atomic stores (LL), polling sentinel-initialized scratch buffers, and bidirectional double buffering where each receive from a peer implicitly permits the next send. These compose into one-shot and two-shot AllReduce kernels whose overhead is within about 7% of their measured s

Load-bearing premise

The load-bearing assumption is that the speed-of-light bound itself is correctly modeled: that all AllReduce buffers stay resident in L2, that stores to all peers can be issued simultaneously with no extra per-rank cost, and that the one-shot push schedule is the absolute minimal data movement—so the reported 'within 7%' is a comparison to this model, not a proven hardware floor.

Editorial extensions

If this is right

  • If the barrier-free kernels work as described, small-message AllReduce on scale-up systems can run close to the hardware floor—2.37 µs versus 11.0 µs for the ring baseline on four GPUs—so latency-bound workloads no longer need to pay a multi-microsecond synchronization tax.
  • Because the same primitives (LL, sentinel, double buffering) are exposed as a small device-side API, other collectives—Broadcast, Reduce, ReduceScatter, AllGather—can be rebuilt with the same barrier-free pattern, not just AllReduce.
  • The kernel-selection rule derived from measurements (one-shot for small messages, two-shot for medium, LL128 atomic for scalable addition-dominated reductions) gives a practical tuning strategy for future workloads: pick by message size and rank count.
  • For long-context LLM serving, these latency improvements translate to 7–13% lower inter-token latency and up to about 15% throughput gains across dense, mixture-of-experts, and hybrid-attention models, with estimated cost savings that grow with generation volume.
  • For traditional HPC, the distributed dense linear algebra experiments show that even without symmetric-memory registration, one-shot kernels improve a dense eigensolver by up to 7%, bridging AI frameworks and HPC libraries.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 'within 7% of the absolute SoL' number is a comparison to the paper's own model of the floor; the practical takeaway—that barrier elimination, not raw data movement, is what remains—would survive even if the absolute floor is later revised.
  • The techniques are described as portable to any platform with GPU-initiated remote writes, device-side polling, and ordering/fence operations; if true, similar gains should appear on other vendors' scale-up interconnects and in software stacks that provide equivalent primitives.
  • The LL128 atomic algorithm's non-deterministic floating-point summation is a real constraint; one can imagine a deterministic variant using per-chunk grouping or hierarchical accumulation that keeps the barrier-free property while restoring reproducibility—an extension the paper does not fully explore.
  • For LLM inference, the cost-per-token savings assume sustained decode throughput at on-demand prices; in disaggregated serving, the benefit would show up mainly in decode-node occupancy, not necessarily as a flat dollar-per-token reduction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents a set of techniques and APIs for building low-latency GPU collective kernels, targeting AllReduce within a single NVLink scale-up domain. The authors identify global memory barriers as a key latency source and propose barrier-free synchronization mechanisms (LL, sentinel, double buffering, and a novel two-shot LL128 atomic algorithm). They implement these as reusable device-side primitives on top of NCCL and evaluate them on GB200 systems. Microbenchmarks show substantial latency reductions for small and medium messages, with a reported overhead of about 7% over the authors' Speed-of-Light (SoL) model at 2 GPUs, and application studies in vLLM and cuSOLVERMp show consistent improvements.

Significance. If the claims hold, this is a significant practical contribution to a growing area: it provides reusable low-latency abstractions and demonstrates large absolute latency reductions (e.g., 11.0 µs to 2.37 µs for small-message AllReduce on 4 GPUs) and measurable application-level gains (7–13% ITL reduction in vLLM, up to 7% in cuSOLVERMp). The paper includes extensive microbenchmarks against six existing implementations, and the authors have released the source code, which is a strong reproducibility asset. The main weakness is that the headline 'absolute SoL lower bound' is an estimate, and the correctness of the LL128 atomic kernel depends on an unverified NVLink hardware guarantee. These issues are fixable but currently weaken the paper's central quantitative claim.

major comments (3)
  1. [Section VI, L_SoL formula and Fig. 11 (bottom)] The claim of 'within 7% of the absolute SoL lower bound' (Abstract and Section VII.C) rests on the SoL model L_SoL = 2·L_L2_RTT + L_remote_store. This is not an 'absolute' lower bound: L_L2_RTT is approximated by the latency of a single __threadfence(), which drains the memory pipeline and is not demonstrably equal to an L2 load-to-use latency; and the model assumes simultaneous stores to all peers with no per-rank fan-out cost, making L_SoL independent of N. The paper's own Fig. 11 (bottom) shows that at 64 GPUs the best one-shot kernels are ~70% above this bound, so the bound does not capture N-dependent costs. The 7% figure is only demonstrated at 2 GPUs. Please either derive a formal lower bound from documented hardware latencies or characterize the model as a design-specific empirical estimate and qualify the abstract and headline claims accordingly.
  2. [Section IV.4, 'LL128 Atomic AllReduce'] Correctness of the ReduceScatter/AllGather protocol depends on the assertion that 'NVLink ensures that these operations are applied atomically at the cache-line level' and that 'NVLink performs 128-byte writes atomically.' No reference or experimental evidence is provided for this hardware guarantee. The completion detection (flag equals N) assumes that all 8 threads' atomic additions to a 128-byte line are applied atomically as a unit; a weaker guarantee (e.g., per-8-byte atomics) would break the algorithm. Please provide a citation to an NVLink specification or a microbenchmark validating 128-byte atomic addition on the target GB200 hardware.
  3. [Section VII.C / Fig. 11 and related text] The paper's own data contradict the unqualified statement that the bound is 'independent of the number of ranks involved' (Section VI). Fig. 11 (bottom) shows a clear increase in latency with GPU count, with 64-GPU latency ~70% above the same bound. This indicates that the SoL model is not a true hardware lower bound for multi-rank collectives. The authors should either extend the model to account for fan-out/contention or explicitly restrict the '7% overhead' claim to the 2-GPU case and present the multi-GPU results as comparisons to the same estimated baseline rather than to an absolute limit.
minor comments (5)
  1. [Abstract and Section I] The phrase 'even µs of overhead' is ungrammatical; use 'even microseconds.' Also 'EveryµsMatters' in the title is stylized but the spacing should be fixed.
  2. [Section IV.4 and Fig. 5] The diagram and description of the 'flag carrier' threads (496–511) versus 'regular threads' (0–495) is somewhat confusing. Please clarify that the displaced element is stored in shared memory and how the extra threads are sized for half-precision.
  3. [Section V.A / Fig. 9] The example uses bcast() to send data to all peers in a one-shot AllReduce. This is a point-to-multipoint push, not the collective Broadcast; please add a sentence to avoid confusion with NCCL's Broadcast collective.
  4. [Section VIII.A] The cost-saving estimate is explicitly labeled as approximate, which is good. However, the conversion uses CoreWeave's on-demand price; a sensitivity analysis (e.g., different cloud providers or on-premises cost) would make the claim more robust, though this is not required for the technical contribution.
  5. [Section IX] The authors note that kernel selection is driven by empirical measurements and that an accurate performance model is future work. This is a welcome and honest limitation; it could be reiterated in the conclusion so readers know the presented thresholds are system-specific.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: latency results are independent measurements against external libraries; the SoL bound is derived from primitive hardware latencies, not fitted to achieved latencies.

full rationale

The paper's central performance claims—microbenchmark latencies, vLLM ITL reductions, and cuSOLVERMp speedups—are external measurements compared against independent baselines (NCCL ring/tree/symmetric kernels, NVSHMEM, MSCCL++, vLLM). The SoL bound in Section VI is not fitted to the achieved latencies: L_SoL = 2·L_L2_RTT + L_remote_store is estimated from separate primitive measurements, namely __threadfence() latency for L_L2_RTT and a two-GPU ping-pong RTT for L_remote_store. Nothing in this construction uses the final kernel latencies as an input, so the 'within 7% of SoL' claim is a comparison to an independently estimated bound, not a renamed fit. The self-citations (e.g., "Demystifying NCCL" [11], NCCL technical blogs [19,22]) are background descriptions of LL protocols, symmetric memory, and device-side APIs; they are not invoked as load-bearing evidence for the new algorithms' correctness or performance. The paper itself discloses the model's idealizations in Section VI: buffers assumed to reside in L2, stores to all peers assumed simultaneous, and instruction scheduling ignored; its own Fig. 11 shows 64-GPU latency about 70% above the bound, indicating the bound is not tight at scale. Similarly, Section IX admits kernel selection is driven by empirical measurements. These are accuracy/tuning caveats, not circular reasoning: the derivation does not reduce to its own outputs by construction.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claim relies on measured hardware latencies (L2 RTT, remote store) and several hardware/software assumptions about L2 coherency, NVLink atomicity, and peer-buffer flow control. No new physical entities are introduced; the free parameters are engineering choices (scratch sizes, selection thresholds) tuned on the same benchmark suite.

free parameters (3)
  • scratch_buffer_size_one_shot = 4 MiB
    Chosen from the scratch-buffer sweep in Fig. 12; one-shot kernels show little benefit beyond this size. This is a tuning parameter, not a fitted constant for a stated law.
  • scratch_buffer_size_two_shot_ll128 = 64 MiB
    Chosen from Fig. 12 as the size at which two-shot and LL128 kernels plateau; affects configuration, not the central claim.
  • kernel_selection_thresholds = e.g., <1 MiB one-shot, 1–2 MiB two-shot at 4 ranks
    NCCL is modified to select algorithms based on the authors' own empirical microbenchmarks (Section VIII); this is empirical tuning of the delivered system.
assumptions (6)
  • domain assumption L2 is the point of coherency and small messages fit entirely in L2; all SoL data movements are L2 hits.
    Section VI and Fig. 10: the SoL bound assumes L2 cache hits for input, scratch, and output; valid for the small/medium messages targeted, but not a universal hardware guarantee.
  • ad hoc to paper A one-shot push AllReduce with simultaneous stores to all peers is the minimal data movement, giving L_SoL = 2 L_L2_RTT + L_remote_store.
    Section VI: presented as the SoL lower bound without a proof that no other schedule (e.g., multicast reduction, load-based) could be faster; this is the key modeling choice behind the 7% headline.
  • domain assumption threadfence latency is a good approximation of L2 RTT.
    Section VI: used to measure L_L2_RTT = 0.306 µs; threadfence enforces visibility at L2 but includes instruction-issue overhead.
  • domain assumption NVLink guarantees cache-line-level atomicity for 128-byte atomic additions, so flag==N implies all contributions for that line have arrived.
    Section IV.B.4 and Fig. 5: correctness of LL128 atomic AllReduce hinges on this; asserted but not documented with a hardware spec or microarchitectural evidence.
  • domain assumption Each rank communicates with a given peer at most once per iteration; receive from a peer grants implicit permission for the next send.
    Section IV.B.3: double-buffering flow control assumes at most one store per remote buffer per iteration; ring-style schedules with forwarding need explicit barriers (stated in Section V.i).
  • domain assumption Users can guarantee that transmitted values never equal the sentinel (e.g., -NaN), and buffers are reset before reuse.
    Section IV.B.2: sentinel mode fails if a legitimate data value matches the sentinel; this is acknowledged as a drawback and a user obligation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Every Microsecond Matters: Achieving Near Speed-of-Light Latency in GPU Collectives." pith.science (2026). https://pith.science/paper/WT4ADOFA

@misc{pith2026260716100,
  author       = {Pith},
  title        = {Pith review of: Every Microsecond Matters: Achieving Near Speed-of-Light Latency in GPU Collectives},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WT4ADOFA}},
  note         = {Machine review of arXiv:2607.16100}
}
read the original abstract

GPU collective communication is typically optimized for bandwidth, yet many emerging workloads are increasingly limited by latency. Long-context decode-heavy large language model (LLM) inference is a prime example, where serving large models requires multiple GPUs, and many small collectives lie directly on the critical path of token generation. Therefore, even microsecond of overhead can impact performance and cost. In this work, we study how to approach the hardware Speed-of-Light (SoL) lower bound for GPU collectives within a scale-up network. We identify key principles for near-optimal designs, including barrier-free synchronization and efficient use of symmetric memory and multicast. Building on NCCL's device-side API, we develop low-latency interfaces for constructing custom collective kernels and use them to implement new symmetric collectives in NCCL. Microbenchmarks show substantial latency reductions for small and medium messages, reducing overhead to within 7% of the absolute SoL lower bound. When integrated into real applications, these kernels improve inter-token latency and throughput in LLM inference and accelerate cuSOLVERMp, demonstrating benefits for both AI inference and traditional HPC workloads.

Figures

Figures reproduced from arXiv: 2607.16100 by the authors.

Figure 1
Figure 1. In long-context, small-batch tensor-parallel (TP) LLM [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of device-initiated communication and sym [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Barrier latency on GB200 as a function of the number [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Example illustrating bidirectional communication with double buffering. Two ranks exchange chunked inputs, where [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Overview of the two-shot LL128 atomic AllReduce [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Overview of the proposed low-latency API, showing device-side and host-side functions. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Example construction and layout of a ncclLLBuffer object. Each CTA is assigned a fixed-size region per epoch, while epochs alternate between sub-buffers. Invoking advanceEpoch() switches the active sub-buffer and incre￾ments the internal epoch value. a) ncclLLBuffer: T…
Figure 8
Figure 8. Figure 8: Illustration of send() and recv() of a ncclLLBuffer for a single CTA. b) send: The send primitive writes a value into a specified peer slot. The destination layout is determined by the type T: each element occupies sizeof(T) bytes, or 2 × sizeof(T) for LL mode. This la…
Figure 9
Figure 9. Figure 9: Example implementation of a one-shot AllReduce using [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Minimal data movement in an AllReduce. All buffers [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: Top plot shows out-of-place AllReduce latency versus message size on GB200 for 2 to 64 GPUs. Each subplot [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 12
Figure 12. Figure 12: Impact of scratch buffer size on AllReduce latency [PITH_FULL_IMAGE:figures/full_fig_p009_12.png]
Figure 13
Figure 13. Figure 13: Effect of low-latency collectives on vLLM inference. Rows report mean inter-token latency (ITL), output throughput, [PITH_FULL_IMAGE:figures/full_fig_p011_13.png]
Figure 14
Figure 14. Figure 14: Performance of mp_sygvd with and without the new low-latency kernels. Bars show mean GFLOPS per GPU over 5 trials, with error bars indicating standard deviation. Percent￾age annotations show the improvement over the baseline. The experiments were conducted on the Alps…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

70 extracted references · 1 canonical work pages

  1. [1]

    Deepseek-v3 technical report,

    DeepSeek-AI, A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Guo, D. Yang, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. Wang, H. Zhang, H. Ding, H. Xin, H. Gao, H. Li, H. Qu, J. L. Cai, J. Liang, J. Guo, J. Ni, J. Li, J. Wang, J. Chen, J. Chen, J. Yuan, J...

    2025

  2. [2]

    Deepseek-v3/r1 671b deployment guide: Gpu require- ments,

    RiseUnion, “Deepseek-v3/r1 671b deployment guide: Gpu require- ments,” 2025. Reports deployments requiring up to 32 accelerators for full-precision inference

    2025

  3. [3]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” inProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  4. [4]

    State of ai: An empirical 100 trillion token study with openrouter,

    M. Aubakirova, A. Atallah, C. Clark, J. Summerville, and A. Midha, “State of ai: An empirical 100 trillion token study with openrouter,” 2026

    2026

  5. [5]

    Sglang: efficient execution of structured language model programs,

    L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y . Sheng, “Sglang: efficient execution of structured language model programs,” inProceedings of the 38th International Conference on Neural Infor- mation Processing Systems, NIPS ’24, (Red Hook, NY , USA), Curran Associates Inc., 2024. h...

    arXiv 2024

  6. [6]

    Tensorrt-llm: A library for optimizing large language model inference,

    N. Corporation, “Tensorrt-llm: A library for optimizing large language model inference,” 2023. Accessed: 2024-05-20

    2023

  7. [8]

    CoreWeave Pricing: Instance Pricing,

    CoreWeave, “CoreWeave Pricing: Instance Pricing,” 2026. Accessed: March 2026

    2026

  8. [9]

    Scaling-up pytorch inference: Serving billions of daily nlp inferences with onnx runtime,

    F. Xu, “Scaling-up pytorch inference: Serving billions of daily nlp inferences with onnx runtime,” 2022. Microsoft Open Source Blog, accessed: March 2026

    2022

Show all 70 references
  1. [10]

    Understanding data movement in tightly coupled heterogeneous systems: A case study with the grace hopper superchip,

    L. Fusco, M. Khalilov, M. Chrapek, G. Chukkapalli, T. Schulthess, and T. Hoefler, “Understanding data movement in tightly coupled heterogeneous systems: A case study with the grace hopper superchip,” 2024

    2024

  2. [11]

    Demystifying NCCL: An In-Depth Analysis of GPU Communication Protocols and Algorithms ,

    Z. Hu, S. Shen, T. Bonato, S. Jeaugey, C. Alexander, E. Spada, J. Dinan, J. Hammond, and T. Hoefler, “ Demystifying NCCL: An In-Depth Analysis of GPU Communication Protocols and Algorithms ,” in2025 IEEE Symposium on High-Performance Interconnects (HOTI), (Los Alamitos, CA, US...

    2025

  3. [12]

    NVSHMEM communication library,

    N. Corporation, “NVSHMEM communication library,” 2022. Accessed on 2024-05-20

    2022

  4. [13]

    ROCm Communication Collectives Library (RCCL) Documentation,

    Advanced Micro Devices, Inc., “ROCm Communication Collectives Library (RCCL) Documentation,” 2024. Accessed: February 2026

    2024

  5. [14]

    AMD ROCm Documentation, 2026

    Advanced Micro Devices, Inc.,rocSHMEM 3.2.0 Documentation. AMD ROCm Documentation, 2026. Accessed: 2026-02-12

    2026

  6. [15]

    Accessed: February 2026

    UXL Foundation,oneAPI Specification 1.3-rev-1, 2024. Accessed: February 2026

    2024

  7. [16]

    Message Passing Interface Forum,MPI: A Message-Passing Interface Standard Version 5.0, June 2025

    2025

  8. [17]

    An introduction to cuda-aware mpi,

    J. Kraus, “An introduction to cuda-aware mpi,”NVIDIA Technical Blog, July 2025

    2025

  9. [19]

    Enabling fast inference and resilient training with nccl 2.27,

    J. Bachan, K. Ouyang, M. Mubarak, T. Gillis, B. Chang, D. Bureddy, G. Congiu, K. Caton, K. Aubrey, and X. Li, “Enabling fast inference and resilient training with nccl 2.27,” Jul 2025. Technical Blog

    2025

  10. [20]

    Gpu-initiated networking for nccl,

    K. Hamidouche, J. Bachan, P. Markthub, P.-J. Gootzen, E. Agostini, S. Jeaugey, A. Shafi, G. Theodorakis, and M. G. Venkata, “Gpu-initiated networking for nccl,” 2025

    2025

  11. [21]

    NVIDIA, 2025

    NVIDIA Corporation,Device-Initiated Communication — NCCL 2.29.1 Documentation. NVIDIA, 2025. Accessed: 2026-02-12

    2025

  12. [22]

    Fusing communication and compute with new device api and copy engine collectives in nvidia nccl 2.28,

    S. Jeaugey, J. Bachan, P. Markthub, Z. He, S. Das, and F. Ghodsian, “Fusing communication and compute with new device api and copy engine collectives in nvidia nccl 2.28,” Nov. 2025. NVIDIA Technical Blog

    2025

  13. [23]

    PyTorch Foundation, 2026

    PyTorch Contributors,Distributed communication package – torch.distributed. PyTorch Foundation, 2026. PyTorch 2.10 documentation, last updated 2026-01-08, accessed 2026-02-12

    2026

  14. [24]

    NHR@FAU, 2026

    Erlangen National High Performance Computing Center (NHR@FAU), PyTorch – NHR@FAU HPC Documentation. NHR@FAU, 2026. Ac- cessed 2026-02-12

    2026

  15. [25]

    Pytorch: An imperative style, high- performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. K ¨opf, E. Z. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, hi...

    1912 arXiv

  16. [26]

    TensorFlow: Large-scale machine learning on heterogeneous systems,

    M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y . Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Man ´e, R. Monga, S. Moore, D. Murray, C. Olah, M...

    2015

  17. [27]

    NVIDIA Corporation,

    NVIDIA Corporation,cuSOLVER API Reference. NVIDIA Corporation,

  18. [28]

    NVIDIA Corporation, 2026

    NVIDIA Corporation,cuBLASMp: A High-Performance CUDA Library for Distributed Dense Linear Algebra. NVIDIA Corporation, 2026. Accessed 2026-02-12

    2026

  19. [29]

    Llm inference beyond a single node: From bottlenecks to mitigations with fast all-reduce communication,

    P. Singhania, S. Singh, L. D. Hough, A. Srivastava, H. Menon, C. F. Jekel, and A. Bhatele, “Llm inference beyond a single node: From bottlenecks to mitigations with fast all-reduce communication,” 2025

    2025

  20. [30]

    Msccl++: Rethinking gpu communication abstractions for ai inference,

    C. Hwang, P. Cheng, R. Dathathri, A. Jangda, S. Maleki, M. Musu- vathi, O. Saarikivi, A. Shah, Z. Yang, B. Li, C. Rocha, Q. Zhou, M. Ghazimirsaeed, S. Anantharamu, and J. Jose, “Msccl++: Rethinking gpu communication abstractions for ai inference,” inProceedings of the 31st ACM...

    2026

  21. [31]

    Nvidia gtc 2025 - built for reasoning, vera rubin, kyber, cpo, dynamo inference, jensen math, feynman,

    D. Patel, M. Xie, D. Nishball, I. Chiam, P. Zhou, Doug, and W. Chu, “Nvidia gtc 2025 - built for reasoning, vera rubin, kyber, cpo, dynamo inference, jensen math, feynman,” Mar. 2025

    2025

  22. [32]

    Colossus: xai’s supercomputer for grok,

    xAI, “Colossus: xai’s supercomputer for grok,” 2024. Describes a large- scale GPU cluster with tightly coupled high-bandwidth interconnect

    2024

  23. [33]

    Recent improvement to open mpi allreduce and the impact to application performance,

    J. Tang, L. Robison, M. Koop, and W. Wang, “Recent improvement to open mpi allreduce and the impact to application performance,” Sept. 2024

    2024

  24. [34]

    Revisiting the time cost model of allreduce,

    D. Xiong, L. Chen, Y . Jiang, D. Li, S. Wang, and S. Wang, “Revisiting the time cost model of allreduce,” 2024

    2024

  25. [35]

    xccl: A survey of industry-led collective communication libraries for deep learning,

    A. Weingram, Y . Li, H. Qi, D. Ng, L. Dai, and X. Lu, “xccl: A survey of industry-led collective communication libraries for deep learning,”J. Comput. Sci. Technol., vol. 38, p. 166–195, Mar. 2023. https://doi.org/10.1007/s11390-023-2894-6

    2023 doi

  26. [36]

    Hear: Homomorphically encrypted allreduce,

    M. Chrapek, M. Khalilov, and T. Hoefler, “Hear: Homomorphically encrypted allreduce,” inProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’23, (New York, NY , USA), Association for Computing Machinery,

  27. [37]

    A survey of mpi usage in the us exascale computing project,

    D. E. Bernholdt, S. Boehm, G. Bosilca, M. Gorentla Venkata, R. E. Grant, T. Naughton, H. P. Pritchard, M. Schulz, and G. R. Vallee, “A survey of mpi usage in the us exascale computing project,”Concurrency and Computation: Practice and Experience, vol. 32, no. 3, p. e4851,

  28. [38]

    A large-scale study of mpi usage in open-source hpc applications,

    I. Laguna, R. Marshall, K. Mohror, M. Ruefenacht, A. Skjellum, and N. Sultana, “A large-scale study of mpi usage in open-source hpc applications,” inProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’19, (New York,...

    2019

  29. [39]

    Short-circuiting rings for low-latency allreduce,

    S.-M. Hammer, S. Schmid, R. Singh, and V . Addanki, “Short-circuiting rings for low-latency allreduce,” 2025

    2025

  30. [40]

    Cuda c++ programming guide

    NVIDIA Corporation, “Cuda c++ programming guide.” https://docs. nvidia.com/cuda/cuda-c-programming-guide/, 2024. CUDA Toolkit Documentation

    2024

  31. [41]

    Parallel thread execution isa version 8.0

    NVIDIA Corporation, “Parallel thread execution isa version 8.0.” https:// docs.nvidia.com/cuda/parallel-thread-execution/, 2023. NVIDIA CUDA PTX Instruction Set Architecture

    2023

  32. [42]

    vllm container (version 26.02-py3),

    NVIDIA Corporation, “vllm container (version 26.02-py3),” 2026. Ac- cessed: 2026-03-29

    2026

  33. [43]

    Network-offloaded bandwidth-optimal broadcast and allgather for distributed ai,

    M. Khalilov, S. D. Girolamo, M. Chrapek, R. Nudelman, G. Bloch, and T. Hoefler, “Network-offloaded bandwidth-optimal broadcast and allgather for distributed ai,” inSC24: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–17,

  34. [44]

    Comparative analysis of large language model inference serving systems: A performance study of vllm and huggingface tgi,

    S. Kolluru, “Comparative analysis of large language model inference serving systems: A performance study of vllm and huggingface tgi,” 2025

    2025

  35. [45]

    vllm – technology radar entry,

    NeoSignal, “vllm – technology radar entry,” 2025. Accessed: 2026-03- 29

    2025

  36. [46]

    The huge potential implications of long-context inference,

    J.-S. Denain and A. Ho, “The huge potential implications of long-context inference,” 2025. Accessed: 2026-03-29

    2025

  37. [47]

    https://dl.acm.org/doi/10.1109/SC41406.2024.00109

    2024 arXiv

  38. [48]

    Chain of agents: Large language models collaborating on long-context tasks,

    Y . Zhang, R. Sun, Y . Chen, T. Pfister, R. Zhang, and S. Arik, “Chain of agents: Large language models collaborating on long-context tasks,” 2024

    2024

  39. [49]

    Breaking the boundaries of long- context llm inference: Adaptive kv management on a single commodity gpu,

    H. Sun, L. Li, M. Xiao, and C. Xu, “Breaking the boundaries of long- context llm inference: Adaptive kv management on a single commodity gpu,” 2025

    2025

  40. [50]

    Prefill-decode disaggregation

    BentoML Team, “Prefill-decode disaggregation.” https://bentoml.com/ llm/inference-optimization/prefill-decode-disaggregation, 2025. Ex- plains motivation and benefits of PD disaggregation. 13

    2025

  41. [51]

    Is long context all you need? leveraging llm’s extended context for nl2sql,

    Y . Chung, G. T. Kakkar, Y . Gan, B. Milne, and F. ¨Ozcan, “Is long context all you need? leveraging llm’s extended context for nl2sql,” Proceedings of the VLDB Endowment, vol. 18, p. 2735–2747, Apr. 2025. http://dx.doi.org/10.14778/3742728.3742761

    2025

  42. [52]

    GROMACS Development Team,GROMACS 2024.3 Installation Guide,

    2024

  43. [53]

    Accessed: March 2026

    LAMMPS Development Team,LAMMPS Documentation: GPU Pack- age, 2026. Accessed: March 2026

    2026

  44. [54]

    Amrex and pyamrex: Looking beyond the exas- cale computing project,

    A. Myers, W. Zhang, A. Almgren, T. Antoun, J. Bell, A. Huebl, and A. Sinn, “Amrex and pyamrex: Looking beyond the exas- cale computing project,”The International Journal of High Perfor- mance Computing Applications, vol. 38, no. 6, pp. 599–611, 2024. https://doi.org/10.1177/10...

    2024 doi

  45. [55]

    Slo-aware compute resource allocation for prefill-decode disaggregated llm inference,

    L. Li, D. Li, B. Gong, and Y . Zhang, “Slo-aware compute resource allocation for prefill-decode disaggregated llm inference,” 2026

    2026

  46. [56]

    Qmcpack: an open source ab initio quantum monte carlo package for the electronic structure of atoms, molecules and solids,

    J. Kim, A. D. Baczewski, T. D. Beaudet, A. Benali, M. C. Bennett, M. A. Berrill, N. S. Blunt, E. J. L. Borda, M. Casula, D. M. Ceperley, S. Chiesa, B. K. Clark, R. C. Clay, K. T. Delaney, M. Dewing, K. P. Esler, H. Hao, O. Heinonen, P. R. C. Kent, J. T. Krogel, I. Kyl ¨anp¨a¨a...

    2018

  47. [57]

    Accessed: March 2026

    2026

  48. [58]

    NVIDIA HPCG Benchmark,

    NVIDIA, “NVIDIA HPCG Benchmark,” 2024. Accessed: 2026-04-08

    2024

  49. [59]

    Accessed: March 2026

    NVIDIA,cuSOLVERMp: A High-Performance CUDA Library for Dis- tributed Dense Linear Algebra, 2026. Accessed: March 2026

    2026

  50. [60]

    AMReX issue #4821: Device-initiated collectives in NCCL/RCCL

    A. Huebl, “AMReX issue #4821: Device-initiated collectives in NCCL/RCCL.” https://github.com/AMReX-Codes/amrex/issues/4821,

  51. [61]

    New research infrastructure: ’alps’ supercomputer inaugurated,

    CSCS, “New research infrastructure: ’alps’ supercomputer inaugurated,” Swiss National Supercomputing Center

  52. [62]

    Flashinfer: Efficient and customizable attention engine for llm inference serving,

    Z. Ye, L. Chen, R. Lai, W. Lin, Y . Zhang, S. Wang, T. Chen, B. Kasikci, V . Grover, A. Krishnamurthy, and L. Ceze, “Flashinfer: Efficient and customizable attention engine for llm inference serving,” 2025

    2025

  53. [63]

    Qmcpack issue #4654,

    QMCPACK Developers, “Qmcpack issue #4654,” 2023. GitHub issue in the QMCPACK repository

    2023

  54. [64]

    Nccl ep: Towards a unified expert parallel communication api for nccl,

    A. Goldman, N. Boker, M. Sheraizin, N. Admoni, A. Polyakov, S. Bhat- tacharya, F. Yu, K. Sun, G. Theodorakis, H.-C. Yin, P.-J. Gootzen, A. Shafi, A. Ravid, S. D. Girolamo, M. G. Venkata, and G. Bloch, “Nccl ep: Towards a unified expert parallel communication api for nccl,” 2026

    2026

  55. [65]

    Enhancing dis- tributed inference performance with the nvidia inference transfer library,

    A. Ranadive, T. Stamler, S. Lee, and M. Khazraee, “Enhancing dis- tributed inference performance with the nvidia inference transfer library,” Mar. 2026. NVIDIA Technical Blog

    2026

  56. [66]

    The elpa library: scal- able parallel eigenvalue solutions for electronic structure theory and computational science,

    A. Marek, V . Blum, R. Johanni, V . Havu, B. Lang, T. Auckenthaler, A. Heinecke, H.-J. Bungartz, and H. Lederer, “The elpa library: scal- able parallel eigenvalue solutions for electronic structure theory and computational science,”Journal of Physics: Condensed Matter, vol. 26...

    2014 doi

  57. [69]

    Deepep: an efficient expert-parallel communication library,

    C. Zhao, S. Zhou, L. Zhang, C. Deng, Z. Xu, Y . Liu, K. Yu, J. Li, and L. Zhao, “Deepep: an efficient expert-parallel communication library,” 2025

    2025

  58. [72]

    Gpt-5.4 chatbot,

    OpenAI, “Gpt-5.4 chatbot,” 2026. 14

    2026

  59. [2020]

    https://doi.org/10.1002/cpe.4851

    doi

  60. [2023]

    https://doi.org/10.1145/3581784.3607099

  61. [2024]

    Version 13.1, accessed 2026-02-12

    2026

  62. [2025]

    GitHub issue, accessed January 2026

    2026

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.