Pith. sign in

REVIEW 2 major objections 5 minor 53 references

A low-overhead bitvector format and adaptive ring algorithms cut GPU collective communication volume for unstructured sparse data, delivering multi-fold speedups over dense NCCL at high sparsity.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 15:20 UTC pith:RGZJGFOX

load-bearing objection Solid systems paper: new low-overhead sparse format plus adaptive ring collectives that deliver real, measured speedups on modern GPUs for unstructured medium-to-high sparsity. the 2 major comments →

arxiv 2607.04676 v1 pith:RGZJGFOX submitted 2026-07-06 cs.DC

Adaptive Space-efficient Collectives for Dynamic and Unstructured Sparsity on GPU Platforms

Lannie Dalton Hough , Emir Gencer , Hoffmann Muki , Abhinav Bhatele This is my paper
classification cs.DC
keywords sparse collectivesGPU communicationall-gatherreduce-scatterall-reducebitvector formatPiciunstructured sparsity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When data on GPUs is sparse—full of zeros—standard collective libraries still ship every element, wasting bandwidth. This paper shows that a new bitvector format called Pici, combined with ring algorithms that can switch between sparse and dense representations mid-collective, lets all-gather, reduce-scatter, and all-reduce send only nonzeros while keeping conversion costs low enough that the savings are real. The adaptation is driven by measured sparsity, network topology (intra-node NVLink versus inter-node fabric), and the phase of the collective, so that densification during reductions does not erase the gains. On a modern GPU cluster the resulting library, SpCCL, is up to five times faster than dense NCCL at 99 percent sparsity and still competitive down to moderate sparsity; the same primitives accelerate end-to-end gradient-pruned language-model training by 13–26 percent. A sympathetic reader cares because communication is already a dominant cost at scale, and many HPC and ML workloads produce exactly the unstructured, medium-to-high sparsity that prior sparse-collective work ignored.

Core claim

At 99 percent input sparsity the authors’ adaptive SpCCL implementations of all-gather, reduce-scatter and all-reduce achieve up to 5.25 imes, 2.5 imes and 2.66 imes speedups over dense NCCL on Perlmutter, with the same primitives yielding 13–26 percent faster end-to-end iterations for gradient-pruned 1.5 B–3.3 B language models; the gains come from a 3.15 percent-overhead bitvector format (Pici) and from topology- and phase-aware decisions that keep (de)compression costs from canceling the reduction in message volume.

What carries the argument

Pici—a flat-index, 64 imes64-tile bitvector format with only 3.15 percent space overhead relative to dense fp32 data—plus CUDA warp-cooperative (de)compression kernels and an adaptive ring algorithm that re-chooses sparse versus dense representation after every ring step according to measured sparsity and link type.

Load-bearing premise

The densification formula and the three empirically tuned thresholds remain near-optimal for sparsity patterns and network topologies that differ from the uniform-random, Slingshot-plus-NVLink setting used to choose them.

What would settle it

Re-run the same 512–2048 MiB microbenchmarks and the gradient-pruned DDP training workload on a different GPU cluster (or with structured rather than uniform sparsity) while freezing the published thresholds; if the reported speedups disappear or reverse, the claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper presents SpCCL, a set of sparsity-aware GPU collectives (all-gather, reduce-scatter, all-reduce) built as an extension of NCCLX. It introduces Pici, a bitvector-based sparse format with ~3.15% overhead relative to dense fp32 data and warp-cooperative CUDA (de)compression kernels optimized for moderate unstructured sparsity. Algorithms adaptively switch between sparse (Pici or COO) and dense representations during ring steps using topology-aware (inter/intra-node) and phase-aware thresholds to handle fill-in densification. Microbenchmarks on Perlmutter (up to 64 A100s) report up to 5.25 imes/2.5 imes/2.66 imes speedups over dense NCCL at 99% sparsity for AG/RS/AR, with end-to-end gains of 13–26% on gradient-pruned DDP training of 1.5B–3.3B models.

Significance. If the reported speedups hold under the evaluated conditions, the work is a solid systems contribution to sparse collective communication for modern GPU clusters and distributed DL. Strengths include a carefully engineered low-overhead format (Pici) with fused kernels, reuse of a production-quality dense baseline (NCCLX) rather than a from-scratch implementation, multi-job averaging with min/max bars, channel/threshold sweeps, and a realistic pruned-LLM case study. The adaptive densification handling and explicit comparison to both dense NCCL/NCCLX and prior sparse work (SparCML) make the empirical claims more credible than many prior sparse-collective papers. The free parameters (thresholds, channel count) are acknowledged and swept; they do not circularly define the headline numbers.

major comments (2)
  1. §IV-C and Eq. (7): The densification extrapolation δ_next = 1−(1−δ_prev)(1−δ_0) and the fixed thresholds (inter_thresh=0.5, intra_thresh=0.6, ag_thresh=0.1) are derived under the assumption of uniform independent random sparsity. While Fig. 2 shows that poorly chosen thresholds hurt, the paper never evaluates non-uniform or structured sparsity patterns that arise in real pruned models or scientific sparse matrices. This is the main load-bearing generality gap for the adaptive algorithm claim; a short additional experiment or explicit caveat with a counter-example pattern would strengthen the central result.
  2. §V-C and Fig. 4: Channel count is treated as a free parameter that is swept offline and then fixed at the best value (often 64). Dense NCCL heuristics are deliberately overridden. Because the reported speedups depend on this elevated channel count (and the associated SM occupancy trade-off), the paper should quantify how much of the gain disappears when the same channel heuristic used by stock NCCL is forced, or state clearly that SpCCL requires a different channel policy.
minor comments (5)
  1. Fig. 3 and §IV-D: COO vs Pici comparison is useful, but the text claims COO is “significantly slower for more moderate sparsities” while the all-gather curves remain competitive down to ~90%. A one-sentence clarification of when decompression cost dominates would help.
  2. §VI-A: The SparCML baseline is host-only and runs on older networks; the order-of-magnitude gap is expected but should be caveated more explicitly so readers do not over-interpret it as a pure algorithmic win.
  3. Eqs. (4)–(6): The extended Hockney model is only used for intuition; the paper never validates the predicted T_cd against measured (de)compression times. A short table or sentence would make the model more than decorative.
  4. Throughout: “NCCLX” is introduced without a citation or version pin until later; a footnote or reference to the arXiv preprint would improve reproducibility.
  5. Fig. 10–11: Error-feedback (EF) overhead is discussed but the absolute iteration times with EF are not plotted against the dense baseline; adding that curve would make the end-to-end claim clearer.

Circularity Check

0 steps flagged

No significant circularity: empirical wall-clock speedups against external dense baselines, with offline-tuned thresholds that do not define the reported results by construction.

full rationale

The paper's central claims are measured performance numbers (up to 5.25 imes/2.5 imes/2.66 imes over NCCL at 99% sparsity for AG/RS/AR, plus 13–26% end-to-end on pruned DDP) obtained from multi-job microbenchmarks and a Megatron-LM case study on Perlmutter. The extended Hockney model (Eqs. 1–6) is used only for intuition about when compression can help; no quantitative predictions are derived from it and then compared to the same data. Thresholds (inter_thresh=0.5, intra_thresh=0.6, ag_thresh=0.1) and the densification extrapolation (Eq. 7) are chosen by offline sweeps and a simple inclusion-exclusion assumption; they are hyperparameters of the adaptive algorithm, not fitted constants that are later re-labeled as predictions. Pici's space overhead (3.15%) follows directly from its bitvector+tile-index layout and is independent of the measured speedups. Related-work citations (SparCML, OmniReduce, etc.) are external and non-load-bearing for the correctness of the reported timings. No equation equates a claimed first-principles result to an input by construction, and there is no self-citation uniqueness chain. The derivation is therefore self-contained empirical engineering.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The central performance claims rest on a small set of empirically chosen thresholds, standard communication models, and the new Pici representation itself. No deep mathematical axioms are required; the work is systems engineering.

free parameters (4)
  • inter_thresh = 0.5
    Sparsity threshold below which inter-node sends switch to dense; chosen by offline sweep (0.5 for Pici).
  • intra_thresh = 0.6
    Sparsity threshold for intra-node (NVLink) sends; chosen by offline sweep (0.6 for Pici).
  • ag_thresh = 0.1
    Threshold deciding whether the all-gather phase of all-reduce stays sparse; set to 0.1 after empirical observation.
  • channel_count = 32-64 (best of sweep)
    Number of NCCL channels (CUDA blocks) raised above library defaults (often 32–64) after sweeps to saturate links under compression overhead.
axioms (4)
  • domain assumption Ring algorithm is bandwidth-optimal and empirically superior to Tree for the large-message regime on Perlmutter.
    Stated in §IV-A; used to justify implementing only Ring variants.
  • domain assumption Densification of independent uniform random sparsity follows the inclusion-exclusion recurrence δ_next = 1-(1-δ_prev)(1-δ_0).
    Eq. 7 in §IV-C; used to extrapolate sparsity across dense intermediate steps.
  • domain assumption Hockney model (α + nβ) plus additive (de)compression terms adequately predicts when sparse communication wins.
    §II-A; guides focus on large messages and threshold design.
  • ad hoc to paper Pici’s 3.15% overhead and warp-cooperative kernels make (de)compression cheaper than the bandwidth saved at moderate-to-high sparsity.
    Core design claim of §III; validated only by the paper’s own kernels and measurements.
invented entities (2)
  • Pici sparse format no independent evidence
    purpose: Low-constant-overhead bitvector representation enabling fast parallel (de)compression at medium sparsities.
    Introduced in §III; no prior format matches the exact tile/index/kernel design. Independent evidence is limited to the paper’s own microbenchmarks.
  • SpCCL library no independent evidence
    purpose: NCCLX fork that implements the adaptive sparse collectives.
    Implementation vehicle; not a scientific entity beyond the algorithms it contains.

pith-pipeline@v1.1.0-grok45 · 23877 in / 2829 out tokens · 26326 ms · 2026-07-11T15:20:29.148556+00:00 · methodology

0 comments
read the original abstract

High-performance collective communication primitives are necessary for a variety of high performance computing (HPC) and machine learning (ML) workloads. State-of-the-art collective communication libraries such as NCCL optimize exclusively for dense data. However, when sending sparse data, we can reduce communication volume by not sending zeros. Unfortunately, explicitly handling sparsity introduces challenges such as format conversion overheads and densification during collectives that involve reductions. In this paper, we introduce sparsity-exploiting algorithms for three collectives that address these challenges: all-gather, reduce-scatter, and all-reduce. Our collective implementations are backed by a new bitvector-based format, Pici, designed for low overhead and fast (de)compression at moderate sparsities. Further, our algorithms adapt to the level of sparsity in data, modifying its representation during the course of the collective. At 99% input sparsity, our collectives achieve up to 5.25x, 2.5x, and 2.66x speedups over NCCL for all-gather, reduce-scatter, and all-reduce, respectively.

Figures

Figures reproduced from arXiv: 2607.04676 by Abhinav Bhatele, Emir Gencer, Hoffmann Muki, Lannie Dalton Hough.

Figure 1
Figure 1. Figure 1: Overview of how Pici bitvector and index components correspond to [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Threshold sweep for reduce-scatter with Pici, 512 MiB collective [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: SpCCL all-gather and all-reduce results on 16 GPUs, using Pici and [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: All-gather channel sweep on 16 GPUs (512 MiB output, 99% sparsity). [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Results for SpCCL all-gather (Pici) versus NCCL (Dense) and NCCLX (Dense) baselines on several GPU counts for 512 and 2048 MiB problem [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: SpCCL sp-all-gather (Pici) speedups over NCCL all-gather (Dense). Speedups are greatest for large problem sizes and at high sparsities, but substantial [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: SpCCL all-reduce (Pici) speedups over NCCL (Dense). At 99% sparsity, substantial speedups can be achieved even at 64 GPUs, while for more [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: SpCCL reduce-scatter (Pici) speedups over NCCL (Dense). At 99% sparsity, substantial speedups can be achieved even at 64 GPUs, while for more [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Results for SpCCL all-reduce (Pici) versus dense baselines and a sparse SparCML baseline on several GPU counts and problem sizes. Speedups are [PITH_FULL_IMAGE:figures/full_fig_p009_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Mean training iteration time for 40 GB and 80 GB GPUs over 8, 16, [PITH_FULL_IMAGE:figures/full_fig_p010_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Breakdown showing all-reduce and pruning time on 32 GPUs. All [PITH_FULL_IMAGE:figures/full_fig_p010_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 11 canonical work pages · 1 internal anchor

  1. [1]

    Cpu- and gpu-initiated communication strategies for conjugate gradient methods on large gpu clusters,

    J. D. Trotter, S. Ekmekc ¸ibas ¸ı, D. Sa ˘gbili, J. Langguth, X. Cai, and D. Unat, “Cpu- and gpu-initiated communication strategies for conjugate gradient methods on large gpu clusters,” inProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’25. New York, NY , USA: Association for Computing...

    doi:10.1145/3712285.3759774 2025

  2. [2]

    Amesos2 and belos: Direct and iterative solvers for large sparse linear systems,

    E. Bavier, M. Hoemmen, S. Rajamanickam, and H. Thornquist, “Amesos2 and belos: Direct and iterative solvers for large sparse linear systems,”Scientific Programming, vol. 20, no. 3, p. 243875, 2012. [Online]. Available: https://doi.org/10.3233/SPR-2012-0352

    work page doi:10.3233/spr-2012-0352 2012

  3. [3]

    Newly released capabilities in the distributed-memory superlu sparse direct solver,

    X. S. Li, P. Lin, Y . Liu, and P. Sao, “Newly released capabilities in the distributed-memory superlu sparse direct solver,”ACM Trans. Math. Softw., vol. 49, no. 1, Mar. 2023. [Online]. Available: https://doi.org/10.1145/3577197

    doi:10.1145/3577197 2023

  4. [4]

    Qiu,Scalable and Efficient Material Point Methods on Modern Computational Platforms

    Y . Qiu,Scalable and Efficient Material Point Methods on Modern Computational Platforms. University of California, Los Angeles, 2024

    2024

  5. [5]

    Exploiting sparsity in pruned neural networks to optimize large model training,

    S. Singh and A. Bhatele, “Exploiting sparsity in pruned neural networks to optimize large model training,” in 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS). Los Alamitos, CA, USA: IEEE Computer Society, may 2023, pp. 245–255. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/IPDPS54959.2023.00033

    doi:10.1109/ipdps54959.2023.00033 2023

  6. [6]

    Deep gradient compression: Reducing the communication bandwidth for distributed training,

    Y . Lin, S. Han, H. Mao, Y . Wang, and W. J. Dally, “Deep gradient compression: Reducing the communication bandwidth for distributed training,”arXiv preprint arXiv:1712.01887, 2017. [Online]. Available: https://doi.org/10.48550/arXiv.1712.01887

    Pith/arXiv arXiv doi:10.48550/arxiv.1712.01887 2017

  7. [7]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks,

    J. Frankle and M. Carbin, “The lottery ticket hypothesis: Finding sparse, trainable neural networks,”arXiv preprint arXiv:1803.03635,

    Pith/arXiv arXiv

  8. [8]

    Available: https://doi.org/10.48550/arXiv.1803.03635

    [Online]. Available: https://doi.org/10.48550/arXiv.1803.03635

    Pith/arXiv arXiv doi:10.48550/arxiv.1803.03635

  9. [9]

    Plexus: Taming billion-edge graphs with 3D parallel full-graph GNN training,

    A. K. Ranjan, S. Singh, C. Wei, and A. Bhatele, “Plexus: Taming billion-edge graphs with 3D parallel full-graph GNN training,” inProceedings of the ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’25. ACM, Nov. 2025. [Online]. Available: https://doi.acm.org/10.1145/3712285.3759890

    doi:10.1145/3712285.3759890 2025

  10. [10]

    Coruscant: Co-designing gpu kernel and sparse tensor core to advocate unstructured sparsity in efficient llm inference,

    D. Joo, H. Hosseini, R. Hadidi, and B. Asgari, “Coruscant: Co-designing gpu kernel and sparse tensor core to advocate unstructured sparsity in efficient llm inference,” inProceedings of the 58th IEEE/ACM International Symposium on Microarchitecture®, 2025, pp. 232–245. [Online]. Available: https://doi.org/10.1145/3725843.3756065

    doi:10.1145/3725843.3756065 2025

  11. [11]

    Sparcml: High-performance sparse communication for machine learning,

    C. Renggli, S. Ashkboos, M. Aghagolzadeh, D. Alistarh, and T. Hoefler, “Sparcml: High-performance sparse communication for machine learning,” inProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2019, pp. 1–15. [Online]. Available: https://doi.org/10.1145/3295500.3356222

    doi:10.1145/3295500.3356222 2019

  12. [12]

    Efficient sparse collective communication and its application to accelerate distributed deep learning,

    J. Fei, C.-Y . Ho, A. N. Sahu, M. Canini, and A. Sapio, “Efficient sparse collective communication and its application to accelerate distributed deep learning,” inProceedings of the 2021 ACM SIGCOMM 2021 Conference, 2021, pp. 676–691. [Online]. Available: https://doi.org/10.1145/3452296.3472904

    doi:10.1145/3452296.3472904 2021

  13. [13]

    Efficient sparse matrix-vector multiplication on cuda,

    N. Bell and M. Garland, “Efficient sparse matrix-vector multiplication on cuda,” Nvidia Technical Report NVR-2008-004, Nvidia Corporation, Tech. Rep., 2008

    2008

  14. [14]

    Yale sparse matrix package. ii. the nonsymmetric codes,

    S. Eisenstat, M. Gursky, M. Schultz, and A. Sherman, “Yale sparse matrix package. ii. the nonsymmetric codes,” Tech. Rep., 1977

    1977

  15. [15]

    Collective communication for 100k+ gpus,

    M. Si, P. Balaji, Y . Chen, C.-H. Chu, A. Gangidi, S. Hasan, S. Iyengar, D. Johnson, B. Liu, J. Renet al., “Collective communication for 100k+ gpus,”arXiv preprint arXiv:2510.20171, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2510.20171

    doi:10.48550/arxiv.2510.20171 2025

  16. [16]

    Perlmutter system architecture,

    NERSC, “Perlmutter system architecture,” https://docs.nersc.gov/systems/perlmutter/architecture/

  17. [17]

    [Online]

    “Nccl,” 2020. [Online]. Avail- able: https://docs.nvidia.com/deeplearning/nccl/user- guide/docs/overview.html

    2020

  18. [18]

    Optimization of collective reduction operations,

    R. Rabenseifner, “Optimization of collective reduction operations,” in International Conference on Computational Science. Springer, 2004, pp. 1–9. [Online]. Available: https://doi.org/10.1007/978-3-540-24685- 5 1

    work page doi:10.1007/978-3-540-24685- 2004

  19. [19]

    The communication challenge for mpp: Intel paragon and meiko cs-2,

    R. W. Hockney, “The communication challenge for mpp: Intel paragon and meiko cs-2,”Parallel Comput., vol. 20, no. 3, p. 389–398, Mar. 1994. [Online]. Available: https://doi.org/10.1016/S0167-8191(06)80021-9

    doi:10.1016/s0167-8191(06)80021-9 1994

  20. [20]

    Pytorch distributed: Experiences on accelerating data parallel training,

    S. Li, Y . Zhao, R. Varma, O. Salpekar, P. Noordhuis, T. Li, A. Paszke, J. Smith, B. Vaughan, P. Damania, and S. Chintala, “Pytorch distributed: Experiences on accelerating data parallel training,”Proc. VLDB Endow., vol. 13, no. 12, p. 3005–3018, Aug. 2020. [Online]. Available: https://doi.org/10.14778/3415478.3415530

    doi:10.14778/3415478.3415530 2020

  21. [21]

    Evaluation criteria for sparse matrix storage formats,

    D. Langr and P. Tvrdik, “Evaluation criteria for sparse matrix storage formats,”IEEE Transactions on parallel and distributed systems, vol. 27, no. 2, pp. 428–440, 2015. [Online]. Available: https://doi.org/10.1109/TPDS.2015.2401575

    work page doi:10.1109/tpds.2015.2401575 2015

  22. [22]

    A systematic survey of general sparse matrix-matrix multiplication,

    J. Gao, W. Ji, F. Chang, S. Han, B. Wei, Z. Liu, and Y . Wang, “A systematic survey of general sparse matrix-matrix multiplication,” ACM Computing Surveys, vol. 55, no. 12, pp. 1–36, 2023. [Online]. Available: https://doi.org/10.1145/3571157

    doi:10.1145/3571157 2023

  23. [23]

    Sparskit: A basic tool kit for sparse matrix computations,

    Y . Saad, “Sparskit: A basic tool kit for sparse matrix computations,” Tech. Rep., 1990

    1990

  24. [24]

    Towards a universal fpga matrix-vector multiplication architecture,

    S. Kestur, J. D. Davis, and E. S. Chung, “Towards a universal fpga matrix-vector multiplication architecture,” in2012 IEEE 20th International Symposium on Field-Programmable Custom Computing Machines. IEEE, 2012, pp. 9–16. [Online]. Available: https://doi.org/10.1109/FCCM.2012.12

    work page doi:10.1109/fccm.2012.12 2012

  25. [25]

    Space-efficient, high- performance rank and select structures on uncompressed bit sequences,

    D. Zhou, D. G. Andersen, and M. Kaminsky, “Space-efficient, high- performance rank and select structures on uncompressed bit sequences,” inInternational Symposium on Experimental Algorithms. Springer, 2013, pp. 151–163. [Online]. Available: https://doi.org/10.1007/978-3- 642-38527-8 15

    doi:10.1007/978-3- 2013

  26. [26]

    Engineering compact data structures for rank and select queries on bit vectors,

    F. Kurpicz, “Engineering compact data structures for rank and select queries on bit vectors,” inInternational Symposium on String Processing and Information Retrieval. Springer, 2022, pp. 257–272. [Online]. Available: https://doi.org/10.1007/978-3-031-20643-6 19

    work page doi:10.1007/978-3-031-20643-6 2022

  27. [27]

    Data parallel algorithms,

    W. D. Hillis and G. L. S. Jr., “Data parallel algorithms,”Communications of the ACM, vol. 29, no. 12, pp. 1170–1183, 1986. [Online]. Available: https://doi.org/10.1145/7902.7903

    doi:10.1145/7902.7903 1986

  28. [28]

    Pat: a new algorithm for all-gather and reduce-scatter operations at scale,

    S. Jeaugey, “Pat: a new algorithm for all-gather and reduce-scatter operations at scale,” 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2506.20252

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.20252 2025

  29. [29]

    Communication algorithm-architecture co-design for distributed deep learning,

    J. Huang, P. Majumder, S. Kim, A. Muzahid, K. H. Yum, and E. J. Kim, “Communication algorithm-architecture co-design for distributed deep learning,” in2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA). IEEE, 2021, pp. 181–194. [Online]. Available: https://doi.org/10.1109/ISCA52012.2021.00023

    doi:10.1109/isca52012.2021.00023 2021

  30. [30]

    Optimization of collective communication operations in mpich,

    R. Thakur, R. Rabenseifner, and W. Gropp, “Optimization of collective communication operations in mpich,”The International Journal of High Performance Computing Applications, vol. 19, no. 1, pp. 49–66,

  31. [31]

    Available: https://doi.org/10.1177/1094342005051521

    [Online]. Available: https://doi.org/10.1177/1094342005051521

    doi:10.1177/1094342005051521

  32. [32]

    The big send-off: Scalable and performant collectives for deep learning,

    S. Singh, K. Pradeep, M. Singh, C. Wei, and A. Bhatele, “The big send-off: Scalable and performant collectives for deep learning,” 2026. [Online]. Available: https://doi.org/10.48550/arXiv.2504.18658

    doi:10.48550/arxiv.2504.18658 2026

  33. [33]

    Gpudirect,

    NVIDIA, “Gpudirect,” https://docs.nvidia.com/cuda/gpudirect- rdma/index.html

  34. [34]

    Nccl tests,

    ——, “Nccl tests,” https://github.com/NVIDIA/nccl-tests, 2017

    2017

  35. [35]

    The mvapich project: Transforming research into high-performance mpi library for hpc community,

    D. K. Panda, H. Subramoni, C.-H. Chu, and M. Bayatpour, “The mvapich project: Transforming research into high-performance mpi library for hpc community,”Journal of Computational Science, vol. 52, p. 101208, 2021. [Online]. Available: https://doi.org/10.1016/j.jocs.2020.101208

    work page doi:10.1016/j.jocs.2020.101208 2021

  36. [36]

    OSU Micro-Benchmarks,

    Network-Based Computing Laboratory, The Ohio State University, “OSU Micro-Benchmarks,” accessed 2026-04-04. [Online]. Available: https://mvapich.cse.ohio-state.edu/benchmarks/

    2026

  37. [37]

    1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs,

    F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu, “1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs,” inInterspeech 2014, 2014, pp. 1058–1062. [Online]. Available: https://doi.org/10.21437/Interspeech.2014-274

    doi:10.21437/interspeech.2014-274 2014

  38. [38]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” Tech. Rep., 2019

    2019

  39. [39]

    Starcoder 2 and the stack v2: The next generation,

    A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y . Wei, T. Liu, M. Tian, D. Kocetkov, A. Zucker, Y . Belkada, Z. Wang, Q. Liu, D. Abulkhanov, I. Paul, Z. Li, W.-D. Li, M. Risdal, J. Li, J. Zhu, T. Y . Zhuo, E. Zheltonozhskii, N. O. O. Dade, W. Yu, L. Krauß, N. Jain, Y . Su, X. He, M. Dey, E. Abati, Y . C...

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.19173 2024

  40. [40]

    Aligning books and movies: Towards story-like visual explanations by watching movies and reading books,

    Y . Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler, “Aligning books and movies: Towards story-like visual explanations by watching movies and reading books,” inarXiv preprint arXiv:1506.06724, 2015. [Online]. Available: https://doi.org/10.48550/arXiv.1506.06724

    Pith/arXiv arXiv doi:10.48550/arxiv.1506.06724 2015

  41. [41]

    Megatron-lm: Training multi-billion parameter language models using model parallelism,

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” Tech. Rep., 2020. [Online]. Available: https://doi.org/10.48550/arXiv.1909.08053

    Pith/arXiv arXiv doi:10.48550/arxiv.1909.08053 2020

  42. [42]

    Gasnet-ex rma communication performance on recent supercomputing systems,

    P. H. Hargrove and D. Bonachea, “Gasnet-ex rma communication performance on recent supercomputing systems,” 2022. [Online]. Available: https://doi.org/10.25344/S40C7D

    work page doi:10.25344/s40c7d 2022

  43. [43]

    omnireduce-experiments,

    SANDS Lab, “omnireduce-experiments,” gitHub repository, commit a2bdc8082c805a1cb86499ce1eae7a9b48fbf8c4, accessed 2026-04-

    2026

  44. [44]

    Available: https://github.com/sands-lab/omnireduce- experiments/commit/a2bdc8082c805a1cb86499ce1eae7a9b48fbf8c4

    [Online]. Available: https://github.com/sands-lab/omnireduce- experiments/commit/a2bdc8082c805a1cb86499ce1eae7a9b48fbf8c4

  45. [45]

    Sparse collectives: Exploiting data sparsity to improve communication efficiency,

    D. Wijerathne, H. Javaid, G. Zhong, D. Wu, X. Y . Kom, and M. Baldi, “Sparse collectives: Exploiting data sparsity to improve communication efficiency,” inProceedings of the 2nd Workshop on Networks for AI Computing, 2025, pp. 64–66. [Online]. Available: https://doi.org/10.1145/3748273.3749206

    doi:10.1145/3748273.3749206 2025

  46. [46]

    Omniccl: Zero-cost sparse allreduce with direct cache access and smartnics,

    T. Gu, J. Fei, and M. Canini, “Omniccl: Zero-cost sparse allreduce with direct cache access and smartnics,” inProceedings of the 2024 SIGCOMM Workshop on Networks for AI Computing, 2024, pp. 75–83. [Online]. Available: https://doi.org/10.1145/3672198.3673804

    work page doi:10.1145/3672198.3673804 2024

  47. [47]

    D-dosa: Dpu-based dataflow offloading and sparse allreduce framework for distributed training,

    Z. Yu, W. Li, S. Guo, Q. Li, F. Qi, and J. Xiu, “D-dosa: Dpu-based dataflow offloading and sparse allreduce framework for distributed training,”IEEE Transactions on Cloud Computing, 2025. [Online]. Available: https://doi.org/10.1109/TCC.2025.3634564

    work page doi:10.1109/tcc.2025.3634564 2025

  48. [48]

    Near-optimal sparse allreduce for distributed deep learning,

    S. Li and T. Hoefler, “Near-optimal sparse allreduce for distributed deep learning,” inProceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, 2022, pp. 135–149. [Online]. Available: https://doi.org/10.1145/3503221.3508399

    doi:10.1145/3503221.3508399 2022

  49. [49]

    A distributed synchronous sgd algorithm with global top-k sparsification for low bandwidth networks,

    S. Shi, Q. Wang, K. Zhao, Z. Tang, Y . Wang, X. Huang, and X. Chu, “A distributed synchronous sgd algorithm with global top-k sparsification for low bandwidth networks,” in2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS). IEEE, 2019, pp. 2238–2247. [Online]. Available: https://doi.org/10.1109/ICDCS.2019.00220

    work page doi:10.1109/icdcs.2019.00220 2019

  50. [50]

    Hecate: Unlocking Efficient Sparse Model Training via Fully Sharded Sparse Data Parallelism

    Y . Qing, G. Zhu, F. Li, L. Lei, Z. Sun, X. Guan, S. Zhao, X. Chen, D. Huang, S. Wanget al., “Hecate: Unlocking efficient sparse model training via fully sharded sparse data parallelism,”arXiv preprint arXiv:2502.02581, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2502.02581

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2502.02581 2025

  51. [51]

    Sparse collective operations for mpi,

    T. Hoefler and J. L. Traff, “Sparse collective operations for mpi,” in2009 IEEE International Symposium on Parallel & Distributed Processing. IEEE, 2009, pp. 1–8. [Online]. Available: https://doi.org/10.1109/IPDPS.2009.5160935

    doi:10.1109/ipdps.2009.5160935 2009

  52. [52]

    Understanding top-k sparsi- fication in distributed deep learning,

    S. Shi, X. Chu, K. C. Cheung, and S. See, “Understanding top-k sparsi- fication in distributed deep learning,”arXiv preprint arXiv:1911.08772,

    Pith/arXiv arXiv 1911

  53. [53]

    Available: https://doi.org/10.48550/arXiv.1911.08772

    [Online]. Available: https://doi.org/10.48550/arXiv.1911.08772

    Pith/arXiv arXiv doi:10.48550/arxiv.1911.08772 1911