Pith. sign in

REVIEW 5 major objections 10 minor 14 references

Prime Collective Communications Library -- Technical Report

T0 review · 5 major / 10 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PCCL is a fault-tolerant collective communication library that uses a master-coordinated micro-consensus protocol to support dynamic peer membership and bit-exact state synchronization for distributed ML over the internet.

desk verdict Master-coordinated WAN collective with dynamic membership is a real contribution, but the master is a single point of failure and the missing artifact keeps this at conditional. read the letter →

arxiv 2505.14065 v1 pith:ADDJPYFA submitted 2025-05-20 cs.DC

Michael Keiblinger , Mario Sieg , Jack Min Ong , Sami Jaghouar , Johannes Hagemann This is my paper
classification cs.DC
keywords pcclcollectiveoperationsacrossenableslibrarycommunicationdistributed
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PCCL is a software library that helps many computers, possibly in different countries, train one machine learning model together over the public internet. In ordinary distributed training, all computers must stay connected and start together, which is hard when machines are spot instances or cross cloud providers. PCCL instead uses a central master process that keeps a list of which computers are active and coordinates a small voting step before each major action. New computers can ask to join, existing computers vote to accept them, and if one computer fails during an operation the whole group can cancel that operation and retry without the failed machine.

The library performs a standard ring-based all-reduce, but splits the data into chunks that travel around a ring of peers. It checks a lightweight master connection for abort signals while sending and receiving, so failures can be unwound quickly. To keep every computer on exactly the same model and optimizer state, PCCL hashes each computer's state and transfers data from any peer that is out of date. The authors also show that a GPU math function used in training, __nv_expf, produces identical bits across five NVIDIA architectures, which supports their claim of deterministic state advancement.

In benchmarks, using many simultaneous connections raises throughput much more than one connection alone. The largest reported result is about 45 Gbit/s across western Europe with 128 concurrent all-reduces, and about 25 Gbit/s across North America and Europe. The report also sketches how to implement DiLoCo and an asynchronous delayed variant on top of PCCL.

Extended reading notes

Core claim

The central claim is that PCCL is a fault-tolerant collective communication library for distributed ML over the public internet: it continues operating when peers fail or join, and it maintains exact bit-parity of shared state across peers under graceful or abrupt peer churn (Abstract; Sections 2.4, 2.5). If true, distributed training can proceed over WANs with dynamic membership.

Load-bearing premise

The master process is assumed to be reliable: PCCL's fault tolerance covers peers, but the master coordinates all micro-consensus, membership, and topology decisions, and the paper does not describe master failure handling or failover (Sections 2.2, 2.5.3). If the master fails, the collective cannot progress, so the system's fault tolerance is bounded by an unaddressed single point of failure.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 10 minor

Summary. This technical report describes PCCL, a master-client collective-communication library for distributed ML training over the public internet. The library allows peers to join or fail during training: a master process tracks membership and ring topology, coordinates 'micro-consensus' phase transitions, and propagates abort signals; shared state is validated by a deterministic GPU/CPU hash and repaired by retransmission from an up-to-date peer; and ring all-reduce is implemented as an interruptible, zero-copy pipelined reduce-scatter/reduce-gather with buffer restoration on abort. The paper presents algorithms for DDP and for DiLoCo and async DiLoCo with peer churn, describes stress tests on Linux/macOS/Windows, empirically verifies bit-identical __nv_expf outputs across five NVIDIA architectures over all 2^32 float bit patterns, and benchmarks WAN all-reduce on GCP (up to 45.74 Gbit/s within Europe and 24.54 Gbit/s across North America+Europe with 128 concurrent connections), with a Gloo comparison. The central claims are: (i) the system continues operating under peer churn; (ii) shared state remains bit-identical in all churn-induced cases; and (iii) multi-connection dispatch efficiently utilizes long-fat-pipe WAN links.

Significance. If the claims hold, PCCL addresses a genuine gap: no widely used collective library (NCCL, Gloo, MPI) natively supports dynamic membership with bit-exact state over WANs, which is a prerequisite for practical spot-instance and multi-cloud training. The paper's strengths are concrete and partly machine-checkable: the exhaustive PTX determinism test in Appendix A.2 with code included; the implementation-agnostic throughput methodology; the internally consistent single-stream benchmark numbers; the stress-test design over three operating systems; and the readable pseudocode for DDP/DiLoCo/async DiLoCo. The unqualified fault-tolerance claim and the 'all cases' bit-parity claim would, however, need to be scoped more carefully before the results can be regarded as established; the master-process single point of failure and the absence of PCCL artifacts are the main limits on verification.

major comments (5)
  1. [§2.2, §2.5.3, §8] The fault-tolerance claims are stated without qualification in the Abstract and Introduction, but §2.2.2 places the master process in the critical path of every collective (it coordinates micro-consensus, membership, and abort propagation; see Algorithm 6, lines 18–21), and §2.5.3 enumerates error paths only for peers. Master failure, replication, and failover are never discussed; §8's 'one non-spot instance, the run is guaranteed stable' is an operational assumption about where the master runs rather than a fault-tolerance mechanism for it. Because a master crash stalls membership changes, shared-state validation, and in-flight collectives simultaneously, the system's availability is bounded by an unaddressed single point of failure. Please either describe a master-failover or recovery mechanism, or explicitly scope all fault-tolerance claims to peer churn under a reliable coordinator.
  2. [Abstract; §2.4; §5.3.1; §5.4; §2.6.1] The Abstract's claim of exact bit-parity 'in all cases induced by graceful or abrupt peer churn' is stronger than the evidence presented. The ring-reduce determinism argument in §5.3.1 is structural rather than formal; the PTX determinism test in §5.4 is exhaustive but covers only __nv_expf, and the extension to other intrinsics is explicitly an extrapolation ('likely many other'); and §2.6.1's stress tests, which are finite, define success only as 'the shared state is advanced correctly' without specifying the oracle that detects incorrect advancement. Moreover, §3.3.1 itself demonstrates that the guarantee is conditional on synchronization strategy: with the default enforcePopular rule, if newcomers outnumber pre-existing peers an 'undesired state' can be selected, so the library requires application-level use of sendOnly/receiveOnly at the right point. Please state the precise conditions under which bit-parity holds, or replace 'all cases' with the tested class, and specify the stress-test correctness oracle and pass criteria.
  3. [§3.3.1, Algorithm 4] Algorithm 4 uses the flags newComerHasJoined (line 23) and selfIsNew (line 27) without defining where or how they are set, and the claim in §3.3.1 that C.are_peers_pending() is 'guaranteed to return the same value for all peers' is asserted without an argument. More importantly, line 17 awaits the previous all-reduce and line 20 then tests isFinished(∆(t−1)), which should always be true, making the else-branch at lines 26–31 either dead code or the locus of the newcomer's receiveOnly path; as written, the reader cannot tell which. Since this algorithm is the paper's only concrete specification of how async DiLoCo composes with membership changes, the branch and flag semantics need to be made explicit and internally consistent.
  4. [Tables 3, 4, 10, 11] Table 3 is titled 'All-Reduce Performance (Europe West, 6 nodes)' but is placed in §6.3.2 (Experiment 1.1, 18 nodes across North America and Europe) and sits next to Table 4, which correctly lists 18 peers; conversely, Table 11 reports 'Number of Peers 12' for Experiment 3.1, which §6.5 defines with 6 nodes. Because the headline WAN figures (24.54 Gbit/s North America+Europe and 45.74 Gbit/s Europe West) come from these two concurrent-connection blocks, the label errors prevent a reader from verifying which experiment produced which number and must be corrected. In the same tables the TX+RX/peer accounting is not defined; please state how ring traffic is counted (for example, whether per-peer contribution size or total bytes moved is reported).
  5. [§2.6.1; availability statement] The paper states that PCCL is available under the MIT license and bases its central fault-tolerance claims on 'extensive long-running stress tests' (§2.6.1), but it provides no repository URL for PCCL or its stress-test harness (only libtsp is linked), and it does not report how many stress runs were executed, at what node counts, or how 'correctly' was determined for the passing criterion. The included PTX test code (Appendix A.2) sets a good precedent for reproducibility; please provide the PCCL source and the stress-test and benchmark scripts, or at minimum the repository link and a precise description of the stress-test configuration and oracle.
minor comments (10)
  1. [§6.5] The first sentence, 'analogous to Experiment 3 of the previous section', appears to be a copy-paste artifact, since §6.4 presents Experiment 2; the sentence should refer to the single-reduce protocol of §6.1 or to Experiments 1–2.
  2. [§5.3 title] The word 'Commmunications' in the section title is misspelled.
  3. [§3.3] The sentence 'major operations (topology updates, shared-state syncs, or collectives) do not overlap with collective operations' is self-contradictory given the 128-way concurrent all-reduces demonstrated in §6.3.2; the exclusivity rule should be stated over operation categories (membership/shared-state vs. collectives) rather than over individual collectives.
  4. [§2.3.2] The rule 'only one major operation ... can be active at any given moment' is ambiguous in light of the per-tag CollectiveCommunicationState; please clarify whether the restriction is one operation per category, since the benchmarks explicitly run many simultaneous all-reduces.
  5. [Tables 4, 8, 11] The row 'Number of Reduce Operations' uses set notation that conflates the number of operations with the concurrency level; please state whether all listed reduces are launched in one wave or serialized.
  6. [Algorithms 1–4] Shared-state arguments such as sharedState, enforcePopular, sendOnly, and receiveOnly are used without a data dictionary; a short table defining these symbols and their semantics would substantially improve readability.
  7. [§5.3.1] The statement that 'the argument can be made that as world size grows, we get a better approximation of the true sum' is a heuristic and should be labeled as such, since it is not required for the paper's determinism claim.
  8. [§3.3.1] 'loose hash popularity' should read 'lose hash popularity.'
  9. [§2.4] The claim that simplehash is 'empirically validated on GPU generations ranging from the GTX 980 Ti to the B200' is not accompanied by the determinism validation data; Appendix A.1 reports performance only, so please add the determinism results or a pointer to the test.
  10. [§6.6, §7] The Gloo comparison is single-stream only, and the comparisons with Hivemind and Horovod+Elastic are purely qualitative; a quantitative comparison (for example, recovery latency under churn at matched world sizes) would substantiate the comparative claims in Section 7.
Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted to data. The report's central claims rest on assumptions about deterministic training, a reliable master, hash collision-freedom, and WAN TCP behavior, rather than on new postulated entities.

assumptions (4)
  • ad hoc to paper The master process is reliable and never fails; it coordinates all membership, topology, and consensus decisions.
    Section 2.2 and 2.5.3: fault tolerance is described for clients/peers, but no master failover or failure handling is specified. The central fault-tolerance claim depends on this.
  • domain assumption Given identical collective results, all peers advance their shared state bit-identically because the training updates are deterministic.
    Section 1 and 3.1: PCCL relies on deterministic optimizers and deterministic GPU intrinsics (e.g. __nv_expf test in 5.4) to guarantee shared-state parity.
  • domain assumption The non-cryptographic simplehash is collision-free enough to detect any shared-state mismatch between peers.
    Section 2.4: the master compares hash values to identify out-of-sync peers; hash collisions could in principle hide mismatches, and the paper provides no collision analysis.
  • domain assumption WAN TCP throughput scales with the number of parallel connections because of per-flow fair queuing and receiver-window auto-scaling.
    Section 6.2: the multiple-connection performance argument relies on this empirical network property, and benchmarks are presented on specific cloud providers (Google Cloud Tier_1, Nebius).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Prime Collective Communications Library -- Technical Report." pith.science (2026). https://pith.science/paper/ADDJPYFA

@misc{pith2026250514065,
  author       = {Pith},
  title        = {Pith review of: Prime Collective Communications Library -- Technical Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ADDJPYFA}},
  note         = {Machine review of arXiv:2505.14065}
}
read the original abstract

This report presents the Prime Collective Communications Library (PCCL), a novel fault-tolerant collective communication library designed for distributed ML workloads over the public internet. PCCL introduces a new programming model that enables dynamic peer joining and failure recovery. The library implements efficient collective operations like all-reduce while providing robust fault tolerance mechanisms that allow the system to continue operating even when peers fail or join during ongoing operations. We demonstrate that PCCL's design enables practical solutions to dynamic membership challenges in workloads with repeated operations and deterministic state advancement. Our implementation passes extensive stress tests across all major operating systems, showing reliable operation even under rapid peer churn and concurrent collective operations. By dispatching to multiple connections, we can efficiently utilize cross-continental long-fat-pipe TCP WAN links, in our experiments achieving up to 45 Gbit/s of bandwidth utilization across Europe and 25 Gbit/s across North America and Europe. PCCL's architecture enables easy implementation of distributed low-communication optimization strategies like DiLoCo, which significantly reduce communication frequency. Combined with quantization, this leads to a significant reduction in the bandwidth required for distributed training workloads. PCCL also allows for concurrent collective operations, which enables optimization strategies like async DiLoCo, which can completely hide communication overhead by implementing one-step delayed parameter updates. PCCL can facilitate exact bit-parity of the shared state across peers in all cases induced by graceful or abrupt peer churn. While PCCL exposes a C99 API, Python bindings are available which are compatible with PyTorch alongside FSDP. PCCL is available under the open source MIT license.

Figures

Figures reproduced from arXiv: 2505.14065 by the authors.

Figure 1
Figure 1. Example flow chart for PCCL states and transitions. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. All Reduce Across North America and Europe [PITH_FULL_IMAGE:figures/full_fig_p020_2.png] view at source ↗
Figure 3
Figure 3. All Reduce Across North America 22 [PITH_FULL_IMAGE:figures/full_fig_p022_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: All Reduce Europe West 24 [PITH_FULL_IMAGE:figures/full_fig_p024_4.png]
Figure 5
Figure 5. Figure 5: RTX 4090: SimpleHash vs. Thrust int32 Reduction [PITH_FULL_IMAGE:figures/full_fig_p029_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 10 canonical work pages

  1. [1]

    Le, Mark Z

    Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, Marc'Aurelio Ranzato, Andrew W. Senior, Paul A. Tucker, Ke Yang, and Andrew Y. Ng. Large scale distributed deep networks. In Advances in Neural Information Processing Systems 25, pages 1223--1231, 2012. URL https://papers.nips.cc/paper/4687-large-scale-distributed-...

    work page 2012

  2. [2]

    Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc'Aurelio Ranzato, Arthur Szlam, and Jiajun Shen

    Arthur Douillard, Qixuan Feng, Andrei A. Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc'Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low-communication training of language models, 2024. URL https://arxiv.org/abs/2311.08105

    arXiv 2024

  3. [3]

    Streaming diloco with overlapping communication: Towards a distributed free lunch, 2025

    Arthur Douillard, Yanislav Donchev, Keith Rush, Satyen Kale, Zachary Charles, Zachary Garrett, Gabriel Teston, Dave Lacey, Ross McIlroy, Jiajun Shen, Alexandre Ramé, Arthur Szlam, Marc'Aurelio Ranzato, and Paul Barham. Streaming diloco with overlapping communication: Towards a distributed free lunch, 2025. URL https://arxiv.org/abs/2501.18512

    arXiv 2025

  4. [4]

    FNV-1a : Fowler--noll--vo non-cryptographic hash function, 1991

    Glenn Fowler, Landon Curt Noll, and Phong Vo. FNV-1a : Fowler--noll--vo non-cryptographic hash function, 1991. URL http://www.isthe.com/chongo/tech/comp/fnv/index.html#FNV-param. Accessed: 2025-05-05

    work page 1991

  5. [5]

    Intellect-1 technical report, 2024

    Sami Jaghouar, Jack Min Ong, Manveer Basra, Fares Obeid, Jannik Straube, Michael Keiblinger, Elie Bakouch, Lucas Atkins, Maziyar Panahi, Charles Goddard, Max Ryabinin, and Johannes Hagemann. Intellect-1 technical report, 2024. URL https://arxiv.org/abs/2412.01152

    arXiv 2024

  6. [6]

    libtsp: A c++ library for the traveling salesman problem, 2025

    Michael Keiblinger. libtsp: A c++ library for the traveling salesman problem, 2025. URL https://github.com/PrimeIntellect-ai/libtsp

    work page 2025

  7. [7]

    Ptx isa 1.4, 2011

    NVIDIA. Ptx isa 1.4, 2011. URL https://www.cs.cmu.edu/afs/cs/academic/class/15668-s11/www/cuda-doc/ptx_isa_1.4.pdf

    work page 2011

  8. [8]

    Thrust : A parallel algorithms library, 2014

    NVIDIA . Thrust : A parallel algorithms library, 2014. URL https://github.com/NVIDIA/thrust. Accessed: 2025-05-05

    work page 2014

Show all 14 references
  1. [9]

    Ptx isa 6.4, 2019

    NVIDIA. Ptx isa 6.4, 2019. URL https://docs.nvidia.com/cuda/archive/10.1/pdf/ptx_isa_6.4.pdf

    2019

  2. [10]

    Ptx isa 8.0, 2020

    NVIDIA. Ptx isa 8.0, 2020. URL https://docs.nvidia.com/cuda/archive/12.0.1/pdf/ptx_isa_8.0.pdf

    2020

  3. [11]

    NCCL : Nvidia collective communications library, 2025

    NVIDIA Corporation . NCCL : Nvidia collective communications library, 2025. URL https://developer.nvidia.com/nccl

    2025

  4. [12]

    Gloo, 2017

    PyTorch. Gloo, 2017. URL https://github.com/pytorch/gloo

    2017

  5. [13]

    H ivemind: D ecentralized D eep L earning in P y T orch, April 2020

    Max Ryabinin, Alexander Borzunov, Michael Diskin, Anton Gusev, Denis Mazur, Vsevolod Plokhotnyuk, Alexey Bukhtiyarov, Pavel Samygin, Anton Sinitsin, and Artem Chumachenko. H ivemind: D ecentralized D eep L earning in P y T orch, April 2020. URL https://github.com/learning-at-h...

    2020

  6. [14]

    Horovod: fast and easy distributed deep learning in tensorflow, 2018

    Alexander Sergeev and Mike Del Balso. Horovod: fast and easy distributed deep learning in tensorflow, 2018. URL https://arxiv.org/abs/1802.05799

    2018 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.