Pith. sign in

REVIEW 4 major objections 6 minor 25 references

Agora claims multi-billion-parameter models can be pretrained permissionlessly over the internet by churning consumer GPUs, and demonstrates an 8.6B, 500B-token run at ~63% of a centralized H100 baseline's efficiency.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-02 05:30 UTC pith:5O2NPYNF

load-bearing objection A credible existence proof for permissionless internet-scale pretraining, with real but fixable inconsistencies in the headline efficiency numbers. the 4 major comments →

arxiv 2607.13332 v1 pith:5O2NPYNF submitted 2026-07-14 cs.LG cs.DC

Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models

Gil Avraham , Violetta Shevchenko , Hadi Mohaghegh Dolatabadi , Karol Pajak , James Snewin , Harry Xi , Rodney O'Donnell , Thalaiyasingam Ajanthan
show 4 more authors
Sameera Ramasinghe Chamin Hewa Koneputugodage Shamane Siriwardhana Alexander Long
This is my paper
classification cs.LG cs.DC
keywords protocol learningdistributed pretrainingpipeline parallelismsubspace networkssparse weight averagingfault-tolerant all-reducepermissionless trainingconsumer GPU training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large-model pretraining has been confined to datacenters because its communication primitives assume high bandwidth, low latency, homogeneous accelerators, and a stable owner. Agora replaces those assumptions with a protocol: the model is split into pipeline stages, inter-stage traffic is compressed roughly 100x by confining activations to a low-rank subspace, and replica consensus is kept by asynchronously averaging a small fraction of weights every 20 steps instead of synchronizing gradients. On this machinery the paper trained Pluralis-8B, an 8.6B-parameter model on 500B tokens, over 40 days with roughly 330 heterogeneous consumer-GPU contributors that joined and left freely, sustaining about 170k tokens/s and about 63% of a centralized H100 baseline's tokens-per-TFLOP efficiency while converging to within a small margin of a centralized reference. If correct, this means the unused GPU pool outside datacenters can be pooled permissionlessly for frontier-scale pretraining, with no single participant ever holding the full weights. The load-bearing question is whether the subspace confinement that enables this remains quality-preserving as models grow.

Core claim

The paper reports the first full-scale demo of its protocol: an 8.6B-parameter decoder-only transformer with 32 layers sharded into seven pipeline stages, trained for 40 days on 500B tokens of an education-filtered web corpus. The compute pool mixed operator-run nodes with 330 external contributor nodes, predominantly consumer GPUs on internet links ranging from about 200 Mbit/s to several Gbit/s with up to 80 ms round-trip latency; 669 nodes joined and 607 departed during the run. The run sustained about 170k tokens/s, reached a steady-state 4.2 tokens per TFLOP of pooled compute (63% of a centralized H100 baseline), and converged to a final training loss of 2.41, close to a centralized ref

What carries the argument

The system rests on two communication-reduction mechanisms. First, reparameterized subspace networks: every layer's output projection and the trainable token embedding are factored through a fixed orthonormal basis, so all hidden states and gradients crossing a pipeline boundary live in a k-dimensional subspace (rank roughly 51 at hidden dimension 5120); a stage sends n×k coordinates per token instead of n×d, a roughly 100x cut in inter-stage traffic. Second, AsyncSPARTA: replicas of a stage take local optimizer steps and, every 20 steps, all-reduce a rotating 5% slice of their weights in the background rather than exchanging gradients before every step; a delta rule preserves each replica's

Load-bearing premise

The load-bearing premise is that confining every layer's output projection and the token embedding to a common low-dimensional subspace (rank ~51 at hidden dimension 5120) preserves model quality at scale; the paper validates this only against a reference run that also enforces the constraint and explicitly declines to claim a scaling law from two configurations.

What would settle it

Train the same 8.6B architecture on the same 500B tokens in a datacenter without the subspace constraint and compare final training loss with 2.41; if the unconstrained model is materially better, the compression premise fails. Alternatively, measure how the minimal usable subspace rank scales with hidden dimension: if rank grows proportionally to d instead of sublinearly (40 at d=2048 vs ~51 at d=5120), the 100x compression and the bandwidth-adequacy argument collapse at frontier scale.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Pretraining compute can be pooled from individually owned, geographically distributed, preemptible GPUs: the run held near-170k tokens/s through 669 joins and 607 departures, with about 60 contributors admitted at a time.
  • The 63%-of-datacenter efficiency and near-centralized convergence indicate that the two communication cuts — subspace-compressed pipeline traffic and sparse asynchronous averaging — are enough to keep internet-grade links off the training critical path at the 8B scale.
  • No node ever holds the full model: workers hold one pipeline stage and trainers hold none, so the trained model is collectively owned; the paper argues this opens an economically sustainable route to open frontier training.
  • The paper's scaling analysis, which treats compression rate as fixed in the pessimistic case, indicates that bandwidth and latency constraints relax as hidden dimension and depth grow, so the protocol is claimed to become easier, not harder, to sustain at larger scale.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the observed sublinear rank growth (rank 40 at d=2048, rank ~51 at d=5120) continues, the compression ratio improves with scale; if rank instead grows proportionally to width, the 100x compression and the admission-bandwidth floor would both need to be revisited.
  • Because latency, not bandwidth or raw GPU FLOPs, governed contributor throughput in the reported measurements, a natural extension not tested here is geographic placement of trainers and reducers near contributors to shorten round trips.
  • The 200 Mbit/s / 80 ms admission floor means the practical 'permissionless' pool is limited to well-connected machines; reach would grow if the compression or averaging could tolerate slower residential links.
  • The convergence comparison is against a reference that also uses subspace compression and sparse averaging, so the run demonstrates decentralized execution matching centralized execution of the same compressed architecture — not that the compressed architecture matches an uncompressed one.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents Agora, a protocol and system for training large language models over a permissionless, heterogeneous, internet-connected pool of GPUs. It combines pipeline-parallel model sharding with reparameterized subspace networks (SSNs) to compress inter-stage activations/gradients, and asynchronous sparse weight averaging (AsyncSPARTA) to keep data-parallel replicas in consensus without per-step gradient all-reduce. The authors report a 40-day, 500B-token pretraining run of an 8.6B-parameter model, Pluralis-8B, on FineWeb-Edu, using ~330 contributor nodes with continuous churn, sustaining ~170k tokens/s and 4.2 tokens/TFLOP of pooled compute, which they state is 63% of a centralized H100 baseline, with training loss converging close to a centralized reference run. The paper also includes system design details, fault-tolerance mechanisms, a scaling analysis, and 1B-scale ablations on convergence robustness.

Significance. If the central claims hold, this is a significant systems contribution: it would be the first demonstration that a multi-billion-parameter LLM can be pretrained at scale over preemptible, individually owned, heterogeneous GPUs connected by the internet, with near-centralized convergence and modest efficiency loss. The engineering is substantial: fault-tolerant sparse all-reduce, admission control and sync phases, deterministic backward pinning, and a load-balanced pipeline over DHT-based peer discovery. The paper also makes code and a live dashboard available, which is a strength for reproducibility. However, the ML-level validation is currently incomplete: the subspace compression that underpins the system is validated only against other subspace-constrained runs, and the efficiency comparison is not model-matched. These gaps are load-bearing for the paper's headline claims and require either additional experiments or a careful restatement of the claims.

major comments (4)
  1. [§4.1, §5.5, Fig. 38] The claim that Pluralis-8B 'converged to within a small margin of a centralized reference' is not validated against a dense, unconstrained transformer. Equation (3) reparameterizes every output projection and the token embedding as W = Z U_k^T, confining the residual stream to a rank-k subspace (k≈51 at d=5120). All ablations and the central reference in Section 5.5 use the same subspace constraint, so they cannot reveal quality loss caused by the constraint itself. Because this premise is what makes the ~100× inter-stage compression and the entire pipeline-parallel design viable, a comparison to a dense baseline—or at least a small-scale dense run under Agora—is needed to separate architecture-induced degradation from system-induced degradation. Without it, the 'small margin' claim is incomplete.
  2. [§5.4, Table 4, §5.1] The 63% efficiency comparison is not model-matched. The centralized baseline is a dense Llama 3.1 8B trained with FSDP2 on H100s, whereas Pluralis-8B's reparameterized SSN has fewer trainable parameters (~8.6B vs ~12B dense-equivalent) and performs fewer FLOPs per token because output projections are low-rank. A higher tokens/TFLOP is therefore expected even at identical system efficiency, so the 63% figure conflates architectural FLOP reduction with system efficiency. Additionally, §5.1 reports 'measured 30.8 ZettaFLOPs (3.08×10^22)' for 500B tokens over 40 days, which implies ~16.2 tokens/TFLOP of actual compute, while Table 4 reports 4.31 tokens/TFLOP using C_pool = 39,749 TFLOP/s spec-sheet capacity. These two metrics should be reconciled; as written, the abstract's efficiency claim is ambiguous.
  3. [§4.2, Eq. (8), §4.2.2] AsyncSPARTA's convergence relies on the delay-corrected sparse averaging scheme, but the convergence guarantee is only cited to 'Maziane et al. (2026)' with no statement of assumptions or proof sketch. Given that the run's convergence under asynchronous averaging with partial all-reduce participation is a central empirical claim, the paper should at least state the conditions under which the guarantee holds (e.g., smoothness, bounded delay, averaging cadence) or provide a proof in an appendix. As it stands, the convergence argument is an appeal to a same-group reference that is not publicly identifiable.
  4. [§3.4.5, §4.1] The scaling analysis concludes that communication constraints relax as models grow, but this relies on the assumption that the required subspace rank grows more slowly than width. The paper explicitly says 'We do not claim a scaling law from two configurations' and reports only rank 40 at d=2048 and rank ≈51 at d=5120. With two points, the trend is not established. Since the introduction and conclusion frame Agora as a path to frontier-scale training, this extrapolation should be either supported with more data or clearly labeled as a conjecture rather than a demonstrated property.
minor comments (6)
  1. [§4.1, Eq. (2)–(3)] The notation for the basis U_k is inconsistent: Eq. (2) sends (X - PE - T_fixed) U_k and reconstructs with U_k^T, while Eq. (3) writes W = Z U_k^T. Clarify whether columns of U_k are the orthonormal basis and ensure the row/column convention is used consistently throughout.
  2. [§5.5] The centralized reference run is only 27k iterations, while Pluralis-8B runs for 60k steps. The paper should justify why a shorter reference is sufficient to support the 'small margin' convergence claim, especially since training dynamics change over the full schedule.
  3. [§3.4.5, §6] The scaling analysis says 'the rank we can use appears to grow more slowly than the width,' but the rank is described as a hyperparameter. Is U_k fixed and unlearned, or is it trained/updated? If fixed and arbitrary, this is a strong architectural assumption that should be stated explicitly and tested.
  4. [Abstract, §1] The term 'Unextractable Protocol Models (UPMs)' is introduced in the abstract/introduction but never defined or used in the system description or experiments. Either define the concept and its role, or remove it from the paper.
  5. [Throughout] Several core methods are cited to unpublished or same-group references without arXiv IDs or version information (Ramasinghe et al. 2025; Beton et al. 2025; Ajanthan et al. 2026; Long et al. 2025; Maziane et al. 2026). Full citations should be added so readers can verify the methods.
  6. [§2.2, §3.2] Typo: 'SW ARM parallelism' should be 'SWARM parallelism' (Section 3.2). Also check formatting of author names (e.g., 'V olz and Brazil, 2018').

Circularity Check

1 steps flagged

Load-bearing subspace premise is imported from same-group prior work and validated only against subspace-constrained references; the run-level empirical claims are not themselves circular.

specific steps
  1. self citation load bearing [Section 4.1, Eqs. (1)–(3); validation in Section 6.1 (Fig. 38) and Section 5.5]
    "Subspace networks (SSNs) (Ramasinghe et al., 2025) reduce this cost by constraining the inter-stage signal to a single low-rank subspace shared across all stages... The reference is a TorchTitan run modified to use the same subspace-network reparameterization (Section 4.1) and SPARTA averaging (Section 4.2) as Pluralis-8B... The reference thus carries the same compression-induced effects on the loss, so any difference between the curves reflects the decentralized execution itself."

    The 100x communication compression that makes internet-scale pipeline parallelism viable assumes all output projections and the token embedding can be confined to a shared rank-k≈51 subspace without an adverse quality effect. That premise is imported from the authors' own Ramasinghe et al. (2025), and the paper's only validation compares subspace-constrained variants: Fig. 38 compares reparameterized vs original SSN, and the Sec. 5.5 central reference uses the same subspace reparameterization. No dense, unconstrained baseline is compared, so the claim that the architecture change does not adversely affect convergence cannot be falsified by these experiments; the system-level convergence/efficiency claims inherit the untested premise from a same-group citation rather than from an independen

full rationale

The headline run-level claims—~170k tokens/s, 4.2 tokens/TFLOP, 63% of a centralized H100 baseline, and loss 2.41—are empirical measurements, not fitted predictions, and the efficiency definition η = TPS/C_pool is a convention rather than a self-fulfilling identity. The 100x communication compression is the algebraic identity W = Z U_k^T; under the stated subspace constraint the encode/decode in Eq. (2) is lossless by construction, which is a theorem rather than circularity. The convergence comparison in Sec. 5.5 deliberately uses a central reference modified to use the same subspace and SPARTA recipe, so it correctly isolates the effect of decentralized execution; the residual claim is measured, not constructed. The paper also includes honest caveats: 'We do not claim a scaling law from two configurations' and 'the rank is a hyperparameter' (Sec. 3.4.5). What prevents a clean 0–2 score is the load-bearing self-citation cluster: the untested premise that rank-k subspace confinement is behavior-preserving at 8B scale comes from Ramasinghe et al. (2025), whose authors overlap with the present paper, and the only convergence evidence for it compares subspace-constrained variants (Fig. 38 and the Sec. 5.5 reference). No dense baseline is reported, so the architecture's quality cost is neither measured nor falsified. AsyncSPARTA's convergence guarantee is likewise delegated to prior/work by Ajanthan et al. (2026) and Maziane et al. (2026), though in-paper 1B ablations provide some empirical support. This is a missing-control and self-citation verifiability gap in the derivation chain, not a definitional equivalence of two equations, hence score 4 rather than 6+.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 1 invented entities

The paper's genuine contribution is the system integration and the 8B-scale demonstration; the theoretical and algorithmic load (SSN validity, SPARTA convergence, delta-rule guarantees, UPM security) is carried by the authors' own prior/concurrent papers, none of which is independently verifiable from this text. Free parameters are largely disclosed and reasonable, but the subspace rank and the clipping thresholds are tuned against the very runs used to validate the system.

free parameters (5)
  • Subspace rank k = 40 (1B ablations), ~51 (Pluralis-8B)
    Sets the 100x/51x inter-stage compression factor; hand-chosen per scale; the authors themselves note the rank is 'a hyperparameter rather than a fixed property of the architecture' (Section 3.4.5).
  • SPARTA sparse share p and cadence N = p = 5%, N = 20 steps
    Control the drift-vs-communication tradeoff on the data-parallel axis (Tables 1, 2); hand-set values, with ablations probing failure rates around them but not the p/N choice itself.
  • Per-stage gradient clip thresholds = 1/√7 ≈ 0.3779 (head/body), 0.7558 (tail)
    Derived from the 1/√S budget but the tail threshold is doubled 'to account for its consistently larger observed gradient norm' (Section 4.3, Figure 21b) — i.e., fitted to the run's own gradient statistics.
  • Admission bandwidth/latency floor = 200 Mbit/s, 80 ms RTT, 24 GB VRAM, 80 GB RAM
    Chosen from an in-house stability grid (Section 3.5.2) that the authors ran; this floor censors the contributor pool and shapes all heterogeneity results in Section 5.3.
  • Sync-phase durations = weight sync 400 steps, optimizer sync 100 steps
    Derived from the 400-step full-parameter averaging cycle and the AdamW β2=0.95 time constant (Section 5.6), but ultimately hand-set; the inter-stage step drift of 540 steps (0.9%) is a direct consequence.
axioms (4)
  • domain assumption Transformer residual streams can be confined, without material quality loss, to a shared k-dimensional subspace; every output projection and the token embedding factor through it (Eqs. 1-3).
    This is the 100x inter-stage compression premise of Section 4.1. Imported from the authors' Ramasinghe et al. 2025 with no external benchmark, and validated only against reference runs that also use the constraint.
  • domain assumption AsyncSPARTA with the delay-corrected delta rule converges for smooth non-convex objectives under communication-computation overlap (Eqs. 6, 8).
    The paper does not prove this; it cites Beton et al. 2025, Ajanthan et al. 2026, and Maziane et al. 2026 (same-group, unverifiable here). The entire data-parallel axis depends on it.
  • domain assumption Kademlia DHT coordination remains stable enough that all-reduce matchmaking groups form within timeouts at ~60-contributor scale.
    Underpins the all-reduce design in Sections 3.3.1 and 5.9; the authors document DHT latency degrading ~9-10x after contributors join (Section 5.9.1), and the run in fact relied on operator-managed services and admission caps.
  • domain assumption A centralized run of the same SSN+SPARTA recipe adequately represents 'centralized training' for the convergence comparison.
    Figure 19's reference is the control for the central convergence claim; it ran only 27k of the 60k steps, and was constructed by the same authors with the same modified model recipe.
invented entities (1)
  • Unextractable Protocol Models (UPMs) no independent evidence
    purpose: Claimed guarantee that trained weights 'can never be extracted from a node at all,' extending the no-single-party-holds-weights property into a security property.
    Introduced only via citation to Long et al. 2025; no security model, proof, or falsifiable handle appears in this text, and the actual run did not implement it (Pluralis nodes hold full stage shards and an Authorizer issues credentials).

reviewed 2026-08-02 · how reviews work

0 comments
read the original abstract

Training large language models at the multi-billion to trillion parameter scale is confined to datacenters, where data-parallel (DP) and model-parallel (MP) techniques presume homogeneous accelerators, high-speed interconnects, and a single orchestrating entity. Frontier model development is thereby concentrated among the few groups able to assemble such clusters. Meanwhile, an enormous pool of compute remains unusable for training: consumer and professional GPUs that are heterogeneous, preemptible, individually owned, and connected only by the internet. We present Agora, a system that makes efficient use of this compute. Agora combines bandwidth-efficient pipeline-parallel model sharding over internet-grade links with multi-party, fault-tolerant collective operations. Each participant holds only one stage of the model, and no single party ever possesses the full weights. We term this setup Protocol Learning: it enables collectively trained, collectively owned models, opening a path to open-source frontier training with economic sustainability. This report presents the outcome of a research effort spanning communication-efficient parallelism, asynchronous optimization, and fault-tolerant systems design. It culminates in the first demonstration of its kind: Pluralis-8B, an open, permissionless pretraining run of an 8.6B-parameter model on 500B tokens of FineWeb-Edu. The model was trained over 40 days by 330 contributor nodes, predominantly consumer GPUs on internet connections, joining and leaving throughout. The run sustained ~170k tokens/s and 4.2 tokens per TFLOP of pooled compute, 63% of the efficiency of a centralized H100 baseline, and converged to within a small margin of a centralized reference run.

Figures

Figures reproduced from arXiv: 2607.13332 by Alexander Long, Chamin Hewa Koneputugodage, Gil Avraham, Hadi Mohaghegh Dolatabadi, Harry Xi, James Snewin, Karol Pajak, Rodney O'Donnell, Sameera Ramasinghe, Shamane Siriwardhana, Thalaiyasingam Ajanthan, Violetta Shevchenko.

Figure 1
Figure 1. Figure 1: Overview of the Agora system, instantiated for the Pluralis-8B run. The model is sharded into seven pipeline stages, each replicated across workers that hold only that stage; trainers route each microbatch through one worker per stage over internet links, and contributors join permissionlessly through the Authorizer. Same-stage replicas are kept in consensus by asynchronous sparse weight averaging, while P… view at source ↗
Figure 2
Figure 2. Figure 2: Diagram of the worker architecture. A worker hosts one replica of a single pipeline stage. Connection handlers receive forward/backward microbatches from the trainers over P2P RPC and enqueue them into forward and backward task queues; the Runtime coalesces queued microbatches and runs them through the stage’s Transformer blocks on the GPU. The optimizer orchestrates training: gradients accumulated on the … view at source ↗
Figure 3
Figure 3. Figure 3: Diagram of the all-reduce averaging algorithm. One chunk iteration of an averaging round with four senders (S1–S4) and three reducers (A–C). Each peer’s flattened parameter vector is partitioned into slices, one per reducer (a, b, c), and every peer streams its copy of each slice, cut into small fixed-size chunks (2 MiB), to the slice’s assigned reducer. A reducer buffers incoming chunks in per-sender queu… view at source ↗
Figure 4
Figure 4. Figure 4: Impact of network quality and trainer count on throughput. (a) A pipeline stage operating in a communications-bound regime: 4 Pluralis (high-bandwidth) nodes and 16 contributor nodes with different latency and bandwidth restrictions (simulated). (b) When communication time exceeds the computation window, utilization and throughput suffer. (c) The total compute capacity of the pipeline needs enough trainer … view at source ↗
Figure 5
Figure 5. Figure 5: All-reduce stability, duration, and completeness under varying network conditions. (a) Percentage of failed all-reduces as a function of RTT latency. (b) Average all-reduce duration (seconds) over the same range. (c) Average percentage of all-reduce completeness over the same range. The solid curve sweeps RTT latency over 0–120 ms at 200 Mbit/s; the dashed horizontal lines mark reference runs at 50 Mbit/s … view at source ↗
Figure 6
Figure 6. Figure 6: All-reduce stability under contributor failure injection. Contributor failure injection via preemption during all-reduce shows no major effect on system all-reduce stability outside of transient drops in completeness. Overall >90% of the required all-reduce payload is successfully communicated across rounds. under the less predictable conditions faced by contributors. However, with Agora’s fault-tolerant a… view at source ↗
Figure 7
Figure 7. Figure 7: All-reduce stability under worker failure injection. Top: all-reduce completeness and percentage of received parameters by reducer under failure 1. Bottom: all-reduce completeness and percentage of received parameters by reducer under failure 2. Both show only transient instability after failure injection, with perfect subsequent all-reduce rounds. Indeed, as shown in [PITH_FULL_IMAGE:figures/full_fig_p02… view at source ↗
Figure 8
Figure 8. Figure 8: Admission flow. A node’s path from a join request to participation in the training network. fourth is accountability: every node authenticates before admission, so a participant that misbehaves can be identified and barred, whether by excluding an individual node or an entire identity. Admission queueing. The Authorizer admits contributors gradually rather than all at once. The reason is the cost that each… view at source ↗
Figure 9
Figure 9. Figure 9: Peer authentication. The Authorizer signs each credential once, and its public key is shared with every peer in advance. Thereafter peers authenticate one another locally. This design keeps the Authorizer off the training data path: verification is local, so no peer-to-peer exchange waits on it. Credentials are short-lived and must be renewed periodically, which gives the operator a direct 24 [PITH_FULL_I… view at source ↗
Figure 10
Figure 10. Figure 10: shows how the monitor joins the run and where its observations are stored [PITH_FULL_IMAGE:figures/full_fig_p025_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: shows the resulting paths [PITH_FULL_IMAGE:figures/full_fig_p027_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Cloud architecture of the Agora system. 4. PLURA LIS-8B To demonstrate the capabilities of Agora under a realistic pretraining workload, we chose to train an 8B￾parameter model, which we call Pluralis-8B. The objective of the run was to demonstrate that the underlying system could sustain a realistic pretraining workload at a meaningful scale. A system of this kind cannot be thoroughly validated through i… view at source ↗
Figure 13
Figure 13. Figure 13: The trainer shard indices during a shard rotation. 5.2 MODEL CARD Pluralis-8B is a decoder-only transformer trained over Agora with reparameterized subspace networks (Section 4.1) on the pipeline-parallel axis and AsyncSPARTA (Section 4.2) on the data-parallel axis. The model is partitioned into seven pipeline stages: a head, five body stages, and a tail, for a total of 32 transformer blocks [PITH_FULL_I… view at source ↗
Figure 14
Figure 14. Figure 14: Hardware heterogeneity of the 330 contributor nodes. (a) Node count by GPU model: participation was restricted to consumer cards plus the L40S, and the RTX 4090 and RTX 5090 together accounted for 55% of nodes. (b) Per-node RAM against VRAM: RAM varied largely independently of GPU class, from the 80 GB admission floor to 2 TB (median 504 GB). (c) Per-node download bandwidth against network latency, with t… view at source ↗
Figure 15
Figure 15. Figure 15: Contributor fleet over the run. (a) Daily join and drop events: 669 joins and 607 drops in total, with an initial wave of 107 joins on 20 May and steady turnover thereafter (a node that dropped and rejoined is counted each time). (b) Distinct contributors seen per day against the concurrent fleet, which reached the 60-node admission cap within two days. (c) Live fleet composition by GPU model: the mix shi… view at source ↗
Figure 16
Figure 16. Figure 16: Join demand and admission over the run. Each client is counted by IP address. (a) Rejection reasons for the 3.33 million rejected join requests (99.98% of all requests): 89% arrived while the run was at capacity, 11% were malformed, and under 0.5% failed for other causes. (b) Cumulative distinct clients that ever attempted to join (612) and that were ever admitted (208): demand kept arriving throughout th… view at source ↗
Figure 17
Figure 17. Figure 17: Throughput and efficiency. (a) Pipeline throughput, per-stage node counts, and trainer count over the run; for the body stages we plot the minimum node count across them. (b) Pipeline efficiency (tokens per pooled TFLOP) vs. a centralized baseline (Llama 3.1 8B, H100, FSDP2). Stage Model layers head 6 transformer blocks + embeddings body-1 4 transformer blocks body-2 4 transformer blocks body-3 4 transfor… view at source ↗
Figure 18
Figure 18. Figure 18: Communication vs. computation time in body stages. Per-microbatch communication (await) and computation time across all body-stage (contributor) nodes, over the stable 20k–30k-step window. In the median, communication was shorter than computation (139 ms vs. 276 ms), so the body stages ran compute￾bound. 5.5 TRAI NI NG LOSS Since the goal of this work is to show that a system trained with Agora matches st… view at source ↗
Figure 19
Figure 19. Figure 19: Pluralis-8B training loss against a central reference curve. (a) The full run, with the average success of workers in averaging their weight shard overlaid above. (b) The same loss curves zoomed in on the y-axis. The early jaggedness coincides with the period of largest weight change; the perturbation near step 40k follows a simultaneous exit of many contributor nodes, from which the loss recovered withou… view at source ↗
Figure 20
Figure 20. Figure 20: The minimum, 10th percentile, and maximum success of workers across the training steps. (a) Averaging their weight shard. (b) Performing the reduce operation of the assigned weight partition. We next observe the gradient norms of the run. In Figure 21a we see the minimum, average, and maximum of the norms across all workers. Figure 21b breaks the average gradient norm down by stage type: head, body, and t… view at source ↗
Figure 21
Figure 21. Figure 21: The gradient norm against the training steps. (a) Across all workers (minimum, average, and maximum). (b) The average gradient norm segmented by stage type: head, body, tail. phase it additionally processes batches, warming up its optimizer state. In this section we analyze whether these phases successfully prevent the negative effects that a stale-weight replica has on convergence, namely gradient and lo… view at source ↗
Figure 22
Figure 22. Figure 22: Sync phases vs. loss and gradient norm. Global training loss (top) and, per body stage, the number of workers in weight sync and optimizer sync, together with the stage’s pre-step gradient norm. Sync activity produces no loss or gradient spikes as workers join. reached only 540 steps, about 0.9% of the 60k steps. This divergence is a matter of step accounting rather than of the data: every stage still pro… view at source ↗
Figure 23
Figure 23. Figure 23: Inter-stage step drift under optimizer sync. The inter-stage step drift (max−min of the per-stage median step) climbs with each burst of optimizer-sync activity (workers in optimizer sync summed over stages), accumulating to 540 steps (0.9%) over the run. The all-reduce step is designed to be fault-tolerant and completes even if some replicas fail to send or receive their complete parameter segment in a g… view at source ↗
Figure 24
Figure 24. Figure 24: All-reduce health for the body-3 stage across the run. Each point is one SPARTA round, plotted at its optimizer step; each panel shows the mean and the worst value over the participating nodes in that round. The brief degradations around steps 10k, 40k, and 50k are most likely network issues on contributor nodes. Observed all-reduce health [PITH_FULL_IMAGE:figures/full_fig_p047_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Inter-replica variance vs. SPARTA segment, for (a) body-3 and (b) head. Each curve is one parameter snapshot, colored by training step. Curves are drawn every 2000 steps so that the 500-step snapshot cadence aligns with the 400-step averaging cycle: in every curve, segment 19 was averaged 20 steps before the snapshot, segment 18 was averaged 40 steps before it, and so on. Variance therefore ramps up linea… view at source ↗
Figure 26
Figure 26. Figure 26: Inter-replica variance vs. optimizer step, one line per SPARTA segment. Segment 19, always the most recently averaged before each snapshot, has the lowest variance; the other segments stack above it in order of averaging recency. Segment 0 is omitted because its variance is noisy due to snapshot timing (see [PITH_FULL_IMAGE:figures/full_fig_p049_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Between-snapshot movement of the consensus mean vs. per-parameter inter-replica diver￾gence. Solid curves show movement and dotted curves divergence, for the body-3 and head stages, with the head token embedding shown separately. Movement and divergence track each other closely for every group, with divergence sitting slightly above movement throughout the run. consensus movement. This indicates that more… view at source ↗
Figure 28
Figure 28. Figure 28: Bandwidth distribution and utilization. 5.8 COM M UNI CATIO N EFFI CI ENCY The communication efficiency of the Pluralis-8B run is highlighted by the ability of the Agora system to multiplex (Section 3.3.1) a node’s forward and backward communication channel with the all-reduce channel, as the latter runs asynchronously. In Figure 28a we show the bandwidth distribution for the contributor nodes, with the b… view at source ↗
Figure 29
Figure 29. Figure 29: Multiplexing computation with the asynchronous all-reduce. For a single body-stage node. available bandwidth avg total comm. (lifecycle) AR spike send AR spike receive 0 250 500 750 1000 1250 1500 Mbit/s 1118 132 507 604 n=28 (a) Head. available bandwidth avg total comm. (lifecycle) AR spike send AR spike receive 0 250 500 750 1000 1250 1500 Mbit/s 714 85 157 165 n=370 (b) Bodies. available bandwidth avg … view at source ↗
Figure 30
Figure 30. Figure 30: Bandwidth usage per stage family. Overall available bandwidth, average bandwidth used, and average upload and download bandwidth during all-reduce. groups, and the sparse state all-reduce itself. Section 3.5.1 characterized their stability under controlled bandwidth constraints and fault injection; here we examine how they behaved during the Pluralis-8B run once contributors with heterogeneous, high-laten… view at source ↗
Figure 31
Figure 31. Figure 31: DHT throughput during the Pluralis-8B training run, measured from Pluralis nodes. Total throughput of DHT operations across all stages; the dashed line marks when contributors were allowed to join. Throughput drops noticeably after the join, despite the additional peers. Between 18 May and the join date the Health Monitor was offline, reducing the operation count over that window. May 18 May 25 Jun 01 Jun… view at source ↗
Figure 32
Figure 32. Figure 32: DHT latency during the Pluralis-8B training run, measured from Pluralis nodes. Latency of DHT operations across all stages; the dashed line marks when contributors were allowed to join, after which latency rises by roughly an order of magnitude. may hold important key-value pairs yet be preempted at any moment; and (2) even fast nodes may need to route through slow contributor nodes to reach a given key. … view at source ↗
Figure 33
Figure 33. Figure 33: Matchmaking statistics during the Pluralis-8B training run. (a) Mean all-reduce matchmaking time per pipeline stage type (head, body, tail), averaged over all stage peers. (b) Group-size deficit, the difference between the expected and formed all-reduce group size, per stage type, averaged over all stage peers. Matchmaking stays stable: times remain within the 40 s timeout on average and the group deficit… view at source ↗
Figure 34
Figure 34. Figure 34: State all-reduce time during the Pluralis-8B training run. Mean and maximum durations of asynchronous state all-reduces per stage type (head, body, tail), over all stage peers. After contributors join, their networking conditions dominate body-stage all-reduce time (up to 8× higher), and the maximum reflects frequent contributor timeouts. stays below 20 seconds, even for body stages, where contributors in… view at source ↗
Figure 35
Figure 35. Figure 35: Maximum time between all-reduce chunks. Maximum time between chunks received by reducers, taken over senders, chunks, and reducers. The “timeout danger” line (6.5 seconds) marks the per-chunk delay beyond which an all-reduce risks timing out. It is rarely crossed, so slow contributor upload alone does not explain the observed timeouts, which likely stem from transient stream failures, peer churn, network … view at source ↗
Figure 36
Figure 36. Figure 36: Mean batch-processing throughput per contributor. (a) Grouped by GPU family (the three RTX PRO 6000 Blackwell variants are merged); bar height is the average over nodes of each node’s lifetime￾mean processed batches per 60 s, with contributor count 𝑛 and the BF16 peak (TFLOP/s) annotated under each card. The ranking is only weakly correlated with peak FLOP/s. (b) Binned by network latency (ms); throughput… view at source ↗
Figure 37
Figure 37. Figure 37: GPU-family composition of contributors within each latency band. Same population as Figure 36b; 𝑛 per band shown in each title. Low-latency nodes are predominantly L40S cards, while high￾latency nodes are almost entirely consumer GPUs, so latency and GPU class are confounded. Setup. For our experiments in this section, we use a 1B reparameterized SSN trained with AdamW and AsyncSPARTA. This is the same tr… view at source ↗
Figure 38
Figure 38. Figure 38: Training loss of the 1B model under reparameterized vs. original SSN. Both variants use the same rank-40 compression and are compared against a central reference. All training runs use AsyncSPARTA for their data-parallel communication. The curves coincide, so reparameterization leaves convergence unchanged. 6.2 ASYNCSPARTA AND BACKWARD STO CHASTI CI TY We saw in Section 3.3.2 that Agora routes each microb… view at source ↗
Figure 39
Figure 39. Figure 39: Per-stage cost of reparameterization, reparameterized vs. original SSN. For the head and body, we report steady-state medians with the reparameterized/original ratio and the per-step trend. The lighter head cuts its all-reduce by about a quarter, and fewer trainable parameters give a faster, cheaper optimizer step. 0 250 500 750 1000 1250 1500 training step 0 10 20 30 40 gradient norm (pre-clip) original … view at source ↗
Figure 40
Figure 40. Figure 40: Gradient magnitude over training. The original SSN spikes early while the reparameterized formulation stays stable because the compression lives inside the computational graph. The ablation run passes through several phases of churn, with about 87 joins and 87 drops in total. We take each body stage from 16 workers to 8 near step 4.5k, then open a slow-join schedule near step 5k across the three slow-join… view at source ↗
Figure 41
Figure 41. Figure 41: Training loss under stochastic vs. deterministic backward. The 1B model runs at AsyncSPARTA cadence 𝑁 = 20, with four pipeline stages over 8 data-parallel pipes. Deterministic routing significantly improves convergence. Contributor tier Upload (Mbit/s) Download (Mbit/s) Latency (ms) Slow-join tiers Fast 270 270 0 Medium 210 210 10 Slow 550 550 35 Stress-join tiers Bare 210 210 80 Asymmetric 210 600 80 Jit… view at source ↗
Figure 42
Figure 42. Figure 42: Achieved heterogeneity ratios across the join tiers. The fast, medium, and slow tiers follow [PITH_FULL_IMAGE:figures/full_fig_p061_42.png] view at source ↗
Figure 43
Figure 43. Figure 43: Training loss across the heterogeneous run. Markers denote the drop (4.5k), slow join (5k), stress swap (6.5k), and drain (8.5k). The loss generally tracks the homogeneous no-churn baseline. During the stress phase, the sharp increase in all-reduce latency and removal of 7 out of 36 stressed workers coincide with a temporary slowing of convergence; the loss resumes its downward trajectory as all-reduce la… view at source ↗
Figure 44
Figure 44. Figure 44: State all-reduce time per stage. The stress-phase kick-out fraction is annotated. The averaging time grows with the worker count and under the 80 ms stress exceeds the timeout for about 19% of stressed workers, which self-terminate, yet the loss in [PITH_FULL_IMAGE:figures/full_fig_p062_44.png] view at source ↗
Figure 45
Figure 45. Figure 45: Convergence under injected all-reduce (AR) failures at fixed 1:1 heterogeneity. The run follows the AR failure schedule of [PITH_FULL_IMAGE:figures/full_fig_p063_45.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 3 linked inside Pith

  1. [1]

    Max Ryabinin and Tim Dettmers and Michael Diskin and Alexander Borzunov , title =

  2. [2]

    Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism , author=

  3. [3]

    Vogels, Thijs and Karimireddy, Sai Praneeth and Jaggi, Martin , booktitle = NeurIPS, year =

  4. [4]

    and Jaggi, Martin , booktitle = ICML, year =

    Karimireddy, Sai Praneeth and Rebjock, Quentin and Stich, Sebastian U. and Jaggi, Martin , booktitle = ICML, year =. Error Feedback Fixes

  5. [5]

    Proceedings of 15th Annual Conference of the International Speech Communication Association (

    Frank Seide and Hao Fu and Jasha Droppo and Gang Li and Dong Yu , title =. Proceedings of 15th Annual Conference of the International Speech Communication Association (

  6. [6]

    Modular, Collaborative and Decentralized Deep Learning (MCDC) Workshop at ICLR , year =

    Improving the Efficiency of Distributed Training using Sparse Parameter Averaging , author =. Modular, Collaborative and Decentralized Deep Learning (MCDC) Workshop at ICLR , year =

  7. [7]

    and Shevchenko, Violetta and Zuo, Yan and Long, Alexander , journal =

    Ajanthan, Thalaiyasingam and Ramasinghe, Sameera and Avraham, Gil and Mohaghegh Dolatabadi, Hadi and Hewa Koneputugodage, Chamin P. and Shevchenko, Violetta and Zuo, Yan and Long, Alexander , journal =

  8. [8]

    2025 , url=

    Node0: Model Parallel Training over the Internet with Protocol Models , author=. 2025 , url=

    2025

  9. [9]

    O'Reilly Media , year =

    Julius Volz and Brian Brazil , title =. O'Reilly Media , year =

  10. [10]

    Guilherme Penedo and Hynek Kydl. The

  11. [11]

    Common Crawl Corpus (CC-MAIN-2025-13) , year=

    2025

  12. [12]

    Wanchao Liang and Tianyu Liu and Less Wright and Will Constable and Andrew Gu and Chien-Chin Huang and Iris Zhang and Wei Feng and Howard Huang and Junjie Wang and Sanket Purandare and Gokul Nadathur and Stratos Idreos , title =

  13. [13]

    Grattafiori, Aaron and Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and Kadian, Abhishek and Al-Dahle, Ahmad and Letman, Aiesha and Mathur, Akhil and Schelten, Alan and Vaughan, Alex and others , journal=. The

  14. [14]

    Liu, Aixin and Feng, Bei and Xue, Bing and Wang, Bingxuan and Wu, Bochao and Lu, Chengda and Zhao, Chenggang and Deng, Chengqi and Zhang, Chenyu and Ruan, Chong and others , journal=

  15. [15]

    Shoeybi, Mohammad and Patwary, Mostofa and Puri, Raul and LeGresley, Patrick and Casper, Jared and Catanzaro, Bryan , journal=

  16. [16]

    Hammond and Torsten Hoefler , title =

    Zhiyi Hu and Siyuan Shen and Tommaso Bonato and Sylvain Jeaugey and Cedell Alexander and Eric Spada and James Dinan and Jeff R. Hammond and Torsten Hoefler , title =. Proceedings of the

  17. [17]

    Le and Yonghui Wu and Zhifeng Chen , title =

    Yanping Huang and Youlong Cheng and Ankur Bapna and Orhan Firat and Dehao Chen and Mia Xu Chen and HyoukJoong Lee and Jiquan Ngiam and Quoc V. Le and Yonghui Wu and Zhifeng Chen , title =

  18. [18]

    Devanur and Gregory R

    Deepak Narayanan and Aaron Harlap and Amar Phanishayee and Vivek Seshadri and Nikhil R. Devanur and Gregory R. Ganger and Phillip B. Gibbons and Matei Zaharia , title =. Proceedings of the

  19. [19]

    Proceedings of the

    Shen Li and Yanli Zhao and Rohan Varma and Omkar Salpekar and Pieter Noordhuis and Teng Li and Adam Paszke and Jeff Smith and Brian Vaughan and Pritam Damania and Soumith Chintala , title =. Proceedings of the

  20. [20]

    Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (

    Samyam Rajbhandari and Jeff Rasley and Olatunji Ruwase and Yuxiong He , title =. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (

  21. [21]

    arXiv preprint arXiv:2304.11277 , year=

    Pytorch fsdp: experiences on scaling fully sharded data parallel , author=. arXiv preprint arXiv:2304.11277 , year=

    Pith/arXiv arXiv

  22. [22]

    Unextractable Protocol Models: Collaborative Training and Inference without Weight Materialization , author =

  23. [23]

    Dirk Groeneveld and Iz Beltagy and Evan Pete Walsh and Akshita Bhagia and Rodney Kinney and Oyvind Tafjord and Ananya Harsh Jha and Hamish Ivison and Ian Magnusson and Yizhong Wang and Shane Arora and David Atkinson and Russell Authur and Khyathi Raghavi Chandu and Arman Cohan and Jennifer Dumas and Yanai Elazar and Yuling Gu and Jack Hessel and Tushar Kh...

  24. [24]

    Team OLMo and Pete Walsh and Luca Soldaini and Dirk Groeneveld and Kyle Lo and Shane Arora and Akshita Bhagia and Yuling Gu and Shengyi Huang and Matt Jordan and Nathan Lambert and Dustin Schwenk and Oyvind Tafjord and Taira Anderson and David Atkinson and Faeze Brahman and Christopher Clark and Pradeep Dasigi and Nouha Dziri and Michal Guerquin and Hamis...

    Pith/arXiv arXiv

  25. [25]

    arXiv preprint arXiv:2605.20866 , year =

    Yassine Maziane and Ammar Mahran and Artavazd Maranjyan and Peter Richt. arXiv preprint arXiv:2605.20866 , year =

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.