Incentivised Orchestrated Training Architecture (IOTA): A Technical Primer for Release

Alan Aboudib; Brian McCrindle; Felix Quinque; Rodrigo Lopez Portillo Alcocer; Steffen Cruz; Szymon Fonau

REVIEW 3 major objections 6 minor 24 references

Incentivised Orchestrated Training Architecture (IOTA): A Technical Primer for Release

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read IOTA claims a permissionless network of heterogeneous miners can pretrain arbitrarily large models by splitting layers, compressing activations by a factor of 128, and rewarding contributors by measured impact.

desk verdict A promising but under-evidenced synthesis: the bottleneck compression idea is worth testing, but the central scaling claim currently rides on a single short run. read the letter →

arxiv 2507.17766 v1 pith:LRUDIIH5 submitted 2025-07-16 cs.DC cs.LG

Felix Quinque , Alan Aboudib , Szymon Fonau , Rodrigo Lopez Portillo Alcocer , Brian McCrindle , Steffen Cruz This is my paper

classification cs.DCcs.LG

keywords decentralizedpretrainingpipelineparallelismactivationcompressionbottlenecktransformerButterflyAll-ReduceShapleyattributionincentivedesign

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that decentralized pretraining does not have to choose between model size and fair incentives. Its architecture, IOTA, splits a single large model across many miners in a pipeline, so the model can grow with the number of participants rather than with any one device's memory. To make that pipeline work over ordinary internet links, it inserts bottleneck transformer blocks that compress the activations and gradients exchanged between miners by up to a factor of 128, with early experiments showing no significant loss in convergence. To keep the system honest, it pairs a constant-bandwidth Butterfly All-Reduce for merging weight shards with a Shapley-based attribution scheme (CLASP) that credits each miner by marginal contribution. If the claims hold, permissionless swarms could pretrain frontier-scale models without data-centre-grade interconnect.

What carries the argument

The load-bearing mechanism is a bottleneck transformer block derived from the Llama3 block but modified so residual connections flow partially into and out of the attention output. This preserves gradient pathways while letting a 2048-dimensional activation be squeezed through a 32-dimensional bottleneck, a compression factor of 128. Around this sit two supporting protocols: Butterfly All-Reduce, whose random all-pairs shard assignment lets every miner share one shard with every other miner, making cheating visible and keeping bandwidth constant as miners are added, and CLASP, whose Shapley-style loss attribution turns per-sample pathway losses into per-miner credit and outlier detection.

What would settle it

Train the proposed bottleneck Llama3 1.5B model at the factor-of-128 compression on tens of billions of tokens, or train the 70B model with eight bottlenecks on a comparable budget, and compare loss against the uncompressed baseline. If the loss gap widens steadily or training diverges once the horizon stretches, the claim that communication can be compressed by a factor of 128 without significant convergence loss is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the obstacles to internet-scale decentralized pretraining, namely memory ceilings, communication bandwidth, and misaligned incentives, can be attacked simultaneously rather than traded off. The central claim is that a modified Llama3 1.5B model with bottleneck blocks that preserve residual connections achieves a symmetric compression of activations and gradients by a factor of 128, with only slight degradation relative to baseline when trained on up to 400 million tokens of FineWeb. The same architecture introduces Butterfly All-Reduce, which merges disjoint shards of miner weights with bandwidth that does not grow with the number of miners while replicating every miner's work across every other miner, giving fault tolerance and collusion detection, and CLASP, which assigns rewards proportional to each miner's marginal utility. The authors are explicit that these are preliminary results and that validation on up to 100 billion tokens is imperative.

Load-bearing premise

The load-bearing premise is that compressed activations and gradients by a factor of 128 keep convergence essentially intact when training runs on realistic, long-horizon corpora; the paper's evidence stops at 400 million tokens, and the authors themselves say validation up to 100 billion tokens is imperative.

Editorial extensions

If this is right

If the compression result holds, a pipeline of miners connected by ordinary internet links can exchange activations without the communication barrier that currently forces model parallelism into data centres.
Model size then scales with the number of participants: adding miners adds layers, so no participant needs to hold more than a slice of the model.
Butterfly All-Reduce keeps merging bandwidth constant regardless of swarm size, and its redundancy preserves over 95 percent of averaged weights even if 10 percent of miners fail.
CLASP ties token emissions to marginal contribution, replacing the winner-takes-all dynamic that encouraged model hoarding with continuous, granular rewards.
Validation by recomputation with cosine similarity, combined with compressed sharing and full sync epochs, gives a practical audit loop for trustless training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Extension: if the factor-of-128 bottleneck compression transfers to larger, deeper models, the paper's own prediction is that a 70B model with eight bottlenecks will be more resilient; the same trick could then also apply to activation offloading during inference, not just training.
Extension: the CLASP pathway-attribution idea is not limited to blockchain training; any federated or pipeline-parallel system that logs per-sample pathways and losses could use the same Shapley-style outlier detection to spot poisoned or free-riding workers.
Extension: a testable next step is to measure how the factor-of-128 compression interacts with DiLoCo-style local optimization steps, since gradients compressed across synchronization gaps may behave differently from the activation-only compression reported here.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read it. It's a systems primer, not a results paper, and that's the right frame. The genuinely new piece is a bottleneck transformer block that compresses activations 128x while preserving residual flow; the rest is a thoughtful integration of known machinery — SWARM, DiLoCo, butterfly all-reduce, Shapley-style attribution — into a blockchain-incentivized pipeline. The authors are unusually honest about what is preliminary, and they cite the right prior work.

What it does well: the Butterfly All-Reduce section cleanly explains how O(1) bandwidth merging can come with redundancy and collusion detection, and the failure-resilience math is correct (modulo a typo in Eq. 1, which should have a range on j). The incentive model is simple but sensible for a trustless setting. The bottleneck block idea is plausible and worth testing. And CLASP, despite the name, is a reasonable direction for attributing credit when per-sample losses are available.

The load-bearing weakness is exactly what your stress-test note says: everything rests on the 128x compression claim, and that claim is supported by a single 1.5B-parameter run on 400M tokens, with no error bars, no seeds, no code, and no comparison to alternatives. The comparison is also confounded — the modified model replaces a third or half of the transformer blocks with bottlenecks, so loss curves don't isolate compression from architecture. The authors themselves say validation up to 100B tokens is imperative, which is honest but also an admission that the headline result isn't demonstrated. If the bottleneck diverges at longer horizons, the 'scale with participants' story collapses.

The CLASP evaluation is nearly as thin: a toy model with hand-set loss distributions and a 10% increase for malicious miners. It shows the mechanism, not the reality. The paper says CLASP won't be in the initial release, so that's a minor issue. No circularity problem; it builds on SN9 but doesn't derive its claims from it.

Overall: this deserves a serious referee. The referee should demand long-horizon experiments (a few billion tokens at least), error bars, code, and an ablation that separates the bottleneck from the compression. The authors can probably do that. Send it to review rather than desk-reject, expecting heavy revision.

Referee Report

3 major / 6 minor

Summary. The paper proposes IOTA, an architecture for incentivized decentralized pretraining of large language models across heterogeneous, unreliable miners. It combines a SWARM-style pipeline-parallel training loop coordinated by a central orchestrator, a bottleneck transformer block that compresses inter-layer activations and gradients by up to 128x, a Butterfly All-Reduce protocol for merging weight shards in per-node O(1) bandwidth with fault tolerance and collusion detection, and a Shapley-inspired attribution scheme (CLASP) for detecting malicious participants. The paper reports early results: a 1.5B Llama3 model with three bottleneck blocks trained on up to 400M FineWeb tokens retains near-baseline loss, and a toy model suggests CLASP can flag outliers. The authors explicitly describe these as preliminary and call for validation on up to 100B tokens.

Significance. If the compression result holds at larger scale, IOTA would address a key obstacle to decentralized model-parallel training: the cost of shipping activations over internet links. The system-level integration of pipeline parallelism, compression, and incentive design is timely, and the paper is candid about the preliminary nature of its results. The evidentiary value is currently limited by the absence of released code, multiple seeds, error bars, and comparisons to alternative compression methods; the central empirical claim rests on a single short training run.

major comments (3)

[Section 4, Figure 5] The central claim that 128x compression yields 'no significant loss in convergence' rests on a single training run (1.5B model, up to 400M tokens) with no seeds, no error bars, and no released code. The comparison is confounded: replacing three of sixteen transformer blocks with bottleneck blocks changes parameter count and per-token compute, so the loss gap does not isolate the effect of compression. Given the authors' own statement that validation up to 100B tokens is 'imperative,' this result does not yet support the abstract's 'scale arbitrarily' promise, which depends on activation bandwidth remaining feasible over internet links.
[Section 5.2, p_valid equation] The formula for p_valid is incorrect as written: p_valid = 1 - (N(N-1)-k(k-1))/(N(N-1)) equals k(k-1)/(N(N-1)), the fraction of invalid shards, not the fraction of valid shards. The correct expression is (N(N-1)-k(k-1))/(N(N-1)). Although the surrounding text and Figure 7b use the correct behavior, the formal derivation of the fault-tolerance claim contains a sign error that must be fixed.
[Section 6] CLASP is presented as a key contribution, but its only evidence is a toy model with normally distributed losses and an assumed 10% loss increase for malicious miners; the two core assumptions (tamper-proof loss reporting and no cross-layer collusion) are stated without validation. Since the paper explicitly says CLASP is not included in the initial release, the abstract should not list it as a key result without a clear caveat that it is speculative.

minor comments (6)

[Section 5, Eq. (1)] The definition of P is incomplete: it should specify 0 ≤ i < j ≤ N-1, not just 0 ≤ i ≤ N-1, otherwise the set and its cardinality are undefined.
[Section 5.3] The comparison 'a central weight merger would need to transfer NmW+3' is unclear; please provide the full expression with units.
[Section 3] The reward formula uses t_decay but the text defines a hyperparameter γ; clarify the relationship between the two.
[Figure 5] Add a legend that clearly identifies the number of bottlenecks and compression rate for each curve, and state the number of seeds used.
[Appendix A] The numerical simulation is described only by a figure; include the simulation details (parameters, code availability) to make the stability claim reproducible.
[Throughout] Please define all symbols (Bmin, γ, N, Nm, W) in one notation table to improve readability.

Circularity Check

0 steps flagged · score 2.0 of 10

No circularity; the only in-house citation (SN9, [1]) motivates but does not derive any IOTA result, and the central claims are self-contained measured or derived results.

full rationale

IOTA does not derive any of its load-bearing claims from fitted parameters or from the authors' own prior results. The 128x compression rate (Section 4) is defined arithmetic (2048-dim fp32 activations to 32-dim bf16 bottleneck gives 64x times 2x = 128x), and the convergence claim is a reported measurement (Figure 5) explicitly flagged by the authors as preliminary and needing validation up to 100B tokens. The Butterfly All-Reduce O(1) bandwidth claim (Section 5.3) is derived from the protocol's own transfer counts (4W + 2W/Nm), and the redundancy formula pvalid follows combinatorially from C(N,2) - C(k,2). CLASP (Section 6) is honestly presented as a toy model whose injected 10% loss increase yields the outlier detection shown; it is not masqueraded as a fitted prediction. The incentive formula in Section 3 is a policy definition, not a discovered equivalence. The only in-house citation, reference [1] to the authors' SN9 report, is used as motivation for why IOTA is needed, and none of the equations or scaling claims depends on it. Concerns about a single 1.5B/400M-token run, no error bars, and no released code are correctness risks, not circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central claim rests on several unvalidated assumptions: compression preserves convergence over long horizons, validators can reliably verify miners, and incentive mechanisms are stable. The only free parameters are hand-chosen hyperparameters and toy-model constants; no inference from data is performed.

free parameters (3)

Bmin (BATCHES_BEFORE_MERGING) = not specified
Hyperparameter controlling how many batches a miner must complete before merging; hand-chosen by the orchestrator in Section 2.1.
Gamma (temporal decay period) = not specified
Hyperparameter in the incentive scheme (Section 3) that determines how long a miner retains score; chosen by design, not inferred from data.
CLASP toy model parameters = mean=4.5, std=0.2, malicious increase=10%
Arbitrary values chosen for the simulation in Section 6 to illustrate outlier detection; not derived from real training runs.

assumptions (4)

domain assumption Stochastic gradient descent with locally compressed activations and infrequent synchronization (DiLoCo) preserves convergence
Assumed throughout Sections 2.1 and 4, based on cited work [6, 15]; not re-derived for the IOTA pipeline.
ad hoc to paper Cosine similarity between validator-reproduced and miner-submitted activations is a reliable honesty signal
Introduced in Section 2.3 without empirical validation of threshold behavior or noise characteristics.
ad hoc to paper Miners cannot effectively collude across layers because they only see black-box activations
Stated in Section 6 as a core assumption for CLASP; no adversarial model or proof is given.
domain assumption Residual connections are crucial for gradient flow and their disruption explains convergence degradation
Cited to a practitioner book [14]; used to justify the bottleneck block design in Section 4.

invented entities (1)

Bottleneck transformer block with partial residual flow
purpose: Compress inter-device activations and gradients by up to 128x
Evaluated only in a short internal experiment (Figure 5); no external replication or code; the paper itself calls results preliminary.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Incentivised Orchestrated Training Architecture (IOTA): A Technical Primer for Release." pith.science (2026). https://pith.science/paper/LRUDIIH5

@misc{pith2026250717766,
  author       = {Pith},
  title        = {Pith review of: Incentivised Orchestrated Training Architecture (IOTA): A Technical Primer for Release},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LRUDIIH5}},
  note         = {Machine review of arXiv:2507.17766}
}

read the original abstract

In August 2024, Bittensor's Subnet 9 (SN9) demonstrated that a distributed network of incentivized, permissionless actors could each pretrain large language models (LLMs) ranging from 700 million to 14 billion parameters, while surpassing established baselines. While that work validated blockchain-based decentralized pretraining as viable, it contained core issues: (i) every miner had to fit an entire model locally, and (ii) "winner-takes-all" rewards encouraged model hoarding. Here we introduce IOTA (Incentivized Orchestrated Training Architecture), an architecture that addresses these limitations by transforming SN9's previously isolated competitors into a single cooperating unit that can scale arbitrarily while still rewarding each contributor fairly. Key preliminary results: (1) Data- and Pipeline-parallel SWARM architecture - An orchestrator distributes model layers across heterogeneous miners and streams activations between them, enabling model sizes to scale with the number of participants rather than being constrained by the VRAM of a single machine; (2) Granular, continuous incentives - Validators measure each miner's contribution and allocate token emissions proportionally; (3) Activation compression - We used model-bottlenecks to cut communication bandwidths of activations by up to 128x, vastly improving training speed; (4) Butterfly All-Reduce - Miners average disjoint parameter slices in O(1) bandwidth, offering linear scalability, redundancy and built-in collusion detection; (5) CLASP (Contribution Loss Assessment via Sampling of Pathways) - A fair attribution scheme assigns credit to miners proportional to their marginal utility and detects exploits, even when contributions are interdependent across the pipeline.

Figures

Figures reproduced from arXiv: 2507.17766 by the authors.

**Figure 1.** Overall system architecture. The orchestrator facilitates the training process by triggering miners to work on [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

**Figure 2.** Timeline of validator-miner tracking, training, compressed sharing, and syncing blocks. Each stage of the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** The two primary stages of the IOTA network. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Bottleneck-LLama with uninterrupted residual flow. Residual connections are partially fed into output [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Convergence rate comparison. Early training loss of the baseline Llama3.2-1.5B model compared to our [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: System schematic of Butterfly All Reduce, utilizing an S3 bucket as an intermediate data storage solution. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

**Figure 7.** Figure 7: a) Illustrating the agreement matrix between miners and b) the resilience of miner failures for Butterfly [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

**Figure 8.** Figure 8: Experiment results for loss contributions using CLASP. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

**Figure 9.** Figure 9: Incentive Stability in Dependence of Monitoring time and Weight Decay Period [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 19 canonical work pages

[1]

LLM pretraining: The use-case blockchain has been waiting for?, 2024

Macrocosmos AI, Taoverse, Const, and Datura. LLM pretraining: The use-case blockchain has been waiting for?, 2024

work page 2024
[2]

Sparse communication for distributed gradient descent.CoRR, abs/1704.05021, 2017

Alham Fikri Aji and Kenneth Heafield. Sparse communication for distributed gradient descent.CoRR, abs/1704.05021, 2017

arXiv 2017
[3]

Bittensor: A peer-to-peer intelligence market, 2023

Bittensor. Bittensor: A peer-to-peer intelligence market, 2023

work page 2023
[4]

Machine learning with adversaries: Byzantine tolerant gradient descent

Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. InAdvances in Neural Information Processing Systems 30 (NeurIPS 2017), 2017

work page 2017
[5]

Aggregathor: Byzantine machine learning via robust gradient aggregation

Georgios Damaskinos, El Mahdi El Mhamdi, Rachid Guerraoui, Arsany Guirguis, and Sébastien Rouault. Aggregathor: Byzantine machine learning via robust gradient aggregation. InProceedings of the 1st SysML Conference, Palo Alto, CA, USA, 2019. SysML Conference. Copyright 2019 by the author(s)

work page 2019
[6]

Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc’Aurelio Ranzato, Arthur Szlam, and Jiajun Shen

Arthur Douillard, Qixuan Feng, Andrei A. Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc’Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low-communication training of language models, 2024

work page 2024
[7]

Secure distributed training at scale

Eduard Gorbunov, Alexander Borzunov, Michael Diskin, and Max Ryabinin. Secure distributed training at scale. InProceedings of the 39th International Conference on Machine Learning, volume 162 ofProceedings of Machine Learning Research, Baltimore, Maryland, USA, 2022. PMLR

work page 2022
[8]

The llama 3 herd of models, 2024

Aaron Grattafiori, Abhimanyu Dubey, and Abhinav Jauhri. The llama 3 herd of models, 2024

work page 2024

Show all 24 references

[9]

Intellect-1: Decentralised large-model training, 2024

Prime Intellect. Intellect-1: Decentralised large-model training, 2024. White paper

2024
[10]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. InAdvances in Neural Information Processing Systems 25, pages 1097–1105. Curran Associates, 2012

2012
[11]

The fineweb datasets: Decanting the web for the finest text data at scale

Guilherme Penedo, Hynek Kydlíˇcek, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Lean- dro V on Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processin...

2024
[12]

Adil, and Esteban La Rocca

Bowen Peng, Jeffrey Quesnelle, Dillon Rolnick, Ari Lotter, Umer H. Adil, and Esteban La Rocca. A preliminary report on distro.https://github.com/NousResearch/DisTrO, 2024. Nous Research Technical Report

2024
[13]

Kakade, and Zaid Harchaoui

Krishna Pillutla, Sham M. Kakade, and Zaid Harchaoui. Robust aggregation for federated learning.IEEE Transactions on Signal Processing, 70:1142–1154, 2022

2022
[14]

Packt Publishing Ltd, 2021

Denis Rothman.Transformers for Natural Language Processing: Build innovative deep neural network ar- chitectures for NLP with Python, PyTorch, TensorFlow, BERT, RoBERTa, and more. Packt Publishing Ltd, 2021

2021
[15]

Swarm parallelism: Training large models can be surprisingly communication-efficient

Max Ryabinin, Tim Dettmers, Michael Diskin, and Alexander Borzunov. Swarm parallelism: Training large models can be surprisingly communication-efficient. InInternational Conference on Machine Learning, pages 29416–29440. PMLR, 2023

2023
[16]

Moshpit SGD: communication-efficient decentralized training on heterogeneous unreliable devices.CoRR, abs/2103.03239, 2021

Max Ryabinin, Eduard Gorbunov, Vsevolod Plokhotnyuk, and Gennady Pekhimenko. Moshpit SGD: communication-efficient decentralized training on heterogeneous unreliable devices.CoRR, abs/2103.03239, 2021

2021 arXiv
[17]

A value for n-person games

Lloyd S Shapley. A value for n-person games. In Harold W. Kuhn and Albert W. Tucker, editors,Contributions to the Theory of Games II, pages 307–317. Princeton University Press, Princeton, 1953

1953
[18]

Richard S. Sutton. The bitter lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html,
[19]

Templar-i: Trustless distributed training for 1b-parameter language models, 2024

Templar. Templar-i: Trustless distributed training for 1b-parameter language models, 2024. Technical report

2024
[20]

Byzantine-robust distributed learning: Towards optimal statistical rates

Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. Byzantine-robust distributed learning: Towards optimal statistical rates. InProceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of Machine Learning Research, pages 5650–565...

2018
[21]

Butterfly mixing: Accelerating incremental-update algorithms on clusters.CoRR, 2013

Huasha Zhao. Butterfly mixing: Accelerating incremental-update algorithms on clusters.CoRR, 2013. 12 Incentivised Orchestrated Training Architecture (IOTA): A Technical Primer for Release A Appendix A: Numerical Simulation of Incentive Stability A.1 Throughput Efficiency and v...

2013
[22]

, N}be the set of miners in the distributed training system, arranged into layers

LetN={1,2, . . . , N}be the set of miners in the distributed training system, arranged into layers
[23]

Each samplex k follows a pathwayπ k ⊆ Nthrough a subset of miners, limited to one miner per layer
[24]

, T} whereπ k is the set of miners that processed samplek, andℓ k is the resulting loss

Each sample produces a lossℓ k ≥0 After processingTsamples, the orchestrator maintains records: D={(π k, ℓk) :k= 1,2, . . . , T} whereπ k is the set of miners that processed samplek, andℓ k is the resulting loss. For each mineri∈ N, compute theiraverage loss: 13 Incentivised O...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

LLM pretraining: The use-case blockchain has been waiting for?, 2024

Macrocosmos AI, Taoverse, Const, and Datura. LLM pretraining: The use-case blockchain has been waiting for?, 2024

work page 2024

[2] [2]

Sparse communication for distributed gradient descent.CoRR, abs/1704.05021, 2017

Alham Fikri Aji and Kenneth Heafield. Sparse communication for distributed gradient descent.CoRR, abs/1704.05021, 2017

arXiv 2017

[3] [3]

Bittensor: A peer-to-peer intelligence market, 2023

Bittensor. Bittensor: A peer-to-peer intelligence market, 2023

work page 2023

[4] [4]

Machine learning with adversaries: Byzantine tolerant gradient descent

Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. InAdvances in Neural Information Processing Systems 30 (NeurIPS 2017), 2017

work page 2017

[5] [5]

Aggregathor: Byzantine machine learning via robust gradient aggregation

Georgios Damaskinos, El Mahdi El Mhamdi, Rachid Guerraoui, Arsany Guirguis, and Sébastien Rouault. Aggregathor: Byzantine machine learning via robust gradient aggregation. InProceedings of the 1st SysML Conference, Palo Alto, CA, USA, 2019. SysML Conference. Copyright 2019 by the author(s)

work page 2019

[6] [6]

Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc’Aurelio Ranzato, Arthur Szlam, and Jiajun Shen

Arthur Douillard, Qixuan Feng, Andrei A. Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc’Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low-communication training of language models, 2024

work page 2024

[7] [7]

Secure distributed training at scale

Eduard Gorbunov, Alexander Borzunov, Michael Diskin, and Max Ryabinin. Secure distributed training at scale. InProceedings of the 39th International Conference on Machine Learning, volume 162 ofProceedings of Machine Learning Research, Baltimore, Maryland, USA, 2022. PMLR

work page 2022

[8] [8]

The llama 3 herd of models, 2024

Aaron Grattafiori, Abhimanyu Dubey, and Abhinav Jauhri. The llama 3 herd of models, 2024

work page 2024