Pith. sign in

REVIEW 3 major objections 7 minor 14 cited by

INTELLECT-1 Technical Report

T0 review · 3 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper reports training a 10-billion-parameter language model across three continents over ordinary internet links, at 83 to 96 percent compute utilization, and claims this makes community-run training of large models feasible.

desk verdict First credible 10B-scale decentralized training run with open artifacts and honest engineering reporting; feasibility holds, but 'comparable performance' is unproven without a matched baseline. read the letter →

arxiv 2412.01152 v1 pith:3UBC6H7B submitted 2024-12-02 cs.DC

Sami Jaghouar , Jack Min Ong , Manveer Basra , Fares Obeid , Jannik Straube , Michael Keiblinger , Elie Bakouch , Lucas Atkins
show 4 more authors
Maziyar Panahi Charles Goddard Max Ryabinin Johannes Hagemann
This is my paper · ORCID
classification cs.DC
keywords decentralizedtrainingdistributedDiLoColow-communicationint8quantizationfaulttolerance10Blanguagemodelcommunity-drivencompute
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

INTELLECT-1 is, by the authors' account, the first 10-billion-parameter language model trained collaboratively across the globe: 30 independent compute providers contributed up to 112 H100 GPUs spread over three continents, and the run completed 1 trillion tokens in 42 days over ordinary internet links. The central claim is that this was not a wasteful exercise — compute utilization stayed at 83 percent in the fully global configuration and 96 percent when nodes were confined to the United States, with model-FLOPS utilization between 36.2 and 41.4 percent. The efficiency comes from synchronizing nodes only once every 100 local training steps and compressing the exchanged gradient information to 8-bit integers, cutting communication volume by a factor of roughly 400 compared with standard data-parallel training. A reader should care because this challenges the assumption that frontier-scale model training requires a single tightly coupled data center, and it turns community-pooled, volunteer compute into a plausible route for open models.

What carries the argument

The load-bearing object is PRIME's ElasticDeviceMesh, a hybrid topology in which every process carries a local rank for fast intra-node collectives and a global rank for a fault-tolerant data-parallel group that spans the internet. Around that mesh, three mechanisms do the real work: the DiLoCo algorithm, which lets each worker run roughly 38 minutes of local AdamW steps before any cross-node communication; the custom int8 ring-all-reduce, which quantizes only the pseudo-gradients (temporal differences between model states, not raw weights) to 8 bits while accumulating in full precision; and a bandwidth-aware ring reordering that continuously solves a maximin Hamiltonian-cycle problem so the ring's slowest link is as fast as possible. A CPU-hosted outer optimizer with Nesterov momentum performs the infrequent global update, so GPU memory is not taxed, and heartbeat-based eviction plus peer-to-peer checkpoint transfer let nodes join and leave mid-run without stopping training.

What would settle it

Train the identical Llama-3-style 10-billion-parameter, 42-layer configuration on precisely the same 1-trillion-token data mix in a centralized data center (or with DiLoCo but full-precision gradients), evaluate on the same eight benchmarks, and compare: a gap of more than a few points on average would refute the claim that the decentralized int8 pipeline preserves centralized quality. A secondary check is to track the int8 pseudo-gradient quantization error over the run — if it grows systematically from step 1,000 to step 80,000, the quantization scheme is accumulating drift and the model's convergence is being silently degraded.

Watch

Extended reading notes

Core claim

The paper's discovery, stated on its own terms, is that the combination of three mechanisms — fully sharded data parallelism inside each node, the DiLoCo local-SGD scheme across nodes with 100 local steps per synchronization, and int8-quantized pseudo-gradient transmission — sustained a full-scale production training run over networks three orders of magnitude slower than a typical HPC cluster. INTELLECT-1, a Llama-3-based model with 42 layers and 10 billion parameters, converged smoothly on 1 trillion tokens while nodes joined, left, and failed, and its benchmark results on MMLU, HellaSwag, GSM8K, and similar suites land roughly in the range of similarly sized centralized open models such as MPT-7B, Falcon-7B, and LLaMA-7B. The authors read these results as evidence that decentralized training is no longer confined to small-scale experiments but is a viable way to pool globally distributed GPU resources for training frontier foundation models.

Load-bearing premise

The claim that decentralized training matches centralized quality rests on comparing INTELLECT-1's benchmarks against other labs' models trained on other data: if the same 10-billion-parameter architecture trained centrally on the very same 1-trillion-token mixture would score clearly higher, the 'comparable performance' conclusion fails even though the efficiency numbers would survive.

Editorial extensions

If this is right

  • Frontier-scale training no longer requires a single data center: a 1-trillion-token, 10-billion-parameter run completed over ordinary internet links at 83 percent global compute utilization, so pooled volunteer GPUs are a credible path to large open models.
  • A roughly 400-fold reduction in communication volume means internet-bandwidth training is feasible where InfiniBand-class interconnect was previously assumed mandatory, provided synchronization happens only every hundred steps.
  • Training survives churn: the run went from 4 to 14 nodes with providers joining and leaving, and only a simultaneous loss of 4 of 12 nodes forced a checkpoint restart, so practical systems must still plan for correlated mass failures.
  • The hybrid recipe — FSDP inside a node, DiLoCo between nodes, int8-compressed deltas on the wire — is reusable, and the released code, checkpoints, data, and public dashboard make it reproducible.
  • Open-source AI gains a counterweight to concentrated compute: if the efficiency holds, community-pooled training could mobilize compute at a scale that individual closed labs cannot match on their own.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the report does not isolate how much of the quality retention comes from the 100-step interval versus the int8 delta quantization; a standalone ablation of int8 pseudo-gradient quantization with the synchronization frequency held fixed would separate the two effects.
  • Editorial extension: the benchmark comparison varies architecture, data, and procedure at once, so 'comparable performance' is a placeholder until a same-architecture, same-data centralized baseline is trained; the efficiency claims, however, do not depend on that baseline.
  • Editorial extension: whether the scheme scales to 100-billion-parameter models is untested here, since synchronization volume grows with parameter count and the 400-fold communication reduction may or may not survive at that scale.
  • Editorial extension: the maximin ring-reordering idea is a general recipe for any internet-scale collective operation, but its standalone contribution is entangled with quantization, checkpointing, and the VPN in this report and is therefore unquantified.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper presents INTELLECT-1, a 10-billion-parameter Llama-3-variant language model trained on 1 trillion tokens using up to 14 nodes distributed across three continents, and PRIME, the distributed training framework used for the run. The main technical claims are: (i) a hybrid FSDP/DiLoCo implementation with an int8 ring all-reduce reduces inter-node communication by 400x relative to standard data-parallel training; (ii) the system tolerates dynamic node joins and departures while sustaining 83-96% compute utilization and 36.2-41.4% model FLOPS utilization; and (iii) the resulting model achieves benchmark performance comparable to centrally trained open models of similar size. The paper includes detailed timing measurements, node-fluctuation statistics, loss curves, and open-source releases of the model, checkpoints, data, and framework.

Significance. If the feasibility claim holds, this is a valuable empirical demonstration: it shows that a 10B-parameter model can be trained over ordinary internet connections with modest wall-clock efficiency loss, which is relevant to community-driven and decentralized training efforts. The paper is strong on reproducibility artifacts: the training dashboard, checkpoints, data, and code are all released, and the measurements of all-reduce times, compute utilization, and node dynamics are reported in unusual detail. The authors also honestly report limitations, including a checkpoint resume after a simultaneous four-node failure. The load-bearing weakness is that the 'comparable performance' claim in the abstract and conclusion is not directly supported by the evaluation, which lacks a matched centralized baseline for the same architecture, data mix, and token budget.

major comments (3)
  1. [§3.6, Table 3] The central claim that decentralized DiLoCo+int8 training delivers 'comparable performance' to centralized training is not supported by the evidence in Table 3. INTELLECT-1 is a 42-layer Llama-3-based 10B model trained on 1T tokens with a specific data mix and WSD schedule, yet it is compared only to MPT-7B, Falcon-7B, Pythia-12B, LLM360-Amber, and LLaMA-7B/13B models that differ in architecture, depth, data composition, token counts, and training recipes. Observed gaps such as HellaSwag 72.26 vs. LLaMA-7B 78.19 and WinoGrande 65.82 vs. LLaMA-7B 72.06 cannot be attributed to the decentralized method without a matched baseline. I request either a same-architecture, same-data, same-optimizer centralized control (at least at a smaller scale or shorter token budget, if full-scale is infeasible) or a revision of the abstract and conclusion to claim 'stable convergence with competitive absolute benchmark scores' rather than 'comparable performance.'
  2. [§3.4, Figure 6] The loss curve in Figure 6 demonstrates stable convergence but does not establish equivalence to centralized training. A smooth perplexity decrease is consistent with a range of possible quality degradations from the H=100 inner-step interval, int8 pseudo-gradient quantization, and dynamic node membership. Without overlaying a centralized-training loss curve on the same data and architecture, the figure cannot serve as evidence for the 'comparable performance' claim; it only supports the weaker claim of training stability.
  3. [§3.5, Table 4] The post-trained evaluation in Table 4 is also not informative for the decentralized-pretraining quality claim. The post-training pipeline includes 16 SFT runs, 8 DPO runs, 16 model merges, and distillation using Llama-3.1-405B logits; these interventions can substantially change benchmark scores independently of the base model. Comparing INTELLECT-1-instruct to MPT-7B-Chat or LLaMA2-7B-chat therefore conflates differences in base pretraining, post-training data, and post-training methodology. The base-model results in Table 3 are the relevant evidence, and they suffer from the missing matched baseline described above.
minor comments (7)
  1. [Abstract and §3.6] The abstract says 'delivering comparable performance,' while §3.6 describes the results as 'promising' and notes the model 'still lags behind current state-of-the-art models'; please align these characterizations to avoid overclaiming.
  2. [§2.2 heading] There is a typo in the heading: 'commmunication' should be 'communication.'
  3. [Table 3 caption] The caption says 'comparable amounts of total tokens,' but the listed models range from 300B tokens (Pythia-12B) to 2T tokens (LLaMA2-7B/13B); please say 'models of similar parameter count' or give exact token counts.
  4. [§3.1] The paper says 'up to eight non-colocated data centers' in §3.1 but the introduction and Figure 5 refer to up to 14 nodes and 112 H100 GPUs; please clarify the relationship between data centers, nodes, and GPUs.
  5. [Table 2] The column 'Inner step time, min' is ambiguous: it appears to be the time for 100 inner steps (the DiLoCo outer interval), not a single step. Please rename it accordingly or add a footnote.
  6. [Figure 6] The left panel is labeled 'Perplexity' but the text refers to loss; please specify whether this is training-set perplexity, held-out perplexity, or something else.
  7. [§2.2] The quantization method is attributed to 'Ryabinin et al. (2020),' but the reference list entry is the Hivemind GitHub repository rather than the associated paper; please cite the appropriate publication or clarify the source.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the 400x bandwidth reduction is arithmetic from the stated H=100 and int8 quantization, and performance claims rest on external benchmarks.

full rationale

The paper's central quantities are either measured or arithmetically derived. The 400x communication reduction claimed in the abstract and Section 3.1 is the product of the stated DiLoCo interval H=100 ('we chose a conservative number of 100 inner steps between each DiLoCo synchronization') and the int8 quantization factor of 4 ('Communicating quantized int8 values instead of the original fp32 values results in a 4 times reduction in communication payload'), i.e., 100 x 4 = 400; this is a bookkeeping calculation, not a fitted parameter renamed as a prediction. Convergence and 'comparable performance' are supported by measured loss curves (Figure 6) and by benchmark comparisons to externally trained models (MPT-7B, Falcon-7B, LLaMA-7B/13B, etc.) in Table 3, so the central empirical claim is not derived from the paper's own inputs. The one self-citation, 'DiLoCo has been shown to enable training across poorly connected devices while maintaining convergence comparable to traditional distributed training (Jaghouar et al., 2024)', is a background motivation; the report's own 1T-token run and external evaluations carry the load, so removing it would not collapse the argument. The absence of a same-architecture centrally trained control (Section 3.6, Table 3) is a genuine limitation on the strength of the 'comparable performance' phrasing, but it is an experimental-design/validity concern, not a case of a prediction being equivalent to its inputs by construction. Likewise, the acknowledged node-loss instability (Section 3.3) and timeout-consistency open challenge (Section 2.4.5) are operational limitations with no circularity content. No fitted input is relabeled as a prediction, no uniqueness theorem is imported from the authors, and no known result is merely renamed.

Assumptions & free parameters 5 free parameters · 3 assumptions · 2 invented entities

The central claim of feasibility depends on several hand-chosen training and quantization hyperparameters (H, clip multiplier, learning rates), which are not derived from first principles. The convergence of DiLoCo and the robustness of int8 pseudo-gradient quantization are assumed from prior work rather than proven here. The main invented entities are software abstractions (ElasticDeviceMesh, PRIME), which have independent evidence through open-source code and the actual training run.

free parameters (5)
  • Inner steps H = 100
    Number of local optimizer steps between DiLoCo outer synchronization. Chosen conservatively; directly sets the communication reduction factor (400x with int8) and the all-reduce interval.
  • Int8 quantization clip multiplier = 6 (sigma)
    Pseudo-gradients are clipped to [mu - 6*sigma, mu + 6*sigma] before uniform int8 quantization. Hand-set multiplier affects quantization error and convergence.
  • Outer Nesterov learning rate = 0.7
    Learning rate for the outer DiLoCo optimizer; taken from the DiLoCo recipe, not derived.
  • Outer Nesterov momentum = 0.9
    Momentum for the outer optimizer; from prior DiLoCo work.
  • Inner learning rate = 7.5e-5
    AdamW LR for inner steps; standard training hyperparameter.
assumptions (3)
  • domain assumption DiLoCo converges comparably to data-parallel training at scale
    The paper relies on prior work (Douillard et al. 2024; Jaghouar et al. 2024) for DiLoCo convergence and assumes it extends to a 10B model trained on 1T tokens. Invoked in Sections 2 and 3.
  • domain assumption Int8 quantization of pseudo-gradients does not significantly degrade training quality
    The paper argues temporal difference quantization is robust, but the claim is supported only by the single training run and prior gradient quantization work (Ryabinin et al. 2020). Invoked in Section 2.2.
  • domain assumption Network conditions during the run (Tailscale VPN, intercontinental links) are representative of internet-scale training environments
    The observed bandwidths of 500 Mb/s to 4 Gb/s and node churn statistics are treated as evidence of general feasibility. Invoked in Sections 2.5 and 3.2.
invented entities (2)
  • ElasticDeviceMesh independent evidence
    purpose: Abstraction managing dynamic global process groups for fault-tolerant inter-node communication and local FSDP process groups for intra-node communication
    The framework is open-sourced at github.com/PrimeIntellect-ai/prime and was used in the training run. It is a software abstraction rather than a physical entity, but it is observable and testable.
  • PRIME framework independent evidence
    purpose: Distributed training framework for fault-tolerant, low-communication training over unreliable global nodes
    Open-source code and the training run constitute evidence of existence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of INTELLECT-1 Technical Report." pith.science (2026). https://pith.science/paper/3UBC6H7B

@misc{pith2026241201152,
  author       = {Pith},
  title        = {Pith review of: INTELLECT-1 Technical Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3UBC6H7B}},
  note         = {Machine review of arXiv:2412.01152}
}
read the original abstract

In this report, we introduce INTELLECT-1, the first 10 billion parameter language model collaboratively trained across the globe, demonstrating that large-scale model training is no longer confined to large corporations but can be achieved through a distributed, community-driven approach. INTELLECT-1 was trained on 1 trillion tokens using up to 14 concurrent nodes distributed across 3 continents, with contributions from 30 independent compute providers dynamically joining and leaving the training process, while maintaining 83-96% compute utilization and 36.2-41.4% model FLOPS utilization. We leverage PRIME, our scalable distributed training framework designed for fault-tolerant, high-performance training on unreliable, globally distributed nodes. Key innovations in PRIME include the ElasticDeviceMesh, which manages dynamic global process groups for fault-tolerant communication across the internet and local process groups for communication within a node, live checkpoint recovery kernels, and a hybrid DiLoCo-FSDP2 implementation. Using PRIME with DiLoCo and our custom int8 all-reduce, we achieve a 400x reduction in communication bandwidth compared to traditional data-parallel training settings while delivering comparable performance. These results demonstrate the feasibility and promise of training frontier foundation models in a decentralized network of global GPU resources.

Figures

Figures reproduced from arXiv: 2412.01152 by the authors.

Figure 1
Figure 1. The topology of the ElasticDeviceMesh. Each process in the ElasticDeviceMesh is assigned a local and global rank. The local rank is used by the FSDP process groups, while the global rank is used by an independent fault-tolerant data-parallel process group. This hybrid approach allows us to leverage both the memory efficiency of FSDP for local training and the communication efficiency of DiLoCo for cross-node synchro… view at source ↗
Figure 2
Figure 2. Locations of the nodes by all 30 compute contributors for [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Distribution of all-reduce operation times across different geographical configurations. The [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Distribution of all-reduce completion times across different geographical setups. The [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Number of active training nodes over training steps. The graph demonstrates [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Training dynamics showing model perplexity and learning rate over training steps, including [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 14 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Can Model Merging Improve Aggregation in DiLoCo?

    cs.LG 2026-07 accept novelty 7.0 of 10

    IsoLoCo, isotropic outer aggregation with Nesterov momentum, outperforms DiLoCo on FLOP-matched LLM pre-training, with the gap widening as the number of workers increases.

  2. Overcoming the Communication-Performance Tradeoff in LLM Pretraining

    cs.LG 2025-08 conditional novelty 7.0 of 10

    SparseLoCo combines error feedback with Top-k sparsification and 2-bit quantization to send 1-3% of the pseudo-gradient during LLM pre-training while matching or beating DiLoCo's dense updates.

  3. Decentralised AI Training and Inference with BlockTrain

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    BlockTrain partitions models into blocks trained on local objectives, reaching CE 1.359 on WikiText within 0.04 of end-to-end baseline while enabling distributed training and inference over TCP for up to 75B-parameter models.

  4. Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models

    cs.CL 2025-09 conditional novelty 6.0 of 10

    TraceRL aligns the RL objective with the DLM's actual step-by-step decoding, producing TraDo-4B/8B models that beat autoregressive baselines on math reasoning.

  5. Compute Requirements for Algorithmic Innovation in Frontier AI Models

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Estimated development compute for 36 LLM pretraining innovations shows half would remain possible under GPT-2-level or 8-H100 compute caps.

  6. DICE: Data Influence Cascade in Decentralized Learning

    cs.LG 2025-07 conditional novelty 6.0 of 10

    DICE defines and approximates multi-hop data influence in decentralized learning, showing that influence is shaped by data, topology, and loss curvature.

  7. NoLoCo: No-all-reduce Low Communication Training Method for Large Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    NoLoCo trains large language models without any all-to-all synchronization by using pairwise weight averaging and random pipeline routing, matching or slightly beating DiLoCo in experiments.

  8. Incentivizing Permissionless Distributed Learning of LLMs

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A deployed incentive mechanism rewarded pseudo-gradient updates with tokens and produced a competitive 1.2B LLM via permissionless distributed training on Bittensor.

  9. Prime Collective Communications Library -- Technical Report

    cs.DC 2025-05 conditional novelty 6.0 of 10

    PCCL is a fault-tolerant collective communication library that uses a master-coordinated micro-consensus protocol to support dynamic peer membership and bit-exact state synchronization for distributed ML over the internet.

  10. Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Streaming DiLoCo trains billion-parameter LLMs at data-parallel quality while cutting the inter-datacenter bandwidth by about two orders of magnitude via partial, overlapped, and 4-bit-quantized synchronization.

  11. Layer-Parallel Inference Reduces Encrypted Nonlinear Depth in Transformers

    cs.LG 2026-07 conditional novelty 5.5 of 10

    SNLP reduces symbolic FHE bootstraps from 53 to 20 on a 0.5B model with +1.2% PPL degradation and lower polynomial-error amplification than sequential inference.

  12. Distributed and Decentralised Training: Technical Governance Challenges in a Shifting AI Landscape

    cs.CY 2025-07 conditional novelty 5.0 of 10

    A policy analysis distinguishing distributed and decentralised AI training, arguing decentralised training may erode detectability and shutdownability while compute controls remain relevant.

  13. MuLoCo: Muon is a practical inner optimizer for DiLoCo

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Using Muon instead of AdamW inside DiLoCo improves worker scaling and critical batch size for LLM pre-training across 150M to 15B parameters.

  14. From Efficiency Gains to Rebound Effects: The Problem of Jevons' Paradox in AI's Polarized Environmental Debate

    cs.CY 2025-01 conditional novelty 4.0 of 10

    Applying Jevons' Paradox to AI, the paper argues that rebound effects mean efficiency gains may raise, not lower, AI's total environmental footprint.

Reference graph

Works this paper leans on

38 extracted references · 5 canonical work pages · cited by 14 Pith papers

  1. [1]

    The falcon series of open language models, 2023

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of open language models, 2023. URL https://arxiv.org/abs/2311.16867

    arXiv 2023

  2. [2]

    Pythia: A suite for analyzing large language models across training and scaling, 2023

    Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023. URL https://arxiv.org/abs/2304.01373

    arXiv 2023

  3. [3]

    Training transformers together, 2022

    Alexander Borzunov, Max Ryabinin, Tim Dettmers, Quentin Lhoest, Lucile Saulnier, Michael Diskin, Yacine Jernite, and Thomas Wolf. Training transformers together, 2022

    work page 2022

  4. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert - Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litw...

    2020

  5. [5]

    Palm: Scaling language modeling with pathways, 2022

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, et al. Palm: Scaling language modeling with pathways, 2022. URL https://arxiv.org/abs/2204.02311

    arXiv 2022

  6. [6]

    Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https://arxiv.org/abs/1803.05457

    arXiv 2018

  7. [7]

    Training verifiers to solve math word problems, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168

    arXiv 2021

  8. [8]

    Llm.int8(): 8-bit matrix multiplication for transformers at scale, 2022

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int8(): 8-bit matrix multiplication for transformers at scale, 2022. URL https://arxiv.org/abs/2208.07339

    arXiv 2022

Show all 38 references
  1. [9]

    Distributed deep learning in open collaborations

    Michael Diskin, Alexey Bukhtiyarov, Max Ryabinin, Lucile Saulnier, Quentin Lhoest, Anton Sinitsin, Dmitry Popov, Dmitriy Pyrkin, Maxim Kashirin, Alexander Borzunov, Albert Villanova del Moral, Denis Mazur, Ilia Kobelev, Yacine Jernite, Thomas Wolf, and Gennady Pekhimenko. Dist...

    2021

  2. [10]

    Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc'Aurelio Ranzato, Arthur Szlam, and Jiajun Shen

    Arthur Douillard, Qixuan Feng, Andrei A. Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc'Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low-communication training of language models, 2024. URL https://arxiv.org/abs/2311.08105

    2024 arXiv

  3. [11]

    A framework for few-shot language model evaluation, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  4. [12]

    The llama 3 herd of models, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  5. [13]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009.03300

    2021 arXiv

  6. [14]

    Scaling laws and compute-optimal training beyond fixed training durations, 2024

    Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations, 2024. URL https://arxiv.org/abs/2405.18392

    2024 arXiv

  7. [15]

    Opendiloco: An open-source framework for globally distributed low-communication training, 2024

    Sami Jaghouar, Jack Min Ong, and Johannes Hagemann. Opendiloco: An open-source framework for globally distributed low-communication training, 2024. URL https://arxiv.org/abs/2407.07852

    2024 arXiv

  8. [16]

    The stack: 3 tb of permissively licensed source code, 2022

    Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. The stack: 3 tb of permissively licensed source code, 2022. URL http...

    2022 arXiv

  9. [17]

    Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan

    Tanishq Kumar, Zachary Ankner, Benjamin F. Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan. Scaling laws for precision, 2024. URL https://arxiv.org/abs/2411.04330

    2024 arXiv

  10. [18]

    Datacomp-lm: In search of the next generation of training sets for language models, 2024

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, et al. D...

    2024 arXiv

  11. [19]

    Truthfulqa: Measuring how models mimic human falsehoods, 2022

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2022. URL https://arxiv.org/abs/2109.07958

    2022 arXiv

  12. [20]

    Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, Zhiqiang Shen, Xuguang Ren, Rob...

    2023 arXiv

  13. [21]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023

    2023

  14. [22]

    Leela chess zero, 2019

    Gian-Carlo Pascutto and Gary Linscott. Leela chess zero, 2019. URL http://lczero.org/

    2019

  15. [23]

    Openwebmath: An open dataset of high-quality mathematical web text, 2023

    Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023. URL https://arxiv.org/abs/2310.06786

    2023 arXiv

  16. [24]

    The fineweb datasets: Decanting the web for the finest text data at scale, 2024

    Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL https://arxiv.org/abs/2406.17557

    2024 arXiv

  17. [25]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A Graduate-Level Google-Proof Q&A Benchmark , 2023. URL https://arxiv.org/abs/2311.12022

    2023 arXiv

  18. [26]

    Towards crowdsourced training of large neural networks using decentralized mixture-of-experts

    Max Ryabinin and Anton Gusev. Towards crowdsourced training of large neural networks using decentralized mixture-of-experts. In Advances in Neural Information Processing Systems, volume 33, 2020. URL https://proceedings.neurips.cc/paper/2020/file/25ddc0f8c9d3e22e03d3076f98d83c...

    2020

  19. [27]

    H ivemind: D ecentralized D eep L earning in P y T orch, April 2020

    Max Ryabinin, Alexander Borzunov, Michael Diskin, Anton Gusev, Denis Mazur, Vsevolod Plokhotnyuk, Alexey Bukhtiyarov, Pavel Samygin, Anton Sinitsin, and Artem Chumachenko. H ivemind: D ecentralized D eep L earning in P y T orch, April 2020. URL https://github.com/learning-at-h...

    2020

  20. [28]

    Winogrande: An adversarial winograd schema challenge at scale, 2019

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019. URL https://arxiv.org/abs/1907.10641

    2019 arXiv

  21. [29]

    Sebastian U. Stich. Local SGD converges fast and communicates little. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=S1g2JnRcFX

    2019

  22. [30]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them, 2022. URL https://arxiv.org/abs/2210.09261

    2022 arXiv

  23. [31]

    Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023

    MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023. URL www.mosaicml.com/blog/mpt-7b. Accessed: 2023-05-05

    2023

  24. [32]

    Optimization of collective communication operations in mpich

    Rajeev Thakur, Rolf Rabenseifner, and William Gropp. Optimization of collective communication operations in mpich. The International Journal of High Performance Computing Applications, 19: 0 49 -- 66, 2005. URL https://api.semanticscholar.org/CorpusID:90404

    2005

  25. [33]

    Llama: Open and efficient foundation language models, 2023 a

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language...

    2023 arXiv

  26. [34]

    Llama 2: Open foundation and fine-tuned chat models, 2023 b

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, et al. Llama 2: Open foundation and fine-tuned chat models, 2023 b . UR...

    2023 arXiv

  27. [35]

    Baichuan 2: Open large-scale language models, 2023

    Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Haizhou Zhao, Hang Xu, Haoze Sun, Hongda Zhang, Hui Liu, Jiaming Ji, Jian Xie, JunTao Dai, Kun Fa...

    2023 arXiv

  28. [36]

    Hellaswag: Can a machine really finish your sentence?, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019. URL https://arxiv.org/abs/1905.07830

    2019 arXiv

  29. [37]

    Pytorch fsdp: Experiences on scaling fully sharded data parallel, 2023

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch fsdp: Experiences on sc...

    2023 arXiv

  30. [38]

    Instruction-following evaluation for large language models, 2023

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models, 2023. URL https://arxiv.org/abs/2311.07911

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.