Pith. sign in

REVIEW 5 cited by

OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.07852 v1 pith:BBLUMRM7 submitted 2024-07-10 cs.LG cs.DC

Sami Jaghouar , Jack Min Ong , Johannes Hagemann This is my paper
classification cs.LGcs.DC
keywords trainingopendilococomputedilocodistributedeffectivenessframeworkimplementation
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

OpenDiLoCo is an open-source implementation and replication of the Distributed Low-Communication (DiLoCo) training method for large language models. We provide a reproducible implementation of the DiLoCo experiments, offering it within a scalable, decentralized training framework using the Hivemind library. We demonstrate its effectiveness by training a model across two continents and three countries, while maintaining 90-95% compute utilization. Additionally, we conduct ablations studies focusing on the algorithm's compute efficiency, scalability in the number of workers and show that its gradients can be all-reduced using FP16 without any performance degradation. Furthermore, we scale OpenDiLoCo to 3x the size of the original work, demonstrating its effectiveness for billion parameter models.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Not Every Sync Is Safe: Calibrated DiLoCo Scheduling for Shared AI Infrastructure

    cs.DC 2026-06 conditional novelty 6.0 of 10

    Matched-random merge placement is the right control for DiLoCo scheduling claims; only calibrated overlap plus EWMA forecasting beats it in real vLLM sidecar replay (6.54%→5.09% SLO violations).

  2. Decentralised AI Training and Inference with BlockTrain

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    BlockTrain partitions models into blocks trained on local objectives, reaching CE 1.359 on WikiText within 0.04 of end-to-end baseline while enabling distributed training and inference over TCP for up to 75B-parameter models.

  3. DiLoCoX: A Low-Communication Large-Scale Training Framework for Decentralized Cluster

    cs.LG 2025-06 conditional novelty 6.0 of 10

    DiLoCoX combines pipeline parallelism with dual optimizers, delayed communication overlap, and adaptive low-rank plus quantization compression to pre-train a 107B model over 1 Gbps links with a claimed 357x speedup ov...

  4. HALoS: Hierarchical Asynchronous Local SGD over Slow Networks for Geo-Distributed Large Language Model Training

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A hierarchical asynchronous local SGD method with regional parameter servers and global model merging is claimed to train small LLMs up to 7.5x faster than DiLoCo in simulated geo-distributed settings.

  5. MuLoCo: Muon is a practical inner optimizer for DiLoCo

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Using Muon instead of AdamW inside DiLoCo improves worker scaling and critical batch size for LLM pre-training across 150M to 15B parameters.

Pith tools