A hierarchical asynchronous local SGD method with regional parameter servers and global model merging is claimed to train small LLMs up to 7.5x faster than DiLoCo in simulated geo-distributed settings.
On the choice of learning rate for local sgd
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
HALoS: Hierarchical Asynchronous Local SGD over Slow Networks for Geo-Distributed Large Language Model Training
A hierarchical asynchronous local SGD method with regional parameter servers and global model merging is claimed to train small LLMs up to 7.5x faster than DiLoCo in simulated geo-distributed settings.