Revisiting Distributed Synchronous SGD
read the original abstract
Distributed training of deep learning models on large-scale training data is typically conducted with asynchronous stochastic optimization to maximize the rate of updates, at the cost of additional noise introduced from asynchrony. In contrast, the synchronous approach is often thought to be impractical due to idle time wasted on waiting for straggling workers. We revisit these conventional beliefs in this paper, and examine the weaknesses of both approaches. We demonstrate that a third approach, synchronous optimization with backup workers, can avoid asynchronous noise while mitigating for the worst stragglers. Our approach is empirically validated and shown to converge faster and to better test accuracies.
This paper has not been read by Pith yet.
Forward citations
Cited by 5 Pith papers
-
Scalable Distributed Stochastic Optimization via Bidirectional Compression: Beyond Pessimistic Limits
Inkheart SGD and M4 use bidirectional compression to achieve time complexities in distributed SGD that improve with worker count n and surpass prior lower bounds under a necessary structural assumption.
-
The Virtual Patch Clamp: Imputing C. elegans Membrane Potentials from Calcium Imaging
A whole-connectome stochastic simulator of C. elegans is used with SMC to impute membrane potentials from calcium fluorescence on synthetic data.
-
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Linear learning-rate scaling plus warmup lets minibatch size 8192 train ResNet-50 on ImageNet in one hour at full small-batch accuracy.
-
FedRef: Bayesian Fine-Tuning using a Reference Model to Mitigate Catastrophic Forgetting for Heterogeneous Federated Learning
FedRef uses a temporally aggregated reference model and MAP regularization for server-side fine-tuning to reduce forgetting and drift in non-IID federated learning, showing better accuracy and lower client compute on ...
-
A Highly Efficient Distributed Deep Learning System For Automatic Speech Recognition
ADPSGD and Hierarchical-ADPSGD support 3x larger batches than SSGD for ASR, training SWB-2000 to 7.6% WER on SWB and 13.2% on CH in 5.2 hours on 64 V100 GPUs.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.