REVIEW 6 cited by
Understanding Top-k Sparsification in Distributed Deep Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Distributed stochastic gradient descent (SGD) algorithms are widely deployed in training large-scale deep learning models, while the communication overhead among workers becomes the new system bottleneck. Recently proposed gradient sparsification techniques, especially Top-$k$ sparsification with error compensation (TopK-SGD), can significantly reduce the communication traffic without an obvious impact on the model accuracy. Some theoretical studies have been carried out to analyze the convergence property of TopK-SGD. However, existing studies do not dive into the details of Top-$k$ operator in gradient sparsification and use relaxed bounds (e.g., exact bound of Random-$k$) for analysis; hence the derived results cannot well describe the real convergence performance of TopK-SGD. To this end, we first study the gradient distributions of TopK-SGD during the training process through extensive experiments. We then theoretically derive a tighter bound for the Top-$k$ operator. Finally, we exploit the property of gradient distribution to propose an approximate top-$k$ selection algorithm, which is computing-efficient for GPUs, to improve the scaling efficiency of TopK-SGD by significantly reducing the computing overhead. Codes are available at: \url{https://github.com/hclhkbu/GaussianK-SGD}.
Forward citations
Cited by 6 Pith papers
-
Overcoming the Communication-Performance Tradeoff in LLM Pretraining
SparseLoCo combines error feedback with Top-k sparsification and 2-bit quantization to send 1-3% of the pseudo-gradient during LLM pre-training while matching or beating DiLoCo's dense updates.
-
Adaptive Space-efficient Collectives for Dynamic and Unstructured Sparsity on GPU Platforms
Pici-backed adaptive sparse collectives on GPUs deliver up to 5.25×/2.5×/2.66× speedups over dense NCCL for all-gather/reduce-scatter/all-reduce at 99% sparsity.
-
Energy Efficiency Maximization for Hybrid RIS-Aided Communications via Deep Unfolding
Deep-unfolded alternating optimization for hybrid RIS mode selection and binary phases yields ~30% higher energy efficiency than plain projected gradient and most of the gain with only ~10% of elements active.
-
Model Parallelism With Subnetwork Data Parallelism
Training each GPU on a fixed overlapping subnetwork and averaging shared parameters cuts per-device memory by up to 60 percent without exchanging activations, matching DDP accuracy under FLOP-matched budgets.
-
Incentivizing Permissionless Distributed Learning of LLMs
A deployed incentive mechanism rewarded pseudo-gradient updates with tokens and produced a competitive 1.2B LLM via permissionless distributed training on Bittensor.
-
MuLoCo: Muon is a practical inner optimizer for DiLoCo
Using Muon instead of AdamW inside DiLoCo improves worker scaling and critical batch size for LLM pre-training across 150M to 15B parameters.
Discussion (0). Sign in to comment.