REVIEW 7 cited by
Sparse Communication for Distributed Gradient Descent
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We make distributed stochastic gradient descent faster by exchanging sparse updates instead of dense updates. Gradient updates are positively skewed as most updates are near zero, so we map the 99% smallest updates (by absolute value) to zero then exchange sparse matrices. This method can be combined with quantization to further improve the compression. We explore different configurations and apply them to neural machine translation and MNIST image classification tasks. Most configurations work on MNIST, whereas different configurations reduce convergence rate on the more complex translation task. Our experiments show that we can achieve up to 49% speed up on MNIST and 22% on NMT without damaging the final accuracy or BLEU.
Forward citations
Cited by 7 Pith papers
-
SiFAR: Synchronization-Free All-Reduce for Low-Latency LLM Inference
SiFAR cuts All-Reduce latency up to 52% and end-to-end decode throughput up to 18.6% at TP=8 by dual buffering, in-switch redundant pull, and speculative reduction with validation.
-
Sketched Gaussian Mechanism for Private Federated Learning
A sketched Gaussian mechanism is shown to have privacy loss that shrinks as the sketch dimension grows, giving communication-efficient federated learning with stronger privacy per noise budget.
-
Incentivised Orchestrated Training Architecture (IOTA): A Technical Primer for Release
IOTA is a decentralized, pipeline-parallel LLM training architecture with per-layer incentives, up to 128x activation compression, and an all-reduce merge that tolerates failures, reported with only preliminary validation.
-
Find a Scapegoat: Poisoning Membership Inference Attack and Defense to Federated Learning
FedPoisonMIA uses angularly-masked poisoned gradients to infer membership in federated learning, and the ATM defense reduces its accuracy by trimming directionally-outlying client updates.
-
Multimodal Federated Learning: A Survey through the Lens of Different FL Paradigms
A paradigm-based taxonomy of multimodal federated learning that assigns each branch a headline challenge: modality heterogeneity (horizontal), privacy leakage (vertical), and efficiency (hybrid).
-
Privacy Preserving Conversion Modeling in Data Clean Room
Batch-level aggregated gradients, LoRA adapters, and de-biased label differential privacy let advertisers and platforms train conversion models in a clean room with modest AUC loss and much lower communication cost.
-
Strategies for Improving Communication Efficiency in Distributed and Federated Learning: Compression, Local Training, and Personalization
A PhD dissertation showing unified compression theory, personalized accelerated local training, and pruning methods that reduce communication costs in federated learning and maintain accuracy in LLM pruning.
Discussion (0). Sign in to comment.