Pith. sign in

REVIEW 6 cited by

Sparse Communication for Distributed Gradient Descent

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1704.05021 v2 pith:UOV4RIQH submitted 2017-04-17 cs.CL cs.DCcs.LG

Alham Fikri Aji , Kenneth Heafield This is my paper
classification cs.CLcs.DCcs.LG
keywords updatesconfigurationsgradientmnistsparsedescentdifferentdistributed
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We make distributed stochastic gradient descent faster by exchanging sparse updates instead of dense updates. Gradient updates are positively skewed as most updates are near zero, so we map the 99% smallest updates (by absolute value) to zero then exchange sparse matrices. This method can be combined with quantization to further improve the compression. We explore different configurations and apply them to neural machine translation and MNIST image classification tasks. Most configurations work on MNIST, whereas different configurations reduce convergence rate on the more complex translation task. Our experiments show that we can achieve up to 49% speed up on MNIST and 22% on NMT without damaging the final accuracy or BLEU.

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SiFAR: Synchronization-Free All-Reduce for Low-Latency LLM Inference

    cs.DC 2026-07 accept novelty 6.5 of 10

    SiFAR cuts All-Reduce latency up to 52% and end-to-end decode throughput up to 18.6% at TP=8 by dual buffering, in-switch redundant pull, and speculative reduction with validation.

  2. Sketched Gaussian Mechanism for Private Federated Learning

    cs.LG 2025-09 conditional novelty 6.0 of 10

    A sketched Gaussian mechanism is shown to have privacy loss that shrinks as the sketch dimension grows, giving communication-efficient federated learning with stronger privacy per noise budget.

  3. Incentivised Orchestrated Training Architecture (IOTA): A Technical Primer for Release

    cs.DC 2025-07 conditional novelty 5.0 of 10

    IOTA is a decentralized, pipeline-parallel LLM training architecture with per-layer incentives, up to 128x activation compression, and an all-reduce merge that tolerates failures, reported with only preliminary validation.

  4. Find a Scapegoat: Poisoning Membership Inference Attack and Defense to Federated Learning

    cs.CR 2025-07 conditional novelty 5.0 of 10

    FedPoisonMIA uses angularly-masked poisoned gradients to infer membership in federated learning, and the ATM defense reduces its accuracy by trimming directionally-outlying client updates.

  5. Multimodal Federated Learning: A Survey through the Lens of Different FL Paradigms

    cs.LG 2025-05 conditional novelty 5.0 of 10

    A paradigm-based taxonomy of multimodal federated learning that assigns each branch a headline challenge: modality heterogeneity (horizontal), privacy leakage (vertical), and efficiency (hybrid).

  6. Strategies for Improving Communication Efficiency in Distributed and Federated Learning: Compression, Local Training, and Personalization

    cs.LG 2025-09 conditional novelty 3.0 of 10

    A PhD dissertation showing unified compression theory, personalized accelerated local training, and pruning methods that reduce communication costs in federated learning and maintain accuracy in LLM pruning.

Pith tools