Pith. sign in

REVIEW 3 cited by

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1905.13727 v3 pith:CJI7KRLF submitted 2019-05-31 cs.LG cs.DCmath.OCstat.ML

PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization

Thijs Vogels , Sai Praneeth Karimireddy , Martin Jaggi This is my paper
classification cs.LG cs.DCmath.OCstat.ML
keywords compressiongradientachievecommunicationdistributedgradientslow-rankoptimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

We study gradient compression methods to alleviate the communication bottleneck in data-parallel distributed optimization. Despite the significant attention received, current compression schemes either do not scale well or fail to achieve the target test accuracy. We propose a new low-rank gradient compressor based on power iteration that can i) compress gradients rapidly, ii) efficiently aggregate the compressed gradients using all-reduce, and iii) achieve test performance on par with SGD. The proposed algorithm is the only method evaluated that achieves consistent wall-clock speedups when benchmarked against regular SGD with an optimized communication backend. We demonstrate reduced training times for convolutional networks as well as LSTMs on common datasets. Our code is available at https://github.com/epfml/powersgd.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. GIFT: Geometry-Informed Low-precision Gradient Communication for LLM Pretraining

    cs.DC 2026-07 conditional novelty 6.0

    Transforming gradients into K-FAC-based coordinates before FP8 quantization reduces communication error and improves downstream task preservation over Euclidean FP8, with a 7.6% end-to-end speedup on 64 GH200 GPUs.

  2. SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication

    cs.LG 2026-07 conditional novelty 6.0

    SCAPE enables 90-99% sparse gradient communication in sharded Adam-style LLM training by deriving masks from first-moment statistics, achieving up to 43.3% faster pre-training on Llama-500M with no loss in validation ...

  3. DBLP: Phase-Aware Bounded-Loss Transport for Burst-Resilient Distributed ML Training

    cs.LG 2026-05 unverdicted novelty 5.0

    DBLP is a training-phase-aware bounded-loss transport protocol that reduces end-to-end distributed ML training time by 24.4% on average (up to 33.9%) and achieves up to 5.88x communication speedup during microbursts w...