Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read NUQSGD replaces QSGD's uniform grid with logarithmic levels, yielding provably lower communication and matching the fast QSGDinf heuristic.

desk verdict Worth refereeing: the variance and code-length analysis is real and checkable, but the paper's 'provably matches QSGDinf' claim is undercut by an unspecified per-bucket normalization and several unproved theorems. read the letter →

arxiv 1908.06077 v2 pith:OSYJYY2Z submitted 2019-08-16 cs.LG stat.ML

Ali Ramezani-Kebrya , Fartash Faghri , Ilya Markov , Vitalii Aksenov , Dan Alistarh , Daniel M. Roy This is my paper · ORCID
classification cs.LGstat.ML
keywords communication-efficientSGDgradientquantizationnonuniformdata-paralleltrainingunbiasedcompressionvarianceboundQSGDdistributeddeeplearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

NUQSGD is a randomized gradient-compression scheme for data-parallel SGD that replaces the uniform quantization grid used by QSGD with nonuniform logarithmic levels concentrated near zero. The paper's central claim is that this simple change is unbiased and provably reduces both quantization variance and total communication: in the overparameterized regime $d \geq 2^{2s+1}$, the variance multiplier is $\varepsilon_Q = 2^{-s}\sqrt{d} - 7/8$, and the bit cost to reach an $\varepsilon$ suboptimality gap is $O(\varepsilon^{-2}\sqrt{d(d-2^{2s})} \log(\sqrt{d}/2^s))$, strictly better than QSGD's $O(\varepsilon^{-2} d \log \sqrt{d})$. The authors prove this for convex, nonconvex, momentum, asynchronous, and decentralized settings, and report experiments on ImageNet and CIFAR where NUQSGD matches the previously heuristic QSGDinf and beats plain QSGD in accuracy and end-to-end wall time. If the analysis holds, NUQSGD would be the first provably unbiased compression method that is also the empirically preferred one, closing the gap between QSGD theory and QSGDinf practice.

What carries the argument

The load-bearing object is the nonuniform logarithmic quantization grid $\mathcal{L} = \{0, 1/2^s, \ldots, (2^s-1)/2^s, 1\}$ with randomized rounding to the two nearest levels. For a normalized coordinate magnitude $r_i = |v_i|/\|v\|$, the random level $h_i$ equals the lower neighbor with probability that makes $E[h_i] = r_i$, which is exactly the minimum-variance unbiased distribution supported on $\mathcal{L}$. The identity doing the work is the coordinate-wise variance formula $E[\|Q_s(v) - v\|^2] = \|v\|^2 \sum_{i=1}^d \tau(r_i)^2 p(r_i)(1-p(r_i))$, combined with the norm constraint that bounds how many coordinates of a unit vector can fall in each bin; that constraint is what turns a $d$-dimensional sum into $\sqrt{d}$ behavior. The same counting feeds the expected code-length bound via a universal integer code, and the combination of the two bounds yields the communication-complexity theorem.

What would settle it

Construct the vector from Theorem 23, with one coordinate 1 and the rest $\Theta(1/d)$, and measure variance of NUQSGD versus QSGDinf for increasing $d$ and $s$: the inequalities stated there predict NUQSGD wins exactly in that regime, so a mismatch would falsify the variance comparison; independently, rerunning the ImageNet benchmark with each 8192-coordinate bucket explicitly L2-normalized and checking whether accuracy and variance still match QSGDinf would test whether the claimed theory-practice closure depends on the unspecified normalization.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a carefully chosen nonuniform level set changes the worst-case variance of unbiased gradient quantization from order $d$ to order $\sqrt{d}$. For a vector $v$, the quantizer sets $Q_s(v)_i = \|v\| \operatorname{sign}(v_i) h_i$, where each $h_i$ is randomly drawn from the adjacent levels in $\mathcal{L} = (0, 1/2^s, \ldots, (2^s-1)/2^s, 1)$ with probabilities chosen so $E[h_i] = |v_i|/\|v\|$. Theorem 4 bounds the quantization variance by $\varepsilon_Q\|v\|^2$ with the piecewise formula above, and the paper's counting lemma bounds the expected number of nonzero entries by $2^{2s} + \sqrt{d}\,2^s$, which drives both the code-length bound and the communication improvement. The authors also give matching lower bounds, an exact worst-case variance characterization as a quadratically constrained program for arbitrary level sets, and convergence theorems for momentum, asynchronous, and decentralized variants; they conclude that NUQSGD closes the gap between QSGD's theoretical guarantees and QSGDinf's empirical performance.

Load-bearing premise

The variance and communication theorems assume the whole gradient vector is rescaled to unit length before quantization; the experiments quantize gradients in 8192-coordinate chunks and never state what normalization each chunk uses, so the headline guarantee may not cover the code that was actually benchmarked.

Editorial extensions

If this is right

  • In the overparameterized regime, the per-iteration communication bound improves from $O(d \log \sqrt{d})$ to $O(\sqrt{d(d-2^{2s})} \log(\sqrt{d}/2^s))$ bits to reach a given suboptimality gap, so the savings grow with model dimension.
  • Users can take full-precision SGD hyperparameters as-is; the paper's experiments reuse the standard learning rate, momentum, and weight decay and still match QSGDinf accuracy on ImageNet.
  • With Huffman coding on the nonzero positions and levels, the implementation sends about 2.7 bits per component and achieves positive strong scaling on ResNet34, ResNet50, and ResNet152 where full-precision SGD's wall time grows with more nodes.
  • The same NUQSGD core plugs into synchronous, momentum, asynchronous parameter-server, and decentralized averaging algorithms, each with its own convergence theorem, so the compression method is not tied to one topology.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's own claims, the strongest theorem treats each vector as a single unit, so bucket-wise quantization changes the effective dimension and normalization; the experiments do not state the per-bucket normalization rule, which means the headline guarantee may not cover the exact code that was benchmarked.
  • The paper's QCQP and LP machinery for arbitrary level sequences shows the logarithmic grid is nearly but not exactly optimal in some regimes; the same machinery could select data-dependent levels that keep the same code cost but lower variance further, a step the authors leave implicit.
  • Because the lower bound shows any unbiased quantizer with fixed levels has variance at least $\Omega(\sqrt{d})$ for some vectors, the $\sqrt{d}$ scaling is a floor for this class of schemes; further communication reductions would need to exploit gradient structure, sparsity, or error feedback rather than level placement alone.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes NUQSGD, an unbiased nonuniform (logarithmically spaced) gradient quantizer for data-parallel SGD. The main theoretical results are a variance bound (Theorem 4) that scales as O(2^{-s}\sqrt{d}) in the overparameterized regime, a matching lower bound (Theorem 7), an expected code-length bound (Theorem 5), and a resulting bound on the total communication bits needed to reach a specified suboptimality gap (Theorem 13) that improves on the corresponding QSGD bound. Convergence guarantees for convex and nonconvex settings, with momentum, asynchronous, and decentralized variants, are stated as extensions. Experiments on CIFAR10 and ImageNet with ResNet models show that NUQSGD matches the accuracy and variance of the QSGDinf heuristic and outperforms QSGD, and a Horovod implementation demonstrates wall-clock speedups.

Significance. If the results hold, NUQSGD would be the first compression scheme that provably matches the practical performance of the L-infinity-normalized QSGDinf heuristic while retaining QSGD-style variance and code-length guarantees. The variance and code-length bounds are derived from first-principles counting and analytic arguments with no fitted constants; the lower bound in Theorem 7 is a genuine tightness result in the dimension; and the communication bound in Theorem 13 is a clean improvement over QSGD in the overparameterized regime. The empirical study is extensive, covering multiple datasets, models, and baselines. However, the stated theory applies to an L2-normalized whole-vector quantizer, whereas the experiments quantize per bucket without specifying the normalization used, so the central 'closes the gap' claim is not yet substantiated by the written manuscript.

major comments (3)
  1. [Section 5 vs. Definition 3/Theorem 4] The quantizer analyzed in Definition 3 and Theorem 4 normalizes the entire gradient vector by its Euclidean norm, with r_i = |v_i|/||v|| and quantization levels in [0,1]. The experiments in Section 5 quantize gradients per bucket of 8192 coordinates but do not state which norm is used for each bucket. If the implementation uses per-bucket L-infinity normalization, as the QSGDinf heuristic does, then the variance of a quantized bucket is controlled by the squared L-infinity norm of the bucket, not its squared L2 norm, and the bound epsilon_Q ||v_bucket||^2 of Theorem 4 does not follow. Since the central claim is that NUQSGD matches QSGDinf while retaining QSGD-style guarantees, the paper must specify the per-bucket normalization used in the experiments and either prove the corresponding variance bound for that normalization or restrict the theoretical claim to the L2-normalized variant. As written, the bridge between the proof and the benchmarked algorithm is missing.
  2. [Theorems 8, 11, 12, 21, 22] These theorems state convergence guarantees for nonconvex optimization, momentum variants, asynchronous training, and decentralized training, but they are presented without proofs or proof sketches, justified only by the phrase 'along the lines of' prior work. Since these guarantees are listed among the paper's contributions, the manuscript should provide complete proofs in the appendix, or at least a precise derivation showing exactly how the cited theorems combine with the NUQSGD variance bound, including the required assumptions and constants. The current level of detail is not sufficient for a journal publication.
  3. [Section 5, 'Efficient Encoding'] The provable code-length bound in Theorem 5 applies to the Elias-coded ENCODE of Appendix A, but the variant that achieves the reported 2.7 bits per component uses Huffman coding. The text should state clearly that the theoretical bit-count guarantee applies to the Elias-coded scheme, and that Huffman coding is an implementation optimization whose performance is empirical. As written, the sentence 'its code-length guarantees also hold in practice' could be read as claiming the theoretical bound covers the Huffman-coded variant, which is not established.
minor comments (5)
  1. [Section 5] The text uses 'k GPUs' inconsistently with the earlier notation K for the number of processors; please unify the notation.
  2. [Section 4.1.1] The sentence 'problem Q1 can be solved efficiently using standard standard interior point-based solvers' contains a duplicated 'standard'.
  3. [Appendix H, Theorem 22] The displayed norm expression in Theorem 22 has malformed notation; please fix the typesetting so that the statement is readable.
  4. [Appendix D] The expression for epsilon_Q in the overparameterized case is unclear as printed ('2−s√d− 22s +O(s)'); it should be typeset consistently with Theorem 4, e.g., 2^{-s}\sqrt{d-2^{2s}} + O(s) if that is the intended bound.
  5. [Section 5] The bucket size is reported as 8192 for the variance/accuracy experiments and 512 for the end-to-end speedup experiments; clarify whether different settings were used and whether the theoretical claims are sensitive to bucket size.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the variance, code-length, and convergence bounds are derived from first-principles arguments and standard external results, with no fitted constants or self-referential reductions.

full rationale

The paper's claimed derivation chain is self-contained. Theorem 4 derives the variance bound from Definition 3 via explicit bin-counting arguments, Hölder's inequality, Jensen's inequality, and an analytic optimization over an auxiliary scalar p; no empirical data, fitted constants, or target-dependent assumptions enter. Theorem 5 derives the code-length bound from the Elias recursive coding properties and a separate nonzero-coordinate count (Lemma 17); again the proof is analytic. The convergence guarantees in Theorems 6, 8, 11, 12, and 13 simply insert the proven variance bound into standard SGD convergence results (Bubeck 2015; Ghadimi and Lan 2013; Yan et al. 2018), which are external parameter-free facts and do not presuppose NUQSGD's conclusions. The empirical section validates the theory after the fact rather than calibrating it: comparisons with QSGD, QSGDinf, DGC, SignSGD, TernGrad, and ATOMO are against independent baselines and are not used to set any theoretical parameter. The paper explicitly acknowledges limitations that are relevant to correctness and novelty but not to circularity: the implementation 'diverges slightly from the theoretical analysis' (Section 5, tensor fusion and full-precision biases), the coded variant uses Huffman coding rather than the analyzed Elias coding, and the footnote discloses earlier independent work by Horváth et al. on exponentially spaced levels. These caveats create a possible gap between the analyzed algorithm and the benchmarked implementation, but they do not reduce a prediction to its own input. Citations to Alistarh et al. (2017) are load-bearing only as standard coding lemmas and convergence corollaries with independently stated assumptions; they do not constitute a self-citation chain that forces the result. Accordingly, the circularity score is 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central derivation uses standard SGD assumptions (beta-smoothness, unbiased gradients with bounded second moment) and prior theorems from the literature. No constants are fitted to data, and no new unverified entities are posited.

assumptions (5)
  • domain assumption The objective f is beta-smooth and stochastic gradients are unbiased with a second-moment bound B.
    Used in the proof of Theorem 6 and Corollary 2, following Alistarh et al. (2017); standard in SGD analysis.
  • domain assumption For momentum results, gradients are uniformly bounded: ||grad f(w)|| <= V.
    Assumed in Theorems 11 and 12, following Yan et al. (2018).
  • standard math Prior convergence theorems from Bubeck, Ghadimi and Lan, Yan et al., Lian et al., and Tang et al. are valid.
    These external results are cited and used to transfer variance bounds into convergence guarantees; they are not re-proved in this paper.
  • domain assumption The quantizer is unbiased conditional on the stochastic gradient, so the cross term in the variance decomposition vanishes.
    Used in equation (5) of the proof of Theorem 6; follows from Definition 3.
  • domain assumption In the decentralized setting, the mixing matrix W is symmetric doubly stochastic with second-largest eigenvalue rho < 1.
    Assumed in Appendix H, following Tang et al. (2018).

how reviews work

0 comments
Cite this review

Pith. "Pith review of NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization." pith.science (2026). https://pith.science/paper/OSYJYY2Z

@misc{pith2026190806077,
  author       = {Pith},
  title        = {Pith review of: NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OSYJYY2Z}},
  note         = {Machine review of arXiv:1908.06077}
}
read the original abstract

As the size and complexity of models and datasets grow, so does the need for communication-efficient variants of stochastic gradient descent that can be deployed to perform parallel model training. One popular communication-compression method for data-parallel SGD is QSGD (Alistarh et al., 2017), which quantizes and encodes gradients to reduce communication costs. The baseline variant of QSGD provides strong theoretical guarantees, however, for practical purposes, the authors proposed a heuristic variant which we call QSGDinf, which demonstrated impressive empirical gains for distributed training of large neural networks. In this paper, we build on this work to propose a new gradient quantization scheme, and show that it has both stronger theoretical guarantees than QSGD, and matches and exceeds the empirical performance of the QSGDinf heuristic and of other compression methods.

Figures

Figures reproduced from arXiv: 1908.06077 by the authors.

Figure 1
Figure 1. An example of nonuniform stochastic quantization with [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Optimal value of problem Q1 versus p ∈ [0, 1] for exponentially spaced collection of levels of the form (0, ps , · · · , p2 , p, 1). Proof The proof follows the steps in the proof of Theorem 9 for the problem of optimizing (d0, · · · , ds) to maximize the following upper bound E[kQs(v) − vk 2 ] ≤ kvk 2Xs j=0 τ 2 j dj/4. (17) The LP bound can be solved exactly in some simple cases. In Appendix E, we present the optim… view at source ↗
Figure 3
Figure 3. Variance upper bounds. 4.3 NUQSGD vs QSGD and QSGDinf How do QSGD and NUQSGD compare in terms of bounds on the expected number of communication bits required to achieve a given suboptimality gap ? The quantity that controls our guarantee on the convergence speed in both algorithms is the variance upper bound, which in turn is controlled by the quantization schemes. Note that the number of quantization levels, s, is… view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: Training loss on CIFAR10 (left) and ImageNet (right) for ResNet models. QSGD, [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Estimated normalized variance on CIFAR10 on the trajectory of single-GPU SGD. [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Accuracy on the hold-out set on CIFAR10 (left) and on ImageNet (right) for [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Scalability behavior for NUQSGD versus the full-precision baseline when training [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: End-to-end training time for ResNet50/ImageNet for NUQSGD and EF-SignSGD [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Training loss (left) and Test accuracy (right) on CIFAR10 for ResNet110. We set [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Test Accuracy on CIFAR10 (left) and CIFAR100 (right) for ResNet18. For [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Estimated variance (left) and normalized variance (right) on CIFAR10 on the [PITH_FULL_IMAGE:figures/full_fig_p038_11.png]
Figure 12
Figure 12. Figure 12: Estimated normalized variance on CIFAR10 (left) and ImageNet (right). For [PITH_FULL_IMAGE:figures/full_fig_p038_12.png]
Figure 13
Figure 13. Figure 13: Comparison with SignSGD on CIFAR10. Training loss (left), training accuracy [PITH_FULL_IMAGE:figures/full_fig_p039_13.png]
Figure 14
Figure 14. Figure 14: Comparison with TernGrad on CIFAR10. Training loss (left), training accuracy [PITH_FULL_IMAGE:figures/full_fig_p039_14.png]
Figure 15
Figure 15. Figure 15: Estimated variance (left) and normalized variance (right) on CIFAR10 for [PITH_FULL_IMAGE:figures/full_fig_p040_15.png]
Figure 16
Figure 16. Figure 16: Training accuracy (left) and validation loss (right) on CIFAR10 for ResNet110. [PITH_FULL_IMAGE:figures/full_fig_p040_16.png]
Figure 17
Figure 17. Figure 17: Scalability behavior for NUQSGD versus the full-precision baseline when training [PITH_FULL_IMAGE:figures/full_fig_p041_17.png]
Figure 18
Figure 18. Figure 18: End-to-end training time for ResNet50/ImageNet for NUQSGD and EF-SignSGD [PITH_FULL_IMAGE:figures/full_fig_p041_18.png]
Figure 19
Figure 19. Figure 19: Estimated variance (left) and normalized variance (right) on CIFAR10 for [PITH_FULL_IMAGE:figures/full_fig_p041_19.png]
Figure 20
Figure 20. Figure 20: Training loss (left) and validation loss (right) on CIFAR10 for ResNet110 with [PITH_FULL_IMAGE:figures/full_fig_p042_20.png]
Figure 21
Figure 21. Figure 21: Training accuracy (left) and test accuracy (right) on CIFAR10 for ResNet110 [PITH_FULL_IMAGE:figures/full_fig_p042_21.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RATQ: A Universal Fixed-Length Quantizer for Stochastic Optimization

    cs.LG 2019-08 conditional novelty 6.0 of 10

    RATQ is a fixed-length quantizer based on random Hadamard rotation and adaptive uniform quantization that nearly attains the information-theoretic lower bound on gradient precision for convex stochastic optimization.

Reference graph

Works this paper leans on

49 extracted references · 40 canonical work pages · cited by 1 Pith paper

  1. [1]

    Abadi, A

    M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, and M. Devin. TensorFlow : Large-scale machine learning on heterogeneous distributed systems. arXiv:1603.04467, 2016

    arXiv 2016

  2. [2]

    Alistarh, D

    D. Alistarh, D. Grubic, J. Z. Li, R. Tomioka, and M. Vojnovic. QSGD : Communication-efficient SGD via gradient quantization and encoding. In Proc. Advances in Neural Information Processing Systems (NIPS), 2017

    work page 2017

  3. [3]

    Bekkerman, M

    R. Bekkerman, M. Bilenko, and J. Langford. Scaling up machine learning: Parallel and distributed approaches. Cambridge University Press, 2011

    work page 2011

  4. [4]

    Bernstein, Y.-X

    J. Bernstein, Y.-X. Wang, K. Azizzadenesheli, and A. Anandkumar. sign SGD : Compressed optimisation for non-convex problems. In Proc. International Conference on Machine Learning (ICML), 2018

    work page 2018

  5. [5]

    Boyd and L

    S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge Univ. Press, 2004

    work page 2004

  6. [6]

    S. Bubeck. Convex optimization: Algorithms and complexity. Foundations and Trends in Machine Learning , 8(3-4): 0 231--358, 2015

    work page 2015

  7. [7]

    K. W. Cattermole. Principles of pulse code modulation. Iliffe, 1969

    work page 1969

  8. [8]

    Chilimbi, Y

    T. Chilimbi, Y. Suzue J. Apacible, and K. Kalyanaraman. Project adam: Building an efficient and scalable deep learning training system. In Proc. USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2014

    work page 2014

Show all 49 references
  1. [9]

    Coates, B

    A. Coates, B. Huval, T. Wang, D. Wu, B. Catanzaro, and A. Ng. Deep learning with cots hpc systems. In Proc. International Conference on Machine Learning (ICML), 2013

    2013

  2. [10]

    J. Dean, G. Corrado, R. Monga K. Chen, M. Devin, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, Q. V. Le, and A. Y. Ng. Large scale distributed deep networks. In Proc. Advances in Neural Information Processing Systems (NIPS), 2012

    2012

  3. [11]

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet : A large-scale hierarchical image database. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009

    2009

  4. [12]

    J. C. Duchi, S. Chaturapruek, and C. R\' e . Asynchronous stochastic convex optimization. In Proc. Advances in Neural Information Processing Systems (NIPS), 2015

    2015

  5. [13]

    P. Elias. Universal codeword sets and representations of the integers. IEEE Transactions on Information Theory , 21(2): 0 194--203, 1975

    1975

  6. [14]

    Ghadimi and G

    S. Ghadimi and G. Lan. Stochastic first- and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4): 0 2341--2368, 2013

    2013

  7. [15]

    Gupta, A

    S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan. Deep learning with limited numerical precision. In Proc. International Conference on Machine Learning (ICML), 2015

    2015

  8. [16]

    Deep speech: Scaling up end-to-end speech recognition

    Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. Deep speech: Scaling up end-to-end speech recognition. arXiv:1412.5567, 2014

    2014 arXiv

  9. [17]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016

    2016

  10. [18]

    Horv\' a th, C.-Y Ho, L

    S. Horv\' a th, C.-Y Ho, L. Horv\' a th, A. N. Sahu, M. Canini, and P. Richt\' a rik. Natural compression for distributed deep learning. arXiv:1905.10988v1, 2019

    1905 arXiv

  11. [19]

    Hou and J

    L. Hou and J. T. Kwok. Loss-aware weight quantization of deep networks. In Proc. International Conference on Learning Representations (ICLR), 2018

    2018

  12. [20]

    Hubara, M

    I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio. Binarized neural networks. In Proc. Advances in Neural Information Processing Systems (NIPS), 2016

    2016

  13. [21]

    S. P. Karimireddy, Q. Rebjock, S. Stich, and M. Jaggi. Error feedback fixes S ign SGD and other gradient compression schemes. In Proc. International Conference on Machine Learning (ICML), 2019

    2019

  14. [22]

    Krizhevsky

    A. Krizhevsky. Learning multiple layers of features from tiny images. 2009. Technical report, University of Toronto

    2009

  15. [23]

    E. H. Lee, D. Miyashita, E. Chai, B. Murmann, and S. S. Wong. LogNet : Energy-efficient neural networks using logarithmic computation. In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017

    2017

  16. [24]

    M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B.-Y. Su. Scaling distributed machine learning with the parameter server. In Proc. USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2014

    2014

  17. [25]

    Li and C

    Z. Li and C. M. De Sa. Dimension-free bounds for low-precision training. In Proc. Advances in Neural Information Processing Systems (NeurIPS), 2019

    2019

  18. [26]

    X. Lian, Y. Huang, Y. Li, and J. Liu. Asynchronous parallel stochastic gradient for nonconvex optimization. In Proc. Advances in Neural Information Processing Systems (NIPS), 2015

    2015

  19. [27]

    Y. Lin, S. Han, H. Mao, Y. Wang, and W. J. Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. In Proc. International Conference on Learning Representations (ICLR), 2018

    2018

  20. [28]

    Miyashita, E

    D. Miyashita, E. H. Lee, and B. Murmann. Convolutional neural networks using logarithmic data representation. arXiv:1603.01025v2, 2016

    2016 arXiv

  21. [29]

    Nesterov

    Y. Nesterov. A method of solving a convex programming problem with convergence O(1/k^2) . Soviet Mathematics Doklady, 27(2): 0 372--376, 1983

    1983

  22. [30]

    J. Park, S. Li, W. Wen, P. Tang, H. Li, Y. Chen, and P. Dubey. Faster CNN s with direct sparse convolutions and guided pruning. In Proc. International Conference on Learning Representations (ICLR), 2017

    2017

  23. [31]

    B. T. Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics , 4(5): 0 1--17, 1964

    1964

  24. [32]

    Recht, C

    B. Recht, C. R\' e , S. Wright, and F. Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Proc. Advances in Neural Information Processing Systems (NIPS), 2011

    2011

  25. [33]

    C. M. D. Sa, Ce. Zhang, K. Olukotun, and C. R\' e . Taming the wild: A unified analysis of hogwild-style algorithms. In Proc. Advances in Neural Information Processing Systems (NIPS), 2015

    2015

  26. [34]

    Seide, H

    F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNN s. In Proc. INTERSPEECH, 2014

    2014

  27. [35]

    Horovod: fast and easy distributed deep learning in TensorFlow

    Alexander Sergeev and Mike Del Balso. Horovod: fast and easy distributed deep learning in TensorFlow . arXiv:1802.05799, 2018

    2018 arXiv

  28. [36]

    N. Strom. Scalable distributed DNN training using commodity GPU cloud computing. In Proc. INTERSPEECH, 2015

    2015

  29. [37]

    H. Tang, S. Gan, C. Zhang, T. Zhang, and J. Liu. Communication compression for decentralized training. In Proc. Advances in Neural Information Processing Systems (NIPS), 2018

    2018

  30. [38]

    H. Wang, S. Sievert, Z. Charles, S. Liu, S. Wright, and D. Papailiopoulos. ATOMO : Communication-efficient learning via atomic sparsification. In Proc. Advances in Neural Information Processing Systems (NIPS), 2018

    2018

  31. [39]

    W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li. Learning structured sparsity in deep neural networks. In Proc. Advances in Neural Information Processing Systems (NIPS), 2016

    2016

  32. [40]

    W. Wen, C. Xu, F. Yan, C. Wu, Y. Wang, Y. Chen, and H. Li. TernGrad : Ternary gradients to reduce communication in distributed deep learning. In Proc. Advances in Neural Information Processing Systems (NIPS), 2017

    2017

  33. [41]

    E. P. Xing, Q. Ho, W. Dai, J. K. Kim, J. Wei, S. Lee, X. Zheng, P. Xie, A. Kumar, and Y. Y. Petuum. Petuum: A new platform for distributed machine learning on big data. IEEE transactions on Big Data , 1(2): 0 49--67, 2015

    2015

  34. [42]

    Y Yan, T. Yang, Q. Lin, Z. Li, and Y Yang. A unified analysis of stochastic momentum methods for deep learning. In Proc. International Joint Conference on Artificial Intelligence (IJCAI), 2018

    2018

  35. [43]

    Zhang, J

    H. Zhang, J. Li, K. Kara, D. Alistarh, J. Liu, and C. Zhang. ZipML : Training linear models with end-to-end low precision, and a little bit of deep learning. In Proc. International Conference on Machine Learning (ICML), 2017

    2017

  36. [44]

    Zhang, A

    S. Zhang, A. E. Choromanska, and Y. LeCun. Deep learning with elastic averaging SGD . In Proc. Advances in Neural Information Processing Systems (NIPS), 2015

    2015

  37. [45]

    S. Zhou, Y. Wu, Z. Ni, X. Zhou, H. Wen, and Y. Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv:1606.06160, 2018

    2018 arXiv

  38. [46]

    Zinkevich, M

    M. Zinkevich, M. Weimer, L. Li, and A. J. Smola. Parallelized stochastic gradient descent. In Proc. Advances in Neural Information Processing Systems (NIPS), 2010

    2010

  39. [47]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  40. [48]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  41. [49]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.