Pith. sign in

REVIEW 5 major objections 4 minor 1 cited by

Performance Analysis and Comparison of Distributed Machine Learning Systems

T0 review · 5 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Ring-allreduce distributed training achieves scalable performance by keeping per-worker communication nearly constant as workers are added, while parameter-server and peer-to-peer architectures saturate from network congestion.

desk verdict The paper's qualitative ranking (RA > P2P > PS) is right, but its quantitative scaling model has a load-bearing unit error in Eq. 20 that invalidates the predicted epoch-time curves. read the letter →

arxiv 1909.02061 v1 pith:GE4WC7UM submitted 2019-09-04 cs.DC

Salem Alqahtani , Murat Demirbas This is my paper · ORCID
classification cs.DC
keywords distributedmachinelearningsynchronousSGDparameterserverpeer-to-peerringallreducecommunicationbottleneckscalabilityperformancemodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Distributed training of deep networks is usually bottlenecked by moving gradients, not by computing them, but the field has lacked a head-to-head account of how the communication architecture itself shapes scaling. This paper builds per-epoch performance models for the three dominant architectures—parameter server (PS), peer-to-peer (P2P), and ring allreduce (RA)—for synchronous stochastic gradient descent, and tests the models on a handwritten-digit classification task with one to seven workers. Its central claim is that architecture is a first-order scaling decision: RA keeps each worker's network traffic roughly constant as workers are added, while PS degrades because all workers share the server's bandwidth and P2P degrades because every peer sends to every other peer. The paper also finds that RA overlaps gradient communication with computation, which the other two architectures fail to do. If correct, the practical implication is that clusters using ring allreduce can add workers without paying a per-worker communication tax, and the bottleneck moves to compute and synchronization rather than the network.

What carries the argument

The carrying object is the ring allreduce communication pattern, a ring of $w$ workers in which each worker sends and receives only with its two immediate neighbors. The load-bearing identity is Eq. 20, $T_{\mathrm{tcp-ring}} = \frac{2(w-1)\,W/w}{\mathrm{Bandwidth}}$, giving the time for one allreduce of a model of size $W$. Because $\frac{2(w-1)}{w}$ approaches 2, the total data each worker moves over the ring stays near $2W$ no matter how many workers participate; this is the mechanism behind the claim that RA decouples network usage from worker count. The model contrasts this with Eq. 8, per-worker bandwidth $B/w$ for PS, and Eq. 16, per-peer bandwidth $B/(2(w-1))$ for P2P, which make those architectures' communication grow with $w$. The equations do the work: they convert the architectural topologies into concrete epoch-time formulas whose predictions the experiments are designed to corroborate.

What would settle it

Take the paper's RA setup with a fixed dataset and a fixed worker count, and run training at two mini-batch sizes, say 50 and 200, so the number of allreduces per epoch differs by a factor of four. If Eq. 20 is already per-epoch, the measured epoch time should stay roughly flat; if the allreduce is paid per mini-batch, epoch time should grow with the number of mini-batches. The paper fixes batch size at 100 in all experiments, so this comparison would settle whether the model's unit accounting supports its scalability conclusion.

Watch

Extended reading notes

Core claim

The paper's discovery is that the ring-allreduce topology, not any framework optimization, is what makes distributed training scalable. In its model, $w$ workers arranged in a ring exchange chunks of size $W/w$ with their two neighbors, so the communication time for an allreduce step is $T_{\mathrm{tcp-ring}} = 2(w-1)W/(wB)$, which tends to $2W/B$ as $w$ grows; the number of workers effectively cancels out of the per-worker cost. For PS, available bandwidth per worker is $B/w$, so both pulling and pushing degrade as the server's link congests; the paper's throughput curves flatten around five workers. For P2P, available bandwidth is $B/(2(w-1))$, so traffic rises with peer count and lands between PS and RA. Experiments with a two-hidden-layer network on 60,000 training images and batch size 100 show RA epoch time falling sublinearly with workers and throughput climbing toward ideal, while PS and P2P plateau. The same experiments show RA overlapping gradient transmission with backward-pass computation, which the paper argues is a second, independent source of its latency advantage.

Load-bearing premise

The load-bearing assumption is that the ring-allreduce communication term in Eq. 20 can be added to the per-epoch compute term as the whole communication cost of one epoch; in synchronous SGD the allreduce is performed once per mini-batch, so if the term is per mini-batch rather than per epoch the per-epoch communication cost must be multiplied by the number of mini-batches, and the paper's claimed worker-count independence would need to be restated.

Editorial extensions

If this is right

  • Clusters running ring allreduce can add workers with per-worker communication time roughly unchanged; beyond compute limits, throughput should continue to climb, as the paper's measured RA throughput does.
  • Parameter-server deployments should expect a throughput ceiling and possibly a decline once the server's link saturates; the paper sees this near five workers in its setup.
  • Peer-to-peer systems without a server are a stopgap: they avoid single-point congestion but still transmit to every peer, so their scaling is bounded by all-to-all traffic.
  • Overlapping gradient communication with backward computation is a measurable performance lever; RA's overlap gives it latency gains that bandwidth alone would not explain.
  • Practitioners can use the paper's equations to estimate epoch time from model size, dataset size, batch size, worker count, and link bandwidth before choosing an architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's experiments keep batch size fixed, so its data cannot distinguish per-epoch from per-mini-batch communication accounting; a mini-batch-size sweep would tighten the model while preserving the architectural comparison.
  • The same bandwidth-sharing comparison extends naturally to asynchronous and stale-synchronous schedules: the PS and P2P communication terms would still grow with worker count, so the architectural ranking is likely stable, though overlap behavior might differ.
  • Because the paper identifies layer-wise load imbalance in PS and P2P, a natural next design is to schedule gradient transmission by layer size rather than by fixed tensor order, effectively giving PS and P2P the overlap benefit RA gets from its pipeline.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. This paper develops analytical performance models for per-epoch training time of distributed synchronous SGD under three communication architectures—parameter server (PS), peer-to-peer (P2P), and ring allreduce (RA)—and tests the models against TensorFlow/Horovod experiments on MNIST with one to seven workers. The authors conclude that RA achieves the best throughput and latency because its communication cost is decoupled from the number of workers and because it overlaps computation and communication, whereas PS and P2P suffer from centralized or all-to-all traffic bottlenecks.

Significance. The paper addresses a practically important question—how the choice of communication architecture affects distributed training performance—and it provides a useful qualitative taxonomy. Its strengths are the head-to-head experimental comparison of PS, P2P, and RA on the same task and cluster, and the attempt to write down explicit per-epoch communication formulas for each topology. If the model were corrected and independently calibrated, the qualitative conclusion that RA is more scalable than PS/P2P would be credible and useful to practitioners. However, the quantitative model as written contains unit inconsistencies and depends on parameters extracted from the same experiments used for validation, so the paper's central quantitative corroboration is not currently established.

major comments (5)
  1. [§III-C, Eqs. (20)–(21)] T(tcp-ring) has units of time per allreduce, but T(cpu-ring) is per-epoch compute time. In synchronous mini-batch SGD each worker executes n/(b*w) local mini-batches per epoch and must perform an allreduce after each one, so the per-epoch communication term should be multiplied by n/(b*w). For the MNIST setup (n=60,000, b=100) this missing factor is 600 for one worker and about 86 for seven workers; the model's epoch-time and throughput predictions in Figs. 17–19 therefore do not follow from Eq. (21). This is an internal inconsistency, not a modeling choice.
  2. [§III-A, Eq. (10)] The push payload is written as W/w, but in data-parallel synchronous SGD each worker computes a local gradient of the full model, so the gradient message size is W (or, for sharded PSs, the portion of W assigned to the corresponding server), not W/w. This underpredicts PS upload traffic by a factor of w and biases the PS-versus-RA comparison in the analytical model.
  3. [§III-A, text after Eq. (11); §III-B, Eq. (15)] Tprocessing is 'defined and extracted from experiments,' and the PS time and update time are unexplained constants. Because the same experimental runs are then used to corroborate the model in Figs. 5, 8, 13, and 19, the reported agreement is partly self-fulfilling. The paper should specify how Tprocessing is measured (ideally on a single worker independent of the distributed runs), report its numerical value, and show the model's sensitivity to this parameter.
  4. [§III-B, Eq. (17)] The P2P derivation states that in every iteration a server sends and receives 2*(w-1) messages of size W/w, but Eq. (17) contains no explicit factor of 2*(w-1) in the message count. As written, the formula does not follow from the stated message count, so the P2P communication model should be rederived from the per-iteration number and size of messages.
  5. [Abstract; §IV] The claim that RA excels by overlapping computation and communication is not supported by the model or by an experiment that isolates overlap. The model sums T(cpu) and T(tcp) for all three architectures (Eqs. 13, 18, 21), so it does not quantify overlap, and the evaluation reports only epoch time and throughput. Please add a direct measurement of overlap (for example, timeline traces) or soften the claim.
minor comments (4)
  1. [Table I] The notation table lists both b (batch size) and m (mini-batch size), but m is never used and the text alternates between 'batch size' and 'mini-batch'; the notation should be made consistent.
  2. [General] Figure references are out of order (for example, Fig. 11 is cited before Figs. 6–10), and several figures have small axis labels that make them hard to read.
  3. [§IV, 'Ease of Development'] The 'Ease of Development' subsection reads as subjective opinion ('I noticed...') rather than a reproducible evaluation; it should be either removed or supported with concrete evidence.
  4. [References] There are multiple typographical and reference issues, including inconsistent 'Tensorflow'/'TensorFlow' spelling and incomplete entries (e.g., [21] and [23]); these should be cleaned up.

Circularity Check

1 steps flagged · score 5.0 of 10

Partial circularity in validation: the compute component of the predicted epoch time is measured from the same experiments; the RA scalability claim itself is not circular.

  1. fitted input called prediction [Section III-A, Eq. 11 (T(cpu-ps)) and Section IV-B validation figures (Figs. 5, 8, 13, 19)]
    "The above formulas for T(cpu−ps) calculate approximate computation time for computing neural network which is a learning variable defined and extracted from experiments because each dataset has a different number of features that lead to different computation time cost."

    T(cpu-ps) in Eq. 11 is constructed from T(processing), which the paper says is 'defined and extracted from experiments' on the same systems and datasets whose runtime it then 'predicts' in T(total)=T(cpu-ps)+T(tcp-ps). The compute half of the predicted epoch time is therefore a measured input, so the model-to-experiment agreement in the epoch-time figures is partly self-fulfilling. The communication terms (Eqs. 9-10, 17, 20) are not fitted, so the architecture comparison retains independent content; this is a partial, not total, circularity.

full rationale

The paper's central scaling claims for RA versus PS/P2P are not circular: Eq. 20's communication cost is the standard bandwidth-optimal allreduce expression attributed to Patarasuk and Yuan [28], not fitted to the authors' experiments, and the qualitative conclusion that RA's per-worker communication is independent of worker count follows from that expression. Neither the self-citation [16] nor any invoked prior result is load-bearing for the architecture comparison. The main circularity lies in the validation loop: Section III-A explicitly defines T(processing) as a 'learning variable defined and extracted from experiments,' then Eq. 11 builds the per-epoch compute time from it, and Eq. 13 defines T(total)=T(cpu-ps)+T(tcp-ps), the quantity compared to measured epoch time in Figures 5, 8, 13, and 19. This makes the compute half of the predicted total equal to a measured input, so the claimed model-to-experiment agreement is partly by construction. The missing n/(b*w) factor in Eq. 20 (a per-allreduce cost added as though per-epoch) is a unit inconsistency that would invalidate the quantitative model, but it is not itself a circularity and is therefore not counted as a circular step. Because the architecture comparisons rest on independent communication terms, the score is 5 rather than higher.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the PS/P2P/RA models in Section III. The key inputs are the per-worker bandwidth formulas (Eqs. 8 and 16), the ring allreduce cost (Eq. 20), and the compute time Tprocessing, which the paper explicitly states is extracted from experiments. No new entities are introduced. The main dependency is the fitted compute time and a unit-inconsistent communication formula.

free parameters (3)
  • Tprocessing = not reported, described as 'extracted from experiments'
    Used in Eqs. 7, 11, 15, 19 as the per-batch compute cost; fitted to the same experiments used for validation, so the model-to-experiment match is partly self-confirming.
  • PS time = constant, value not reported
    Introduced in Eq. 11 as the time for the parameter server to update the model; no measurement method or value is given.
  • update time = not specified
    Appears in Eqs. 15 and 19 for P2P and RA compute terms; never defined or measured independently.
assumptions (5)
  • domain assumption Available bandwidth per worker in PS is TotalBandwidth/w, scaled by the number of parameter servers.
    Eq. 8. Assumes equal bandwidth sharing and that more PSs linearly increase per-worker bandwidth, which conflicts with the paper's own 2PS/4PS results showing degradation.
  • standard math Ring allreduce communication cost for one update is 2(w-1)W/(wB).
    Eq. 20, from Patarasuk and Yuan (2009). Correct for a single allreduce, but the paper treats it as the per-epoch communication cost.
  • ad hoc to paper Tprocessing is independent of the number of workers and architecture.
    Implicit in Eqs. 11, 15, 19, which reuse the same fitted compute term for all architectures and worker counts.
  • domain assumption Synchronous SGD is the appropriate focus for DNN training.
    Section I, citing [6]. This scopes the study but is not load-bearing for the architecture comparison.
  • domain assumption The per-node network bandwidth B available to ring allreduce stays constant as workers are added.
    Eq. 20. The scalability claim that RA is nearly independent of w depends on B not degrading with ring size.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Performance Analysis and Comparison of Distributed Machine Learning Systems." pith.science (2026). https://pith.science/paper/GE4WC7UM

@misc{pith2026190902061,
  author       = {Pith},
  title        = {Pith review of: Performance Analysis and Comparison of Distributed Machine Learning Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GE4WC7UM}},
  note         = {Machine review of arXiv:1909.02061}
}
read the original abstract

Deep learning has permeated through many aspects of computing/processing systems in recent years. While distributed training architectures/frameworks are adopted for training large deep learning models quickly, there has not been a systematic study of the communication bottlenecks of these architectures and their effects on the computation cycle time and scalability. In order to analyze this problem for synchronous Stochastic Gradient Descent (SGD) training of deep learning models, we developed a performance model of computation time and communication latency under three different system architectures: Parameter Server (PS), peer-to-peer (P2P), and Ring allreduce (RA). To complement and corroborate our analytical models with quantitative results, we evaluated the computation and communication performance of these system architectures of the systems via experiments performed with Tensorflow and Horovod frameworks. We found that the system architecture has a very significant effect on the performance of training. RA-based systems achieve scalable performance as they successfully decouple network usage from the number of workers in the system. In contrast, 1PS systems suffer from low performance due to network congestion at the parameter server side. While P2P systems fare better than 1PS systems, they still suffer from significant network bottleneck. Finally, RA systems also excel by virtue of overlapping computation time and communication time, which PS and P2P architectures fail to achieve.

Figures

Figures reproduced from arXiv: 1909.02061 by the authors.

Figure 1
Figure 1. Deep Neural Networks B. Distributed Neural Network Training In recent years, the advance of hardware, training methods, and network architectures have enabled distributed training which minimizes the training time for DNNs training. Instead of restricted to a single machine, now we can scale to as many resources as required. In this paper, we choose to perform on data parallel distributed training, but we also expla… view at source ↗
Figure 2
Figure 2. Model Parallelism [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Data Parallelism. 2) Data Parallelism: In the data parallelism scheme, as shown in [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: PS Architecture. availableB = (( 1 w ∗ T otalBandwidth) ∗ ps) (8) Notice that the availableB is the available bandwidth between every client and PS, T otalBandwidth represents the total bandwidth available for all clients, and w indicates all active workers who communi…
Figure 5
Figure 5. Figure 5: Estimated epoch time for 1PS. The formula below calculates the ideal samples per second with respect to number of workers. Ideal = (Tsingle ∗ n) ∗ w (14) In [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Measured training throughput of 1PS [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 9
Figure 9. Figure 9: Measured training throughput of 2PS. read whole model size from the same machine while in writing the workers update subset of the model size through network. Recently, most DNN frameworks overlap computation time with gradients updates. T(cpu−p2p) = (epoch ∗ ( n b w )…
Figure 10
Figure 10. Figure 10: Epoch time for 2PS [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]
Figure 14
Figure 14. Figure 14: Epoch time of P2P system. C. Distributed Training with Ring-allreduce In this system architecture, as shown in [PITH_FULL_IMAGE:figures/full_fig_p007_14.png]
Figure 12
Figure 12. Figure 12: P2P based Architecture [PITH_FULL_IMAGE:figures/full_fig_p007_12.png]
Figure 15
Figure 15. Figure 15: Measured training throughput of P2P system. [PITH_FULL_IMAGE:figures/full_fig_p008_15.png]
Figure 16
Figure 16. Figure 16: RA Architecture [PITH_FULL_IMAGE:figures/full_fig_p008_16.png]
Figure 17
Figure 17. Figure 17: Epoch time of RA. IV. EVALUATION A. Experimental Environment Here, we run a set of experiments in distributed ML system introduced in Section II. In order to provide a quantitative evaluation of 1PS, 2PS, 4PS, RA (Horovod), and P2P Systems, [PITH_FULL_IMAGE:figures/f…
Figure 18
Figure 18. Figure 18: Measured training throughput of RA [PITH_FULL_IMAGE:figures/full_fig_p009_18.png]
Figure 19
Figure 19. Figure 19: Estimated epoch time for RA. we evaluated the performance of these system architectures with the same basic classification ML tasks. The system per￾formance has two dimensions, latency metric and throughput metric. All of our experiments were conducted in an Amazon EC…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Accelerated Methods with Compressed Communications for Distributed Optimization Problems under Data Similarity

    math.OC 2024-12 conditional novelty 6.0 of 10

    OLGA and EF-OLGA are the first accelerated distributed optimization methods to combine compression with Hessian similarity, achieving record communication-time complexity.

Reference graph

Works this paper leans on

37 extracted references · 30 canonical work pages · cited by 1 Pith paper

  1. [1]

    B. Zhang. A solution to the memory limit chal- lenge in big data machine learning. [Online]. Avail- able: https://medium.com/@Petuum/a-solution-to-the-memory-limit- challenge-in-big-data-machine-learning-49783a72088b

    work page

  2. [2]

    Tensorflow: A system for large- scale machine learning,

    M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard et al., “Tensorflow: A system for large- scale machine learning,” in 12th {USENIX} Symposium on Operating Systems Design and Implementation ( {OSDI} 16), 2016, pp. 265–283

    work page 2016

  3. [3]

    Horovod: fast and easy distributed deep learning in tensorflow,

    A. Sergeev and M. Del Balso, “Horovod: fast and easy distributed deep learning in tensorflow,” arXiv preprint arXiv:1802.05799 , 2018

    arXiv 2018

  4. [4]

    Zhang, Intro to Distributed Deep Learning Systems

    H. Zhang, Intro to Distributed Deep Learning Systems . [Online]. Available: https://medium.com/@Petuum/intro-to-distributed- deep-learning-systems-a2e45c6b8e7

    work page

  5. [5]

    Demystifying parallel and distributed deep learning: An in-depth concurrency analysis,

    T. Ben-Nun and T. Hoefler, “Demystifying parallel and distributed deep learning: An in-depth concurrency analysis,” arXiv preprint arXiv:1802.09941, 2018

    arXiv 2018

  6. [6]

    Revisiting distributed synchronous sgd,

    J. Chen, X. Pan, R. Monga, S. Bengio, and R. Jozefowicz, “Revisiting distributed synchronous sgd,” arXiv preprint arXiv:1604.00981 , 2016

    arXiv 2016

  7. [7]

    Artificial neural network — Wikipedia, the free encyclopedia,

    Wikipedia contributors, “Artificial neural network — Wikipedia, the free encyclopedia,” 2018, [Online; accessed 01-Dec-2018]. [Online]. Available: https://en.wikipedia.org/wiki/Artificial neural network

    work page 2018

  8. [8]

    C. M. B. et al, Pattern Recognition and Machine Learning . Springer, 2006

    work page 2006

Show all 37 references
  1. [9]

    Learning in- ternal representations by error propagation,

    D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning in- ternal representations by error propagation,” in Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Volume 1: Foundations, D. E. Rumelhart and J. L. Mcclelland, Eds. Cambridge, MA: ...

    1986

  2. [10]

    Malt: distributed data- parallelism for existing ml applications,

    H. Li, A. Kadav, E. Kruus, and C. Ungureanu, “Malt: distributed data- parallelism for existing ml applications,” in Proceedings of the Tenth European Conference on Computer Systems . ACM, 2015, p. 3

    2015

  3. [11]

    Large-scale machine learning with stochastic gradient de- scent,

    L. Bottou, “Large-scale machine learning with stochastic gradient de- scent,” in Proceedings of COMPSTAT’2010. Springer, 2010, pp. 177– 186

    2010

  4. [12]

    A parameter communication optimization strategy for distributed machine learning in sensors,

    J. Zhang, H. Tu, Y . Ren, J. Wan, L. Zhou, M. Li, J. Wang, L. Yu, C. Zhao, and L. Zhang, “A parameter communication optimization strategy for distributed machine learning in sensors,” Sensors, vol. 17, no. 10, p. 2172, 2017

    2017

  5. [13]

    A bridging model for parallel computation,

    L. G. Valiant, “A bridging model for parallel computation,” Communi- cations of the ACM , vol. 33, no. 8, pp. 103–111, 1990

    1990

  6. [14]

    Strategies and principles of distributed machine learning on big data,

    E. P. Xing, Q. Ho, P. Xie, and D. Wei, “Strategies and principles of distributed machine learning on big data,” Engineering, vol. 2, no. 2, pp. 179–195, 2016

    2016

  7. [15]

    More effective distributed ml via a stale synchronous parallel parameter server,

    Q. Ho, J. Cipar, H. Cui, S. Lee, J. K. Kim, P. B. Gibbons, G. A. Gibson, G. Ganger, and E. P. Xing, “More effective distributed ml via a stale synchronous parallel parameter server,” in Advances in neural information processing systems , 2013, pp. 1223–1231

    2013

  8. [16]

    A comparison of distributed machine learning platforms,

    K. Zhang, S. Alqahtani, and M. Demirbas, “A comparison of distributed machine learning platforms,” inComputer Communication and Networks (ICCCN), 2017 26th International Conference on. IEEE, 2017, pp. 1–9

    2017

  9. [17]

    An architecture for parallel topic models,

    A. Smola and S. Narayanamurthy, “An architecture for parallel topic models,” Proceedings of the VLDB Endowment , vol. 3, no. 1-2, pp. 703–710, 2010

    2010

  10. [18]

    Large scale distributed deep networks,

    J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. V . Le et al. , “Large scale distributed deep networks,” in Advances in neural information processing systems , 2012, pp. 1223–1231

    2012

  11. [19]

    Scaling distributed machine learning with the parameter server

    M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V . Josifovski, J. Long, E. J. Shekita, and B.-Y . Su, “Scaling distributed machine learning with the parameter server.” in OSDI, vol. 14, 2014, pp. 583– 598

    2014

  12. [20]

    Communication efficient distributed machine learning with the parameter server,

    M. Li, D. G. Andersen, A. J. Smola, and K. Yu, “Communication efficient distributed machine learning with the parameter server,” in Advances in Neural Information Processing Systems , 2014, pp. 19–27

    2014

  13. [21]

    W. C. I. Dropbox Inc, Google Inc. Grpc. [Online]. Available: https://grpc.io/

  14. [22]

    Firecaffe: near-linear acceleration of deep neural network training on compute clusters,

    F. N. Iandola, M. W. Moskewicz, K. Ashraf, and K. Keutzer, “Firecaffe: near-linear acceleration of deep neural network training on compute clusters,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2592–2600

    2016

  15. [23]

    Petuum: A framework for iterative-convergent distributed ml,

    W. Dai, J. Wei, J. K. Kim, S. Lee, J. Yin, Q. Ho, and E. P. Xing, “Petuum: A framework for iterative-convergent distributed ml,” 2013

    2013

  16. [24]

    Exploiting bounded staleness to speed up big data analytics

    H. Cui, J. Cipar, Q. Ho, J. K. Kim, S. Lee, A. Kumar, J. Wei, W. Dai, G. R. Ganger, P. B. Gibbons et al. , “Exploiting bounded staleness to speed up big data analytics.” in USENIX Annual Technical Conference, 2014, pp. 37–48

    2014

  17. [25]

    Big data: the driver for innovation in databases,

    B. Cui, H. Mei, and B. C. Ooi, “Big data: the driver for innovation in databases,” National Science Review , vol. 1, no. 1, pp. 27–30, 2014

    2014

  18. [26]

    Effect of data skewness and workload balance in parallel data mining,

    D. W. Cheung, S. D. Lee, and Y . Xiao, “Effect of data skewness and workload balance in parallel data mining,” IEEE Transactions on Knowledge and Data Engineering , vol. 14, no. 3, pp. 498–514, 2002

    2002

  19. [27]

    Scalable inference in latent variable models,

    A. Ahmed, M. Aly, J. Gonzalez, S. Narayanamurthy, and A. J. Smola, “Scalable inference in latent variable models,” in Proceedings of the fifth ACM international conference on Web search and data mining . ACM, 2012, pp. 123–132

    2012

  20. [28]

    Bandwidth optimal all-reduce algorithms for clusters of workstations,

    P. Patarasuk and X. Yuan, “Bandwidth optimal all-reduce algorithms for clusters of workstations,” Journal of Parallel and Distributed Comput- ing, vol. 69, no. 2, pp. 117–124, 2009

    2009

  21. [29]

    baidu-research/tensorflow-allreduce,

    baidu research, “baidu-research/tensorflow-allreduce,” Aug 2017. [On- line]. Available: https://github.com/baidu-research/tensorflow-allreduce

    2017

  22. [30]

    Open mpi: Goals, concept, and design of a next generation mpi implementation,

    E. Gabriel, G. E. Fagg, G. Bosilca, T. Angskun, J. J. Dongarra, J. M. Squyres, V . Sahay, P. Kambadur, B. Barrett, A. Lumsdaine, R. H. Castain, D. J. Daniel, R. L. Graham, and T. S. Woodall, “Open mpi: Goals, concept, and design of a next generation mpi implementation,” in In ...

    2004

  23. [31]

    A. Sergeev. An uber journey in distributed deep learning. Youtube. [Online]. Available: https://www.youtube.com/watch?v=SphfeTl70MI

  24. [32]

    MNIST handwritten digit database,

    Y . LeCun and C. Cortes, “MNIST handwritten digit database,” 2010. [Online]. Available: http://yann.lecun.com/exdb/mnist/

    2010

  25. [33]

    Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems,

    T. Chen, M. Li, Y . Li, M. Lin, N. Wang, M. Wang, T. Xiao, B. Xu, C. Zhang, and Z. Zhang, “Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems,” arXiv preprint arXiv:1512.01274, 2015

    2015 arXiv

  26. [34]

    Petuum: A new platform for distributed machine learning on big data,

    E. P. Xing, Q. Ho, W. Dai, J. K. Kim, J. Wei, S. Lee, X. Zheng, P. Xie, A. Kumar, and Y . Yu, “Petuum: A new platform for distributed machine learning on big data,” IEEE Transactions on Big Data , vol. 1, no. 2, pp. 49–67, 2015

    2015

  27. [35]

    Under- standing and optimizing the performance of distributed machine learning applications on apache spark,

    C. D ¨unner, T. Parnell, K. Atasu, M. Sifalakis, and H. Pozidis, “Under- standing and optimizing the performance of distributed machine learning applications on apache spark,” arXiv preprint arXiv:1612.01437 , 2016

    2016 arXiv

  28. [36]

    Performance modeling of distributed deep neural networks,

    S. H. Hashemi, S. A. Noghabi, W. Gropp, and R. H. Campbell, “Performance modeling of distributed deep neural networks,” arXiv preprint arXiv:1612.00521, 2016

    2016 arXiv

  29. [37]

    Optimizing network performance in distributed machine learning

    L. Mai, C. Hong, and P. Costa, “Optimizing network performance in distributed machine learning.” in HotCloud, 2015

    2015

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.