Pith. sign in

REVIEW 4 major objections 4 minor 39 references

On the Acceleration of Deep Learning Model Parallelism with Staleness

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Diversely Stale Parameters removes forward, backward, and update locking in parallel deep learning and provably converges to critical points, with measured speedups up to 4.8x.

desk verdict A genuinely new layer-wise staleness mechanism with solid-looking experiments, but the convergence theorem doesn't control the bias it explicitly permits, so the central claim doesn't stand. read the letter →

arxiv 1909.02625 v3 pith:4TS6E76O submitted 2019-09-05 cs.LG cs.DCstat.ML

An Xu , Zhouyuan Huo , Heng Huang This is my paper · ORCID
classification cs.LGcs.DCstat.ML
keywords modelparallelismlayer-wisestalenessasynchronoustrainingbackpropagationlockingsnon-convexconvergenceconvolutionalneuralnetworksrecomputationpipeline
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Backpropagation imposes three waits when a network is split across devices: data cannot move up until lower blocks finish, gradients cannot move down until upper blocks finish, and no parameter update can start until both sweeps are done. The paper claims that all three waits can be eliminated by giving each block a fixed, different staleness level, so lower blocks update with older parameters while upper blocks use fresher ones. Its algorithm, Diversely Stale Parameters (DSP), computes each block's gradient from a mix of current activations and deliberately stale weights, with recomputation keeping memory low. The authors prove that with plain SGD and with momentum SGD, DSP converges to points where the gradient is zero (critical points) for non-convex problems. In experiments on ResNet, VGG-19, ResNeXt-29, and ImageNet, DSP trains 1.5x to 4.8x faster than standard backpropagation or Features Replay baselines, with final test accuracy that matches or slightly exceeds them.

What carries the argument

The load-bearing object is Layer-wise Staleness, $\Delta t_k = t_{2K-1-k} - t_k$, the preset gap between the timestamp at which block $k$'s parameters are used in the forward pass and the timestamp at which they are updated in the backward pass. DSP combines this with three FIFO queues per block (input, output, gradient) whose lengths satisfy $q_k = m_{k-1} - p_{k-1} - m_k > 0$, a constraint that makes the error gradient meet the activation of the same data item when the backward pass runs. The DSP gradient (Eq. 5) replaces the unavailable future-parameter gradient of Eq. 4 by the parameters already seen plus the parameters being updated, using recomputation to supply missing activations. That replacement is what removes forward, backward, and update lockings in one mechanism, while the queue constraints keep the computation correct.

What would settle it

Train the same model with DSP and with standard backpropagation from the same initialization on a problem where the stale forward parameters start far from any critical point, and track the per-block norm $\|G_{\mathrm{DSP}} - G_{\mathrm{BP}}\|$ in the first hundred steps. If that gap does not shrink rapidly as training proceeds, or if DSP's loss fails to decrease while backpropagation's does, the equivalence behind Eq. (5) is not doing its load-bearing work.

Watch

Extended reading notes

Core claim

The paper claims that a neural network split into $K$ blocks can be trained in parallel without synchronization barriers, as long as each block $k$ uses parameters with different ages. The DSP gradient (Eq. 5) computes the forward pass with the freshest available activations and the backward pass with deliberately older parameters; recomputation of a block's forward output is overlapped with the forward pass of other data, so memory stays low. Under Lipschitz-gradient, bounded-variance, and bounded-error-gradient assumptions, Theorem 1 (SGD) and Theorem 2 (momentum SGD) give $\mathcal{O}(1/\sqrt{N})$ convergence rates to critical points for non-convex objectives, even though the gradient is biased. The experiments report speedups between 1.5x and 4.8x over standard backpropagation or Features Replay on ResNet, VGG-19, ResNeXt-29, and ImageNet models, with best test accuracy at least matching and sometimes beating those baselines. The paper also suggests that stale-gradient noise, which shrinks as training proceeds, acts as a regularizer and helps explain the occasional accuracy gains.

Load-bearing premise

The convergence proof rests on the unproved premise that computing gradients with stale forward parameters is equivalent to differentiating at a parameter configuration where the true gradient is zero; if the stale forward parameters are not close to such a zero-gradient configuration, the gradient bias is uncontrolled and the convergence guarantee does not follow.

Editorial extensions

If this is right

  • Model parallelism no longer needs to wait for whole-network forward and backward sweeps, so device idle time in a K-block split can be reduced to near zero.
  • The same convergence guarantee holds for both plain SGD and momentum SGD, with an $\mathcal{O}(1/\sqrt{N})$ rate to critical points for non-convex problems.
  • DSP's memory footprint stays linear in network size plus small queue buffers, because recomputation is overlapped with the forward pass instead of storing all intermediate activations.
  • Longer FIFO queues translate into resilience to random slow devices (stragglers): in the paper's slowdown experiments DSP degrades far less than Features Replay.
  • Final test accuracy can match or exceed standard backpropagation, meaning the speedup does not have to be traded away against model quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The mechanism should transfer to transformer-style and other sequential architectures, since the proof only assumes layerwise Lipschitz gradients and bounded error gradients, not convolution structure; testing DSP on language models would check that scope.
  • Because the convergence bound depends on maximum Layer-wise Staleness rather than on the per-block arrangement, an adaptive schedule that shrinks staleness near the end of training could keep the early speedup while tightening the gradient bias later.
  • The regularizer interpretation suggests a controlled experiment: train the same architecture with DSP and with backpropagation plus injected noise matched in magnitude to the measured DSP-BP gradient gap, to see whether the accuracy gain is reproduced.
  • The paper leaves queue depths as hand-set hyperparameters; a practical extension would be to size them automatically from measured device speeds and layer compute times.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript proposes Layer-wise Staleness and a training algorithm called Diversely Stale Parameters (DSP) for model-parallel CNN training. DSP decouples the forward and backward passes across blocks with preset staleness values and uses recomputation to reduce memory, with the goal of breaking forward, backward, and update locking while remaining robust to stragglers. The paper claims convergence of DSP to critical points for non-convex problems when used with SGD (Theorem 1, Corollary 1.2) and momentum SGD (Theorem 2), and reports experiments on CIFAR, ImageNet, and several architectures showing speedups up to 4.8x and test accuracy comparable to or better than backpropagation and Features Replay.

Significance. If the convergence guarantee were valid, this would be a useful contribution: DSP attacks all three lockings simultaneously, overlaps recomputation with the forward pass, and the experiments indicate practical speedups and straggler robustness. The empirical study is reasonably broad and the gradient-difference measurements in Figure 5 provide an interesting check of Lemma 1. However, the advertised theoretical guarantee is the paper's central novelty, and the convergence claim as stated is not supported: the analysis allows a biased gradient estimator without proving that the bias disappears at critical points, and the central theorems are asserted without derivations. The contribution in its current form cannot be accepted as a rigorous convergence paper.

major comments (4)
  1. [Section 3.2, Eq. (5)] The justification of the DSP gradient is circular. The text says that Eq. (5) is equivalent to Eq. (4) at a zero-gradient parameter point x*, and that this is reasonable because parameters gradually converge to the optima. That is exactly the convergence behavior that must be proved. The manuscript provides no bound showing that the difference between the biased DSP gradient and the true gradient of the actual objective tends to zero along the DSP trajectory; rather, it assumes the conclusion in the motivation of the algorithm.
  2. [Section 4, Assumption 1 and Theorems 1-2] Assumption 1 explicitly permits E[G(x;\xi)] = G(x) \neq \nabla f(x), so the estimator can be biased by a nonzero amount that is not controlled by the variance bound. The theorems bound E||\nabla f(x'_n)||^2, where x'_n are the forward parameters, while the update uses the backward parameters x_n with the biased gradient G. No assumption or lemma forces the bias G(x_n) - \nabla f(x_n) to vanish as n grows. Consequently, Corollary 1.2's conclusion lim E||\nabla f(x_n)||^2 = 0 does not follow from the stated theorem: a constant nonzero bias satisfies Assumption 1 and yields a fixed point of E[G] = 0 that is not a critical point of f. This is a load-bearing gap in the central claim.
  3. [Section 4, Lemma 1 and Theorems 1-2] The main theoretical results are stated without proofs. Lemma 1 bounds the per-sample deviation of the DSP gradient from the BP gradient by sums of staleness gaps, but the theorems do not show how these lags are controlled along the trajectory, how they enter the learning-rate condition, or how they vanish in the limit. Because the paper's main contribution is the convergence guarantee, omitting the derivations makes the technical conditions unverifiable. The authors should provide complete proofs or a clearly identified supplement.
  4. [Section 3.3, Eq. (6)] The constraint m_k > 0 for all k \in {0,...,K-1} is inconsistent with the experimental configurations. For example, with K=3 the configurations DSP(1,1,0;4,2,0), DSP(2,2,0;6,3,0), and DSP(5,5,0;14,7,0) all set m_2 = 0. If the last block is allowed to have zero Layer-wise Staleness, the constraint and the definition of Layer-wise Staleness need to be revised; if not, the experiments are run outside the formally analyzed setting.
minor comments (4)
  1. [Eq. (4), third line] The expression for G_hK appears to be missing a derivative operator in the denominator; it reads like a fraction with 'F' instead of an appropriate variation, which makes the formula ambiguous.
  2. [Theorem 1 and Corollary 1.2] The notation switches from x'_n in Theorem 1 to x_n in Corollary 1.2 without explicitly defining the relationship between the forward and backward parameter sequences in the conclusion.
  3. [Figure 4] The legend uses inconsistent list separators, e.g., 'DSP(1,1,0,4,2,0)' versus 'DSP(1,1,0;4,2,0)', which should be unified.
  4. [Table 2] The definition of 'Slow down percentage' is not given; the text should state how the GPU slowdown is injected and how the slowdown percentage is computed.

Circularity Check

1 steps flagged · score 4.0 of 10

The convergence theorems are formally independent, but Section 3.2 justifies the DSP gradient by assuming the convergence that Theorem 1 and Corollary 1.2 are meant to prove.

  1. other [Section 3.2, DSP Gradient, paragraph following Eq. (5)]
    "The intuition behind the DSP gradient of Eq. (5) is that it is equivalent to Eq. (4) with parameters x* where the gradient is zero (xtk_k = xt2K-1-k_k afterwards), and as the training proceeds the parameters gradually converge to the optima."

    This is the paper's own justification for replacing the unavailable ideal gradient (Eq. 4) with the biased DSP gradient (Eq. 5). The justification asserts that the two coincide at a zero-gradient point and that training drives the parameters to that point, which is exactly the convergence result Theorem 1/Corollary 1.2 are supposed to establish. Thus the validity of the estimator is made to depend on the conclusion of the convergence proof. Assumption 1 explicitly permits E[G(x;ξ)] = G(x) ≠ ∇f(x), and Lemma 1 only bounds the gradient bias by staleness gaps; neither statement proves those gaps vanish along the DSP trajectory. Without such a proof, the step from 'updates driven by a biased estimator converge' to 'lim E||∇f(x_n)||² = 0' rests on the very convergence being derived.

full rationale

The paper is not fitting parameters and calling them predictions, and its experimental evaluation against BP, FR, DNI, and GPipe is self-contained external benchmarking. The citations to the authors' earlier DDG and FR work are used as comparison baselines and motivation, not as a load-bearing uniqueness or derivation chain. The formal convergence theorems (Theorem 1, Theorem 2, and corollaries) are stated with explicit assumptions (bounded variance, Lipschitz gradients, bounded error gradients) and are not circular by construction. The one genuinely circular element is the informal justification of the DSP gradient in Section 3.2: Eq. (5) is said to be acceptable because it matches Eq. (4) at a zero-gradient configuration and because 'as the training proceeds the parameters gradually converge to the optima.' That is a self-referential premise — it assumes the convergence that the theorem is meant to prove. The formal proof might in principle supply an independent bound on the staleness gaps via Lemma 1, but the paper as written does not exhibit such a bound, and Assumption 1 explicitly allows a persistent bias. I therefore flag this as a partial circularity in the derivation narrative while recognizing that the central theorems and experiments have substantial independent content.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a set of smoothness and boundedness assumptions (Assumptions 1-3), on the unproved gradient-equivalence premise in Section 3.2, and on the validity of queue configurations. The layer-wise staleness values are user-chosen hyperparameters, not derived quantities. No new physical entities are introduced; the main new object is the biased DSP gradient estimator, which is represented by the axioms above.

free parameters (3)
  • Layer-wise Staleness / queue lengths (p_k, m_k; Delta t_k) = e.g., (1,1,0;4,2,0), (2,2,0;6,3,0), (5,5,0;14,7,0)
    Chosen by hand for each model; controls the trade-off between speedup, memory, and accuracy. No selection rule or sensitivity analysis is provided.
  • Learning rate schedule = 0.01 or 0.1 with decays at epochs 150/225, 100/150/200, or 30/60/80
    Standard training hyperparameter, but final accuracy numbers depend on it; no tuning analysis is provided.
  • Number of blocks K and split positions = K=3 or 4 in the experiments
    Manually chosen; affects the convergence bound through Delta t and determines the speedup numbers.
assumptions (5)
  • domain assumption Each block's output and the full loss have Lipschitz continuous gradients (Assumption 2).
    Assumed for all block outputs and the loss; needed for Lemma 1 and Theorems 1-2. Standard smoothness but not guaranteed for ReLU networks with batch normalization.
  • domain assumption Error gradients received by each block are bounded by M (Assumption 3).
    Bounds the propagation Jacobian to prevent explosion; not verified empirically; needed for the bias bound in Lemma 1.
  • domain assumption DSP stochastic gradient has bounded variance and expectation G(x) not equal to the true gradient (Assumption 1).
    The bias E[G]=G is not equal to grad f is acknowledged; convergence to exact critical points requires the bias to vanish, which is not shown.
  • ad hoc to paper DSP gradient in Eq. (5) approximates the unavailable BP gradient because it would equal Eq. (4) at a zero-gradient parameter point.
    Section 3.2 states this equivalence without proof; it is the core justification for replacing future parameters with stale ones.
  • domain assumption Queue-length constraints Eq. (6) can be satisfied and ensure each error gradient meets the activation of the same data.
    The constraints are assumed to define valid DSP configurations; examples include m_{K-1}=0, contradicting m_k>0, so the constraint set is internally inconsistent.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Acceleration of Deep Learning Model Parallelism with Staleness." pith.science (2026). https://pith.science/paper/4TS6E76O

@misc{pith2026190902625,
  author       = {Pith},
  title        = {Pith review of: On the Acceleration of Deep Learning Model Parallelism with Staleness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4TS6E76O}},
  note         = {Machine review of arXiv:1909.02625}
}
read the original abstract

Training the deep convolutional neural network for computer vision problems is slow and inefficient, especially when it is large and distributed across multiple devices. The inefficiency is caused by the backpropagation algorithm's forward locking, backward locking, and update locking problems. Existing solutions for acceleration either can only handle one locking problem or lead to severe accuracy loss or memory inefficiency. Moreover, none of them consider the straggler problem among devices. In this paper, we propose Layer-wise Staleness and a novel efficient training algorithm, Diversely Stale Parameters (DSP), to address these challenges. We also analyze the convergence of DSP with two popular gradient-based methods and prove that both of them are guaranteed to converge to critical points for non-convex problems. Finally, extensive experimental results on training deep learning models demonstrate that our proposed DSP algorithm can achieve significant training speedup with stronger robustness than compared methods.

Figures

Figures reproduced from arXiv: 1909.02625 by the authors.

Figure 1
Figure 1. Comparison of different methods with three blocks. The forward and recomputation are overlapped in DSP. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. A DSP data traversal in a K-block neural network. Red arrows denote the forward pass; blue arrows denote [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. DSP(p0, p1, 0; m0, m1, m2) for parallel training (K=3). The input queue is at the L.H.S of a block, while the output queue is at the R.H.S. The gradient queue is below the input queue. because each block cannot compute gradients until having received the error gradient Gh from the upper block. Besides, the backward process can not start until the whole forward process is completed, which is known as the update locki… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Training loss (solid line) and testing loss (dash line) for ResNet98, ResNet164 on CIFAR-10. The first row [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Top left: Average difference of DSP and BP gradient regarding the number of parameters. The rest: Training [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 24 canonical work pages

  1. [1]

    Abadi, P

    M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI’ 16), pages 265–283, 2016

    work page 2016

  2. [2]

    Decoupled Greedy Learning of CNNs

    E. Belilovsky, M. Eickenberg, and E. Oyallon. Decoupled greedy learning of cnns. arXiv preprint arXiv:1901.08164, 2019

    work page Pith review arXiv 1901

  3. [3]

    L. Bottou. Large-scale machine learning with stochastic gradient descent. In Y . Lechevallier and G. Saporta, editors, Proceedings of COMPSTAT’2010, pages 177–186, Heidelberg, 2010. Physica-Verlag HD

    work page 2010

  4. [4]

    T. Chen, M. Li, Y . Li, M. Lin, N. Wang, M. Wang, T. Xiao, B. Xu, C. Zhang, and Z. Zhang. Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv:1512.01274, 2015

    arXiv 2015

  5. [5]

    T. Chen, B. Xu, C. Zhang, and C. Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016

    arXiv 2016

  6. [6]

    Choromanska, M

    A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, and Y . LeCun. The loss surfaces of multilayer networks. In Artificial Intelligence and Statistics, pages 192–204, 2015

    2015

  7. [7]

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  8. [8]

    Griewank

    A. Griewank. An implementation of checkpointing for the reverse or adjoint model of differentiation. ACM Trans. Math. Software, 26(1):1–19, 1999. 9 A PREPRINT - S EPTEMBER 25, 2019

    work page 1999

Show all 39 references
  1. [9]

    K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015

    2015

  2. [10]

    Q. Ho, J. Cipar, H. Cui, S. Lee, J. K. Kim, P. B. Gibbons, G. A. Gibson, G. Ganger, and E. P. Xing. More effective distributed ml via a stale synchronous parallel parameter server. In Advances in neural information processing systems, pages 1223–1231, 2013

    2013

  3. [11]

    J. Hu, L. Shen, and G. Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018

    2018

  4. [12]

    Huang, Y

    Y . Huang, Y . Cheng, D. Chen, H. Lee, J. Ngiam, Q. V . Le, and Z. Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. arXiv preprint arXiv:1811.06965, 2018

    2018 arXiv

  5. [13]

    Huang, X

    Y . Huang, X. Yan, G. Jiang, T. Jin, J. Cheng, A. Xu, Z. Liu, and S. Tu. Tangram: bridging immutable and mutable abstractions for distributed data analytics. In 2019{USENIX} Annual Technical Conference ({USENIX}{ATC} 19), pages 191–206, 2019

    2019

  6. [14]

    Z. Huo, B. Gu, and H. Huang. Training neural networks using features replay. In Advances in Neural Information Processing Systems, pages 6659–6668, 2018

    2018

  7. [15]

    Z. Huo, B. Gu, qian Yang, and H. Huang. Decoupled parallel backpropagation with convergence guarantee. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2098–2106,...

    2018

  8. [16]

    Ioffe and C

    S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015

    2015 arXiv

  9. [17]

    Jaderberg, W

    M. Jaderberg, W. M. Czarnecki, S. Osindero, O. Vinyals, A. Graves, D. Silver, and K. Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1627–1635. JMLR. org, 2017

    2017

  10. [18]

    Kawaguchi

    K. Kawaguchi. Deep learning without poor local minima. In Advances in neural information processing systems, pages 586–594, 2016

    2016

  11. [19]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  12. [20]

    Krizhevsky

    A. Krizhevsky. One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997, 2014

    2014 arXiv

  13. [21]

    LeCun, B

    Y . LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541–551, 1989

    1989

  14. [22]

    S. Lee, J. K. Kim, X. Zheng, Q. Ho, G. A. Gibson, and E. P. Xing. On model parallelization and scheduling strategies for distributed machine learning. In Advances in neural information processing systems, pages 2834– 2842, 2014

    2014

  15. [23]

    M. Li, D. G. Andersen, A. J. Smola, and K. Yu. Communication efficient distributed machine learning with the parameter server. In Advances in Neural Information Processing Systems, pages 19–27, 2014

    2014

  16. [24]

    T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015

    2015 arXiv

  17. [25]

    Y . Liu, A. Xu, and Z. Chen. Map-based deep imitation learning for obstacle avoidance. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8644–8649. IEEE, 2018

    2018

  18. [26]

    Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell. Rethinking the value of network pruning. arXiv preprint arXiv:1810.05270, 2018

    2018 arXiv

  19. [27]

    Nesterov

    Y . Nesterov. Introductory lectures on convex optimization: A basic course , volume 87. Springer Science & Business Media, 2013

    2013

  20. [28]

    A. Nøkland. Direct feedback alignment provides learning in deep neural networks. In Advances in neural information processing systems, pages 1037–1045, 2016

    2016

  21. [29]

    Robbins and S

    H. Robbins and S. Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400–407, 1951

    1951

  22. [30]

    D. E. Rumelhart, G. E. Hinton, R. J. Williams, et al. Learning representations by back-propagating errors. Cognitive modeling, 5(3):1, 1988

    1988

  23. [31]

    Simonyan and A

    K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 10 A PREPRINT - S EPTEMBER 25, 2019

    2014 arXiv

  24. [32]

    Szegedy, W

    C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V . Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015

    2015

  25. [33]

    Szegedy, V

    C. Szegedy, V . Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818–2826, 2016

    2016

  26. [34]

    Tamar, Y

    A. Tamar, Y . WU, G. Thomas, S. Levine, and P. Abbeel. Value iteration networks. In D. D. Lee, M. Sugiyama, U. V . Luxburg, I. Guyon, and R. Garnett, editors,Advances in Neural Information Processing Systems 29, pages 2154–2162. Curran Associates, Inc., 2016

    2016

  27. [35]

    Tieleman and G

    T. Tieleman and G. Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26–31, 2012

    2012

  28. [36]

    L. G. Valiant. A bridging model for parallel computation. Communications of the ACM, 33(8):103–111, 1990

    1990

  29. [37]

    S. Xie, R. B. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5987–5995, 2017

    2017

  30. [38]

    Q. Yang, Z. Huo, W. Wang, H. Huang, and L. Carin. Ouroboros: On accelerating training of transformer-based language models. arXiv preprint arXiv:1909.06695, 2019

    1909 arXiv

  31. [39]

    T. Yang, Q. Lin, and Z. Li. Unified convergence analysis of stochastic momentum methods for convex and non-convex optimization. arXiv preprint arXiv:1604.03257, 2016. 11

    2016 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.