Accelerated CNN Training Through Gradient Approximation

Sree Harsha Nelaturu; Ziheng Wang

REVIEW 3 major objections 5 minor 33 references

Accelerated CNN Training Through Gradient Approximation

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Selectively approximating filter gradients—keeping one scaled entry per output channel—can speed up CNN training in wall-clock time while keeping validation accuracy nearly unchanged.

desk verdict A real GPU-kernel contribution with a small, plausible speedup, but the reporting is sloppy: no error bars, an abstract/body speedup inconsistency, and an unbiasedness claim that is mathematically wrong but blessedly not load-bearing. read the letter →

arxiv 1908.05460 v1 pith:MKNUG7SU submitted 2019-08-15 cs.CV cs.LG

Ziheng Wang , Sree Harsha Nelaturu This is my paper · ORCID

classification cs.CVcs.LG

keywords gradientapproximationCNNtrainingaccelerationsparsebackpropagationCUDAkernelsschedulesResNet-20VGG-19CIFAR-10

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that deep convolutional networks can be trained faster in real wall-clock time by computing only a very sparse approximation of the gradients on a subset of layers and batches, rather than using the exact dense gradient. It proposes three approximation methods—zeroing the gradient, replacing it with fresh random noise, and keeping only the largest-magnitude value per output channel scaled by the channel sum—and shows that they all let ResNet-20, VGG-19, and a small 2-layer CNN converge on CIFAR-10 with little validation-accuracy loss. The main payoff is the approximated-gradient method, whose custom CUDA kernel turns the sparse gradient computation into fast patch extractions and reports wall-clock training speedups over full cuDNN-based training, with the modest speedup being a practical consequence of approximating only a quarter of the gradient computations.

What carries the argument

The load-bearing object is the max-gradient approximation: for each output channel and each batch element, keep only the largest-magnitude entry of the $H\times W$ output-gradient map, scale it by the sum of all entries in that map, and zero everything else, so the filter gradient becomes a sum of $K\times K$ input patches weighted by those scaled scalars. The argument wraps this in an 'approximation schedule'—a per-layer, per-batch choice of which approximation to apply—which lets the method touch only a quarter of filter-gradient computations. The CUDA kernel that interprets the sparse convolution as patch extraction is what converts the algorithmic saving into actual wall-clock speed.

What would settle it

Compare, over many random input/backprop pairs at an untrained layer, the average approximated filter gradient with the average full filter gradient; a persistent nonzero difference would show the estimate is biased and that convergence is driven by something other than unbiased gradient estimation.

Watch

Extended reading notes

Core claim

The central claim, stated as the authors would state it: training deep CNNs does not require exact dense filter gradients at every layer and every batch. On a schedule that approximates a quarter of the filter-gradient computations (on every fourth convolutional layer, every batch, starting from the second layer), keeping only the single largest-magnitude value in each output channel's gradient map—scaled by that map's total sum and all other values zeroed—is enough to train ResNet-20 and VGG-19 to validation accuracy within a small margin of full-gradient training. Implemented as a CUDA patch-extraction kernel, this approximation is faster than cuDNN's dense gradient convolution in the tested cases, and the measured wall-clock speedups range from 3.5% for ResNet-20 up to over 7% for the two networks, with accuracy loss as low as 0.1 percentage point under some schedules.

Load-bearing premise

The load-bearing premise is that the scaled top-1 approximation produces an unbiased estimate of the true filter gradient; if that premise fails—and deterministic retention of the largest element does not by itself make each position unbiased—the theoretical justification collapses even though the training curves may still converge.

Editorial extensions

If this is right

Approximating one quarter of the filter-gradient computations gives a theoretical speedup ceiling of roughly 8 percent, and measured wall-clock speedups on CIFAR-10 range from 3.5 percent on ResNet-20 to over 7 percent across the two networks.
The approximated-gradient method keeps validation accuracy within about 0.1 percentage point of full-gradient training under at least one schedule, making the accuracy loss smaller than the loss from substituting a smaller ResNet-14 baseline.
The approximation methods require no network structure changes or explicit sparsity inputs, so they can be dropped into standard VGG and ResNet training pipelines.
Using approximations early in training and switching to full gradients later is a promising schedule direction for VGG-like networks, since full-gradient training overtakes the approximations only at the end.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: The deterministic top-1 selection scaled by channel sum is not unbiased per-position, so if the method trains well, the explanation likely lies in SGD optimization dynamics rather than in unbiasedness; averaging over many layers and batches may hide systematic bias.
Editorial inference: The success of freshly regenerated random gradients on some architectures, with no fixed alignment, suggests a testable hypothesis that stochastic gradient noise itself, not directional alignment, keeps the network trainable.
Editorial inference: Fusing the input transposes into the CUDA kernel could remove more than half the reported overhead, indicating that larger speedups are available without changing the approximation itself.
Editorial inference: The same approximation could be applied to activation gradients in deeper layers, but error propagation to shallower layers would need to be controlled; the paper's own reason for avoiding that direction sets a boundary for future work.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The genuinely new thing here is the CUDA kernel: a patch-extraction implementation that computes the top-1 sparse filter gradient faster than cuDNN in isolation, with coalesced memory access and a reasonable treatment of transpose overhead. That is a concrete systems contribution, and the paper earns credit for it. The approximation schedules framing is also useful, even if the underlying methods (top-k, zero, random) are all known. The authors are honest about negative results: the 2-layer CNN slowdown, the inefficient random-gradient implementation, and the fact that a smaller ResNet-14 is often competitive. Citation patterns look fair; there is no circularity.

The soft spots are real but not all equally soft. The Sec. 2.3 unbiasedness claim is indeed wrong: deterministic top-1 selection scaled by the channel sum does not produce an unbiased per-weight estimator. However, the experiments are empirical convergence demonstrations; nothing in the measured speedups depends on that claim. It is a theoretical framing error, easily fixed by calling it a biased but effective estimator. The bigger problem is the lack of uncertainty quantification: single runs, no seeds, validation accuracy sampled every ten epochs, and the abstract's "upwards of 7%" speedup is never reconciled with the 3.5% reported for ResNet-20 in the body. A few-percent wall-clock speedup is well within run-to-run noise on a shared GPU, so the central claim is under-evidenced as written. The tables that would let a reader check the kernel benchmarking are referenced but not shown in the text I have, which makes independent verification harder. This is a reporting problem, not a fatal flaw: the kernel benchmark and the measured accuracy differences are plausible.

Who is this for? People working on gradient sparsification, GPU kernels for training, or practical techniques to shave a few percent off training time. It is not a field-shaping result. With code, seed-averaged speedups, and a cleaned-up abstract, it could be a solid workshop or short-paper contribution. I would send it to a serious reviewer with a request for those revisions; it deserves referee time, but not a clean acceptance as-is.

Referee Report

3 major / 5 minor

Summary. The paper proposes three gradient approximation methods for CNN training — zero gradient, random gradient, and a top-1 approximated gradient — and introduces the notion of approximation schedules that specify when each approximation is applied per layer and per training batch. A custom CUDA kernel implements the approximated gradient, and the paper reports wall-clock training speedups on CIFAR-10 for ResNet-20 and VGG-19, claiming speedups 'upwards of 7%' with minimal validation accuracy loss. The evaluation includes kernel timing comparisons against cuDNN, convergence curves for three network architectures, and robustness checks across three approximation schedules for ResNet-20.

Significance. If the reported speedups and accuracy results are reproducible, the paper makes a modest but useful systems contribution: a GPU-friendly sparse gradient approximation that can reduce training time for standard deep CNNs without significant accuracy loss. The paper's strengths include a nontrivial CUDA kernel implementation, explicit accounting for overheads such as data transposes, and transparent reporting of negative results (e.g., the slowdown on a small 2-layer CNN). The approximation-schedule abstraction is also a useful conceptual contribution. However, the central empirical claim is not yet convincing because the reported speedups are small, the experiments appear to be single runs without error bars, and the abstract's headline speedup is not reconciled with the body.

major comments (3)

[Section 2.3, Algorithm 1] The claim that the approximated gradient estimate is unbiased is mathematically incorrect. The retained element is selected deterministically as the largest-magnitude gradient, and scaling by the channel sum preserves only the total sum, not the per-coordinate expectations. For an unbiased estimator, the selection would need to be randomized with an appropriate inverse-probability rescaling, as in standard gradient sparsification methods. This error undermines the theoretical justification for the approximated gradient method, even though the empirical convergence curves might still be valid; the authors should either provide a correct unbiasedness argument, characterize the bias, or remove the claim.
[Abstract and Section 3.3] The abstract's headline claim of wall-clock speedup 'upwards of 7%' is not supported by the body: the only explicitly reported speedup in the text is 3.5% for ResNet-20 with the approximated gradient, and no VGG-19 speedup is stated in prose or referenced to a table. Moreover, Tables 1–5 are referenced but their contents are not present in the manuscript, so the underlying timing and accuracy numbers cannot be checked. The authors must include the actual table data and reconcile the abstract with the reported results.
[Section 3 (Evaluation)] All experiments appear to be single runs with no error bars, standard deviations, or confidence intervals. Since the reported speedups are only a few percent, they are within the typical run-to-run variation of GPU training time. In addition, validation accuracy is recorded only every ten epochs, so the 'best validation accuracy' is a coarse sample. The authors should run multiple seeds for each configuration and report the mean and variance (or at least the observed range) for both wall-clock time and validation accuracy. Without this, the central empirical claim is not established.

minor comments (5)

[Section 2.2] 'tf.py func' should be written as 'tf.py_func'.
[Section 3.4] The phrase 'in now line with its poor loss curve' appears to be a typo; it should likely be 'in line with' or 'no longer in line with'.
[Section 4] 'its performance serious lags other methods' should be 'its performance seriously lags other methods'.
[Section 3] The training setup does not specify key hyperparameters such as batch size, initial learning rate, learning rate decay schedule, weight decay, or the exact number of training steps; providing these details would improve reproducibility.
[Section 2.4] The notation for the filter tensor dimension 'KKC iCo' is unclear without explicit multiplication signs or subscripts; please define the dimensions unambiguously.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the speedup and convergence claims are benchmarked against external baselines, and no prediction reduces to a fitted input or self-citation.

full rationale

The paper's derivation chain is self-contained and does not reduce to its inputs by construction. The central speedup claim is measured against NVIDIA cuDNN v7.4.2, an external baseline, using wall-clock timing on a Titan-Xp GPU, so the headline result is not equivalent to a fitted parameter or a self-citation. The gradient approximation methods (zero, random, and top-k approximated gradients) are defined algorithmically, and the scale factor 1/128 and the 25% approximation schedules are hand-chosen and openly stated rather than fitted to the final validation accuracy or speedup. The paper contains no load-bearing self-citation: the cited works on sparsified backpropagation and meProp are prior external work, not the authors' own uniqueness claim. The Section 2.3 assertion that channel-sum scaling makes the top-k gradient estimate unbiased is a mathematical claim that may be incorrect, but incorrectness is a correctness risk, not circularity, because the claim is not derived from the target result and the experiments do not depend on it for their empirical comparison. The discrepancy between the abstract's 'upwards of 7%' and the body's 3.5% ResNet-20 speedup, and the absence of multiple seeds or error bars, are evidence-quality concerns, not circular construction. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper's central claim rests on a small number of hand-chosen hyperparameters (scaling, schedule) and on two assumptions that are not proven: the locality of approximation error and the unbiasedness of the top-1 scaled estimate. The unbiasedness assumption is questionable.

free parameters (3)

Gradient scaling factor = 1/128
Used to scale both the random and approximated gradients (Sections 2.2, 2.3). The paper states it is proportional to batch size, but no derivation is given; with Adam it may be absorbed, but it is still a hand-chosen constant.
Approximation schedule = every fourth conv layer starting from second, all batches (schedule 1); other schedules in Section 3.4
The fraction of approximated computation (25%) and the layer/batch selection are chosen by hand, and the paper notes that determining the optimal schedule is future work.
Training epochs = 500/250/200 for the three networks
Training length is chosen as the point where validation accuracy stabilizes; this is standard but adds flexibility in the results.

assumptions (3)

domain assumption Approximating weight gradients is safe because errors remain local and do not propagate to other layers.
Stated in Section 5 (Future Work): 'if we approximate the hidden activations at a deeper layer of the network, the approximation error will be propagated to the shallower layers. Due to this concern, we start with approximating filter weight gradients, where the effect of errors are local.' This assumption underlies why the method can be applied selectively.
ad hoc to paper The top-1 selection with channel-sum scaling yields an unbiased gradient estimate.
Claimed in Section 2.3 without proof. Deterministic selection of the largest magnitude element and scaling by the channel sum preserves the sum but not the individual coordinates, so the resulting filter gradient is generally biased.
domain assumption cuDNN v7.4.2 is a representative baseline for full gradient computation.
Used in Section 3.1 for all timing benchmarks; the speedup claims are relative to this single library.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Accelerated CNN Training Through Gradient Approximation." pith.science (2026). https://pith.science/paper/MKNUG7SU

@misc{pith2026190805460,
  author       = {Pith},
  title        = {Pith review of: Accelerated CNN Training Through Gradient Approximation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MKNUG7SU}},
  note         = {Machine review of arXiv:1908.05460}
}

read the original abstract

Training deep convolutional neural networks such as VGG and ResNet by gradient descent is an expensive exercise requiring specialized hardware such as GPUs. Recent works have examined the possibility of approximating the gradient computation while maintaining the same convergence properties. While promising, the approximations only work on relatively small datasets such as MNIST. They also fail to achieve real wall-clock speedups due to lack of efficient GPU implementations of the proposed approximation methods. In this work, we explore three alternative methods to approximate gradients, with an efficient GPU kernel implementation for one of them. We achieve wall-clock speedup with ResNet-20 and VGG-19 on the CIFAR-10 dataset upwards of 7%, with a minimal loss in validation accuracy.

Figures

Figures reproduced from arXiv: 1908.05460 by the authors.

**Figure 3.** Example approximation schedule for a 5-layer network [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

**Figure 4.** The three approximation schedules studied for the 2- [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

Figures from the paper (3 more)

**Figure 6.** Figure 6: a) Training loss of ResNet-20 with different approxima [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]

**Figure 7.** Figure 7: a) Training loss of VGG-19 model with different approx [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

**Figure 8.** Figure 8: a) Training loss of ResNet-20 with different approxima [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 15 canonical work pages

[1]

Tensorﬂow: A system for large-scale machine learning

Mart ´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghe- mawat, Geoffrey Irving, Michael Isard, et al. Tensorﬂow: A system for large-scale machine learning. In 12th {USENIX} Symposium on Operating Systems Design and Implementa- tion ({OSDI} 16), pages 265–283, 2016

2016
[2]

Faster Neural Network Training with Approximate Tensor Operations

Menachem Adelman and Mark Silberstein. Faster neural network training with approximate tensor operations. arXiv preprint arXiv:1805.08079, 2018

work page Pith review arXiv 2018
[3]

Ex- tremely large minibatch sgd: Training resnet-50 on imagenet in 15 minutes

Takuya Akiba, Shuji Suzuki, and Keisuke Fukuda. Ex- tremely large minibatch sgd: Training resnet-50 on imagenet in 15 minutes. arXiv preprint arXiv:1711.04325, 2017

arXiv 2017
[4]

Assessing the scalability of biologically-motivated deep learning algo- rithms and architectures

Sergey Bartunov, Adam Santoro, Blake Richards, Luke Mar- ris, Geoffrey E Hinton, and Timothy Lillicrap. Assessing the scalability of biologically-motivated deep learning algo- rithms and architectures. In Advances in Neural Information Processing Systems, pages 9368–9378, 2018

work page 2018
[5]

Escort: Efﬁcient sparse convolutional neural networks on gpus

Xuhao Chen. Escort: Efﬁcient sparse convolutional neural networks on gpus. arXiv preprint arXiv:1802.10280, 2018

arXiv 2018
[6]

cudnn: Efﬁcient primitives for deep learning

Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. cudnn: Efﬁcient primitives for deep learning. arXiv preprint arXiv:1410.0759, 2014

arXiv 2014
[7]

Drop- back: Continuous pruning during training

Maximilian Golub, Guy Lemieux, and Mieszko Lis. Drop- back: Continuous pruning during training. arXiv preprint arXiv:1806.06949, 2018

arXiv 2018
[8]

Sub- manifold sparse convolutional networks

Benjamin Graham and Laurens van der Maaten. Sub- manifold sparse convolutional networks. arXiv preprint arXiv:1706.01307, 2017

arXiv 2017

Show all 33 references

[9]

Gpu kernels for block-sparse weights

Scott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights. arXiv preprint arXiv:1711.09224, 2017

2017 arXiv
[10]

Efﬁcient convolutional neural network training with direct feedback alignment

Donghyeon Han and Hoi-jun Yoo. Efﬁcient convolutional neural network training with direct feedback alignment. arXiv preprint arXiv:1901.01986, 2019

1901 arXiv
[11]

Deep com- pression: Compressing deep neural networks with pruning, trained quantization and huffman coding

Song Han, Huizi Mao, and William J Dally. Deep com- pression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015

2015 arXiv
[12]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

2016
[13]

Batch normalization: Accelerating deep network training by reducing internal co- variate shift

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal co- variate shift. arXiv preprint arXiv:1502.03167, 2015

2015 arXiv
[14]

Highly scalable deep learn- ing training system with mixed-precision: Training imagenet in four minutes

Xianyan Jia, Shutao Song, Wei He, Yangzihao Wang, Haidong Rong, Feihu Zhou, Liqiang Xie, Zhenyu Guo, Yuanzhou Yang, Liwei Yu, et al. Highly scalable deep learn- ing training system with mixed-precision: Training imagenet in four minutes. arXiv preprint arXiv:1807.11205, 2018

2018 arXiv
[15]

In-datacenter per- formance analysis of a tensor processing unit

Norman P Jouppi, Cliff Young, Nishant Patil, David Patter- son, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter per- formance analysis of a tensor processing unit. In 2017 ACM/IEEE 44th Annual International Symposium on ...

2017
[16]

Learning multiple layers of features from tiny images

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Cite- seer, 2009

2009
[17]

Random synaptic feedback weights support error backpropagation for deep learning

Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature communications, 7:13276, 2016

2016
[18]

Efﬁ- cient sparse-winograd convolutional neural networks

Xingyu Liu, Jeff Pool, Song Han, and William J Dally. Efﬁ- cient sparse-winograd convolutional neural networks. arXiv preprint arXiv:1802.06367, 2018

2018 arXiv
[19]

Prunetrain: Gradual structured pruning from scratch for faster neural network training

Sangkug Lym, Esha Choukse, Siavash Zangeneh, Wei Wen, Mattan Erez, and Sujay Shanghavi. Prunetrain: Gradual structured pruning from scratch for faster neural network training. arXiv preprint arXiv:1901.09290, 2019

1901 arXiv
[20]

Nvidia tensor core pro- grammability, performance & precision

Stefano Markidis, Steven Wei Der Chien, Erwin Laure, Ivy Bo Peng, and Jeffrey S Vetter. Nvidia tensor core pro- grammability, performance & precision. In 2018 IEEE In- ternational Parallel and Distributed Processing Symposium Workshops (IPDPSW), pages 522–531. IEEE, 2018

2018
[21]

Mixed precision training

Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv:1710.03740, 2017

arXiv
[22]

Direct feedback alignment provides learning in deep neural networks

Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. In Advances in neural information processing systems, pages 1037–1045, 2016

2016
[23]

Faster cnns with di- rect sparse convolutions and guided pruning

Jongsoo Park, Sheng Li, Wei Wen, Ping Tak Peter Tang, Hai Li, Yiran Chen, and Pradeep Dubey. Faster cnns with di- rect sparse convolutions and guided pruning. arXiv preprint arXiv:1608.01409, 2016. 8

2016 arXiv
[24]

Sbnet: Sparse blocks network for fast inference

Mengye Ren, Andrei Pokrovsky, Bin Yang, and Raquel Urta- sun. Sbnet: Sparse blocks network for fast inference. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8711–8720, 2018

2018
[25]

Very deep convo- lutional networks for large-scale image recognition

Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014

2014 arXiv
[26]

meprop: Sparsiﬁed back propagation for accelerated deep learning with reduced overﬁtting

Xu Sun, Xuancheng Ren, Shuming Ma, and Houfeng Wang. meprop: Sparsiﬁed back propagation for accelerated deep learning with reduced overﬁtting. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 3299–3308. JMLR. org, 2017

2017
[27]

Training simpliﬁcation and model simpliﬁcation for deep learning: A minimal effort back propagation method.IEEE Transactions on Knowledge and Data Engineering, 2018

Xu Sun, Xuancheng Ren, Shuming Ma, Bingzhen Wei, Wei Li, Jingjing Xu, Houfeng Wang, and Yi Zhang. Training simpliﬁcation and model simpliﬁcation for deep learning: A minimal effort back propagation method.IEEE Transactions on Knowledge and Data Engineering, 2018

2018
[28]

Gra- dient sparsiﬁcation for communication-efﬁcient distributed optimization

Jianqiao Wangni, Jialei Wang, Ji Liu, and Tong Zhang. Gra- dient sparsiﬁcation for communication-efﬁcient distributed optimization. In Advances in Neural Information Processing Systems, pages 1306–1316, 2018

2018
[29]

Minimal effort back propagation for convolutional neural networks

Bingzhen Wei, Xu Sun, Xuancheng Ren, and Jingjing Xu. Minimal effort back propagation for convolutional neural networks. arXiv preprint arXiv:1709.05804, 2017

2017 arXiv
[30]

Terngrad: Ternary gradi- ents to reduce communication in distributed deep learning

Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Terngrad: Ternary gradi- ents to reduce communication in distributed deep learning. In Advances in neural information processing systems, pages 1509–1519, 2017

2017
[31]

Biologically-plausible learning algorithms can scale to large datasets

Will Xiao, Honglin Chen, Qianli Liao, and Tomaso Poggio. Biologically-plausible learning algorithms can scale to large datasets. arXiv preprint arXiv:1811.03567, 2018

2018 arXiv
[32]

Imagenet training in minutes

Yang You, Zhao Zhang, Cho-Jui Hsieh, James Demmel, and Kurt Keutzer. Imagenet training in minutes. In Proceedings of the 47th International Conference on Parallel Processing, page 1. ACM, 2018

2018
[33]

Structurally sparsiﬁed backward propagation for faster long short-term memory training

Maohua Zhu, Jason Clemons, Jeff Pool, Minsoo Rhu, Stephen W Keckler, and Yuan Xie. Structurally sparsiﬁed backward propagation for faster long short-term memory training. arXiv preprint arXiv:1806.00512, 2018. 9

2018 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Tensorﬂow: A system for large-scale machine learning

Mart ´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghe- mawat, Geoffrey Irving, Michael Isard, et al. Tensorﬂow: A system for large-scale machine learning. In 12th {USENIX} Symposium on Operating Systems Design and Implementa- tion ({OSDI} 16), pages 265–283, 2016

2016

[2] [2]

Faster Neural Network Training with Approximate Tensor Operations

Menachem Adelman and Mark Silberstein. Faster neural network training with approximate tensor operations. arXiv preprint arXiv:1805.08079, 2018

work page Pith review arXiv 2018

[3] [3]

Ex- tremely large minibatch sgd: Training resnet-50 on imagenet in 15 minutes

Takuya Akiba, Shuji Suzuki, and Keisuke Fukuda. Ex- tremely large minibatch sgd: Training resnet-50 on imagenet in 15 minutes. arXiv preprint arXiv:1711.04325, 2017

arXiv 2017

[4] [4]

Assessing the scalability of biologically-motivated deep learning algo- rithms and architectures

Sergey Bartunov, Adam Santoro, Blake Richards, Luke Mar- ris, Geoffrey E Hinton, and Timothy Lillicrap. Assessing the scalability of biologically-motivated deep learning algo- rithms and architectures. In Advances in Neural Information Processing Systems, pages 9368–9378, 2018

work page 2018

[5] [5]

Escort: Efﬁcient sparse convolutional neural networks on gpus

Xuhao Chen. Escort: Efﬁcient sparse convolutional neural networks on gpus. arXiv preprint arXiv:1802.10280, 2018

arXiv 2018

[6] [6]

cudnn: Efﬁcient primitives for deep learning

Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. cudnn: Efﬁcient primitives for deep learning. arXiv preprint arXiv:1410.0759, 2014

arXiv 2014

[7] [7]

Drop- back: Continuous pruning during training

Maximilian Golub, Guy Lemieux, and Mieszko Lis. Drop- back: Continuous pruning during training. arXiv preprint arXiv:1806.06949, 2018

arXiv 2018

[8] [8]

Sub- manifold sparse convolutional networks

Benjamin Graham and Laurens van der Maaten. Sub- manifold sparse convolutional networks. arXiv preprint arXiv:1706.01307, 2017

arXiv 2017