Pith. sign in

REVIEW 2 major objections 5 minor 36 references

Towards Better Generalization: BP-SVRG in Training Deep Neural Networks

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that switching the sign of SVRG's control variate, turning variance reduction into explicit variance addition, yields an optimizer (BP-SVRG) that generalizes better than B-SVRG and often better than Nesterov-accelerated…

desk verdict The sign-flip idea is genuinely novel and the BP-SVRG vs B-SVRG result is consistent, but the compute equalization with SGD is miscalculated, so the headline 'even SGD' claim is not established. read the letter →

arxiv 1908.06395 v1 pith:JILYEZXI submitted 2019-08-18 stat.ML cs.LG

Hao Jin , Dachao Lin , Zhihua Zhang This is my paper · ORCID
classification stat.MLcs.LG
keywords SVRGvariancereductionsignswitchgeneralizationdeepneuralnetworkssharpminimaPolyak-Lojasiewiczconditionmini-batching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish why SVRG, despite its optimization guarantees, generalizes poorly in deep learning, and to fix it. It proposes that two quantities—the average norm of per-sample gradients and the norm of the average gradient—track how flat the loss landscape is and bound the generalization gap under the Polyak-Lojasiewicz condition. Observing that B-SVRG keeps these metrics large, the authors flip the sign of the SVRG control variate, producing BP-SVRG. Numerically, BP-SVRG attains lower test loss and often higher test accuracy than B-SVRG, and matches or beats Nesterov SGD while traversing the dataset fewer times.

What carries the argument

The mechanism is the control variate in the SVRG update, whose sign is reversed: instead of subtracting the snapshot correction $(\nabla f_i(w)-\mu)$, BP-SVRG adds it. The argument that this helps generalization runs through two metrics: $\mathbb{E}_i\|\nabla f_i(w)\|^2$, which proxies data-relevant expected sharpness along gradient directions, and $\|\nabla F(w)\|^2$, the norm of the full average gradient. Under the Polyak-Lojasiewicz condition—a gradient-dominance inequality saying $\|\nabla f(x)\|^2 \ge 2\mu(f(x)-f(x^*))$—these two metrics upper-bound the generalization gap, so lowering them is claimed to lower test error. The sign flip is a deliberate variance-injection step, analogous to large learning rates or small batches in SGD.

What would settle it

A direct test: on a deep network where the Polyak-Lojasiewicz condition visibly fails, record $\mathbb{E}_i\|\nabla f_i(w)\|^2$, $\|\nabla F(w)\|^2$, and the true generalization gap; if the two metrics move opposite to the gap, the bound's premise is broken. A sharper comparison: train BP-SVRG and SGD(NAG) with exactly equal numbers of gradient evaluations and no batchnorm-statistics tricks; if BP-SVRG's test-loss advantage disappears, the claimed data-traversal efficiency is an artifact of the comparison.

Watch

Extended reading notes

Core claim

The central discovery is a sign switch: B-SVRG updates with the snapshot correction subtracted from the stochastic gradient, while BP-SVRG adds it, explicitly injecting variance instead of canceling it. Under Assumption 1, the generalization gap $|F(w)-\bar{F}(w)|$ is bounded by $\frac{1}{2\mu}\mathbb{E}_i\|\nabla f_i(w)\|^2 + \frac{1}{2\mu}\|\nabla F(w)\|^2$ plus a dataset-dependent constant, so smaller gradient norms should mean better generalization. Empirically, B-SVRG has large values of both metrics, and the sign-flipped variant reduces them, which the paper ties to a flatter landscape via a data-relevant sharpness representation. On VGG, ResNet, and DenseNet over CIFAR and SVHN, BP-SVRG generally achieves the smallest test loss throughout training and top accuracy while using only two-thirds as many dataset traversals as SGD(NAG).

Load-bearing premise

The claim rests on assuming that every per-sample loss and the overall risk satisfy a strong gradient-dominance inequality (the Polyak-Lojasiewicz condition) with the same constant, and that the per-sample minimal-loss term stays roughly equal across the solutions being compared; deep networks are not known to satisfy either premise, and the paper concedes this.

Editorial extensions

If this is right

  • BP-SVRG can be preferred over SGD(NAG) when dataset traversal is costly, since it matches or beats NAG accuracy while seeing the data only two-thirds as often.
  • The two gradient metrics give a practical, cheaply computable training signal: if an SVRG variant keeps $\mathbb{E}_i\|\nabla f_i(w)\|^2$ and $\|\nabla F(w)\|^2$ low, it is likely to generalize better.
  • Mini-batching and learning-rate decay together make SVRG usable on deep networks, with learning-rate decay closing most of the generalization gap caused by approximate outer batches.
  • SVRG's theoretical convergence advantage does not automatically transfer to generalization; variance reduction can even hurt, and deliberate variance addition can help.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the sign flip acts as an implicit flat-minima regularizer, so BP-SVRG should help most on tasks where sharp minima overfit and least where added noise is harmful; this can be tested by varying label noise or batch size.
  • Beyond the paper: the two gradient metrics could serve as a cheap early-training diagnostic for stopping or hyperparameter selection on deep networks; the paper does not test this use.
  • Beyond the paper: the outer/inner batch reuse suggests a family of 'Plus' variants for any variance-reduced method, with the control-variate sign possibly annealed from subtraction to addition over training; that family is not explored here.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes BatchPlus-SVRG (BP-SVRG), obtained from Batch-SVRG (B-SVRG) by switching the sign of the control-variate correction, with the motivation that the average per-sample gradient norm and the norm of the average gradient serve as proxies for generalization. The authors first study mini-batching and learning-rate decay for SVRG variants, then present a heuristic derivation linking these gradient-norm metrics to the generalization gap under a Polyak-Lojasiewicz assumption, and finally report experiments on CIFAR-10, CIFAR-100, and SVHN with VGG, ResNet, and DenseNet architectures. The main empirical claim is that BP-SVRG consistently outperforms B-SVRG in test accuracy and test loss, and is competitive with or better than Nesterov-accelerated SGD while using fewer training epochs.

Significance. If the empirical claims hold under a correct compute budget, BP-SVRG would be a practically interesting optimizer: it is a simple modification of B-SVRG, it is evaluated across 24 model-dataset cells with repeated runs and reported means and standard deviations, and the B-SVRG versus BP-SVRG comparison is internally controlled. The paper is also useful for documenting the effect of mini-batching and learning-rate decay on SVRG-type methods. However, the comparison with SGD is not compute-matched as reported, so the headline 'even SGD' claim is currently unsupported. The theoretical motivation is explicitly heuristic and rests on assumptions that are unlikely to hold for deep networks, a limitation the authors themselves acknowledge in Appendix B. The strongest dependable contribution is the consistent BP-SVRG improvement over B-SVRG, which does not depend on the disputed SGD compute accounting.

major comments (2)
  1. [§4.1, Algorithm 1] The compute-equalization premise in §4.1 is incorrect. In each outer iteration, Algorithm 1 evaluates gradients on the B samples of I to form µ, and then evaluates gradients on the same B samples in the inner loop; under the most favorable assumption that the outer-batch snapshot gradients are reused, this is 2B per-sample gradient evaluations per outer iteration, i.e., 2n per full dataset pass. Equalizing total gradient evaluations against SGD therefore requires 2N SGD epochs, not 1.5N. The stated rule 'B(P)-SVRG of N epochs with SGD (NAG) of 1.5N epochs' gives BP-SVRG 4/3 times the gradient budget of SGD if 'epoch' means a full pass, and if 'epoch' in Algorithm 1 means a single outer iteration the comparison is not compute-matched at all. Consequently, the claims in §4.2 that BP-SVRG beats SGD and is 'apparently superior in dataset traversal time' are not supported by Table 3 and Figure 2 as reported; the SGD comparison must be rerun at equal compute or explicitly qualified.
  2. [§3.3 and Appendix B] The generalization bound in Eq. (7) and Eq. (10) depends on Assumption 1, namely the P-L condition with a common µ for F and every fi, and on the approximations E≈sample and E∇≈∇E. These conditions are not satisfied by deep networks in general, as the authors concede in Appendix B ('Although deep neural networks may not meet the assumptions...'). The derivation is therefore not an upper bound for the settings in which BP-SVRG is tested. This does not invalidate the empirical BP-SVRG-versus-B-SVRG comparison, but the paper should present the theoretical motivation as heuristic intuition and explicitly condition the statement 'so smaller Ei... may lead to better generalization' on unverified assumptions rather than presenting it as a proven bridge between optimization and generalization.
minor comments (5)
  1. [Algorithm 1] The pseudocode box contains two update lines without labels; the B-SVRG line and the BP-SVRG line should be clearly distinguished, for example by labeling the two variants explicitly.
  2. [Table 3] The text in §4.2 says results in which BP-SVRG obviously outperforms SGD (NAG) are marked in boldface, but no boldface is visible in the typeset table; please fix the formatting or clarify the marking convention.
  3. [Eq. (6)] The step marked '≲' in Eq. (6) suppresses the Hessian-spectral factor λ_Hw and the o(η²‖∇fi(w)‖²) terms; the statement that S_φ± (w) is upper bounded by S_hat_φ± (w) should be qualified as holding only under additional smoothness and small-step-size assumptions.
  4. [Section 4.1] The sentence 'one additional gradient computation is required for the parameter update on one outer batch' is ambiguous; please state explicitly whether the snapshot gradients ∇fi(w) used inside the inner mini-batches are reused from the outer-batch computation or recomputed, and give the per-sample gradient-evaluation count for one outer iteration.
  5. [Appendix A] There are several typos, including 'learning raet' in A.2 and 'generalizatino', 'Speicifically', and 'stablizes' in A.4; these should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; BP-SVRG is evaluated on held-out test data external to its metric-based motivation.

full rationale

The paper's derivation chain does not reduce to its own inputs. It motivates two gradient-norm metrics through a P-L-condition upper bound on the generalization gap (Eq. 7), observes empirically that B-SVRG has relatively large values of these metrics, then constructs BP-SVRG by switching the sign of the control variate. The central empirical claims are then tested on held-out test accuracy and test loss (Table 3, Figures 2-3), which are external to the metric definitions and are not fitted quantities. No parameter is fitted to the target result and then renamed as a prediction. The Appendix B caveat that deep networks may not satisfy P-L weakens the theoretical motivation but does not make the empirical comparison circular. The Sec. 4.1 compute-equalization choice is a comparison-fairness concern rather than a circularity. No load-bearing self-citations are used: the B-SVRG baseline builds on prior work cited as [20], and the sign-switch is presented as a new algorithmic choice, not as a consequence of a self-cited uniqueness theorem.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central claim rests on modest mathematical scaffolding: the P-L condition (admitted inapplicable to deep networks), Taylor-expansion and expectation-interchange steps in Eqs. (6) and (7), and the treatment of a dataset-dependent term as roughly constant. The empirical comparison rests on hand-selected learning rates, batch sizes, and epoch budgets. The only newly introduced quantity is the data-relevant sharpness measure S_phi+-, which is measurable on any training run.

free parameters (3)
  • BP-SVRG initial learning rate = 0.5 or 1.0, chosen per model and dataset
    Section 4.1: 'We display the better result of BP-SVRG with initial learning rate shown in parenthesis.' This hand-selected hyperparameter favors BP-SVRG in Table 3, while the SGD baseline uses a fixed 0.1.
  • Inner and outer batch sizes (b, B = 2b) = b = 128 (64 for DenseNet on CIFAR); B = 2b
    Design choices that set the cost of one epoch and the snapshot-gradient budget; the B = 2b choice is what makes the compute-fairness accounting in Section 4.1 ambiguous.
  • Epoch budget N and learning rate decay schedule = N = 40 (SVHN), 200 or 250 (CIFAR); LR divided by 10 at 40/60/80% or 50/75%
    Hand-chosen training budgets and schedules; N defines the comparison of BP-SVRG against 1.5N epochs of SGD.
assumptions (4)
  • domain assumption Each per-sample loss f_i and the population risk F satisfy the P-L condition with a common mu > 0 (Assumption 1).
    Used to derive the generalization bound Eq. (7). The authors admit in Appendix B: 'Although deep neural networks may not meet the assumptions, we believe that such heuristic results are not trivial.'
  • standard math Expectation and derivative interchange: nabla_w E f_xi(w) = E nabla_w f_xi(w).
    Invoked in Appendix B 'by exchanging the expectation and the derivative in common cases'; requires regularity conditions that are not stated or verified.
  • domain assumption Second-order Taylor expansion of F around w with higher-order terms dropped, and the largest Hessian singular value lambda_Hw treated as comparable across compared methods.
    Eq. (6) bounds data-relevant sharpness by eta^2 lambda_Hw E_i ||nabla f_i||^2; first-order terms cancel only in the symmetric +- sum, and the o(eta^2) terms plus lambda_Hw variation are discarded.
  • domain assumption The dataset-dependent term (1/n) sum_i |f_i(w*_i) - F(w*)| is roughly constant across compared solutions.
    Eq. (7) keeps this term aside as depending only on model and dataset, so the gradient-norm terms are used to rank solutions; if this term varies across minima, the ranking argument fails.
invented entities (1)
  • Data-relevant sharpness S_phi+- (w) independent evidence
    purpose: A generalization and flatness proxy defined as the expected change in loss when shifting the weights by +-eta nabla f_xi(w); its empirical upper bound E_i ||nabla f_i(w)||^2 motivates BP-SVRG.
    Not a physical entity but a newly introduced measurable quantity. Independent evidence exists because anyone can compute it on a training run, and the paper's correlation between this metric and generalization gaps is falsifiable by new experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Better Generalization: BP-SVRG in Training Deep Neural Networks." pith.science (2026). https://pith.science/paper/JILYEZXI

@misc{pith2026190806395,
  author       = {Pith},
  title        = {Pith review of: Towards Better Generalization: BP-SVRG in Training Deep Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JILYEZXI}},
  note         = {Machine review of arXiv:1908.06395}
}
read the original abstract

Stochastic variance-reduced gradient (SVRG) is a classical optimization method. Although it is theoretically proved to have better convergence performance than stochastic gradient descent (SGD), the generalization performance of SVRG remains open. In this paper we investigate the effects of some training techniques, mini-batching and learning rate decay, on the generalization performance of SVRG, and verify the generalization performance of Batch-SVRG (B-SVRG). In terms of the relationship between optimization and generalization, we believe that the average norm of gradients on each training sample as well as the norm of average gradient indicate how flat the landscape is and how well the model generalizes. Based on empirical observations of such metrics, we perform a sign switch on B-SVRG and derive a practical algorithm, BatchPlus-SVRG (BP-SVRG), which is numerically shown to enjoy better generalization performance than B-SVRG, even SGD in some scenarios of deep neural networks.

Figures

Figures reproduced from arXiv: 1908.06395 by the authors.

Figure 1
Figure 1. Eik∇fi(w)k 2 and k∇F(w)k 2 by SGD, Momentum, Nesterov momentum, B-SVRG and BP-SVRG on VGG11 with dropout on CIFAR10, without learning rate decay. Initial learning rates are shown in the legends. Other experimental settings are same as Section 4. Curves are smoothed with a moving average over 5 points. Loss gap in (b) refers to testing loss minus training loss. Accuracy gap in (e) displays the difference between trai… view at source ↗
Figure 2
Figure 2. Testing loss with std on VGG and ResNet in CIFAR-10. Here testing loss is the softmax [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Testing loss (an average ± standard deviation) on VGG, ResNet, and DenseNet on CIFAR and SVHN. Curves are smoothed with a moving average over 5 points. where . means approximately less-than under some conditions. Moreover, since k · k2 is convex, and use training data S to approximate the expectation, then kEξ∼D∇fξ(w)k 2 ≤ Eξ∼Dk∇fξ(w)k 2 ≈ Eik∇fi(w)k 2 . (9) Therefore, |F(w) − F(w)| . 1 µ Eik∇fi(w)k 2 + 1 n Xn i=1 |… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: (a) exhibits the metrics on VGG11, CIFAR10. Experimental settings are same as Section 4. [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 18 canonical work pages

  1. [1]

    Katyusha: The first direct acceleration of stochastic gradient methods

    Zeyuan Allen-Zhu. Katyusha: The first direct acceleration of stochastic gradient methods. The Journal of Machine Learning Research, 18(1):8194–8244, 2017

    work page 2017

  2. [2]

    Improved svrg for non-strongly-convex or sum-of-non- convex objectives

    Zeyuan Allen-Zhu and Yang Yuan. Improved svrg for non-strongly-convex or sum-of-non- convex objectives. In International conference on machine learning, pages 1080–1089, 2016

    work page 2016

  3. [3]

    On the optimization of deep networks: Implicit acceleration by overparameterization

    Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. arXiv preprint arXiv:1802.06509, 2018

    arXiv 2018

  4. [4]

    Understanding batch normalization

    Nils Bjorck, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normalization. In Advances in Neural Information Processing Systems, pages 7694–7705, 2018

    work page 2018

  5. [5]

    On the ineffectiveness of variance reduced optimization for deep learning

    Aaron Defazio and Léon Bottou. On the ineffectiveness of variance reduced optimization for deep learning. arXiv preprint arXiv:1812.04529, 2018

    arXiv 2018

  6. [6]

    Saga: A fast incremental gradient method with support for non-strongly convex composite objectives

    Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. In Advances in neural information processing systems, pages 1646–1654, 2014

    work page 2014

  7. [7]

    Accelerated gradient methods for nonconvex nonlinear and stochastic programming

    Saeed Ghadimi and Guanghui Lan. Accelerated gradient methods for nonconvex nonlinear and stochastic programming. Mathematical Programming, 156(1-2):59–99, 2016

    work page 2016

  8. [8]

    Deep learning

    Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016

    2016

Show all 36 references
  1. [9]

    Accurate, large minibatch sgd: Training imagenet in 1 hour

    Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017

    2017 arXiv

  2. [10]

    Stopwasting my gradients: Practical svrg

    Reza Harikandeh, Mohamed Osama Ahmed, Alim Virani, Mark Schmidt, Jakub Koneˇcn`y, and Scott Sallinen. Stopwasting my gradients: Practical svrg. In Advances in Neural Information Processing Systems, pages 2251–2259, 2015

    2015

  3. [11]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  4. [12]

    Train longer, generalize better: closing the generalization gap in large batch training of neural networks

    Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. InAdvances in Neural Information Processing Systems, pages 1731–1741, 2017

    2017

  5. [13]

    Deep networks with stochastic depth

    Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pages 646–661. Springer, 2016

    2016

  6. [14]

    Densely connected convolutional networks

    Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017

    2017

  7. [15]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015

    2015 arXiv

  8. [16]

    Linear convergence of gradient and proximal- gradient methods under the polyak-łojasiewicz condition

    Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal- gradient methods under the polyak-łojasiewicz condition. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 795–811. Springer, 2016

    2016

  9. [17]

    On large-batch training for deep learning: Generalization gap and sharp minima

    Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016

    2016 arXiv

  10. [18]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009

    2009

  11. [19]

    Imagenet classification with deep convolutional neural networks

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012

    2012

  12. [20]

    Non-convex finite-sum optimization via scsg methods

    Lihua Lei, Cheng Ju, Jianbo Chen, and Michael I Jordan. Non-convex finite-sum optimization via scsg methods. In Advances in Neural Information Processing Systems, pages 2348–2358, 2017

    2017

  13. [21]

    Introductory lectures on convex optimization: A basic course , volume 87

    Yurii Nesterov. Introductory lectures on convex optimization: A basic course , volume 87. Springer Science & Business Media, 2013. 9

    2013

  14. [22]

    Reading digits in natural images with unsupervised feature learning

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011

    2011

  15. [23]

    Data-dependent path normalization in neural networks

    Behnam Neyshabur, Ryota Tomioka, Ruslan Salakhutdinov, and Nathan Srebro. Data-dependent path normalization in neural networks. arXiv preprint arXiv:1511.06747, 2015

    2015 arXiv

  16. [24]

    Norm-based capacity control in neural networks

    Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory, pages 1376–1401, 2015

    2015

  17. [25]

    Exploring generalization in deep learning

    Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. In Advances in Neural Information Processing Systems, pages 5947–5956, 2017

    2017

  18. [26]

    Automatic differentiation in pytorch

    Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017

    2017

  19. [27]

    Gradient methods for minimizing functionals

    Boris Teodorovich Polyak. Gradient methods for minimizing functionals. Zhurnal Vychisli- tel’noi Matematiki i Matematicheskoi Fiziki, 3(4):643–653, 1963

    1963

  20. [28]

    Stochastic variance reduction for nonconvex optimization

    Sashank J Reddi, Ahmed Hefny, Suvrit Sra, Barnabas Poczos, and Alex Smola. Stochastic variance reduction for nonconvex optimization. InInternational conference on machine learning, pages 314–323, 2016

    2016

  21. [29]

    Minimizing finite sums with the stochastic average gradient

    Mark Schmidt, Nicolas Le Roux, and Francis Bach. Minimizing finite sums with the stochastic average gradient. Mathematical Programming, 162(1-2):83–112, 2017

    2017

  22. [30]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014

    2014 arXiv

  23. [31]

    A bayesian perspective on generalization and stochastic gradient descent

    Samuel L Smith and Quoc V Le. A bayesian perspective on generalization and stochastic gradient descent. arXiv preprint arXiv:1710.06451, 2017

    2017 arXiv

  24. [32]

    Dropout: a simple way to prevent neural networks from overfitting

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958, 2014

    1929

  25. [33]

    Training very deep networks

    Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. Training very deep networks. In Advances in neural information processing systems, pages 2377–2385, 2015

    2015

  26. [34]

    On the importance of initialization and momentum in deep learning

    Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139–1147, 2013

    2013

  27. [35]

    Wide residual networks

    Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016. 10 A Experiments details A.1 Networks and devices Experiments are run using PyTorch [26] on the machine with 128 GB RAM, and NVIDIA TESLA P100 GPU. We use VGG from PyTorch modu...

    2016 arXiv

  28. [64]

    The initial learning raet of SGD (NAG) is set to 0.1 on all of the three datasets. For all of the optimization methods, the learning rate is divided by 10 at 40%, 60% and 80% of the total number of training epochs on CIFAR datasets excluding Densenet, while the learning rate i...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.