Pith. sign in

REVIEW 4 major objections 8 minor 32 references

Insights from Gradient Dynamics: Gradient Autoscaled Normalization

T0 review · 4 major / 8 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that the global standard deviation of gradients falls steadily through CNN training, and that a hyperparameter-free multiplier derived from that trend improves ResNet test accuracy on CIFAR-100 while matching the baseline o

desk verdict A simple, plausibly useful gradient autoscaler, but the accuracy gains are unverified and the convergence proof has a load-bearing gap; needs more evidence before the claims hold. read the letter →

arxiv 2509.03677 v2 pith:2WHIM4UH submitted 2025-09-03 cs.LG cs.AIcs.CVcs.ITmath.IT

Vincent-Daniel Yun This is my paper
classification cs.LGcs.AIcs.CVcs.ITmath.IT MSC 68T07
keywords gradientdynamicsnormalizationglobalstandarddeviationhyperparameter-freeoptimizationconvergenceguaranteesconvolutionalneuralnetworksCIFAR-100AdamW
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish two linked claims: that the global standard deviation of gradients in convolutional networks falls consistently as training proceeds, even though layer-wise gradient variance rises and falls unevenly, and that this global trend can be turned into a working optimizer component. The proposed component removes each layer's gradient mean, then multiplies all eligible gradients by a single scalar computed from the global gradient standard deviation through a simple log-scale map — no hyperparameters, no per-layer division. Because it never divides by a layer's local standard deviation, it avoids the amplification that the paper identifies in z-score gradient normalization for low-variance layers. On CIFAR-100 with AdamW plus label smoothing and CutMix, the method improves test accuracy on ResNet-20 (61.3% versus 59.3%) and ResNet-56 (71.3% versus 70.0%) and matches the baseline on VGG-16-BN, where alternative gradient normalizations degrade. The paper also shows the transformation preserves standard SGD convergence guarantees when the effective step size η·a_t is at most 1/β.

What carries the argument

The load-bearing object is the global gradient standard deviation s_t, computed each iteration by concatenating all layer gradients with more than one dimension. From it, the autoscale map a_t = (4/(|log s_t| + ε))^{p_t} produces a scalar multiplier whose size falls as s_t falls; the exponent p_t ∈ {1, 2} is fixed once at the first iteration so that a small initial s_t does not trigger overly aggressive downscaling. The transform itself is layer-wise mean removal (the gradient-centralization operation) followed by multiplication of every eligible layer by the same a_t, so the method never divides by a local standard deviation and cannot blow up low-variance layers. The theoretical work is th

What would settle it

Record s_t = Std of the concatenated eligible gradients every epoch on a setting outside the paper's three CNNs — for example, a Vision Transformer or a ResNet trained with a learning rate several times larger — and plot it; if s_t trends upward for a sustained stretch, a_t = (4/(|log s_t| + ε))^{p_t} grows and the claimed a_t ∈ (0,1] premise fails, so the convergence bound no longer applies. The minimal check is to publish the s_t and a_t curves for the three reported CIFAR-100 runs that the paper says it measured.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a regularity and a method built on it. Tracking layer-wise gradient statistics in ResNet-20, ResNet-56, and VGG-16-BN shows heterogeneous behavior — some layers' gradient standard deviation falls, some rises — but the global standard deviation of all eligible gradients follows a consistent decreasing trend over 300 epochs on CIFAR-10. The paper claims this monotone global decay is the right signal to scale gradients by: it defines a global autoscale a_t = (4/(|log s_t| + ε))^{p_t}, with the integer exponent p_t chosen once at the first iteration to avoid over-shrinking when the initial std is small. Applying layer-wise mean removal followed

Load-bearing premise

The whole method rests on the premise that a network's global gradient standard deviation falls steadily across training, so a time-shrinking multiplier is always the right correction; the paper supports that trend only with plots it does not include, and the autoscale formula can exceed 1 when |log s_t| is small, a regime the convergence proof does not cover.

Editorial extensions

If this is right

  • The method introduces no hyperparameters: the multiplier and its exponent are fully determined by the measured gradient statistics, so the transform can be dropped into an existing AdamW or SGD pipeline without tuning.
  • The transform changes only the gradient, not the update rule, so the paper's construction carries over directly to other optimizers such as Adam and RMSProp.
  • Under the stated assumptions, scaled SGD retains its convergence guarantee: the only condition new to the analysis is η·a_t ≤ 1/β, and the multiplier acts purely as an adaptively shrinking effective step size.
  • The accuracy improvements on ResNet-20 and ResNet-56 (about +2.0 and +1.3 points on CIFAR-100) arise precisely in the strong-generalization setting (label smoothing and CutMix) where competing normalizations (GradNorm, gradient centralization, ZNorm) often lose accuracy; on VGG-16-BN the method ties the baseline instead of degrading.
  • The claimed empirical regularity — layer-wise gradient std is heterogeneous while global std decays monotonically — is itself a general observation about CNNs, offering a design principle for future optimizers: scale by global, not local, gradient statistics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to log s_t over the paper's own CIFAR-100 runs; whenever |log s_t| < 4, Eq. (1) yields a_t > 1, a regime the convergence proof does not cover — clipping a_t at 1 would make the theory match the implementation.
  • The performance pattern (gains on ResNets, tie on VGG) suggests the assumed global-std decay may be stronger in residual networks; applying the autoscaler to a Vision Transformer or a wide ResNet, where gradient dynamics differ, would test how far the regularity extends.
  • Since the method only rescales gradients, combining it with gradient clipping or with per-tensor norms is a natural untested direction; the paper's reported stability gains might interact with clipping thresholds in ways neither method alone shows.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper proposes Gradient Autoscaled Normalization (GAN), a gradient scaling method that (i) removes the per-layer mean of gradients and (ii) multiplies them by a global time-varying factor a_t derived from the global standard deviation of the gradient vector. The authors argue that, unlike layer-wise Z-score normalization, this avoids amplification in low-variance layers. They provide an empirical study of layer-wise and global gradient standard deviation on CIFAR-10, present CIFAR-100 results for ResNet-20, ResNet-56, and VGG-16-BN with AdamW, and include a convergence analysis for SGD. The central claims are that the method is hyperparameter-free, stabilizes optimization, preserves convergence guarantees, and improves accuracy on ResNet architectures.

Significance. If rigorously established, the paper would offer a simple, parameter-light normalization scheme that uses global gradient statistics and a time-varying scaling factor, and the empirical observation that global gradient standard deviation decreases over training is potentially useful. The proposed method is easy to implement and could be tested broadly. However, the current manuscript's main theoretical contribution—the convergence guarantee—is not valid as stated, and the empirical evidence is based on single runs without error bars. The idea is interesting and potentially salvageable, but the present form does not meet the standard of a rigorous journal publication.

major comments (4)
  1. [Section 4 / Appendix B, Eqs. (26)–(33)] Theorem 2's proof is invalid for the proposed method because a_t is time-varying. In Eq. (29) the factor η a_t / 2 is moved outside the sum over t, and in Eq. (33) division by η a_t is performed. This is only legitimate if a_t is constant over t, but Eq. (1) defines a_t as a function of s_t, the global gradient standard deviation at iteration t, which changes every iteration. The correct averaging must handle sums of a_t and a_t^2, and deriving a rate would require a lower bound on a_t. As written, the theorem does not establish convergence for the proposed update rule.
  2. [Section 4, after Eq. (1)] The paper states 'Since a_t ∈ (0,1]' but Eq. (1) does not enforce this. For p_t = 1, if |log s_t| + ε < 4, then a_t > 1; for p_t = 2, if |log s_t| + ε < 16, then a_t > 1. No constraint or clipping on s_t is given. Therefore the effective step size η a_t may violate the condition η a_t ≤ 1/β used in Lemma 1, and the claim that the method 'prevents unintended amplification' is not guaranteed by the formula.
  3. [Section 3.2, Eq. (2) and Section 4, Theorem 2] The convergence analysis omits the zero-centering step. Lemma 1 and Theorem 2 analyze the update w_{t+1} = w_t − η a_t ∇L_t(w_t), but the actual update is w_{t+1} = w_t − η a_t (G_t^{(l)} − μ_t^{(l)} 1) for each layer. After subtracting the per-layer mean, the transformed gradient is not an unbiased estimator of ∇L(w_t): its expectation is a_t (∇L(w_t) − mean(∇L(w_t))) in each layer, which is generally nonzero. Hence the assumption E[∇L_t(w_t)] = ∇L(w_t) does not hold for the method, and the proof does not apply to the proposed update.
  4. [Section 5, Table 1] The experimental results are reported for a single run per configuration, with no error bars, seeds, or statistical significance tests. The claimed gains are small (e.g., ResNet-56: +0.0128; VGG-16-BN: 0.0000) and could be within run-to-run variance. Additionally, the theory is for SGD, while the experiments use AdamW; no argument connects the convergence proof for scaled SGD to the AdamW setting. To support the empirical claim, the authors should provide means and standard deviations over multiple seeds and clarify the relationship between the theoretical update and the optimizer used.
minor comments (8)
  1. [Section 3.2] The phrase 'hyperparameter-free' is overstated: Eq. (1) contains fixed constants (4, 0.5, 2, ε) and the p_t rule has a threshold. Even if these are not tuned per dataset, they are still design choices; the text should say 'fixed' or 'hand-set' rather than 'hyperparameter-free'.
  2. [Section 3.2, Eq. (1)] The definition of p_t is ambiguous: the text says 'p_t = 1 for all t≥2 if a_1 < 0.5, otherwise 2', but does not define p_1 clearly. Please state p_1 explicitly.
  3. [Section 5] Details of label smoothing and CutMix are missing: the smoothing parameter and CutMix alpha are not reported. Also, the number of runs (apparently one) should be stated.
  4. [Appendix B, Lemma 1 proof] Equations (12)–(21) are duplicated and written in a confusing loop; the proof of E[||∇L_t(w_t)||^2] can be compressed to a single derivation.
  5. [Figures] Figures 1, 2, and 4 are referenced but the actual plots are not included in the provided text. The captions mention mean/std information, but the curves are needed to verify the asserted trends, and error bars should be added if possible.
  6. [Section 2.1] The paper cites [18] as a GitHub repo for ViT-CIFAR, but the experiments use CNNs. Please clarify which repository or implementation was used for the CNN models.
  7. [Section 5, baselines] The 'AdamW + Gradient Normalization' baseline [6] is applied as a single-task gradient normalization method, but GradNorm is originally designed for multitask balancing. The exact variant used here (e.g., scaling the whole gradient to a target norm) should be specified.
  8. [Section 2.1] Minor typos: 'Gradient Gradient Autoscaled Normalization' and 'Autocaled' in Section 3.2; 'our proposed method' appears in lowercase inconsistently. Please proofread.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the scaling rule is constructed from observed gradient statistics, the convergence claim is a standard SGD reduction, and the sole self-citation (ZNorm) is a non-load-bearing baseline.

full rationale

The paper's central derivation is the transform a_t = (4/(|log s_t|+epsilon))^{p_t} (Eq. 1). This is an explicit construction from the global gradient standard deviation; no test-accuracy value or target outcome enters the formula, so it is not a fitted parameter renamed as a prediction. The convergence guarantee (Lemma 1, Theorem 2, Appendix B) is the classical SGD proof with an extra scalar factor eta*a_t; it reduces to the standard argument rather than assuming the paper's empirical conclusion. The asserted range a_t in (0,1] and the time-varying treatment of a_t in the averaging step are mathematically unsupported (the proof pulls eta*a_t out of the time sum as if constant), but that is a correctness/validity gap, not a definitional circularity. The motivation cites ZNorm [25], which shares the author's identity, but ZNorm is used only as a comparative baseline and failure-mode illustration, not as the evidence for the new method's accuracy; the experimental comparison is against external CIFAR-100 baselines. The observation that global std decreases motivates a deliberately decreasing scaling map, and the scaling factor's decrease is a design consequence, not an empirical prediction used to certify the method. No load-bearing step reduces by construction to its own input, so circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical entities, but it does rest on hand-selected constants and an unvalidated empirical trend. The convergence proof assumes conditions that the adaptive formula does not guarantee.

free parameters (3)
  • log-std scaling constant 4 = 4
    Numerator in a_t = (4/(|log s_t|+epsilon))^{p_t}; chosen by hand, not derived. It is not shown to guarantee a_t in (0,1].
  • p_t safeguard rule = p_t in {1,2} with threshold a_1 < 0.5
    Adaptively chosen at the first iteration based on a_1; an arbitrary hand-designed rule to avoid excessive downscaling, not a learned parameter.
  • epsilon stabilizer = 1e-8
    Numerical stabilizer used in Eq. (1). Standard in normalization methods, but still a fixed constant.
assumptions (3)
  • standard math Loss L is beta-smooth and stochastic gradients are unbiased with bounded variance (Assumptions 1 and 2, Appendix B).
    Used to prove Lemma 1 and Theorem 2 in Appendix B; standard machinery for SGD analysis.
  • domain assumption Global gradient standard deviation s_t consistently decreases during training (Section 2.2, Figure 2).
    This empirical trend motivates the diminishing scaling factor. It is only claimed for three CNN architectures on CIFAR-10 and is not supported by released data or error bars.
  • domain assumption a_t in (0,1] and eta a_t <= 1/beta (Section 4).
    Needed for the convergence bound, but Eq. (1) does not mathematically force a_t <= 1 for all possible s_t, especially when s_t is near 1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Insights from Gradient Dynamics: Gradient Autoscaled Normalization." pith.science (2026). https://pith.science/paper/2WHIM4UH

@misc{pith2026250903677,
  author       = {Pith},
  title        = {Pith review of: Insights from Gradient Dynamics: Gradient Autoscaled Normalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2WHIM4UH}},
  note         = {Machine review of arXiv:2509.03677}
}
read the original abstract

Gradient dynamics play a central role in determining the stability and generalization of deep neural networks. In this work, we provide an empirical analysis of how variance and standard deviation of gradients evolve during training, showing consistent changes across layers and at the global scale in convolutional networks. Motivated by these observations, we propose a hyperparameter-free gradient normalization method that aligns gradient scaling with their natural evolution. This approach prevents unintended amplification, stabilizes optimization, and preserves convergence guarantees. Experiments on the challenging CIFAR-100 benchmark with ResNet-20, ResNet-56, and VGG-16-BN demonstrate that our method maintains or improves test accuracy even under strong generalization. Beyond practical performance, our study highlights the importance of directly tracking gradient dynamics, aiming to bridge the gap between theoretical expectations and empirical behaviors, and to provide insights for future optimization research.

Figures

Figures reproduced from arXiv: 2509.03677 by the authors.

Figure 1
Figure 1. Layer-wise Gradients Standard Deviation for ResNet-20 [ [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Global gradients standard deviation (top) and corresponding scaling factor (bottom) for [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Scaling factor on VGG-16-BN [21] with different settings: (A) Pt = 2, (B) Pt = 1. Autoscale via log–std. Our proposed method maps the global scale st to a scalar multiplier at ∈ (0, +∞) through a smooth, hyperparameter-free transform of |log st |: at = 0.5 + 1 0.5 (|log st | + ϵ)  − 0.5  · 2 pt , pt = ( 1, for all t ≥ 2 if a1 < 0.5, 2, otherwise. (1) Algebraically, eq 1 simplifies to at =  4 |log st|+ϵ pt wher… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Top-1 Test accuracy comparison on ResNet-56 [11] Our method consistently improves accuracy on ResNet architectures, while on VGG it achieves performance comparable to the baseline. Notably, alternative normalization techniques often lead to degraded results, underscori…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

32 extracted references · 20 canonical work pages

  1. [1]

    Layer normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. 2016. doi: 10.48550/arXiv.1607.06450. 6 GRADIENTAUTOSCALEDNORMALIZATION

    arXiv doi:10.48550/arxiv.1607.06450 2016

  2. [2]

    Large-scale machine learning with stochastic gradient descent

    L ´eon Bottou. Large-scale machine learning with stochastic gradient descent. InProceedings of COMPSTAT’2010, pages 177–186. Springer, 2010

    2010

  3. [3]

    The tradeoffs of large scale learning

    Leon Bottou and Olivier Bousquet. The tradeoffs of large scale learning. InAdvances in Neural Information Processing Systems, volume 20, 2007

    work page 2007

  4. [4]

    Curtis, and Jorge Nocedal

    Leon Bottou, Frank E. Curtis, and Jorge Nocedal. Optimization methods for large-scale ma- chine learning.SIAM Review, 60(2):223–311, 2018. doi: 10.1137/16M1080173

    doi:10.1137/16m1080173 2018

  5. [5]

    Entropy-sgd: biasing gradient descent into wide valleys.Journal of Statistical Mechanics: Theory and Experiment, 2019 (12):124018, 2019

    Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: biasing gradient descent into wide valleys.Journal of Statistical Mechanics: Theory and Experiment, 2019 (12):124018, 2019. doi: 10.1088/1742-5468/ab39d9

    doi:10.1088/1742-5468/ab39d9 2019

  6. [6]

    Gradnorm: Gra- dient normalization for adaptive loss balancing in deep multitask networks

    Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gra- dient normalization for adaptive loss balancing in deep multitask networks. InProceedings of the 35th International Conference on Machine Learning (ICML), volume 80 ofProceedings of Machine Learning Research (PMLR), pages 794–803, Stockholm, Sweden, July 2018. PMLR

    work page 2018

  7. [7]

    Fleet, and Jimmy Ba

    Fartash Faghri, David Duvenaud, David J. Fleet, and Jimmy Ba. A study of gradient variance in deep learning.arXiv preprint arXiv:2007.04532, 2020. doi: 10.48550/arXiv.2007.04532

    arXiv doi:10.48550/arxiv.2007.04532 2007

  8. [8]

    Understanding the difficulty of training deep feedfor- ward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Yee Whye Teh and Mike Titterington, editors,Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 ofPro- ceedings of Machine Learning Research, pages 249–256, Chia Laguna Resort, Sardinia, It...

    work page 2010

Show all 32 references
  1. [9]

    Take a shortcut back: Mitigating the gradient vanishing for training spiking neural networks

    Yufei Guo, Yuanpei Chen, Zecheng Hao, Weihang Peng, Zhou Jie, Yuhan Zhang, Xiaode Liu, and Zhe Ma. Take a shortcut back: Mitigating the gradient vanishing for training spiking neural networks. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang...

    2024

  2. [10]

    Stable architectures for deep neural networks.Inverse Prob- lems, 34(1):014004, 2018

    Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks.Inverse Prob- lems, 34(1):014004, 2018. doi: 10.1088/1361-6420/aa9a90

    2018 doi

  3. [11]

    Deep residual learning for im- age recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016

    2016

  4. [12]

    Densely con- nected convolutional networks

    Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely con- nected convolutional networks. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4700–4708, 2017

    2017

  5. [13]

    Batch normalization: accelerating deep network training by reducing internal covariate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift. page 448–456, 2015

    2015

  6. [14]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InInterna- tional Conference on Learning Representations, 2015. 7 GRADIENTAUTOSCALEDNORMALIZATION

    2015

  7. [15]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. Technical Report

    2009

  8. [16]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternational Conference on Learning Representations, 2019

    2019

  9. [17]

    Rafael Muller, Simon Kornblith, and Geoffrey Hinton.When does label smoothing help? 2019

    2019

  10. [18]

    ViT-CIFAR: PyTorch implementation for Vision Transformer on CIFAR datasets.https://github.com/omihub777/ViT-CIFAR, 2021

    OmiHub777. ViT-CIFAR: PyTorch implementation for Vision Transformer on CIFAR datasets.https://github.com/omihub777/ViT-CIFAR, 2021. Accessed: 2025- 08-15

    2021

  11. [19]

    On the difficulty of training recur- rent neural networks

    Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recur- rent neural networks. InProceedings of the 30th International Conference on International Conference on Machine Learning - Volume 28, ICML’13, page III–1310–III–1318. JMLR.org, 2013

    2013

  12. [20]

    How does batch normalization help optimization? InProceedings of the 32nd International Conference on Neural Information Processing Systems, page 2488–2498, 2018

    Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Mkadry. How does batch normalization help optimization? InProceedings of the 32nd International Conference on Neural Information Processing Systems, page 2488–2498, 2018

    2018

  13. [21]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. InInternational Conference on Learning Representations, 2015

    2015

  14. [22]

    Lecture 6.5 - RMSProp: Divide the gradient by a running average of its recent magnitude

    Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5 - RMSProp: Divide the gradient by a running average of its recent magnitude. Coursera: Neural Networks for Machine Learning,

  15. [23]

    Aggregated residual transformations for deep neural networks

    Saining Xie, Ross Girshick, Piotr Doll ´ar, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. InProceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 1492–1500, 2017

    2017

  16. [24]

    Gradient centraliza- tion: A new optimization technique for deep neural networks.Proceedings of the Eu- ropean Conference on Computer Vision (ECCV), pages 635–651, 2020

    Hongwei Yong, Jianqiang Huang, Xiansheng Hua, and Lei Zhang. Gradient centraliza- tion: A new optimization technique for deep neural networks.Proceedings of the Eu- ropean Conference on Computer Vision (ECCV), pages 635–651, 2020. doi: 10.1007/ 978-3-030-58568-6 37

    2020

  17. [25]

    Znorm: Z-score gradient normalization accelerating skip-connected network training without architectural modification

    Juyoung Yun. Znorm: Z-score gradient normalization accelerating skip-connected network training without architectural modification. InAI for Research and Scalable, Efficient Systems, pages 240–254, Singapore, 2025. Springer Nature Singapore

    2025

  18. [26]

    Cutmix: Regularization strategy to train strong classifiers with localizable features

    Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6022–6031, 2019

    2019

  19. [27]

    Wide residual networks

    Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. InProceedings of the British Machine Vision Conference (BMVC), 2016. 8 GRADIENTAUTOSCALEDNORMALIZATION

    2016

  20. [28]

    When will gradient regularization be harmful? In Forty-first International Conference on Machine Learning

    Yang Zhao, Hao Zhang, and Xiuyuan Hu. When will gradient regularization be harmful? In Forty-first International Conference on Machine Learning

  21. [29]

    Penalizing gradient norm for efficiently improving generalization in deep learning

    Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. InInternational Conference on Machine Learning, pages 26982–26992. PMLR, 2022

    2022

  22. [30]

    Recurrent neural networks: vanishing and exploding gradients are not the end of the story

    Nicolas Zucchet and Antonio Orvieto. Recurrent neural networks: vanishing and exploding gradients are not the end of the story. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 9 GRADIENTAUTOSCALEDNORMALIZATION Appendix A. Related Works Res...

    2024

  23. [32]

    Therefore, E[L(wt+1)]≤E[L(w t)]− ηat 2 E[∥∇L(wt)∥2] +(ηat)2βσ 2 2b .(25) Theorem 2Let the loss functionLbeβ-smooth, and assume the stochastic gradient is unbiased with bounded variance, i.e.,E[∇L t(wt)] =∇L(w t)andE[∥∇L t(wt)− ∇L(wt)∥2]≤ σ2 b . If the effective step size satis...

  24. [2012]

    University of Toronto

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.