Pith. sign in

REVIEW 4 major objections 5 minor 51 references

An Adaptive Method Stabilizing Activations for Enhanced Generalization

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AdaAct stabilizes network activations by scaling each neuron's learning rate with its activation variance, giving SGD-level generalization with Adam-like convergence speed on image benchmarks.

desk verdict AdaAct is a plausible new optimizer with clean CIFAR results, but the generalization proof has a real gap and the ImageNet comparison is not controlled. read the letter →

arxiv 2506.08353 v1 pith:D2DSRQ5B submitted 2025-06-10 cs.LG cs.CV

Hyunseok Seung , Jaewoo Lee , Hyunsuk Ko This is my paper
classification cs.LGcs.CV MSC 68T0790C26
keywords AdaActactivationvarianceadaptiveoptimizationgeneralizationlearningrateadaptationdeepoptimizerImageNetnon-convex
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AdaAct is an optimizer that makes the learning rate for a neuron depend on how stable that neuron's activations are: high activation variance means a smaller step, low variance means a larger step. The paper argues that this neuron-wise adaptivity stabilizes the network's internal outputs during training and that this stability is what lets AdaAct generalize like tuned SGD while still converging as fast as Adam. On CIFAR-10/100 it matches or exceeds SGD and KFAC for the architectures tested, and on ImageNet ResNet-50 it reports 77.6% top-1 accuracy, above the quoted Adam, AdamW, and SGD numbers. If the claim holds, activation-variance adaptation is a viable third path between per-parameter adaptive learning rates and a single global learning rate.

What carries the argument

The load-bearing object is the variance-adaptive preconditioner: at each step the algorithm maintains $V_t=\beta_2 V_{t-1}+(1-\beta_2)\widehat{A}_t$, where $\widehat{A}_t$ is the diagonal of the empirical activation covariance $\frac{1}{|B_t|}\sum_i \tilde{a}^{(\ell-1)}_i(\tilde{a}^{(\ell-1)}_i)^{\top}$, and updates $\theta_t=\theta_{t-1}-\eta_t(\widehat{M}_t(\sqrt{\widehat{V}_t}+\epsilon I)^{-1}+\lambda\theta_{t-1})$. This makes the effective learning rate shared across all parameters that feed the same neuron—a middle ground between SGD's single global rate and Adam's per-parameter rates. The theoretical machinery is a generalized Adam convergence framework and uniform-stability generalization analysis; both rely on assumptions that activation variances stay between fixed bounds and that effective step sizes do not increase after an early iteration, which the paper substantiates with LeNet-5 and Fashion-MNIST activation traces rather than proofs.

What would settle it

Train a large transformer or a deep CNN with AdaAct while logging per-neuron activation variance every iteration; if variance leaves the assumed $[c_L, c_U]$ band for any layer, or if the effective step-size differences $\left|\eta_t/\sqrt{\hat{v}_t}-\eta_{t-1}/\sqrt{\hat{v}_{t-1}}\right|$ never settle near zero, then the $O(\log T/\sqrt{T})$ convergence bound and the generalization bound no longer follow, because the paper's proofs require exactly those conditions.

Watch

Extended reading notes

Core claim

The paper's central claim is that replacing the gradient second moment in Adam with the second moment of activations—formally, scaling the momentum update by $(\sqrt{\hat{V}_t}+\epsilon I)^{-1}$, where $\hat{V}_t$ is a bias-corrected exponential moving average of per-neuron activation variances—stabilizes neuron outputs and improves generalization relative to standard adaptive methods. The authors show this empirically on CIFAR-10/100 with ResNet-20/32/34 and DenseNet-121, and on ImageNet with ResNet-50/101 and ViT-S, reporting AdaAct at 77.6% top-1 on ResNet-50, matching LAMB on ViT-S, and matching tuned SGD on CIFAR while converging faster. They also prove an $O(\log T/\sqrt{T})$ convergence rate for non-convex objectives and a uniform-stability generalization bound, conditional on activation variances staying bounded and effective step sizes not increasing after a fixed iteration.

Load-bearing premise

The guarantees rest on activation variances staying within a fixed low-and-high band and on effective step sizes never starting to increase again after an early iteration, and the paper supports these only with trace plots on two small networks rather than with a proof or large-scale measurements.

Editorial extensions

If this is right

  • Adam's generalization gap is not unavoidable: a per-neuron adaptive rate can match tuned SGD and KFAC on the CIFAR architectures tested.
  • Large-batch training can use AdaAct at a learning rate of 4.0 and reach 77.6% top-1 on ImageNet ResNet-50, above the quoted Adam and AdamW baselines.
  • AdaAct reaches KFAC-level accuracy without storing or inverting full covariance matrices, so it offers a cheaper route to second-order-like generalization.
  • AdaAct partially substitutes for batch normalization: removing BN costs AdaAct far less than it costs SGD on ResNet-20/CIFAR-10.
  • Worst-case convergence is not sacrificed: under the paper's assumptions AdaAct matches AMSGrad's $O(\log T/\sqrt{T})$ rate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A reader might expect AdaAct's benefit to be largest where Adam's generalization gap is largest, such as small datasets, networks without normalization, and early-training instability, and smallest where SGD already converges well.
  • The mechanism suggests a testable link to sharpness: if stabilized activations are what helps generalization, AdaAct minima should be flatter than Adam minima under the same schedule, but the paper does not measure sharpness.
  • The same diagonal variance preconditioner could be extended to recurrent and language architectures, where activation stability is a known failure mode, though the paper evaluates only image classifiers.
  • If one clipped the variance estimate to a fixed interval explicitly, the bounded-variance assumption would hold by construction, turning an empirical observation into a built-in guarantee; the paper mentions clipping as a way to satisfy that assumption but does not adopt it in the main algorithm.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes AdaAct, an optimizer that scales parameter updates by the inverse square root of an EMA of per-neuron activation variances, aiming to stabilize neuron outputs and improve generalization relative to Adam-family optimizers while retaining Adam-like convergence speed. The authors provide a convergence analysis built on an existing Adam-type framework (Chen et al., 2019) with additional assumptions on activation variance bounds, and a generalization analysis via uniform stability of Hardt et al. Experiments cover CIFAR-10/100 with ResNet/DenseNet and ImageNet with ResNet-50/101 and ViT-S, including CIFAR runs with five seeds. The main theoretical support for the generalization claim is a stability bound whose proof, as written, does not close the recurrence on the parameter-distance term.

Significance. If the empirical claims hold, AdaAct is a practically interesting optimizer: it is simpler than KFAC/FOOF, has memory comparable to Adam, and its CIFAR results show accuracy close to tuned SGD and KFAC while converging faster in early epochs. The paper ships a clear algorithmic description, public code, and properly reported multi-seed CIFAR experiments, which are strengths. However, the theoretical sections are the main advertised justification for better generalization, and the stability proof contains a genuine gap. The ImageNet comparison is weakened by using published baselines from a different pipeline and by changing the loss function for AdaAct. The paper is therefore a potentially valuable empirical contribution that needs a corrected or substantially reframed theoretical analysis.

major comments (4)
  1. [§5.2, Theorem 5.5] The proof of Theorem 5.5 does not establish epsilon-uniform stability. The displayed inequality has term B = (eta L / c_L) * sum_t E[ sum_{k<=t} beta_1^{t-k}(1-beta_1) Delta_k ], which depends on the quantity Delta_k being bounded. The text dismisses this as the EMA of Delta_t, but a bound containing the target quantity on the right-hand side is not a closed-form bound. Swapping sums gives D_{T+1} <= A_T + (eta L/(c_L(1-beta_1))) sum_{k<=T} D_k, which can grow exponentially in T unless an additional condition such as eta L < c_L(1-beta_1) is imposed; no such condition is stated or verified. Consequently, the epsilon-uniform stability claim and the generalization bound via Theorem 5.4 do not follow from the proof as written. This is an internal gap independent of Assumptions A4/A5, and it directly undermines the paper's theoretical support for the claimed generalization benefit.
  2. [§5.1, Assumptions A4 and A5] Corollary 5.2's O(log T / sqrt T) rate depends entirely on Assumption A4 (activation variances bounded between c_L and c_U) and Assumption A5 (effective learning rates do not increase after t_0). The paper does not prove these assumptions for the architectures and batch sizes used in the experiments; it only points to LeNet-5 training traces on CIFAR-10 and Fashion-MNIST (Figures 4 and 5). Moreover, these traces are generated by AdaAct itself, so the assumptions are verified on the very method whose guarantees they support. The proposed clipping fix is not part of Algorithm 1. A convergence claim that holds only under unverified, architecture-specific conditions should be stated as conditional, and the authors should either prove the assumptions for the practical settings or report them as empirical observations rather than theorem hypotheses.
  3. [§6.2, Tables 2 and 3] The ImageNet comparisons are not apples-to-apples. AdaAct is run with a different loss (cross-entropy) than the stated A2 setting (binary cross-entropy), a batch size of 2,048 from a timm template, and the baselines SGD, Adam, AdamW, LAMB, and SAM are quoted from [45] rather than run in the same pipeline. Accuracy differences of 0.5-1.0% on ResNet-50 could easily arise from these pipeline differences (augmentation, loss, training length, weight decay). The paper should either run the baselines in the same codebase with the same loss and schedule, or explicitly frame Table 2/3 as a comparison to published numbers with the caveat that settings differ.
  4. [§4, Figure 3 and hyperparameter p] The defining exponent p = 0.5 is selected by comparing test accuracy in Figure 3 on ResNet-20/CIFAR-10, and the same dataset is then used in the main CIFAR-10 evaluation table. This means the reported recipe is partly selected on the target metric, which weakens the claim that p = 0.5 is the natural or generally best choice. The authors should disclose this selection explicitly and, ideally, validate p = 0.5 on a held-out architecture/dataset or show that results are insensitive to p in the range shown.
minor comments (5)
  1. [Algorithm 1] Line 5 defines V_t but the bias-corrected quantity is written as bV_t; the notation is clear in context but would be easier to read with a consistent hat notation throughout, including in the convergence analysis where v-hat is used.
  2. [Appendix C, equation (i)] The proof of Theorem 5.5 uses both n and N for the dataset size; for example the term 2H/(n c_L) appears with n, while the theorem and text use N. This should be unified to avoid confusion.
  3. [§3.2 and §4] The connection to KFAC/FOOF is explained via the covariance matrix, but the diagonal approximation and the neuron-wise sharing of learning rates in convolutional layers could be stated more precisely; the im2col description is helpful, but the exact grouping of parameters that share a learning rate is implicit.
  4. [Figure 3] The legend item Full cov is not labeled as a method or a value of p; the reader should be told that this is the full-covariance variant of AdaAct, not a separate baseline.
  5. [Table 4] The hyperparameter table has a row for SGD with a dot in the beta columns and an apparent lambda value of 5e-4; it would be helpful to state explicitly which entries are not applicable rather than using dots without a key.

Circularity Check

2 steps flagged · score 6.0 of 10

The generalization bound in Theorem 5.5 is self-referential (term B is a sum of the Δ_k being bounded), and the defining exponent p=0.5 is selected on the same CIFAR10 test metric that is later reported as evidence.

  1. self definitional [Section 5.2, Theorem 5.5 and the paragraph immediately after it; detailed in Appendix C.]
    "Then we have E[∆T+1] ≤ ηH(N−1)/N ΣTt=1 E[|1/√v̂t − 1/√v̂′t|] (A) + ηL/cL ΣTt=1 E[Σtk=1 βt−k1(1−β1)∆k] (B) + 2ηHT/(N cL). As shown in Figure 5, the term A in Theorem 5.5 is small enough (almost zero across iterations). The term B is the EMA of ∆t and the last term is small for datasets of moderate size. The generalization analysis demonstrates that ADAACT maintains a bounded generalization error, attributable to its ε-uniform stability and the Lipschitz continuity of the loss function."

    The displayed inequality is a recurrence: term B is a weighted sum of E[Δ_k] for k ≤ t, which is exactly the quantity the theorem sets out to bound. The proof never closes the recurrence; no Gronwall step or condition such as ηL < c_L(1−β_1) is stated or verified. The text then treats B as merely 'the EMA of Δt' and concludes bounded generalization error, which assumes the boundedness of Δ that is the theorem's conclusion. Thus the claimed ε-uniform stability and the resulting generalization bound are not derived from the assumptions; the conclusion is fed back into the justification.

  2. fitted input called prediction [Section 4, 'Scaled Activation Variance' (Figure 3); the chosen configuration is reported in Section 6.1, Table 1.]
    "The use of square root was derived in AdaGrad by considering the optimal step size in hindsight to minimize the regret in online learning. Through experiments, we observed that p= 0.5 achieves better performance than other values, even better than when the full covariance matrix is used. See Figure 3."

    Figure 3 is a test-accuracy comparison on ResNet-20/CIFAR10 across p ∈ {1, 1/2, 1/3, 1/4, 1/6}; p=1/2 is selected because it yields the highest test accuracy. Table 1 then reports AdaAct's CIFAR10 ResNet-20 accuracy (92.49) as evidence of generalization. The benchmark result is produced by the same configuration selected on that benchmark's target metric, so on this dataset the reported accuracy is a selection outcome rather than an independent prediction of a pre-specified algorithm.

full rationale

The paper's empirical ImageNet and CIFAR comparisons are external benchmarks and are not themselves circular: AdaAct is trained and evaluated under standard protocols, and there is no load-bearing self-citation chain or uniqueness theorem imported from the authors. However, the theoretical support for the central 'better generalization' claim is partially circular. Theorem 5.5's displayed bound contains term B = (ηL/c_L)Σ_t E[Σ_{k≤t}β_1^{t−k}(1−β_1)Δ_k], a weighted sum of the very differences Δ_k it is supposed to bound; the paragraph after the theorem asserts bounded generalization error by referring to B simply as 'the EMA of Δt', without closing the recurrence. This makes the stability conclusion self-referential unless an additional, unstated condition is imposed. Separately, the algorithm's defining exponent p=0.5 is chosen by test accuracy in Figure 3, and the same CIFAR10 ResNet-20 result is later reported in Table 1, so part of the headline empirical claim is a fitted input rather than an independent prediction. These two issues do not make the entire paper circular, but they do make the claimed derivation of bounded generalization partially reduce to its own inputs; hence a score of 6 rather than 0-2.

Assumptions & free parameters 3 free parameters · 8 assumptions · 0 invented entities

The central claim draws on standard stochastic optimization assumptions, two empirically verified assumptions specific to this method, and a diagonal approximation of the activation covariance. No invented entities are introduced.

free parameters (3)
  • p (variance scaling exponent) = 0.5
    Selected by comparing test accuracy in Figure 3; the paper states p=0.5 outperforms p=1/4, 1/3, 1/6 and even the full covariance matrix.
  • beta2 (EMA decay for activation variance) = 0.999
    Inherited from Adam's default without a derivation specific to activation statistics; the method's behavior depends on this smoothing coefficient.
  • base learning rates = 0.1 (CIFAR), 4.0 (ImageNet)
    Chosen via grid search; the paper notes AdaAct works well with relatively high learning rates, unlike typical adaptive optimizers.
assumptions (8)
  • domain assumption A1: f is differentiable with L-Lipschitz gradient and lower bounded
    Standard smoothness assumption used in Section 5.1.
  • domain assumption A2: true and noisy gradients are bounded by H
    Standard bounded-gradient assumption used in the convergence proof.
  • domain assumption A3: noisy gradient is unbiased and noise is independent
    Standard noise assumption in stochastic optimization.
  • ad hoc to paper A4: activation variances are bounded between c_L and c_U
    Verified only empirically in Figure 4 with LeNet-5; no proof is given, and it is not checked for ViT or large-batch runs.
  • ad hoc to paper A5: effective learning rates do not increase after t0
    Supported only by Figure 5 from AdaAct's own training; asserted as a condition for the O(logT/sqrtT) rate.
  • domain assumption Diagonal approximation of the activation covariance preserves the benefit of full covariance preconditioning
    Motivated by ReLU sparsity and memory constraints, but not proven; central to the algorithm's design.
  • ad hoc to paper Term A in Theorem 5.5 is small enough to ignore
    The generalization bound depends on this term being near zero, shown only on a LeNet-5 curve in Figure 5.
  • standard math Hardt et al. uniform stability framework applies
    Used as the theoretical backbone for the generalization analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Adaptive Method Stabilizing Activations for Enhanced Generalization." pith.science (2026). https://pith.science/paper/D2DSRQ5B

@misc{pith2026250608353,
  author       = {Pith},
  title        = {Pith review of: An Adaptive Method Stabilizing Activations for Enhanced Generalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D2DSRQ5B}},
  note         = {Machine review of arXiv:2506.08353}
}
read the original abstract

We introduce AdaAct, a novel optimization algorithm that adjusts learning rates according to activation variance. Our method enhances the stability of neuron outputs by incorporating neuron-wise adaptivity during the training process, which subsequently leads to better generalization -- a complementary approach to conventional activation regularization methods. Experimental results demonstrate AdaAct's competitive performance across standard image classification benchmarks. We evaluate AdaAct on CIFAR and ImageNet, comparing it with other state-of-the-art methods. Importantly, AdaAct effectively bridges the gap between the convergence speed of Adam and the strong generalization capabilities of SGD, all while maintaining competitive execution times. Code is available at https://github.com/hseung88/adaact.

Figures

Figures reproduced from arXiv: 2506.08353 by the authors.

Figure 1
Figure 1. (Left) Averaged activation variance from each hidden layer and (Right) test accuracy training LeNet-5 on CIFAR10. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Heatmap of the activation covariance from the hidden layers of LeNet-5 trained on CIFAR10. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Train and test accuracy of ResNet-20 on CIFAR10 with varying value of [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Difference in effective stepsizes: (Left) between itera [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Test accuracy of ResNet-20 on CIFAR10: A [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Comparison of test accuracy and training time among methods on CIFAR datasets [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Activation variance resulted from training LeNet-5 on CIFAR10 [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 41 canonical work pages

  1. [45]

    Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models

    Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, and Shuicheng YAN. Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models. InHas it Trained Yet? NeurIPS 2022 Workshop, 2022

    work page 2022

  2. [1]

    Ehsan Amid, Rohan Anil, and Manfred K. Warmuth. Locoprop: Enhancing backprop via local loss optimization. InInternational Conference on Artificial Intelligence and Statistics, 2021

    work page 2021

  3. [2]

    Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization.ArXiv, 2016

    work page 2016

  4. [3]

    Gradient descent on neurons and its link to approximate second-order optimization

    Frederik Benzing. Gradient descent on neurons and its link to approximate second-order optimization. In International Conference on Machine Learning, 2022

    work page 2022

  5. [4]

    Closing the generalization gap of adaptive gradient methods in training deep neural networks

    Jinghui Chen and Quanquan Gu. Closing the generalization gap of adaptive gradient methods in training deep neural networks. InInternational Joint Conference on Artificial Intelligence, 2018

    work page 2018

  6. [5]

    Closing the generalization gap of adaptive gradient methods in training deep neural networks

    Jinghui Chen, Dongruo Zhou, Yiqi Tang, Ziyan Yang, Yuan Cao, and Quanquan Gu. Closing the generalization gap of adaptive gradient methods in training deep neural networks. InInternational Joint Conference on Artificial Intelligence, 2020

    work page 2020

  7. [6]

    When vision transformers outperform resnets without pretraining or strong data augmentations.International Conference on Learning Representations, 2022

    Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform resnets without pretraining or strong data augmentations.International Conference on Learning Representations, 2022

    work page 2022

  8. [7]

    On the convergence of a class of adam-type algorithms for non-convex optimization

    Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of a class of adam-type algorithms for non-convex optimization. InInternational Conference on Learning Representations, 2019

    work page 2019

Show all 51 references
  1. [8]

    Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V . Le. Randaugment: Practical automated data augmentation with a reduced search space.IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2019

    2019

  2. [9]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. InIEEE conference on computer vision and pattern recognition, 2009

    2009

  3. [10]

    Regularizing activation distribution for training binarized deep networks.IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019

    Ruizhou Ding, Ting-Wu Chin, Zeye Dexter Liu, and Diana Marculescu. Regularizing activation distribution for training binarized deep networks.IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019

    2019

  4. [11]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  5. [12]

    Incorporating nesterov momentum into adam.ICLR Workshop, 2016

    Timothy Dozat. Incorporating nesterov momentum into adam.ICLR Workshop, 2016

    2016

  6. [13]

    Adaptive subgradient methods for online learning and stochastic optimization.Journal of Machine Learning Research, 2011

    John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization.Journal of Machine Learning Research, 2011

    2011

  7. [14]

    Sharpness-aware minimization for efficiently improving generalization

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. InInternational Conference on Learning Representations, 2021. 10 An Adaptive Method Stabilizing Activations for Enhanced Generalization

    2021

  8. [15]

    Neuron with steady response leads to better generalization

    Qiang Fu, Lun Du, Haitao Mao, Xu Chen, Wei Fang, Shi Han, and Dongmei Zhang. Neuron with steady response leads to better generalization. InNeural Information Processing Systems, 2022

    2022

  9. [16]

    Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He

    Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour.ArXiv, 2017

    2017

  10. [17]

    Train faster, generalize better: Stability of stochastic gradient descent

    Moritz Hardt, Benjamin Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. InInternational Conference on Machine Learning, 2016

    2016

  11. [18]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InIEEE Conference on Computer Vision and Pattern Recognition, 2016

    2016

  12. [19]

    Weinberger

    Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. Densely connected convolutional networks.IEEE Conference on Computer Vision and Pattern Recognition, 2016

    2016

  13. [20]

    Weinberger

    Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. InEuropean Conference on Computer Vision, 2016

    2016

  14. [21]

    Adaptive learning rate via covariance matrix based preconditioning for deep neural networks

    Yasutoshi Ida, Yasuhiro Fujiwara, and Sotetsu Iwamura. Adaptive learning rate via covariance matrix based preconditioning for deep neural networks. InInternational Joint Conference on Artificial Intelligence, 2016

    2016

  15. [22]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. InInternational Conference on Machine Learning, 2015

    2015

  16. [23]

    Regularizing activations in neural networks via distribution matching with the wasserstein metric

    Taejong Joo, Donggu Kang, and Byunghoon Kim. Regularizing activations in neural networks via distribution matching with the wasserstein metric. InInternational Conference on Learning Representations, 2020

    2020

  17. [24]

    Improving generalization performance by switching from adam to sgd

    Nitish Shirish Keskar and Richard Socher. Improving generalization performance by switching from adam to sgd. arXiv, 2017

    2017

  18. [25]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InInternational Conference on Learning Representations, 2015

    2015

  19. [26]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009

    2009

  20. [27]

    Regularizing rnns by stabilizing activations

    David Krueger and Roland Memisevic. Regularizing rnns by stabilizing activations. InAdvances in Neural Information Processing Systems, 2015

    2015

  21. [28]

    Lecun, L

    Y . Lecun, L. Bottou, Y . Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998

    1998

  22. [29]

    Regularizing by the variance of the activations’ sample-variances

    Etai Littwin and Lior Wolf. Regularizing by the variance of the activations’ sample-variances. InConference on Neural Information Processing Systems, 2018

    2018

  23. [30]

    On the variance of the adaptive learning rate and beyond

    Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. InInternational Conference on Learning Representations, 2020

    2020

  24. [31]

    Adam+: A stochastic method with adaptive variance reduction, 2021

    Mingrui Liu, Wei Zhang, Francesco Orabona, and Tianbao Yang. Adam+: A stochastic method with adaptive variance reduction, 2021

    2021

  25. [32]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternational Conference on Learning Representations, 2019

    2019

  26. [33]

    Adaptive gradient methods with dynamic bound of learning rate

    Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. InInternational Conference on Learning Representations, 2019

    2019

  27. [34]

    Optimizing neural networks with kronecker-factored approximate curvature

    James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International Conference on Machine Learning, 2015

    2015

  28. [35]

    Revisiting activation regularization for language rnns.ArXiv, 2017

    Stephen Merity, Bryan McCann, and Richard Socher. Revisiting activation regularization for language rnns.ArXiv, 2017

    2017

  29. [36]

    Reddi, Satyen Kale, and Sanjiv Kumar

    Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. InInternational Conference on Learning Representations, 2019

    2019

  30. [37]

    Dropout: A simple way to prevent neural networks from overfitting.Journal of Machine Learning Research, 2014

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting.Journal of Machine Learning Research, 2014

    2014

  31. [38]

    Dahl, and Geoffrey E

    Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. On the importance of initialization and momentum in deep learning. InInternational Conference on Machine Learning, 2013

    2013

  32. [39]

    Tieleman and G

    T. Tieleman and G. Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012. 11 An Adaptive Method Stabilizing Activations for Enhanced Generalization

    2012

  33. [40]

    Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N

    Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InNeural Information Processing Systems, 2017

    2017

  34. [41]

    Pytorch image models

    Ross Wightman. Pytorch image models. https://github.com/rwightman/pytorch-image-models , 2019

    2019

  35. [42]

    Resnet strikes back: An improved training procedure in timm

    Ross Wightman, Hugo Touvron, and Herve Jegou. Resnet strikes back: An improved training procedure in timm. InNeurIPS 2021 Workshop on ImageNet: Past, Present, and Future, 2021

    2021

  36. [43]

    Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht

    Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. InNeural Information Processing Systems, 2017

    2017

  37. [44]

    Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017

    Han Xiao, Kashif Rasul, and Roland V ollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017

    2017

  38. [46]

    Adaptive inertia: Disentangling the effects of adaptive learning rate and momentum

    Zeke Xie, Xinrui Wang, Huishuai Zhang, Issei Sato, and Masashi Sugiyama. Adaptive inertia: Disentangling the effects of adaptive learning rate and momentum. InInternational Conference on Machine Learning, 2020

    2020

  39. [47]

    Large batch optimization for deep learning: Training bert in 76 minutes

    Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. InInternational Conference on Learning Representations, 2020

    2020

  40. [48]

    Cutmix: Regularization strategy to train strong classifiers with localizable features.IEEE/CVF International Conference on Computer Vision, 2019

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Young Joon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features.IEEE/CVF International Conference on Computer Vision, 2019

    2019

  41. [49]

    Dauphin, and David Lopez-Paz

    Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. InInternational Conference on Learning Representations, 2018

    2018

  42. [50]

    Eva: Practical second-order optimization with kronecker-vectorized approximation

    Lin Zhang, Shaohuai Shi, and Bo Li. Eva: Practical second-order optimization with kronecker-vectorized approximation. InInternational Conference on Learning Representations, 2023

    2023

  43. [51]

    TX t=1 ηtgt/ p bvt 2 # ≤E

    Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients.Advances in Neural Information Processing Systems, 2020. 12 An Adaptive Method Stabil...

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.