Pith. sign in

REVIEW 3 major objections 4 minor 39 references

A Method for Enhancing Generalization of Adam by Multiple Integrations

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding integrals to Adam's update finds flatter minima

desk verdict A simple, plausible optimizer with substantial empirical results, but the central escape-time theorem is based on a continuum limit that doesn't match its own update rule. read the letter →

arxiv 2412.12473 v1 pith:ZPA2TNBD submitted 2024-12-17 cs.LG

Long Jin , Han Nong , Liangming Chen , Zhenming Su This is my paper · ORCID
classification cs.LG
keywords MIAdamAdamoptimizerflatminimageneralizationmultipleintegrationlabelnoisediffusiontheoryadaptivelearningrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes MIAdam, a version of the Adam optimizer whose parameter update includes a multiple integral term: several nested sums over the gradient history are added to Adam's first moment estimate. The authors argue that this term acts as a low-pass filter on the optimizer's trajectory, so sharp minima, viewed as high-frequency features, are smoothed away and the optimizer drifts toward flat minima. The paper's claim is that flat minima generalize better, and therefore MIAdam improves test accuracy and robustness to label noise while retaining Adam's fast convergence. The authors prove, under a diffusion-theory approximation, that the first-order integral version escapes sharp minima faster than Adam, and they prove that the integral phase alone does not converge, which motivates a switch back to Adam after a fixed number of epochs. A sympathetic reader would take the central claim to be that a simple summation added to Adam's update is enough to bias deep-network training toward flat, well-generalizing solutions.

What carries the argument

The carrying object is the $n$-th-order multiple summation term $m^{(n)}_{t,i}$, built by recursively accumulating the first moment $m_{t,i}$ $n$ times, each accumulation weighted by a rate $\kappa$. In the continuous-time limit this term becomes an $n$-fold integral of the gradient signal, and integration acts as a low-pass filter: high-frequency fluctuations, identified with sharp minima, are attenuated. In the discrete algorithm the term is computed by $n$ extra running sums per parameter, so the added cost is small. Its role is to give an early exploration phase that gravitates toward flat basins; because the term alone prevents convergence (Theorem 2), the algorithm switches to standard Adam updates after $\zeta$ steps, and the final phase converges quickly.

What would settle it

A concrete test on the paper's two-parameter landscapes is to count, over many random initializations, how often MIAdam reaches the flat basin compared with Adam, and to compute the Hessian trace at each final point; if MIAdam does not reach the flat basin more often, or if its final Hessian trace is not smaller, the claim that the integral term filters sharp minima is unsupported.

Watch

Extended reading notes

Core claim

The central claim is that adding an $n$-th-order multiple integral of past gradients to Adam's update, before switching back to Adam at step $\zeta$, filters out sharp minima and guides the optimizer toward flat minima, improving generalization and label-noise robustness without sacrificing Adam's convergence speed. The update in Eq. (12) replaces the first moment $m_{t,i}$ with a nested summation $m^{(n)}_{t,i}=\sum_{t_1=0}^{t}\kappa^{t-t_1}\sum_{t_2=0}^{t_1}\cdots\sum_{t_n=0}^{t_{n-1}}\beta_1^{t_{n-1}-t_n}g_{t_n,i}$, where $\kappa$ is the multiple-integration rate. Under the diffusion-theory approximation, Theorem 1 gives the mean escape time $\phi_{\mathrm{MIAdam1}}$ from a sharp minimum through a saddle point; it is smaller than the corresponding Adam escape time for $\tilde{t}>1$, meaning the integrated optimizer is more likely to abandon sharp minima. Theorem 2 shows that before the switch the algorithm has non-vanishing average regret, so the integral phase is intentionally non-convergent, and the later switch to Adam restores convergence. Empirically, MIAdam1 achieves higher top-1 accuracy than Adam and its variants on CIFAR-10/100, ImageNet-1k, and text classification datasets, and higher accuracy under 20--80% label noise; Hessian eigenvalue densities show flatter final minima.

Load-bearing premise

The whole case rests on treating sharp minima as high-frequency components of the optimizer trajectory that a multiple integral can filter out, together with the diffusion-theory assumptions of a locally quadratic loss, quasi-equilibrium, and low gradient noise; if any of these fail, the escape-time comparison does not establish that MIAdam finds flatter minima.

Editorial extensions

If this is right

  • MIAdam is a drop-in replacement for Adam: it uses the same learning rate and decay hyperparameters, adds only a multiple-integration rate $\kappa$ and a switch step $\zeta$, and its per-iteration cost is $n$ extra summations per parameter.
  • If the central claim holds, training with MIAdam instead of Adam should consistently produce solutions with smaller top Hessian eigenvalues and traces, that is, flatter loss landscapes, across a wide range of architectures and datasets.
  • The label-noise experiments imply that the filtering effect is most valuable when gradients are corrupted, because the integral averages out spurious high-frequency gradient fluctuations; this extends naturally to noisy or small-sample training regimes.
  • Since the integral phase is provably non-convergent, the switch time $\zeta$ is the key hyperparameter: too short a switch gives Adam-like behavior, too long delays convergence, so an automatic or scheduled switch could be a practical improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's theory covers only the first-order integral (MIAdam1), while experiments include second- and third-order versions; a natural inference is that higher orders filter more aggressively but also cause larger oscillations near flat regions, consistent with the reported MIAdam3 instability and suggesting an optimal finite order.
  • The signal-processing analogy suggests the same integration trick could be applied to other adaptive optimizers, such as AdamW or AdaBound, or even to SGD with momentum; the mechanism is not obviously specific to Adam's moment estimates.
  • One testable extension is to make the integration rate $\kappa$ or the switch time $\zeta$ adaptive to gradient statistics, rather than fixed grid-searched values, which could remove the two new hyperparameters.
  • Because the theoretical escape-time comparison uses a locally quadratic loss, the practical claim is stronger than the proof: the empirical flatness gains on deep networks suggest the mechanism survives far outside the quadratic regime, but that generalization is not established by Theorem 1.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes MIAdam, an Adam variant that replaces the momentum update with an n-th order cumulative (multiple-integral) sum of past momentum for the first ζ iterations and then switches to standard Adam. The paper claims that the multiple integral acts as a low-pass filter that filters out sharp minima, guiding the optimizer to flat minima and thereby improving generalization and robustness to label noise while retaining Adam's fast convergence. The theoretical section uses a diffusion-theory mean-escape-time calculation to argue that MIAdam1 escapes sharp minima faster than Adam, together with a regret analysis showing non-convergence before the switch. The empirical section reports image classification (CIFAR and ImageNet), text classification (BERT/RoBERTa), and label-noise experiments, plus Hessian eigenvalue diagnostics, and the paper provides a code link.

Significance. If the theoretical claims were valid, MIAdam would be a simple, cheap drop-in optimizer with practical gains, and the paper's empirical scope is indeed substantial: multiple architectures on CIFAR, ImageNet results, text fine-tuning, label-noise experiments, and Hessian diagnostics, with code provided. The paper also honestly acknowledges that the multiple-integral phase is non-convergent and must be followed by a switch to Adam. However, the central theoretical justification is load-bearing and, on inspection, does not describe the algorithm's dynamics: the continuum limit of the MIAdam1 update is different from the equation used in Theorem 1, the escape-time formula contains unexplained scaling, and the convergence section neither covers the switched algorithm nor supports the claimed divergence. Because the main novel claims rest on these derivations, I cannot recommend publication in the current form.

major comments (3)
  1. [Appendix, 'Generalization Proof', Eqs. (18)-(20)] The continuum limit of the MIAdam1 update in Eq. (18) is not the deformed motion equation in Eq. (20). Eliminating m_t and M_t from Eq. (18) gives θ_{t+1} - (2+β_1)θ_t + (1+2β_1)θ_{t-1} - β_1θ_{t-2} = -α^2(1-β_1)g_t. Setting α=h and β_1=1-δh and letting h→0 yields δθ'' + θ''' = -δg, i.e. χθ''' + θ'' = -g with χ=h/(1-β_1). Equation (20) instead asserts χθ''' = δχθ'' + f, which with δχ=1 and f=-g gives χθ''' - θ'' = -g, the opposite sign for θ''. The later rescaling in Eq. (21) changes the noise/diffusion term but does not repair this deterministic mismatch. Since Eq. (20) is the starting point of the Fokker-Planck reduction, the escape time in Eq. (27) is not a consequence of the MIAdam1 dynamics in Eq. (18).
  2. [Generalization Analyses, Eq. (14) and Eq. (27)] Even if Eq. (20) were accepted, the comparison φ_MIAdam1 < φ_Adam is not supported by the displayed formulas. The factor 1/˜t appears in both the prefactor and the exponent of Eq. (14)/(27) without derivation, and the learning-rate scaling is inconsistent with the α^2 update in Eq. (18): Theorem 1 uses α as if it were the small step in the diffusion term, while the deterministic update already scales with α^2. The claimed inequality is therefore an artifact of an unexplained ˜t-dependent rescaling and the chosen effective learning-rate convention, rather than a property of the multiple integral term.
  3. [Convergence Analyses and Appendix, 'Convergence Proof', Eqs. (36)-(40)] The claimed divergence of term {3} in the proof of Theorem 2 rests on an extra factor t. For κ_1∈(0,1), the sum ∑_{r=1}^{t-1} κ_1^{t-r} is bounded by κ_1/(1-κ_1) and does not grow with t, whereas Eq. (36) as written multiplies this bounded sum by t and then concludes divergence; for κ_1=1 the bound with denominator 1-κ_1 is not valid. Moreover, even a correct divergence proof would only establish Theorem 2's statement of non-convergence before the switch, while the actual MIAdam algorithm switches to Adam at ζ; no regret bound after ζ is supplied. The conclusion that MIAdam provides 'a guarantee of convergence' is therefore unsupported.
minor comments (4)
  1. [Conclusion] The conclusion states that the paper has 'provided a guarantee of convergence,' but Theorem 2 only proves non-convergence before the switch and no post-switch regret bound is given; this wording should be corrected.
  2. [Experiments, 'Robustness Against Label Noises' and Table 2] The dataset name is misspelled as 'CIAFR10' in the label-noise section, and Table 2's header says 'mean±std' while the table entries are single numbers; the reporting format should be made consistent.
  3. [Appendix, 'Generalization Proof'] The proof of Theorem 1 sets κ=1 'without loss of generality,' but κ is a tuned hyperparameter in Algorithm 1 and in the grid search; either a justification of the WLOG reduction or an explicit restriction of the theorem to κ=1 is needed.
  4. [Experiments, Table 3] The text says MIAdam 'significantly outperforms' Adam on text classification, but many differences in Table 3 are within 0.1-0.3 percentage points and no significance tests are reported; the claim of significance should be softened or supported.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the generalization claim is supported by independent benchmark evaluations and an external diffusion-theory framework, not by definition or by load-bearing self-citation.

full rationale

MIAdam's central claims are (i) that the multiple-integral update escapes sharp minima and hence reaches flatter regions, and (ii) that this translates into improved test accuracy and label-noise robustness. Claim (ii) is verified against standard CIFAR, ImageNet, and text benchmarks and compared with Adam and its variants; these empirical numbers are not equal by construction to the algorithm's update. Claim (i) is argued through the diffusion-theory escape-time comparison. The proof in the appendix substitutes the MIAdam1 update into the framework of Xie, Sato, and Sugiyama (2020) and Xie et al. (2022), which are external sources, not the authors' own prior work; the resulting formula (14) is compared with Adam's escape-time formula (15) from the same external framework. The comparison is quantitative and could in principle have shown the opposite ordering, so the conclusion is not built into the definitions. The only self-citation is Jin, Zhang, and Li (2015), and it is used solely to motivate the noise-filtering intuition behind integration; it is not used to justify the flat-minima or generalization result, nor to rule out alternative mechanisms. The grid search for kappa and zeta is a hyperparameter-selection issue that could affect the strength of the empirical comparison, but without evidence that the selection was made on the reported test labels it is not a demonstration that a fitted parameter was renamed as a prediction. A reviewer-identified discrepancy between the discrete update (18) and the continuum equation (20) would, if correct, undermine Theorem 1's proof as a matter of mathematical validity rather than circularity: it would make the escape-time comparison unsupported, not make the claim an input. Under the stated standard, no circular step is exhibited, so the appropriate score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The ledger shows that the method introduces two tuned hyperparameters (κ and ζ) and an order n chosen empirically. The theoretical results rest on standard diffusion-theory approximations and an unproven analogy between sharp minima and high-frequency signal components. No new physical entities are introduced.

free parameters (3)
  • κ (multiple integration rate) = 0.98
    Grid-searched over [0.01, 0.02, ..., 0.99] on the target benchmarks (Appendix, Experiment Settings).
  • ζ (switching moment) = 20 (image/text), 40 (label noise)
    Grid-searched over [1, 2, ..., 150]; the value controls when MIAdam reverts to Adam and directly determines whether convergence is retained.
  • n (integration order) = 1 (MIAdam1 best)
    The order is a design choice; results show MIAdam3 oscillates more and the order is not derived from theory.
assumptions (4)
  • domain assumption Loss landscape is locally quadratic around critical points (Assumption 1, Eq. (13)).
    Needed for the diffusion-theory escape-time analysis in Theorem 1.
  • domain assumption Quasi-equilibrium near minima and low-temperature (small gradient noise) approximations (Assumptions 2-3).
    Inherited from Xie et al. 2020/2022; required to derive the Fokker-Planck escape-time formula.
  • domain assumption Flat minima generalize better than sharp minima.
    Cited from Hochreiter and Schmidhuber 1997 and others; the paper's goal is built on this correlation.
  • ad hoc to paper Sharp minima correspond to high-frequency components of the optimizer trajectory, and multiple integration filters them out.
    Asserted in Section MIAdam after Eq. (10); this signal-processing analogy is the motivation for the algorithm but is not derived from loss-landscape geometry.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Method for Enhancing Generalization of Adam by Multiple Integrations." pith.science (2026). https://pith.science/paper/ZPA2TNBD

@misc{pith2026241212473,
  author       = {Pith},
  title        = {Pith review of: A Method for Enhancing Generalization of Adam by Multiple Integrations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZPA2TNBD}},
  note         = {Machine review of arXiv:2412.12473}
}
read the original abstract

The insufficient generalization of adaptive moment estimation (Adam) has hindered its broader application. Recent studies have shown that flat minima in loss landscapes are highly associated with improved generalization. Inspired by the filtering effect of integration operations on high-frequency signals, we propose multiple integral Adam (MIAdam), a novel optimizer that integrates a multiple integral term into Adam. This multiple integral term effectively filters out sharp minima encountered during optimization, guiding the optimizer towards flatter regions and thereby enhancing generalization capability. We provide a theoretical explanation for the improvement in generalization through the diffusion theory framework and analyze the impact of the multiple integral term on the optimizer's convergence. Experimental results demonstrate that MIAdam not only enhances generalization and robustness against label noise but also maintains the rapid convergence characteristic of Adam, outperforming Adam and its variants in state-of-the-art benchmarks.

Figures

Figures reproduced from arXiv: 2412.12473 by the authors.

Figure 1
Figure 1. The idea of this work and the filtering effect of in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Simulations of trajectory of Adam and MIAdam on 2-parameter loss landscapes. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Comparisons of top Hessian eigenvalues λtop, Hessian traces λtrace, and full Hessian eigenvalue densities for loss landscapes on the CIFAR-10 dataset using ResNet18. Optimizer ResNet18 ResNet50 CIFAR-10(%) Time CIFAR-100(%) Time CIFAR-10(%) Time CIFAR-100(%) Time Adam 93.89±0.18 47m 73.17±0.26 47m 93.69±0.28 2h 45m 75.24±0.62 2h 47m NAdam 93.92±0.11 46m 73.08±0.31 46m 94.03±0.07 2h 57m 74.95±0.08 2h 58m AdamW 93.74±… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparisons of the sum of the absolute values of the eigenvalues of the Hessian matrix at the convergence points of [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Training and testing comparations of Adam, MIAdam1, MIAdam2, and MIAdam3 on CIFAR-100 using [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 30 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Chaudhari, P.; et al. 2019. Entropy- SGD : Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12): 124018

    work page 2019

  4. [4]

    Deng, X.; Sun, T.; Li, S.; and Li, D. 2023. Stability-based generalization analysis of the asynchronous decentralized SGD. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 7340--7348

    work page 2023

  5. [5]

    Dinh, L.; Pascanu, R.; Bengio, S.; and Bengio, Y. 2017. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, 1019--1028

    work page 2017

  6. [6]

    Dozat, T. 2016. Incorporating N esterov momentum into A dam. In International Conference on Machine Learning

    work page 2016

  7. [7]

    Du, J.; Zhou, D.; Feng, J.; Tan, V.; and Zhou, J. T. 2022. Sharpness-aware training for free. Advances in Neural Information Processing Systems, 23439--23451

    work page 2022

  8. [8]

    Han, D.; Kim, J.; and Kim, J. 2017. Deep pyramidal residual networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5927--5935

    work page 2017

Show all 39 references
  1. [9]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE C onference on C omputer V ision and P attern R ecognition , 770--778

    2016

  2. [10]

    Hochreiter, S.; and Schmidhuber, J. 1995. Simplifying neural nets by discovering flat minima. Advances in N eural I nformation P rocessing S ystems , 7: 529--536

    1995

  3. [11]

    Hochreiter, S.; and Schmidhuber, J. 1997. Flat minima. Neural Computation, 9(1): 1--42

    1997

  4. [12]

    Huang, G.; Liu, Z.; van der Maaten, L.; and Weinberger, K. Q. 2017. Densely Connected Convolutional Networks. In Proceedings of the IEEE C onference on C omputer V ision and P attern Recognition

    2017

  5. [13]

    Jiang, Y.; Neyshabur, B.; Mobahi, H.; Krishnan, D.; and Bengio, S. 2020. Fantastic Generalization Measures and Where to Find Them. In International Conference on Learning Representations

    2020

  6. [14]

    Jin, L.; Zhang, Y.; and Li, S. 2015. Integration-enhanced Z hang neural network for real-time-varying matrix inversion in the presence of various kinds of noises. IEEE Transactions on Neural Networks and Learning Systems, 27(12): 2615--2627

    2015

  7. [15]

    Johnson, T.; Agrawal, P.; Gu, H.; and Guestrin, C. 2020. AdaScale SGD : A user-friendly algorithm for distributed training. In International Conference on Machine Learning, 4911--4920

    2020

  8. [16]

    Kalinay, P.; and Percus, J. K. 2012. Phase space reduction of the one-dimensional F okker- P lanck ( K ramers) equation. Journal of Statistical Physics, 148(6): 1135--1155

    2012

  9. [17]

    S.; Mudigere, D.; Nocedal, J.; Smelyanskiy, M.; and Tang, P

    Keskar, N. S.; Mudigere, D.; Nocedal, J.; Smelyanskiy, M.; and Tang, P. T. P. 2017. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations

    2017

  10. [18]

    S.; and Socher, R

    Keskar, N. S.; and Socher, R. 2017. Improving generalization performance by switching from A dam to SGD . arXiv preprint arXiv:1712.07628

    2017 arXiv

  11. [19]

    P.; and Ba, J

    Kingma, D. P.; and Ba, J. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations

    2015

  12. [20]

    Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012. Image N et classification with deep convolutional neural networks. Advances in N eural I nformation P rocessing S ystems , 1097--1105

    2012

  13. [21]

    Lin, Y.; Meng, Y.; Sun, X.; Han, Q.; Kuang, K.; Li, J.; and Wu, F. 2021. BERTGCN : Transductive text classification by combining GCN and BERT . arXiv preprint arXiv:2105.05727

    2021 arXiv

  14. [22]

    B.; and Ueda, M

    Liu, Z.; Li, B.; Simon, J. B.; and Ueda, M. 2021. SGD can converge to local maxima. In International Conference on Learning Representations

    2021

  15. [23]

    Loshchilov, I.; and Hutter, F. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101

    2017 arXiv

  16. [24]

    Luo, L.; Xiong, Y.; Liu, Y.; and Sun, X. 2019. Adaptive gradient methods with dynamic bound of learning rate. arXiv preprint arXiv:1902.09843

    2019 arXiv

  17. [25]

    Mulayoff, R.; and Michaeli, T. 2020. Unique properties of flat minima in deep networks. In International C onference on M achine L earning , 7108--7118

    2020

  18. [26]

    Petzka, H.; Kamp, M.; Adilova, L.; Sminchisescu, C.; and Boley, M. 2021. Relative flatness and generalization. Advances in Neural Information Processing Systems, 18420--18432

    2021

  19. [27]

    Qian, N. 1999. On the momentum term in gradient descent learning algorithms. Neural Networks, 12(1): 145--151

    1999

  20. [28]

    A.; and Mullis, C

    Roberts, R. A.; and Mullis, C. T. 1987. Digital signal processing. Addison-Wesley Longman Publishing Co., Inc

    1987

  21. [29]

    Savarese, P. 2019. On the convergence of A da B ound and its connection to SGD . arXiv preprint arXiv:1908.04457

    2019 arXiv

  22. [30]

    Sun, Y.; Shen, L.; Chen, S.; Ding, L.; and Tao, D. 2023. Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscape. In International Conference on Machine Learning

    2023

  23. [31]

    S.; Xiong, C.; and Socher, R

    Wang, H.; Keskar, N. S.; Xiong, C.; and Socher, R. 2018. Identifying generalization properties in neural networks. arXiv preprint arXiv:1809.07402

    2018 arXiv

  24. [32]

    C.; Roelofs, R.; Stern, M.; Srebro, N.; and Recht, B

    Wilson, A. C.; Roelofs, R.; Stern, M.; Srebro, N.; and Recht, B. 2017. The marginal value of adaptive gradient methods in machine learning. Advances in Neural I nformation P rocessing S ystems , 4148--4158

    2017

  25. [33]

    Xie, Z.; Sato, I.; and Sugiyama, M. 2020. A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. In International Conference on Machine Learning

    2020

  26. [34]

    Xie, Z.; Wang, X.; Zhang, H.; Sato, I.; and Sugiyama, M. 2022. Adaptive inertia: Disentangling the effects of adaptive learning rate and momentum. In International Conference on Machine Learning, 24430--24459

    2022

  27. [35]

    Yang, X. 2020. Stochastic gradient variance reduction by solving a filtering problem. arXiv preprint arXiv:2012.12418

    2020 arXiv

  28. [36]

    Yao, Z.; Gholami, A.; Keutzer, K.; and Mahoney, M. W. 2020. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE International Conference on Big Data (Big Data), 581--590

    2020

  29. [37]

    Yao, Z.; Gholami, A.; Shen, S.; Mustafa, M.; Keutzer, K.; and Mahoney, M. 2021. Adahessian: An adaptive second order optimizer for machine learning. In proceedings of the AAAI conference on artificial intelligence, volume 35, 10665--10673

    2021

  30. [38]

    Zhang, Z. 2018. Improved A dam optimizer for deep neural networks. In 2018 IEEE/ACM 26th I nternational S ymposium on Q uality of S ervice (IWQoS) , 1--2

    2018

  31. [39]

    Zou, D.; Cao, Y.; Li, Y.; and Gu, Q. 2021. Understanding the generalization of A dam in learning neural networks with proper regularization. arXiv preprint arXiv:2108.11371

    2021 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.