Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

The fixed global momentum coefficient is why mSGD lags behind Adam on transformers; replacing it with per-parameter kinetic-energy friction closes most of the gap.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 06:05 UTC pith:SC6SW2NP

load-bearing objection New optimizer dynamics worth refereeing, but the headline empirical claim is undercut by an unequal tuning protocol and the stochastic convergence statement as written is false. the 4 major comments →

arxiv 2602.00334 v2 pith:SC6SW2NP submitted 2026-01-30 cs.LG math.OC

Adaptive Momentum and Nonlinear Damping for Neural Network Training

Aikaterini Karoni , Rajit Rajpal , Benedict Leimkuhler , Gabriel Stoltz This is my paper
classification cs.LG math.OC MSC 65K1068T0790C26
keywords adaptive momentumcubic dampingkinetic energytransformer optimizationmomentum SGDAdamcontinuous-time optimizationstrong convexity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the fixed, global momentum coefficient of momentum SGD is the reason it lags behind Adam on transformer models, and replaces it with a per-parameter friction coefficient that adapts to each coordinate's kinetic energy. This adaptive friction is shown to be equivalent, near equilibrium, to a cubic damping term, leading to two new optimizers: iKFAD and cubically damped mSGD (CD). On TinyViT, DistilBERT, and GPT2 tasks, the authors report that iKFAD and CD match or beat Adam's test loss without using per-parameter adaptive learning rates, closing much of the Adam–mSGD gap. The paper also proves exponential convergence for the continuous and discrete dynamics of both methods under strong convexity assumptions. A sympathetic reader would care because it suggests a simple, memory-cheap modification to momentum methods can capture much of the benefit of adaptive optimizers.

Core claim

The central claim is that the failure of momentum SGD on transformers stems from a single damping coefficient applied uniformly to all coordinates, and that per-coordinate adaptive damping — regulated by the kinetic energy p_i²/ρ of each momentum component, or its near-equilibrium cubic form −c[p]³ — restores stability and speed without adaptive learning rates. In the proposed iKFAD dynamics, an auxiliary variable ξ tracks an exponentially weighted average of past squared momenta and multiplies p componentwise; in CD, the cubic term acts the same way. The authors demonstrate on TinyViT, DistilBERT, and GPT2 that both methods reach test losses competitive with or better than Adam while mSGD s

What carries the argument

The load-bearing object is a dissipative Hamiltonian reformulation of momentum: mSGD is an Euler discretization of linearly damped Hamiltonian dynamics ẋ=p, ṗ=−∇f(x)−γp, where the friction γ and the momentum μ are linked by μ≈1−γ√δt. The paper's new step is to make friction coordinate-wise and state-dependent: iKFAD sets γ_i=γ+ξ_i with ξ̇_i=p_i²/ρ−αξ_i, so each coordinate's damping grows with its recent kinetic energy; near equilibrium this reduces to cubic damping −c[p]³, which is gentler than linear damping at low momenta and stronger at high momenta. The proofs of exponential convergence use Lyapunov functions that mix position, momentum, and the auxiliary friction variable.

Load-bearing premise

The convergence proofs assume a strongly convex (bowl-shaped, curvature-bounded) objective with γ>0 for iKFAD, while real transformer losses are nonconvex; moreover, the discrete guarantees are proved for a simplified Euler scheme, not the implemented operator-splitting updates.

What would settle it

Train CD and iKFAD (with γ=0) on a transformer task above roughly 100M parameters under the same hyperparameter budget as Adam: if either fails to match Adam's test loss or diverges where Theorem 4's step-size condition is satisfied, the claim that these mechanisms close the Adam–mSGD gap would be weakened. Alternatively, on a strongly convex quadratic with known m and M, check whether the implemented splitting scheme converges for step sizes that the Euler-based Theorem 4 allows; a counterexample would show the discrete analysis does not cover the deployed algorithm.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Momentum methods can rival adaptive optimizers on transformer architectures without storing or rescaling per-parameter learning rates, so the Adam–mSGD gap is not intrinsic to momentum.
  • CD with γ=0 keeps the same memory footprint as mSGD (two state variables) while adding only element-wise cubing, making it a cheap drop-in alternative.
  • The near-equilibrium link between iKFAD and CD gives a principled way to tune cubic damping: c≈1/(αρ).
  • The continuous-time analysis provides a Lyapunov-function template for proving exponential convergence of momentum variants under strong convexity, a setting typical of quadratics and well-conditioned problems.
  • Setting γ=0 does not degrade iKFAD/CD in the reported experiments, removing one hyperparameter.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A reader could test a simpler hypothesis the paper leaves implicit: since iKFAD behaves like CD near equilibrium, the extra ξ state may be unnecessary for many tasks, and CD alone is the minimal method worth adopting.
  • The reported sensitivity of optimal c and ρ across tasks suggests gradient-scale normalization would make these hyperparameters portable; the paper notes this but does not implement it.
  • Because the adaptive friction acts like a thermostat on momenta, it may also improve robustness to heavy-tailed gradient noise — a property the paper does not directly test.
  • The theory only covers strongly convex f, so the empirical success on nonconvex transformers relies on the damping mechanism's stabilizing effect; a natural extension is analyzing convergence on nonconvex landscapes with a Polyak–Łojasiewicz condition.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces three continuous-time optimizers: iKFAD, which replaces the scalar friction of dissipative Hamiltonian dynamics with a per-parameter kinetic-energy-controlled friction; CD, which augments mSGD with cubic damping; and CADAM, which adds cubic damping to a continuous Adam model. The authors argue that near the iKFAD steady state ξ̇≈0 the adaptive friction reduces to cubic damping, motivate CD as a practical discretization-friendly version, and provide Lyapunov-based exponential convergence theorems for continuous and discrete iKFAD and CD dynamics under strong-convexity-type conditions. Empirically, on TinyViT, DistilBERT, GPT2-Nano, and GPT2-XS, CD and iKFAD are reported to match or outperform Adam and to close much of the Adam–mSGD gap without per-parameter adaptive learning rates.

Significance. If the empirical and theoretical claims are upheld, the paper would make a useful contribution: it offers a simple continuous-time mechanism for coordinate-wise momentum adaptation that does not require per-parameter learning rates, and it connects an adaptive thermostat to cubic damping in a way that is easy to implement and to analyze. The paper also has positive methodological elements: the operator-splitting updates in Table 3 are explicit and inexpensive, the Lyapunov proofs for the continuous models are at least sketched, and the low-dimensional illustrations (Figures 2–3) give intuition. However, the significance is conditional. The central empirical claim rests on a hyperparameter-selection protocol that gives the proposed optimizers an extra best-of-two sweep, and the discrete convergence theorems analyze discretizations that are not the ones used in the experiments. The stochastic convergence statement in Proposition 4 is, as written, incompatible with constant step size and bounded noise. These issues must be resolved before the paper's main conclusions can be accepted.

major comments (4)
  1. [§6.1, Table 1, Fig. 4, Appendix G] The empirical comparison is not currently fair. The text states that CD/iKFAD were evaluated with two distinct sweeps — one with γ>0 and one with γ=0 — and the top-performing configuration was selected for each. Appendix G confirms: “There were some curves with γ=0 that ended up doing better than γ>0. We ended up choosing those as the overall best hyperparameters when presenting them in Figure 4.” Adam and mSGD, by contrast, receive a single 80-trial Optuna sweep. This gives CD/iKFAD strictly more tuning budget and selects the best of two independent runs on the test set. Since the paper's headline claim is that CD/iKFAD “match or outperform Adam” on transformer tasks, this protocol can fully explain the gains in Table 1 and Figure 4. The central empirical claim is therefore not established by the current data. The authors should rerun the comparison with an identical protocol for all op
  2. [§5.2, Eq. (12) vs Table 2 (C’D’BA), §3] Theorem 4 analyzes the Euler discretization in Eq. (12), while the experiments for CD use the C’D’BA operator-splitting scheme described in Appendix B and Table 2. These are different algorithms; the theorem provides no convergence guarantee for the implemented method. The paper even notes that Euler is “simpler to work with than the splitting scheme proposed in Table 2,” but then uses the splitting scheme in the experiments. The same gap appears for iKFAD: Theorem 2 analyzes the CDBA splitting only after the proof is deferred to self-cited reference [9], and the implemented iKFAD uses a different composition (C’DBA). The theory therefore does not cover the optimizers whose performance is reported. At minimum, the authors should state this limitation explicitly in the main text and prove convergence for the actual implemented schemes, or implement and evaluate the analyzed Euler schemes.
  3. [Appendix I.4, Proposition 4] Proposition 4 claims exponential convergence to the exact minimizer for stochastic CD with a constant step size δt and uniformly bounded gradient noise variance D>0: E[f(x_n)−f(x*)]+E||p_n||² ≤ C e^{−κ n δt}. With constant step size and bounded noise, a stochastic optimizer cannot converge to the exact minimizer in general; it converges to a stationary distribution or bounded neighborhood whose radius depends on D and δt. The proof itself introduces D-dependent terms (e.g., Cδt²D in the bound of E||p_{n+1}||²) that are not present in the theorem statement. As written, the proposition is incorrect. It should either be removed, or restated as convergence to an O(Dδt)-neighborhood / biased stationary state, with the D-dependent terms retained in the rate.
  4. [§5.1–5.2, Theorems 1–4, Appendix G, Table 4] The theoretical results require strong convexity (or condition (9)) and, for iKFAD, γ>0. Yet the best reported configurations for CD and iKFAD in Table 4 and Appendix G often set γ=0 (e.g., CD on SST-2, QNLI, NanoGPT; iKFAD on GPT2-XS). For γ=0, Theorems 1 and 2 do not apply, and transformer losses are not strongly convex, so none of the theorems cover the settings where the empirical claims are made. This is not by itself fatal — many optimizer papers use theory for convex toy models — but the paper should clearly state the mismatch between the theoretical hypotheses and the experimental regime, and should not imply that the reported transformer results are backed by the exponential convergence theorems.
minor comments (5)
  1. [§6.1, Figure 4 caption] The caption lists “ADAM mSGD iKFAD CD” but the surrounding text and Section 6.1 mention CADAM curves; Appendix E states that CADAM was included in Figure 10. Please make the caption consistent with the actual contents.
  2. [Appendix B, Table 2] The table has rendering artifacts with stray symbols in the operator-splitting decomposition; the sub-operators would be easier to check if presented as plain equations.
  3. [§5.2, Eq. (12) and Theorem 4 proof (Appendix I.3)] The proof of Theorem 4 uses several generic constants (Q, K, ν) without explicit definitions in the statement; please state the dependencies explicitly or clarify that they are universal constants depending only on m, M, γ, c, and R.
  4. [§6, text after Table 1] The sentence “Across all benchmarks, iKFAD, CD, and CADAM consistently achieve lower training and test losses than Adam and mSGD” is contradicted by the CADAM results in Appendix E, where CADAM is reported as generally inferior to Adam. Please restrict the claim to iKFAD/CD or qualify it.
  5. [Throughout] There are several typos and notation slips: “notoceably” (Section 6.1), “mGD” vs “mSGD” (Figures 2–3 and text), and “Leftr⫯g⊸tl⫯ne⇒” in Appendix A. A careful proofread is recommended.

Circularity Check

0 steps flagged

No circularity: the theoretical derivations are self-contained and the iKFAD-CD link is a stated steady-state approximation, not a fitted prediction.

full rationale

The derivation chain is not circular. The iKFAD-to-CD connection is obtained by an explicit quasi-steady-state reduction of the paper's own ODEs: setting ξ̇≈0 in (6c) gives ξ≈(αρ)^{-1}[p]^2, and substituting into (6b) yields the CD momentum equation with c=(αρ)^{-1}. This is an algebraic approximation, not a parameter fitted to data and then renamed as a prediction. The convergence theorems for iKFAD and CD are supported by explicit Lyapunov-function proofs in Appendices H.2, H.3, I.2, and I.3; the references to the authors' earlier FAD paper [9] are proof-strategy pointers, not the sole justification, and the current proofs are actually written out. No uniqueness theorem is imported from the authors' prior work to force the choice of iKFAD or CD, and no known empirical pattern is merely relabeled. The main validity concerns—best-of-two-sweep selection for CD/iKFAD versus a single sweep for Adam/mSGD, strong-convexity assumptions that do not hold for transformer losses, and Theorem 4 analyzing Euler discretization (12) rather than the deployed C'D'BA splitting—are matters of experimental design and theory-to-algorithm coverage, not circularity. The stated limitation that CADAM convergence is not proved is also a missing-support issue, not a circular step. Overall, the paper's central derivations do not reduce to their own inputs by construction.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 1 invented entities

The central empirical claim is supported by the open scheduler; the deterministic theory rests on strong convexity/condition (9), smoothness, and bounded noise assumptions; the analysis discretizes the ODEs with schemes that do not always match the deployed algorithms; the adaptive friction variable ξ is an introduced algorithmic state with no external evidence.

free parameters (5)
  • γ (linear friction) = varies per task; often ~0 for CD/iKFAD, large for CADAM (e.g., 9.22 for SST-2)
    Friction coefficient in (6b),(7b); swept in [1e-8,10] and chosen by Optuna; shown unnecessary for CD/iKFAD, essential for CADAM.
  • α (friction decay rate) = e.g., 0.0465 (SST-2 iKFAD), 0.0476 (GPT2-XS iKFAD), 0.005-8.8 for CADAM
    Memory decay in (6c),(8c); controls timescale of kinetic-energy averaging.
  • ρ (kinetic-energy scale in iKFAD) = ranges e.g. 3.61e-7 to 6.71 across tasks
    Normalizes [p]^2 in (6c); strongly task-dependent per Table 4/5.
  • c (cubic damping coefficient) = ranges e.g. 5.68e4 to 1.90e8
    Coefficient of [p]^3 in (7b),(8b); chosen per task by Optuna; authors note wide range and suggest gradient normalization.
  • δt (learning rate) = see Table 4
    Discretization step for all schemes; tuned per task.
axioms (4)
  • domain assumption f ∈ C^2 with strong convexity / condition (9): a[f−f*]+b||x−x*||^2 ≤ (x−x*)·(∇f−∇f*)
    Used in Theorems 1-4 and Lemma 1; neural network losses are not strongly convex, so the theory does not cover the empirical regime.
  • domain assumption Bounded gradient noise variance E||G(x)−∇f(x)||^2 ≤ D (Assumption 1)
    Used in stochastic CD proof; with constant step size this prevents exact convergence to the minimizer, so Proposition 4 is doubtful.
  • ad hoc to paper Continuous-time ODEs (6)-(8) faithfully represent the implemented split-discretization optimizers
    The theoretical results are for the continuous dynamics or for Euler/CDBA schemes; the implemented CD uses C'D'BA splitting, which is not analyzed.
  • ad hoc to paper Steady-state approximation ξ̇ ≈ 0 to relate iKFAD to cubic damping
    Justifies the link −ξp ≈ −(αρ)^{-1}[p]^3 (Eq. after (6c)); used as a motivating explanation, not a theorem.
invented entities (1)
  • ξ (per-parameter adaptive friction vector) no independent evidence
    purpose: Dynamically adjusts damping per parameter based on kinetic energy (6c)
    Algorithmic state variable; no independent empirical evidence outside the paper's own experiments.

pith-pipeline@v1.3.0-alltime-deepseek · 37937 in / 19711 out tokens · 204998 ms · 2026-08-03T06:05:37.814831+00:00 · methodology

0 comments
read the original abstract

Momentum Stochastic Gradient Descent (mSGD) relies on a fixed momentum coefficient shared across all parameters, failing to account for the heterogeneous structure of modern loss landscapes. In this work, we adopt a continuous-time formulation to introduce individual, adaptive momentum coefficients regulated by the kinetic energy of each model parameter. This mechanism automatically adjusts to evolving training dynamics to maintain stability without sacrificing convergence speed. We demonstrate that this adaptive friction is inextricably linked to cubic damping, a suppression mechanism from structural dynamics. We additionally introduce two optimization schemes by augmenting the continuous dynamics of mSGD and Adam with a cubic damping term. Empirically, our methods demonstrate robustness and match or outperform Adam on training ViT, BERT, and GPT2 tasks where mSGD typically struggles. We further provide theoretical results establishing the exponential convergence of the proposed schemes.

Figures

Figures reproduced from arXiv: 2602.00334 by Aikaterini Karoni, Benedict Leimkuhler, Gabriel Stoltz, Rajit Rajpal.

Figure 1
Figure 1. Figure 1: Adam-mSGD gap: Selected experiments demonstrating that CD and iKFAD can close the gap between Adam and mSGD without per-parameter adaptive learning rates on language modeling tasks using Transformers. the full gradient setting, where it behaves similarly to sign descent with momentum. This suggests that Adam’s strength lies in its coordinate-wise normalization, which allows it to tackle the Hessian heterog… view at source ↗
Figure 2
Figure 2. Figure 2: Phase portraits for a two-dimensional Rosenbrock function: Each red grid point corresponds to a different initialization in the (x, y) domain and each blue line corresponds to a different optimization trajectory. The minimum (xmin, ymin) = (1, 1), is denoted by a green star. We set γ = 1, h = 0.005, and α = ρ = c = 1. mSGD exhibits significant oscillations and overshooting of the minimum (note the differen… view at source ↗
Figure 3
Figure 3. Figure 3: Effect of adaptive friction and cubic damping on a two-hundred dimensional anisotropic quadratic: f(x) = 1 2 x T Ax, with eigenvalues ranging between 1 and 104 . For momentum gradient descent (mGD), we choose the theoretically optimal learning rate h = 2/ √ M and friction γ = 2 √ m, where m = 1 and M = 104 are the minimum and maximum eigenvalues of f respectively. Left: Objective function value f(x) across… view at source ↗
Figure 4
Figure 4. Figure 4: Training and test losses for iKFAD, CD, CADAM, Adam, and mSGD across ResNet-18, DistilBERT, and GPT2. All curves are averaged over 10 random seeds. Standard deviations are omitted for readability. All experiments showcase a substantial gap in performance between Adam and mSGD. Both iKFAD and CD are able to overcome this gap despite not using per-parameter learning rates. Hyperparameter optimization for all… view at source ↗
Figure 5
Figure 5. Figure 5: Corresponding training and test accuracies for the experiments in [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Effect of removing linear damping. Hyperparameter sweeps for CD and iKFAD comparing γ = 0 against γ ∈ [10−6 , 10]. Setting γ = 0, does not worsen performance, allowing us to eliminate γ as a hyperparameter. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: γ − δt grid. Test accuracies/losses for iKFAD, CD, LDHD, and mSGD on various experiments as functions of learning rate δt and linear friction γ. Higher color intensity indicates better performance. γ for mSGD is obtained by the correspondence derived in Appendix A. LDHD uses splitting for discretization. The best CD and iKFAD results from the current figure (either γ = 0 or γ > 0) were selected to present … view at source ↗
Figure 8
Figure 8. Figure 8: Left: Friction schedules corresponding to [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Complete Results on ResNet-18. First row has train + test accuracy and losses. Second row is a grid search [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Complete results of losses including CADAM. This includes standard deviations. CADAM (yellow) performs competitively to Adam in some cases. However, more often than not, the performance of CADAM is slightly worse. Despite having the same number of optimizer states and hyperparameters as Adam, with the advantage of per-parameter adaptive learning rates, CADAM still does not perform as well as iKFAD and CD.… view at source ↗
Figure 11
Figure 11. Figure 11: CADAM when γ = 0. We conducted a hyperparameter sweep when γ = 0 and when γ ∈ [1e − 6, 10] for CADAM. This is the same experiment as done in [PITH_FULL_IMAGE:figures/full_fig_p017_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Accelerated sampling using SamAdams variable timesteps and position-adaptive Langevin dynamics

    math.NA 2026-06 unverdicted novelty 7.0

    SA-PAL combines adaptive timesteps and position-dependent friction in Langevin dynamics, reporting 1.5-3x faster mixing on Rosenbrock and Mueller-Brown potentials plus order-of-magnitude efficiency gains on other test...

  2. Critical Damping as a Momentum Schedule: Multi-Seed Validation, a Hybrid Recipe, and an Exhaustive Negative Result on Surgical Layer Selection

    cs.LG 2026-03 unverdicted novelty 6.0

    A momentum schedule from critical damping speeds convergence and yields an optimizer-invariant diagnostic for locating and correcting specific underperforming layers in trained networks.

  3. Critical Damping as a Momentum Schedule: Multi-Seed Validation, a Hybrid Recipe, and an Exhaustive Negative Result on Surgical Layer Selection

    cs.LG 2026-03 conditional novelty 5.5

    Critical-damping momentum mu(t)=1-2*sqrt(alpha(t)) speeds ResNet-18/CIFAR-10 to 90% by ~2.3x; a hybrid switch recovers final accuracy; surgical layer selection by gradient attribution is no better than random.

  4. Why That Robot? A Qualitative Analysis of Justification Strategies for Robot Color Selection Across Occupational Contexts

    cs.RO 2026-03 unverdicted novelty 5.0

    Open-ended justifications for robot color across occupations are dominated by functionalism that systematically tracks racial and occupational stereotypes, while primes shift choices without changing stated reasons.

Reference graph

Works this paper leans on

36 extracted references · 5 linked inside Pith · cited by 3 Pith papers

  1. [1]

    Babister

    A. Babister. Non-linear differential equations having both cubic damping and stiffness. Technical Report 7601, University of Glasgow, 1976. Department of Aeronautics and Fluid Mechanics

    1976

  2. [2]

    Shallue, Zachary Nado, Jaehoon Lee, Chris J

    Dami Choi, Christopher J. Shallue, Zachary Nado, Jaehoon Lee, Chris J. Maddison, and George E. Dahl. On empirical comparisons of optimizers for deep learning, 2020

    2020

  3. [3]

    The Loss Surfaces of Multilayer Networks

    Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Ben Arous, and Yann LeCun. The Loss Surfaces of Multilayer Networks. In Guy Lebanon and S. V . N. Vishwanathan, editors,Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, volume 38 ofProceedings of Machine Learning Research, pages 192–204, San Diego...

    2015

  4. [4]

    A general system of differential equations to model first-order adaptive algorithms.The Journal of Machine Learning Research, 21(1):5072–5113, 2020

    André Belotto Da Silva and Maxime Gazeau. A general system of differential equations to model first-order adaptive algorithms.The Journal of Machine Learning Research, 21(1):5072–5113, 2020

    2020

  5. [5]

    Skeel, and Hartmut Neven

    Nan Ding, Youhan Fang, Ryan Babbush, Changyou Chen, Robert D. Skeel, and Hartmut Neven. Bayesian sampling using stochastic gradient thermostats. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors,Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014

    2014

  6. [6]

    On the connections between optimization algorithms, lyapunov functions, and differential equations: theory and insights.arXiv preprint arXiv:2305.08658, 2023

    Paul Dobson, Jesus Maria Sanz-Serna, and Konstantinos Zygalakis. On the connections between optimization algorithms, lyapunov functions, and differential equations: theory and insights.arXiv preprint arXiv:2305.08658, 2023

    Pith/arXiv arXiv 2023

  7. [7]

    X. Gao, M. Gurbuzbalaban, and L. Zhu. Global convergence of stochastic gradient hamiltonian monte carlo for non-convex stochastic optimization: Non-asymptotic performance bounds and momentum-based acceleration, 2018

    2018

  8. [8]

    PhD thesis, The University of Edinburgh, 2024

    Aikaterini Karoni.Higher-order damping mechanisms with applications in optimisation and machine learning. PhD thesis, The University of Edinburgh, 2024

    2024

  9. [9]

    Friction-adaptive descent: A family of dynamics- based optimization methods.Journal of Computational Dynamics, 2023

    Aikaterini Karoni, Benedict Leimkuhler, and Gabriel Stoltz. Friction-adaptive descent: A family of dynamics- based optimization methods.Journal of Computational Dynamics, 2023. 11 Adaptive Momentum and Nonlinear Damping for Neural Network TrainingPREPRINT

    2023

  10. [10]

    nanogpt: The simplest, fastest repository for training/finetuning medium-sized gpts

    Andrej Karpathy. nanogpt: The simplest, fastest repository for training/finetuning medium-sized gpts. https: //github.com/karpathy/nanoGPT, 2022. Accessed: 2026-01-22

    2022

  11. [11]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors,3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015

    2015

  12. [12]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    2009

  13. [13]

    Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be.arXiv preprint arXiv:2304.13960, 2023

    Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be.arXiv preprint arXiv:2304.13960, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    Cambridge Monographs on Applied and Computational Mathematics

    Benedict Leimkuhler and Sebastian Reich.Simulating Hamiltonian Dynamics. Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, 2005

    2005

  15. [15]

    Visualizing the loss landscape of neural nets.Advances in neural information processing systems, 31, 2018

    Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets.Advances in neural information processing systems, 31, 2018

    2018

  16. [16]

    Maddison, D

    C.J. Maddison, D. Paulin, Y .W. Teh, B. O’Donoghue, and A. Doucet. Hamiltonian descent methods.arXiv preprint, 1809.05042, 2018

    Pith/arXiv arXiv 2018

  17. [17]

    Small batch size training for language models: When vanilla SGD works, and why gradient accumulation is wasteful

    Martin Marek, Sanae Lotfi, Aditya Somasundaram, Andrew Gordon Wilson, and Micah Goldblum. Small batch size training for language models: When vanilla SGD works, and why gradient accumulation is wasteful. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  18. [18]

    Mattingly, A.M

    J.C. Mattingly, A.M. Stuart, and D.J. Higham. Ergodicity for SDEs and approximations: locally Lipschitz vector fields and degenerate noise.Stochastic Processes and their Applications, 101(2):185–232, 2002

    2002

  19. [19]

    A systematic approach to Lyapunov analyses of continuous-time models in convex optimization.SIAM Journal on Optimization, 33(3):1558–1586, 2023

    Céline Moucer, Adrien Taylor, and Francis Bach. A systematic approach to Lyapunov analyses of continuous-time models in convex optimization.SIAM Journal on Optimization, 33(3):1558–1586, 2023

    2023

  20. [20]

    A method of solving a convex programming problem with convergence rate o( 1 k2 ).Doklady Akademii Nauk SSSR, 269(3):543, 1983

    Yurii Nesterov. A method of solving a convex programming problem with convergence rate o( 1 k2 ).Doklady Akademii Nauk SSSR, 269(3):543, 1983

    1983

  21. [21]

    Springer Science & Business Media, 2013

    Yurii Nesterov.Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2013

    2013

  22. [22]

    Adaptive restart for accelerated gradient schemes.Found

    Brendan O’Donoghue and Emmanuel Candès. Adaptive restart for accelerated gradient schemes.Found. Comput. Math., 15(3):715–732, 2015

    2015

  23. [23]

    Panananda, N.S

    N. Panananda, N.S. Ferguson, and T.P. Waters. The effect of cubic damping in an automotive vehicle suspension model. InComputational Modelling and Analysis of Vehicle Body Noise and Vibration, 2012

    2012

  24. [24]

    Z. K. Peng, Z. Q. Lang, X. J. Jing, S. A. Billings, G. R. Tomlinson, and L. Z. Guo. The transmissibility of vibration isolators with a nonlinear antisymmetric damping characteristic.Journal of Vibration and Acoustics, 132(1):014501, 01 2010

    2010

  25. [25]

    B. T. Poljak. Some methods of speeding up the convergence of iterative methods.Ž. Vyˇ cisl. Mat i Mat. Fiz., 4:791–803, 1964

    1964

  26. [26]

    Covariance-controlled adaptive langevin thermostat for large-scale bayesian sampling

    Xiaocheng Shang, Zhanxing Zhu, Benedict Leimkuhler, and Amos J Storkey. Covariance-controlled adaptive langevin thermostat for large-scale bayesian sampling. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015

    2015

  27. [27]

    Randomised splitting methods and stochastic gradient descent

    Luke Shaw and Peter A Whalley. Randomised splitting methods and stochastic gradient descent. arXiv preprint arXiv:2504.04274, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    Simsekli, L

    U. Simsekli, L. Zhu, Y .-W. Teh, and M. Gurbuzbalaban. Fractional underdamped langevin dynamics: Retargeting sgd with momentum under heavy-tailed gradient noise. InInternational conference on machine learning, pages 8970–8980. PMLR, 2020

    2020

  29. [29]

    Weijie Su, Stephen Boyd, and Emmanuel J. Candès. A differential equation for modeling Nesterov’s accelerated gradient method: theory and insights.J. Mach. Learn. Res., 17:Paper No. 153, 43, 2016

    2016

  30. [30]

    On the importance of initialization and momentum in deep learning

    Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. InInternational conference on machine learning, pages 1139–1147. PMLR, 2013

    2013

  31. [31]

    Understanding why adam outperforms sgd: Gradient heterogeneity in transformers.arXiv preprint arXiv:2502.00213, 2025

    Akiyoshi Tomihari and Issei Sato. Understanding why adam outperforms sgd: Gradient heterogeneity in transformers.arXiv preprint arXiv:2502.00213, 2025. 12 Adaptive Momentum and Nonlinear Damping for Neural Network TrainingPREPRINT

    arXiv 2025

  32. [32]

    Glue: A multi-task benchmark and analysis platform for natural language understanding.arXiv preprint arXiv:1804.07461, 2018

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding.arXiv preprint arXiv:1804.07461, 2018

    Pith/arXiv arXiv 2018

  33. [33]

    Tinyvit: Fast pretraining distillation for small vision transformers

    Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. InEuropean conference on computer vision (ECCV), 2022

    2022

  34. [34]

    Why are adaptive methods good for attention models?Advances in Neural Information Processing Systems, 33:15383–15393, 2020

    Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models?Advances in Neural Information Processing Systems, 33:15383–15393, 2020

    2020

  35. [35]

    Why transformers need adam: A hessian perspective

    Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, and Zhi-Quan Luo. Why transformers need adam: A hessian perspective. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances in Neural Information Processing Systems, volume 37, pages 131786–131823. Curran Associates, Inc., 2024

    2024

  36. [36]

    artificial

    YP. Zhu and Z.Q. Lang. Beneficial effects of antisymmetric nonlinear damping with application to energy harvesting and vibration isolation under general inputs.Nonlinear Dynamics, 108:2917–2933, 2022. A Equivalence between mSGD and LDHD In this section we will show how the mSGD discrete equations (1a), (1b) can be obtained as an Euler discretisation of li...

    arXiv 2022