Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Torque-Aware Momentum

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Torque-Aware Momentum damps misaligned gradients and improves generalization across vision, language, and online learning.

desk verdict Useful new momentum variant with small but real gains, but the convergence-inheritance argument in §3 does not hold as stated. read the letter →

arxiv 2412.18790 v1 pith:GVGAP36Y submitted 2024-12-25 cs.LG cs.AI

Pranshu Malviya , Goncalo Mordido , Aristide Baratin , Reza Babanezhad Harikandeh , Gintare Karolina Dziugaite , Razvan Pascanu , Sarath Chandar This is my paper · ORCID
classification cs.LGcs.AI
keywords Torque-AwareMomentumoptimizergradientalignmentcosinesimilaritydampingexplorationdistributionshiftrobustnesswarm-upstrategyAdaTAM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a single scalar can fix a known weakness of momentum in deep learning: when a new gradient badly misaligns with the accumulated momentum, standard momentum lets that gradient yank the update into an oscillating, sharp region. Torque-Aware Momentum (TAM) instead scales each new gradient by a factor between zero and one, computed from a smoothed measure of how well the gradient agrees with the current momentum direction, so misaligned gradients are damped while aligned ones pass through. The paper reports that this damping improves or preserves accuracy on image classification and large-language-model fine-tuning, and that its biggest wins come under distribution shifts such as label flipping in online learning. TAM adds no extra memory and works as a standalone optimizer, as an adaptive variant of Adam, and as a short warm-up phase before switching to another optimizer. The paper also derives a learning-rate transfer rule, double a tuned SGDM learning rate, and argues that near an optimum TAM inherits SGDM's convergence behavior.

What carries the argument

The load-bearing object is the smoothed alignment statistic $\hat{s}_t = \gamma \hat{s}_{t-1} + (1-\gamma) S_t$, where $S_t = (m_{t-1} \cdot g_t)/(\|m_{t-1}\| \|g_t\|)$ is the cosine similarity between previous momentum and current gradient, wrapped into the damping factor $d_t = (1+\hat{s}_t)/2$. That factor continuously scales each gradient's contribution from full (aligned) to near zero (opposing), without any extra per-parameter state. This scalar also produces the effective-learning-rate identity $\eta_{\mathrm{TAM}} = 2/(1+s^*) \eta_{\mathrm{SGDM}}$ once $\hat{s}_t$ is assumed to settle at constant $s^* \approx 0$, which is the step that lets the paper transfer tuned SGDM learning rates and claim inheritance of SGDM convergence guarantees.

What would settle it

On a standard training run (for example ResNet18 on CIFAR10), record the alignment statistic $\hat{s}_t$ at every step and check whether it converges to a constant near $s^* = 0$ over the whole run, including the early phase where gradients are largest. If $\hat{s}_t$ instead tracks the data or oscillates, or if the learning rate that actually performs best for TAM departs systematically from the $2\eta_{\mathrm{SGDM}}$ rule once $s^*$ is measured, the paper's rate-transfer and convergence-inheritance claims are contradicted.

Watch

Extended reading notes

Core claim

The central claim is that the harmful effect of 'torqued' gradients, large updates that oppose or sharply cross the current momentum, can be damped without weakening momentum's benefits. TAM changes the momentum recurrence from $m_t = \beta m_{t-1} + g_t$ to $m_t = \beta m_{t-1} + (\epsilon + d_t) g_t$, where $d_t = (1+\hat{s}_t)/2$ and $\hat{s}_t$ is an exponentially smoothed cosine similarity between the previous momentum and the current gradient. The factor $d_t$ lies in $[0,1]$, so aligned gradients receive nearly full weight and opposing gradients nearly none. The paper treats this as anisotropic friction, arguing it stabilizes the update direction, delays the oscillation that SGDM shows at larger learning rates, and lets the optimizer explore wider basins early in training. The paper then argues that when $\hat{s}_t$ stabilizes to a constant $s^* \approx 0$ near the optimum, TAM's effective learning rate is $2/(1+s^*)$ times SGDM's, which justifies setting $\eta_{\mathrm{TAM}} = 2\eta_{\mathrm{SGDM}}$ and inheriting SGDM's convergence guarantees. Empirical sections claim parity or gains over SGDM and Adam across CIFAR, ImageNet, MTEB fine-tuning of BERT-family models, online label-flipping tasks, and GNN link-prediction warm-up.

Load-bearing premise

The argument rests on a running average of how well each new gradient agrees with the current momentum direction settling down to a constant near zero as training proceeds; if that average keeps moving or settles somewhere far from zero, the learning-rate doubling rule and the claim of matching SGDM's convergence behavior no longer follow.

Editorial extensions

If this is right

  • TAM is a drop-in replacement for momentum: it keeps the memory footprint of SGDM, fixes its two new hyperparameters ($\gamma = 0.9$, $\epsilon = 10^{-8}$), and can be paired with either SGD or Adam.
  • A tuned SGDM learning rate transfers to TAM by doubling it near an optimum, and TAM retains SGDM's convergence guarantees in that regime.
  • Across the reported benchmarks, CIFAR10/100, ImageNet, MobileNet, ViT fine-tuning, and MTEB fine-tuning of BERT, DeBERTa, and RoBERTa, TAM and AdaTAM match or beat SGDM, Adam, and AngularGrad on accuracy.
  • In online learning with label flipping, TAM remains more stable and accurate than SGDM when shifts are drastic, 80% and 100% flips.
  • Using TAM for the first phase of training and then switching to SGDM or Adam reaches lower validation error and lower loss barriers than starting with the base optimizer, in the paper's warm-up experiments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that if the benefit really comes from damping misaligned gradients, TAM's advantage should grow as the frequency or magnitude of gradient misalignment increases; a direct test would compare SGDM and TAM under controlled noise levels or sharp-region loss surfaces.
  • We infer that the stabilizing value of $\hat{s}_t \to s^* \approx 0$ is only demonstrated near convergence on stationary image tasks, and in non-stationary settings, where the loss landscape shifts between tasks, the same stabilization may not hold, so the paper's convergence argument may not transfer.
  • We infer that the damping mechanism is a scalar gate on the gradient-momentum angle and could be combined with other adaptive optimizers beyond Adam, such as AdamW variants with different second-moment or normalization schemes, which the paper does not test.
  • We infer that the warm-up results suggest a cheaper way to use TAM: run it for the first tens of epochs only, then switch to the default optimizer, converting an exploration benefit into a final-model gain without paying TAM's behavior for the whole run.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Torque-Aware Momentum (TAM), which modifies the standard SGDM update by replacing the fixed gradient coefficient in the momentum recursion with a state-dependent damping factor built from a smoothed cosine similarity between the current momentum and the new gradient. Concretely, with m_t = β m_{t-1} + (ε + (1+ŝ_t)/2) g_t and θ_{t+1} = θ_t − η m_t, TAM down-weights 'torqued' gradients that are misaligned with the momentum direction. The authors also introduce AdaTAM by inserting the same damping into the momentum accumulator of Adam. The paper claims that TAM stabilizes updates, improves exploration and generalization, transfers learning rates from SGDM via an effective-learning-rate heuristic with η_TAM ≈ 2 η_SGDM, and 'inherits the well-established convergence guarantees of SGDM' near the optimum. The empirical section compares TAM/AdaTAM with SGD, SGDM, Adam, and AngularGrad on CIFAR10/100, ImageNet, MobileNet/ViT, MTEB-based LLM fine-tuning, online label-flipping benchmarks, and warm-up ablations on CIFAR and a GNN link-prediction task.

Significance. If the empirical findings hold, TAM is an attractive drop-in optimizer: it has the same memory footprint as SGDM, adds only one cosine-similarity computation and two fixed hyperparameters (γ=0.9, ε=1e−8), and shows consistent gains in several settings, most notably online learning with large label-flip shifts and LLM fine-tuning with DeBERTa/RoBERTa-base. The paper is also reasonably transparent: it reports five-seed averages, includes ablations on γ and on the momentum recursion, and states where results are mixed (e.g., AdaTAM on CIFAR10/ResNet34, RoBERTa-large on MTEB). The main weakness is that the paper's theoretical safety net—the convergence-inheritance claim—is currently unsupported, and the learning-rate transfer heuristic rests on an empirical stabilization assumption verified only on two CIFAR datasets. These issues are fixable by rewriting the theory section as a heuristic and adding evidence for the ŝ_t-stabilization assumption in the other experimental regimes.

major comments (3)
  1. [Section 3, Eqs. (4)–(6) and following paragraph] The claim that TAM 'inherits the well-established convergence guarantees of SGDM' is not established for the actual update rule. If ŝ_t were exactly constant, say ŝ_t = s*, then with m_0 = 0 the TAM recursion is a rescaling of the SGDM recursion: m_t = a m'_t with a = (1+s*)/2, so TAM is exactly SGDM with learning rate η a. Thus the specific concern that 'TAM is not SGDM with any single learning rate' is answerable in the exactly-constant case. However, in the real algorithm d_t = (1+ŝ_t)/2 is a function of m_{t-1} and g_t and is therefore correlated with the stochastic gradient noise. Boundedness of d_t alone does not place the recursion in the class covered by the cited SGDM analyses (Yan et al. 2018; Liu et al. 2020), which rely on a fixed gain. The sentence after Eq. (6) should either be replaced by a precise statement about the constant-gain idealization, or supported by a convergence proof for the state-dependent gain, or removed in favor of an explicitly empirical claim.
  2. [Appendix A.2.1, Fig. 8; Tables 4–5; Figs. 3, 5, 7] The learning-rate transfer η_TAM = 2 η_SGDM depends on the observation that ŝ_t stabilizes to s* ≈ 0, but this observation is shown only for ResNet18 on CIFAR10 and CIFAR100. The same rule is then applied to ImageNet, MTEB fine-tuning, online label-flipping, and GNN warm-up experiments, where no ŝ_t trajectories are reported. Moreover, the paper's exploration argument concerns early training, exactly the regime where Fig. 8 shows ŝ_t first rising and then dipping negative before flattening near zero. The authors should either report ŝ_t dynamics for representative non-CIFAR runs, or explicitly frame the 2× rule as a heuristic whose domain of validity is currently unknown.
  3. [Section 4.1, Table 1; Section 4.2, Figs. 3–4, Tables 15–16] The abstract's 'improves generalization performance across various tasks' is stronger than the data shown. In Table 1, AdaTAM underperforms Adam on CIFAR10/ResNet34 (93.3 vs. 93.6), and in the MTEB experiments RoBERTa-large is consistently worse with AdaTAMW than with AdamW on a substantial fraction of datasets, including AmazonCounterfactualClassification, EmotionClassification, and ImdbClassification at 3 epochs in Table 16. The paper already acknowledges 'mixed' results in places, so the main text and abstract should be reworded to say that TAM/AdaTAM 'often matches or improves upon' baselines, with the consistent gains appearing in online learning and in several LLM fine-tuning configurations, rather than claiming universal improvement.
minor comments (6)
  1. [Section 4.2, paragraph 1] There is a typo: 'varying numbes of epochs' should be 'varying numbers of epochs'.
  2. [Section 5, Conclusion] 'Additionnaly' should be 'Additionally', and 'capabilites' should be 'capabilities'.
  3. [Section 2, Related Work] 'more criticial for stability' should be 'more critical for stability'.
  4. [Figure 3 caption] The phrase 'The the y-axis labels' contains a duplicated article and should be corrected.
  5. [Appendix A.2.7, Figure 10 vs. Figure 4] The definition of 'similar performance' changes between Figure 4 (threshold 0.2%) and Figure 10 (threshold 1%); the main text should state which threshold is used for the summary percentages.
  6. [Section 4.4, paragraph 1] When switching from TAM to SGDM 'while keeping the effective learning rate and optimizer state same', the paper should specify how the TAM momentum state is mapped to SGDM; since the damping factor is not constant at the switch time, this mapping is not automatic.

Circularity Check

1 steps flagged · score 2.0 of 10

Mild self-referential LR transfer: the 2x learning-rate rule and the SGDM-convergence inheritance claim both reduce to the authors' own observation that s*≈0; empirical accuracy claims are otherwise self-contained.

  1. fitted input called prediction [Section 3 'Learning Rate Transfer', Eq. 5-6 and the paragraph after Eq. 6; Appendix A.2.1 / Fig. 8.]
    "Assuming that, as t increases, the cosine similarity ˆst stabilizes to a constant value s∗, TAM’s effective learning rate becomes: ηeff TAM ≈ (1+s∗)/(2(1−β)) η (5) ... In practice, we observed that s∗ ≈ 0 as t increases (see Appendix A.2.1 for empirical evidence). In our experiments, we set βTAM = βSGDM, and found that η∗TAM = 2η∗SGDM consistently yields optimal performance. This equivalence means that in the neighborhood of optima, where ˆst has stabilized, TAM inherits the well-established convergence guarantees of SGDM."

    The claimed LR transfer is not an independent prediction: Eq. 6 collapses to η*_TAM = 2η*_SGDM only after substituting s*≈0, and s*≈0 is an empirical value read off the authors' own CIFAR-10/100 training runs (Fig. 8). The same self-measured stabilization is then used as the basis for the statement that TAM 'inherits the well-established convergence guarantees of SGDM.' Thus both the transfer rule and the convergence-inheritance claim are downstream of the authors' own observation rather than derived from TAM's update rule alone. The accuracy comparisons are not circular, since they come from grid search against held-out baselines; this is a mild self-referential element, not a forced fit.

full rationale

The central empirical evaluation is self-contained: TAM and AdaTAM are compared with SGDM, Adam, and AngularGrad on held-out test sets under learning-rate grid search, and the reported gains are not constructed from the method's own parameters. The LR-transfer section contains the only self-referential step: the 2x learning-rate rule and the SGDM-convergence inheritance statement both rely on s*≈0, which is measured on the authors' own runs rather than established independently. The convergence claim is additionally weak because d_t is state-dependent and correlated with the gradient noise, so fixed-gain SGDM guarantees (Yan et al. 2018; Liu et al. 2020) do not automatically apply; however, that is an evidence gap, not a circular reduction. Coauthor citations (Frankle et al. 2020; Lyle et al. 2024a,b) are used only for experimental setup and are not load-bearing. Overall score 2.

Assumptions & free parameters 3 free parameters · 2 assumptions · 0 invented entities

The method introduces two hyperparameters (γ, ε) plus an observed steady-state cosine s* that drives the learning-rate transfer. It also relies on two unproved premises: that ŝ_t converges to s*, and that the bounded damping factor lets TAM inherit SGDM convergence. No new entities are postulated.

free parameters (3)
  • gamma = 0.9 (default)
    Smoothing decay for the cosine similarity in Eq. 3; the paper fixes it by default and reports an ablation showing low sensitivity.
  • epsilon = 1e-8
    Small constant in Eq. 4 to keep a floor on gradient influence.
  • s_star = ≈0 (observed)
    Steady-state cosine similarity value used in the LR transfer formula (Eq. 6); measured in the authors' own CIFAR10/100 runs in Appendix A.2.1, not predicted by theory.
assumptions (2)
  • domain assumption The smoothed cosine similarity ŝ_t converges to a constant s* over training.
    Invoked in Section 3 'Learning Rate Transfer' to derive Eq. 5 and the LR transfer, and to justify convergence inheritance; only empirical evidence is Figure 8 (CIFAR10/100).
  • ad hoc to paper TAM with bounded damping factor (1+ŝ_t)/2 inherits SGDM convergence guarantees near the optimum.
    Claimed in Section 3 after Eq. 6 without a proof; the time-varying damping changes the momentum dynamics, so the standard SGDM analysis does not directly apply.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Torque-Aware Momentum." pith.science (2026). https://pith.science/paper/GVGAP36Y

@misc{pith2026241218790,
  author       = {Pith},
  title        = {Pith review of: Torque-Aware Momentum},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GVGAP36Y}},
  note         = {Machine review of arXiv:2412.18790}
}
read the original abstract

Efficiently exploring complex loss landscapes is key to the performance of deep neural networks. While momentum-based optimizers are widely used in state-of-the-art setups, classical momentum can still struggle with large, misaligned gradients, leading to oscillations. To address this, we propose Torque-Aware Momentum (TAM), which introduces a damping factor based on the angle between the new gradients and previous momentum, stabilizing the update direction during training. Empirical results show that TAM, which can be combined with both SGD and Adam, enhances exploration, handles distribution shifts more effectively, and improves generalization performance across various tasks, including image classification and large language model fine-tuning, when compared to classical momentum-based optimizers.

Figures

Figures reproduced from arXiv: 2412.18790 by the authors.

Figure 1
Figure 1. Comparing momentum up￾dates obtained using SGDM and TAM for a given SGD trajectory. While TAM results in more stable directions pointing to a lower loss basin, SGDM has higher magnitude updates susceptible to mis￾aligned gradients. While SGD with momentum (SGDM) has shown remark￾able success in various scenarios, particularly in computer vision (Sutskever et al., 2013), it remains vulnerable to the adverse effects o… view at source ↗
Figure 2
Figure 2. TAM controls update magnitude (red) based on the alignment between momentum and new gradients. The angle (α1, α2) between previous momentum (green) and new gradients (white) determines the magnitude of the update (red). When g1 aligns well with m0, the resulting momentum m1 has a higher magnitude. In con￾trast, when the misalignment between g2 and m1 results in a smaller magnitude m2. To increase robustness against … view at source ↗
Figure 3
Figure 3. Percentage improvement in the average scores of AdaTAMW compared to AdamW across [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Percentage of times when AdaTAMW performs better (or similar/better) than AdamW on various LLMs across 56 MTEB datasets. Green indicates that AdaTAMW achieves similar or bet￾ter performance, while red indicates worse perfor￾mance. Except for BERT models with 10 epochs …
Figure 5
Figure 5. Figure 5: Comparing online accuracy of TAM with SGDM and SGD on label flipping benchmark for [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: (i) Comparing the performance of TAM and SGDM while training ResNet18 on CIFAR10 [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Evaluating warmup with TAM on Link prediction task using GNNs for different switching [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Evolution of sˆt during training ResNet18 on CIFAR10 and CIFAR100 datasets. We observe that after starting from positive value, sˆt drops to negative, fluctuates and eventually saturates near s ∗ = 0 in both cases. A.2.2 IMAGE CLASSIFICATION We evaluate TAM on the foll…
Figure 9
Figure 9. Figure 9: 1. On SVHN (Netzer et al., 2011), we observe a pattern similar to CIFAR10 in [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 9
Figure 9. Figure 9: Comparing gradient norm observed during training ResNet18 similar to [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Percentage of times when AdaTAMW performs better (or similar/better) than AdamW on [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A meta-pipeline plus LMO four-axis view yields a dual taxonomy of 108 optimizers, and a multi-objective LLM/vision benchmark shows no single family dominates the quality–cost–memory frontier.

Reference graph

Works this paper leans on

53 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Momentum improves normalized sgd

    Ashok Cutkosky and Harsh Mehta. Momentum improves normalized sgd. In International conference on machine learning, pp.\ 2260--2268. PMLR, 2020 a

    work page 2020

  3. [3]

    Momentum improves normalized SGD

    Ashok Cutkosky and Harsh Mehta. Momentum improves normalized SGD . In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp.\ 2260--2268. PMLR, 13--18 Jul 2020 b . URL https://proceedings.mlr.press/v119/cutkosky20b.html

    work page 2020

  4. [4]

    ImageNet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, 2009

    work page 2009

  5. [5]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. 2018

    work page 2018

  6. [6]

    Sutton, , and A

    Shibhansh Dohare, Richard S. Sutton, , and A. Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.06325, 2021

    arXiv 2021

  7. [7]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021

    2021

  8. [8]

    Addressing loss of plasticity and catastrophic forgetting in continual learning

    Mohamed Elsayed and A Rupam Mahmood. Addressing loss of plasticity and catastrophic forgetting in continual learning. arXiv preprint arXiv:2404.00781, 2024

    arXiv 2024

Show all 53 references
  1. [9]

    Sharpness-aware minimization for efficiently improving generalization

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021

    2021

  2. [10]

    Linear mode connectivity and the lottery ticket hypothesis

    Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp.\ 3259--3269. PMLR, 2020

    2020

  3. [11]

    Identification of mass, damping, and stiffness matrices of mechanical systems

    Claus-Peter Fritzen. Identification of mass, damping, and stiffness matrices of mechanical systems. 1986

    1986

  4. [12]

    When and why momentum accelerates sgd: An empirical study

    Jingwen Fu, Bohan Wang, Huishuai Zhang, Zhizheng Zhang, Wei Chen, and Nanning Zheng. When and why momentum accelerates sgd: An empirical study. arXiv preprint arXiv:2306.09000, 2023

    2023 arXiv

  5. [13]

    A loss curvature perspective on training instability in deep learning

    Justin Gilmer, Behrooz Ghorbani, Ankush Garg, Sneha Kudugunta, Behnam Neyshabur, David Cardoze, George Dahl, Zachary Nado, and Orhan Firat. A loss curvature perspective on training instability in deep learning. arXiv preprint arXiv:2110.04369, 2021

    arXiv

  6. [14]

    The movielens datasets: History and context

    F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5 0 (4): 0 1--19, 2015

    2015

  7. [15]

    Deberta: Decoding-enhanced bert with disentangled attention

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=XPZIaotutsD

    2021

  8. [16]

    Mobilenets: Efficient convolutional neural networks for mobile vision applications

    Andrew G Howard. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017

    2017 arXiv

  9. [17]

    Accelerated gradient descent escapes saddle points faster than gradient descent

    Chi Jin, Praneeth Netrapalli, and Michael I Jordan. Accelerated gradient descent escapes saddle points faster than gradient descent. In Conference On Learning Theory, pp.\ 1042--1085. PMLR, 2018

    2018

  10. [18]

    When do flat minima optimizers work? Advances in Neural Information Processing Systems, 35: 0 16577--16595, 2022

    Jean Kaddour, Linqing Liu, Ricardo Silva, and Matt J Kusner. When do flat minima optimizers work? Advances in Neural Information Processing Systems, 35: 0 16577--16595, 2022

    2022

  11. [19]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015

    2015

  12. [20]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009

    2009

  13. [21]

    Maintaining plasticity in continual learning via regenerative regularization

    Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity in continual learning via regenerative regularization. arXiv preprint arXiv:2308.11958, 2023

    2023 arXiv

  14. [22]

    Learning continually by spectral regularization

    Alex Lewandowski, Saurabh Kumar, Dale Schuurmans, Andr \'a s Gy \"o rgy, and Marlos C Machado. Learning continually by spectral regularization. arXiv preprint arXiv:2406.06811, 2024

    2024 arXiv

  15. [23]

    Learning without forgetting

    Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40 0 (12): 0 2935--2947, 2017

    2017

  16. [24]

    The clear benchmark: Continual learning on real-world imagery

    Zhiqiu Lin, Jia Shi, Deepak Pathak, and Deva Ramanan. The clear benchmark: Continual learning on real-world imagery. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021

    2021

  17. [25]

    An improved analysis of stochastic gradient descent with momentum

    Yanli Liu, Yuan Gao, and Wotao Yin. An improved analysis of stochastic gradient descent with momentum. Advances in Neural Information Processing Systems, 33: 0 18261--18271, 2020

    2020

  18. [26]

    Aggregated momentum: Stability through passive damping

    James Lucas, Shengyang Sun, Richard Zemel, and Roger Grosse. Aggregated momentum: Stability through passive damping. arXiv preprint arXiv:1804.00325, 2018

    2018 arXiv

  19. [27]

    Normalization and effective learning rates in reinforcement learning

    Clare Lyle, Zeyu Zheng, Khimya Khetarpal, James Martens, Hado van Hasselt, Razvan Pascanu, and Will Dabney. Normalization and effective learning rates in reinforcement learning. arXiv preprint arXiv:2407.01800, 2024 a

    2024 arXiv

  20. [28]

    Disentangling the causes of plasticity loss in neural networks

    Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado van Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the causes of plasticity loss in neural networks. arXiv preprint arXiv:2402.18762, 2024 b

    2024 arXiv

  21. [29]

    The wikitext long term dependency language modeling dataset

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. The wikitext long term dependency language modeling dataset. Salesforce Metamind, 9, 2016

    2016

  22. [30]

    Mteb: Massive text embedding benchmark

    Niklas Muennighoff, Nouamane Tazi, Lo \" c Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316, 2022

    2022 arXiv

  23. [31]

    A method for solving the convex programming problem with convergence rate O (1/k^2)

    Yurii Nesterov. A method for solving the convex programming problem with convergence rate O (1/k^2) . Proceedings of the USSR Academy of Sciences, 269: 0 543--547, 1983. URL https://api.semanticscholar.org/CorpusID:145918791

    1983

  24. [32]

    Reading digits in natural images with unsupervised feature learning, 2011

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning, 2011

    2011

  25. [33]

    Catastrophic overfitting is a bug but also a feature

    Guillermo Ortiz-Jim \'e nez, Pau de Jorge, Amartya Sanyal, Adel Bibi, Puneet K Dokania, Pascal Frossard, Gregory Rog \'e z, and Philip HS Torr. Catastrophic overfitting is a bug but also a feature. arXiv preprint arXiv:2206.08242, 2022

    2022 arXiv

  26. [34]

    Some methods of speeding up the convergence of iteration methods

    Boris T Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 1964

    1964

  27. [35]

    On the momentum term in gradient descent learning algorithms

    Ning Qian. On the momentum term in gradient descent learning algorithms. Neural networks, 12 0 (1): 0 145--151, 1999

    1999

  28. [36]

    Angulargrad: A new optimization technique for angular convergence of convolutional neural networks

    Swalpa Kumar Roy, Mercedes Eugenia Paoletti, Juan Mario Haut, Shiv Ram Dubey, Purbayan Kar, Antonio Plaza, and Bidyut B Chaudhuri. Angulargrad: A new optimization technique for angular convergence of convolutional neural networks. arXiv preprint arXiv:2105.10190, 2021

    2021 arXiv

  29. [37]

    The dormant neuron phenomenon in deep reinforcement learning

    Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning, 2023

    2023

  30. [38]

    On the importance of initialization and momentum in deep learning

    Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pp.\ 1139--1147. PMLR, 2013

    2013

  31. [39]

    Inversion of friction anisotropy in a bio-inspired asymmetrically structured surface

    Halvor T Tramsen, Stanislav N Gorb, Hao Zhang, Poramate Manoonpong, Zhendong Dai, and Lars Heepe. Inversion of friction anisotropy in a bio-inspired asymmetrically structured surface. Journal of the Royal Society Interface, 15 0 (138): 0 20170629, 2018

    2018

  32. [40]

    Better sgd using second-order momentum

    Hoang Tran and Ashok Cutkosky. Better sgd using second-order momentum. Advances in Neural Information Processing Systems, 35: 0 3530--3541, 2022

    2022

  33. [41]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  34. [42]

    Positive-negative momentum: Manipulating stochastic gradient noise to improve generalization

    Zeke Xie, Li Yuan, Zhanxing Zhu, and Masashi Sugiyama. Positive-negative momentum: Manipulating stochastic gradient noise to improve generalization. In International Conference on Machine Learning, pp.\ 11448--11458. PMLR, 2021

    2021

  35. [43]

    A walk with sgd: How sgd explores regions of deep network loss? 2018

    Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio. A walk with sgd: How sgd explores regions of deep network loss? 2018

    2018

  36. [44]

    A unified analysis of stochastic momentum methods for deep learning

    Yan Yan, Tianbao Yang, Zhe Li, Qihang Lin, and Yi Yang. A unified analysis of stochastic momentum methods for deep learning. arXiv preprint arXiv:1808.10396, 2018

    2018 arXiv

  37. [45]

    Adahessian: An adaptive second order optimizer for machine learning

    Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer, and Michael Mahoney. Adahessian: An adaptive second order optimizer for machine learning. In proceedings of the AAAI conference on artificial intelligence, volume 35, pp.\ 10665--10673, 2021

    2021

  38. [46]

    Why gradient clipping accelerates training: A theoretical justification for adaptivity

    Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. arXiv preprint arXiv:1905.11881, 2019

    1905 arXiv

  39. [47]

    Link prediction based on graph neural networks

    Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. Advances in neural information processing systems, 31, 2018

    2018

  40. [48]

    Do you remember? overcoming catastrophic forgetting for fake audio detection

    Xiaohui Zhang, Jiangyan Yi, Jianhua Tao, Chenglong Wang, and Chu Yuan Zhang. Do you remember? overcoming catastrophic forgetting for fake audio detection. In International Conference on Machine Learning, pp.\ 41819--41831. PMLR, 2023

    2023

  41. [49]

    A robustly optimized BERT pre-training approach with post-training

    Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. A robustly optimized BERT pre-training approach with post-training. In Sheng Li, Maosong Sun, Yang Liu, Hua Wu, Kang Liu, Wanxiang Che, Shizhu He, and Gaoqi Rao (eds.), Proceedings of the 20th Chinese National Conference on Computat...

    2021

  42. [50]

    Laprop: Separating momentum and adaptivity in adam

    Liu Ziyin, Zhikang T Wang, and Masahito Ueda. Laprop: Separating momentum and adaptivity in adam. arXiv preprint arXiv:2002.04839, 2020

    2002 arXiv

  43. [51]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  44. [52]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  45. [53]

    njd G ( 1 O.fT GS

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.