{"id":"026183a4-7e4b-46f5-ad4a-1ee067c74333","arxiv_id":"2412.18790","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Torque-Aware Momentum damps momentum updates by the alignment between new gradients and previous momentum, giving small gains on some benchmarks but mixed results on large model fine-tuning.","lead":"The paper proposes a new optimizer twist, Torque-Aware Momentum, which scales the gradient contribution to a momentum buffer by the cosine similarity between the gradient and the current momentum. It reports small accuracy gains on several image classification tasks, mixed results when fine-tuning large language models, and better robustness on small label-flipping online learning benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Convergence inheritance claim in §3 is unsupported: the bounded damping factor (1+ŝ_t)/2 does not make TAM a reparameterized SGDM because d_t is state-dependent and the update is not SGDM with any single learning rate.","rationale":"The reader's conditional verdict is appropriate. The paper has real empirical breadth (CIFAR/ImageNet, MTEB, online learning, warm-up) and the TAM update is simple, memory-neutral, and plausibly useful; the learning-rate transfer heuristic is a practical contribution if s* ≈ 0 is verified. However, the convergence claim is not a corollary of boundedness: d_t is a stochastic, state-dependent coefficient correlated with gradients and momentum, so existing SGDM proofs do not apply. The specific non-equivalence point (the momentum coefficient differs from SGDM at any learning rate) is checkable analytically on a quadratic and would settle the issue. The empirical evidence for s* ≈ 0 is limited to two CIFAR settings; extending Fig. 8 to ImageNet and MTEB would address the transfer concern. A minor but real implementation issue: with m0 = 0, Eq. 2 is 0/0 at t = 1 and needs a patch. None of this overturns the empirical claims, so the reader's CONDITIONAL verdict stands without change.","tokens_in":26581,"tokens_out":6726,"duration_ms":64734,"concrete_test":"Compute the mean-squared-error contraction rate for TAM vs SGDM on a one-dimensional quadratic with stochastic gradient noise, e.g. f(θ)=aθ²/2 with g_t = aθ_t + ξ_t. Fix β and set η_TAM = 2η_SGDM. Derive or simulate E[‖θ_t - θ*‖²] over 10^4 steps. If TAM's contraction rate differs from SGDM's beyond a benign constant-factor learning-rate change, the Section 3 convergence-inheritance claim is disproved. Independently, log ŝ_t during an ImageNet or MTEB fine-tuning run; if |ŝ_t| does not approach 0 within the first few epochs, the 2× learning-rate transfer in Eq. 6 is not valid in those regimes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"TAM's theoretical safety net is the statement after Eq. 6 that it 'inherits the well-established convergence guarantees of SGDM'. This rests on two assumptions: (A) ŝ_t stabilizes to a constant s* (Eq. 5), and (B) with s* fixed, TAM is equivalent to SGDM with learning rate 2η/(1+s*). Assumption (A) is only checked on CIFAR-10/100 (Fig. 8); assumption (B) is not established. With ŝ_t = s*, Eq. 4 gives θ_{t+1} = θ_t - η(β m_{t-1} + (ε+(1+s*)/2)g_t). This is not SGDM with any learning rate: compared to SGDM with η' = 2η/(1+s*), the momentum term has coefficient η rather than η', so the recursion differs in every transient step. Eq. 5 matches only the fixed point under a constant gradient, not the stochastic recursion; the cited SGDM convergence proofs (Yan et al. 2018; Liu et al. 2020) rely on a fixed gain and do not cover a gain d_t that is a function of m_{t-1} and g_t and hence correlated with the noise. Boundedness of d_t is insufficient. Moreover, early training is exactly when ŝ_t deviates from 0 (Fig. 8 shows ŝ_t dropping to roughly -0.03), so the transfer heuristic η_TAM = 2η_SGDM has no support in the regime where TAM's exploration benefits are claimed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Torque-Aware Momentum (TAM), which modifies the standard SGDM update by replacing the fixed gradient coefficient in the momentum recursion with a state-dependent damping factor built from a smoothed cosine similarity between the current momentum and the new gradient. Concretely, with m_t = β m_{t-1} + (ε + (1+ŝ_t)/2) g_t and θ_{t+1} = θ_t − η m_t, TAM down-weights 'torqued' gradients that are misaligned with the momentum direction. The authors also introduce AdaTAM by inserting the same damping into the momentum accumulator of Adam. The paper claims that TAM stabilizes updates, improves exploration and generalization, transfers learning rates from SGDM via an effective-learning-rate heuristic with η_TAM ≈ 2 η_SGDM, and 'inherits the well-established convergence guarantees of SGDM' near the optimum. The empirical section compares TAM/AdaTAM with SGD, SGDM, Adam, and AngularGrad on CIFAR10/100, ImageNet, MobileNet/ViT, MTEB-based LLM fine-tuning, online label-flipping benchmarks, and warm-up ablations on CIFAR and a GNN link-prediction task.","tokens_in":26957,"tokens_out":6048,"duration_ms":56021,"significance":"If the empirical findings hold, TAM is an attractive drop-in optimizer: it has the same memory footprint as SGDM, adds only one cosine-similarity computation and two fixed hyperparameters (γ=0.9, ε=1e−8), and shows consistent gains in several settings, most notably online learning with large label-flip shifts and LLM fine-tuning with DeBERTa/RoBERTa-base. The paper is also reasonably transparent: it reports five-seed averages, includes ablations on γ and on the momentum recursion, and states where results are mixed (e.g., AdaTAM on CIFAR10/ResNet34, RoBERTa-large on MTEB). The main weakness is that the paper's theoretical safety net—the convergence-inheritance claim—is currently unsupported, and the learning-rate transfer heuristic rests on an empirical stabilization assumption verified only on two CIFAR datasets. These issues are fixable by rewriting the theory section as a heuristic and adding evidence for the ŝ_t-stabilization assumption in the other experimental regimes.","major_comments":[{"comment":"The claim that TAM 'inherits the well-established convergence guarantees of SGDM' is not established for the actual update rule. If ŝ_t were exactly constant, say ŝ_t = s*, then with m_0 = 0 the TAM recursion is a rescaling of the SGDM recursion: m_t = a m'_t with a = (1+s*)/2, so TAM is exactly SGDM with learning rate η a. Thus the specific concern that 'TAM is not SGDM with any single learning rate' is answerable in the exactly-constant case. However, in the real algorithm d_t = (1+ŝ_t)/2 is a function of m_{t-1} and g_t and is therefore correlated with the stochastic gradient noise. Boundedness of d_t alone does not place the recursion in the class covered by the cited SGDM analyses (Yan et al. 2018; Liu et al. 2020), which rely on a fixed gain. The sentence after Eq. (6) should either be replaced by a precise statement about the constant-gain idealization, or supported by a convergence proof for the state-dependent gain, or removed in favor of an explicitly empirical claim.","section":"Section 3, Eqs. (4)–(6) and following paragraph"},{"comment":"The learning-rate transfer η_TAM = 2 η_SGDM depends on the observation that ŝ_t stabilizes to s* ≈ 0, but this observation is shown only for ResNet18 on CIFAR10 and CIFAR100. The same rule is then applied to ImageNet, MTEB fine-tuning, online label-flipping, and GNN warm-up experiments, where no ŝ_t trajectories are reported. Moreover, the paper's exploration argument concerns early training, exactly the regime where Fig. 8 shows ŝ_t first rising and then dipping negative before flattening near zero. The authors should either report ŝ_t dynamics for representative non-CIFAR runs, or explicitly frame the 2× rule as a heuristic whose domain of validity is currently unknown.","section":"Appendix A.2.1, Fig. 8; Tables 4–5; Figs. 3, 5, 7"},{"comment":"The abstract's 'improves generalization performance across various tasks' is stronger than the data shown. In Table 1, AdaTAM underperforms Adam on CIFAR10/ResNet34 (93.3 vs. 93.6), and in the MTEB experiments RoBERTa-large is consistently worse with AdaTAMW than with AdamW on a substantial fraction of datasets, including AmazonCounterfactualClassification, EmotionClassification, and ImdbClassification at 3 epochs in Table 16. The paper already acknowledges 'mixed' results in places, so the main text and abstract should be reworded to say that TAM/AdaTAM 'often matches or improves upon' baselines, with the consistent gains appearing in online learning and in several LLM fine-tuning configurations, rather than claiming universal improvement.","section":"Section 4.1, Table 1; Section 4.2, Figs. 3–4, Tables 15–16"}],"minor_comments":[{"comment":"There is a typo: 'varying numbes of epochs' should be 'varying numbers of epochs'.","section":"Section 4.2, paragraph 1"},{"comment":"'Additionnaly' should be 'Additionally', and 'capabilites' should be 'capabilities'.","section":"Section 5, Conclusion"},{"comment":"'more criticial for stability' should be 'more critical for stability'.","section":"Section 2, Related Work"},{"comment":"The phrase 'The the y-axis labels' contains a duplicated article and should be corrected.","section":"Figure 3 caption"},{"comment":"The definition of 'similar performance' changes between Figure 4 (threshold 0.2%) and Figure 10 (threshold 1%); the main text should state which threshold is used for the summary percentages.","section":"Appendix A.2.7, Figure 10 vs. Figure 4"},{"comment":"When switching from TAM to SGDM 'while keeping the effective learning rate and optimizer state same', the paper should specify how the TAM momentum state is mapped to SGDM; since the damping factor is not constant at the switch time, this mapping is not automatic.","section":"Section 4.4, paragraph 1"}],"recommendation":"major_revision","confidential_remarks":"This is a useful empirical paper with a simple, memory-neutral modification and several well-executed experiments. The main risk is that the theory section overclaims a convergence guarantee that the current analysis does not provide; if the authors reframe Section 3 as a heuristic and provide the requested ŝ_t evidence, the paper could be acceptable. I would not recommend reject, because the empirical contribution is substantial and the theoretical flaw appears fixable by rewriting rather than by new mathematics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Torque-Aware Momentum is a genuinely new twist on momentum: scale each gradient by (1+cos(momentum,gradient))/2 before adding it to the momentum buffer. That is a simple, cheap modification, and it is not the same as AngularGrad, which uses the angle between consecutive gradients. The paper deserves credit for a clean mechanism and a broad empirical sweep: CIFAR, ImageNet, MTEB fine-tuning, online label-flipping, and warmup experiments. The gains are mostly small (77.0 to 77.1 on ImageNet, around 0.5 on CIFAR), and there are a few losses (AdaTAM on CIFAR10 ResNet34, several MTEB cells), but the direction is consistent enough that this looks like a real effect worth understanding.\n\nThe soft spot is the theoretical framing in §3. The claim that TAM \"inherits the well-established convergence guarantees of SGDM\" does not follow from the bounded damping factor. With ŝ_t stabilized at s*, the TAM update is θ_{t+1} = θ_t - η(β m_{t-1} + d g_t), d=(1+s*)/2. That is not SGDM with any fixed learning rate: the momentum and gradient terms have different effective gains. The cited SGDM proofs (Yan et al. 2018, Liu et al. 2020) are for a fixed-gain recursion; here d_t is a function of m_{t-1} and g_t, hence correlated with the noise. Boundedness of d_t alone does not give convergence. The effective-LR transfer heuristic also leans on s*≈0 observed from the authors' own runs on two CIFAR datasets; early in training, where TAM claims its exploration benefit, Fig. 8 shows ŝ_t deviating from 0. So the LR transfer is a heuristic, not a theorem, and the paper should present it as such.\n\nOther minor issues: no code is available, and the MTEB gains are uneven enough that \"AdaTAMW consistently outperforms\" is too strong. The warmup experiments are a nice addition, and the mode-connectivity analysis is suggestive.\n\nBottom line: this is a plausible, testable variant with a small but apparently real edge in several settings. The empirical work is substantial; the theory section overpromises and should be rewritten as a heuristic argument. That is very fixable. I would send it to review.","headline":"Useful new momentum variant with small but real gains, but the convergence-inheritance argument in §3 does not hold as stated.","tokens_in":27502,"tokens_out":2688,"would_cite":false,"duration_ms":24236,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Torque-Aware Momentum damps misaligned gradients and improves generalization across vision, language, and online learning.","keywords":["Torque-Aware Momentum","momentum optimizer","gradient alignment","cosine similarity damping","optimizer exploration","distribution shift robustness","warm-up strategy","AdaTAM"],"falsifier":"On a standard training run (for example ResNet18 on CIFAR10), record the alignment statistic $\\hat{s}_t$ at every step and check whether it converges to a constant near $s^* = 0$ over the whole run, including the early phase where gradients are largest. If $\\hat{s}_t$ instead tracks the data or oscillates, or if the learning rate that actually performs best for TAM departs systematically from the $2\\eta_{\\mathrm{SGDM}}$ rule once $s^*$ is measured, the paper's rate-transfer and convergence-inheritance claims are contradicted.","tokens_in":26424,"feed_emoji":"🎯","tokens_out":8382,"duration_ms":66853,"temperature":0.7,"pith_summary":"This paper claims that a single scalar can fix a known weakness of momentum in deep learning: when a new gradient badly misaligns with the accumulated momentum, standard momentum lets that gradient yank the update into an oscillating, sharp region. Torque-Aware Momentum (TAM) instead scales each new gradient by a factor between zero and one, computed from a smoothed measure of how well the gradient agrees with the current momentum direction, so misaligned gradients are damped while aligned ones pass through. The paper reports that this damping improves or preserves accuracy on image classification and large-language-model fine-tuning, and that its biggest wins come under distribution shifts such as label flipping in online learning. TAM adds no extra memory and works as a standalone optimizer, as an adaptive variant of Adam, and as a short warm-up phase before switching to another optimizer. The paper also derives a learning-rate transfer rule, double a tuned SGDM learning rate, and argues that near an optimum TAM inherits SGDM's convergence behavior.","feed_headline":"Momentum that damps misaligned gradients lifts accuracy","feed_subtitle":"A cosine-similarity damping term lets TAM match SGDM's convergence while exploring wider basins and surviving label flips.","key_machinery":"The load-bearing object is the smoothed alignment statistic $\\hat{s}_t = \\gamma \\hat{s}_{t-1} + (1-\\gamma) S_t$, where $S_t = (m_{t-1} \\cdot g_t)/(\\|m_{t-1}\\| \\|g_t\\|)$ is the cosine similarity between previous momentum and current gradient, wrapped into the damping factor $d_t = (1+\\hat{s}_t)/2$. That factor continuously scales each gradient's contribution from full (aligned) to near zero (opposing), without any extra per-parameter state. This scalar also produces the effective-learning-rate identity $\\eta_{\\mathrm{TAM}} = 2/(1+s^*) \\eta_{\\mathrm{SGDM}}$ once $\\hat{s}_t$ is assumed to settle at constant $s^* \\approx 0$, which is the step that lets the paper transfer tuned SGDM learning rates and claim inheritance of SGDM convergence guarantees.","core_discovery":"The central claim is that the harmful effect of 'torqued' gradients, large updates that oppose or sharply cross the current momentum, can be damped without weakening momentum's benefits. TAM changes the momentum recurrence from $m_t = \\beta m_{t-1} + g_t$ to $m_t = \\beta m_{t-1} + (\\epsilon + d_t) g_t$, where $d_t = (1+\\hat{s}_t)/2$ and $\\hat{s}_t$ is an exponentially smoothed cosine similarity between the previous momentum and the current gradient. The factor $d_t$ lies in $[0,1]$, so aligned gradients receive nearly full weight and opposing gradients nearly none. The paper treats this as anisotropic friction, arguing it stabilizes the update direction, delays the oscillation that SGDM shows at larger learning rates, and lets the optimizer explore wider basins early in training. The paper then argues that when $\\hat{s}_t$ stabilizes to a constant $s^* \\approx 0$ near the optimum, TAM's effective learning rate is $2/(1+s^*)$ times SGDM's, which justifies setting $\\eta_{\\mathrm{TAM}} = 2\\eta_{\\mathrm{SGDM}}$ and inheriting SGDM's convergence guarantees. Empirical sections claim parity or gains over SGDM and Adam across CIFAR, ImageNet, MTEB fine-tuning of BERT-family models, online label-flipping tasks, and GNN link-prediction warm-up.","pith_inferences":["We infer that if the benefit really comes from damping misaligned gradients, TAM's advantage should grow as the frequency or magnitude of gradient misalignment increases; a direct test would compare SGDM and TAM under controlled noise levels or sharp-region loss surfaces.","We infer that the stabilizing value of $\\hat{s}_t \\to s^* \\approx 0$ is only demonstrated near convergence on stationary image tasks, and in non-stationary settings, where the loss landscape shifts between tasks, the same stabilization may not hold, so the paper's convergence argument may not transfer.","We infer that the damping mechanism is a scalar gate on the gradient-momentum angle and could be combined with other adaptive optimizers beyond Adam, such as AdamW variants with different second-moment or normalization schemes, which the paper does not test.","We infer that the warm-up results suggest a cheaper way to use TAM: run it for the first tens of epochs only, then switch to the default optimizer, converting an exploration benefit into a final-model gain without paying TAM's behavior for the whole run."],"forward_implications":["TAM is a drop-in replacement for momentum: it keeps the memory footprint of SGDM, fixes its two new hyperparameters ($\\gamma = 0.9$, $\\epsilon = 10^{-8}$), and can be paired with either SGD or Adam.","A tuned SGDM learning rate transfers to TAM by doubling it near an optimum, and TAM retains SGDM's convergence guarantees in that regime.","Across the reported benchmarks, CIFAR10/100, ImageNet, MobileNet, ViT fine-tuning, and MTEB fine-tuning of BERT, DeBERTa, and RoBERTa, TAM and AdaTAM match or beat SGDM, Adam, and AngularGrad on accuracy.","In online learning with label flipping, TAM remains more stable and accurate than SGDM when shifts are drastic, 80% and 100% flips.","Using TAM for the first phase of training and then switching to SGDM or Adam reaches lower validation error and lower loss barriers than starting with the base optimizer, in the paper's warm-up experiments."],"supporting_citations":[{"why":"Defines the momentum update that TAM modifies.","marker":"Qian, 1999"},{"why":"Establishes momentum and initialization as key to deep learning, motivating preservation of momentum benefits.","marker":"Sutskever et al., 2013"},{"why":"Supplies the effective-learning-rate comparison and the abrupt-sharpening analysis used for TAM's learning-rate transfer.","marker":"Fu et al., 2023"},{"why":"AngularGrad is the closest angle-based optimizer and serves as a direct empirical baseline.","marker":"Roy et al., 2021"},{"why":"Adam is the adaptive baseline that AdaTAM extends by modifying only the momentum term.","marker":"Kingma & Ba, 2015"},{"why":"Provides the SGDM convergence analysis that TAM claims to inherit near the optimum.","marker":"Yan et al., 2018"},{"why":"Gives an improved stochastic-momentum convergence analysis also invoked for TAM's convergence inheritance.","marker":"Liu et al., 2020"},{"why":"Provides the mode-connectivity and loss-barrier measurement used in the warm-up experiments.","marker":"Frankle et al., 2020"},{"why":"Supplies the online-learning and effective-learning-rate setup that TAM's distribution-shift experiments follow.","marker":"Lyle et al., 2024a;b"}],"fun_headline_variants":["Damping torqued gradients boosts deep net generalization","Angle-aware momentum damps oscillations, widens basins","Torque-Aware Momentum: align gradients, explore better","Cosine damping for momentum stabilizes and generalizes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument rests on a running average of how well each new gradient agrees with the current momentum direction settling down to a constant near zero as training proceeds; if that average keeps moving or settles somewhere far from zero, the learning-rate doubling rule and the claim of matching SGDM's convergence behavior no longer follow.","fun_headline_variants_meta":{"raw":{"variants":["Damping torqued gradients boosts deep net generalization","Angle-aware momentum damps oscillations, widens basins","Torque-Aware Momentum: align gradients, explore better","Cosine damping for momentum stabilizes and generalizes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000113,"raw_usage":{"total_tokens":1070,"prompt_tokens":956,"completion_tokens":114,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":51}},"tokens_in":572,"tokens_out":114,"duration_ms":2185,"temperature":1.0,"reasoning_tokens":51,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:27:47.846496+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a standard training run (for example ResNet18 on CIFAR10), record the alignment statistic $\\hat{s}_t$ at every step and check whether it converges to a constant near $s^* = 0$ over the whole run, including the early phase where gradients are largest. If $\\hat{s}_t$ instead tracks the data or oscillates, or if the learning rate that actually performs best for TAM departs systematically from the $2\\eta_{\\mathrm{SGDM}}$ rule once $s^*$ is measured, the paper's rate-transfer and convergence-inheritance claims are contradicted.","supporting_citations":[{"cited_title":"An improved analysis of stochastic gradient descent with momentum","cited_arxiv_id":null,"evidence_quote":"Gives an improved stochastic-momentum convergence analysis also invoked for TAM's convergence inheritance."}],"review_version":1}