{"id":"364f1dc7-5bf4-4b69-bfb5-eca5ea6dcc0b","arxiv_id":"2412.17107","paper_version":3,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Grams is an optimizer that forces the update direction to match the current gradient's sign while scaling by the magnitude of Adam's momentum, and the paper claims faster loss descent and global convergence.","lead":"This paper proposes Grams, an optimizer that uses the sign of the current gradient as the update direction and the magnitude of an Adam-style momentum term as the step size. The authors claim Grams converges faster than Adam, Lion, and their cautious variants, and has a global convergence guarantee, supported by experiments on small LLM and vision tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4.3/B.3 has an unsatisfiable step-size hypothesis: the min includes a non-positive term, so no positive η_t exists; the faster-descent guarantee is vacuous as stated.","rationale":"The reader and I identify the same load-bearing defect: the comparison theorem that supports the faster-descent claim has a hypothesis that is unsatisfiable for positive step sizes. This is not a matter of disagreeing with an established convention; it is an internal inconsistency in the stated theorem and its proof. The informal theorem 4.3 and the formal theorem B.3 both require η_t to be bounded above by a non-positive quantity, so the claimed implication holds only vacuously. The proof itself, if repaired by replacing the min with a sum over the conflicting-sign coordinates, could yield a meaningful statement, but the published version does not contain that repair. The global convergence theorem inherits the problem because its proof invokes Theorem B.3, and it also misapplies the comparison by treating Adam iterates as if they were covered by a Grams-versus-Cautious result. Thus both headline theoretical contributions — faster descent and global convergence — are unsupported as written. I credit the paper for a clear algorithm definition, a valid elementary descent lemma for Grams alone (Lemma B.2), and a public code repository. However, those do not rescue the specific advertised guarantees. The empirical section also reports single runs without error bars or ablations, so it cannot independently establish the headline speed advantage. Given that the central theorems are not sound in their current form, the REJECT verdict is appropriate, and my stress-test does not change it.","tokens_in":20215,"tokens_out":5037,"duration_ms":47025,"concrete_test":"Instantiate the proof of Theorem B.3 in d=1. Let L(w)=0.5 w², so L is 1-smooth; set w_1=1, hence g_1=1, and choose momentum u_1=-1. Then ⟨u∘g, 1_{u∘g≥0}⟩=0, ⟨u∘g, 1_{u∘g<0}⟩=-1, min=-1, and the printed condition requires η_1 ≤ -2, impossible for η_1>0. The proof's own derived sufficient condition requires η_1 ≤ 2, so the printed hypothesis and the proof's algebra are contradictory. Recomputing lines (11)-(12) of B.3 with this example settles whether the theorem is vacuous: it is, as printed. A secondary check: in Theorem D.1, replace the cited comparison by the actual Grams-vs-C-Adam statement of B.3 and verify whether the step '≥ 2µ(∆L^{w_t}+...)' follows; it does not, because B.3 never compares Grams to Adam.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central 'faster descent' claim rests on Theorem 4.3 (formal Theorem B.3). Its hypothesis is η_t ≤ 2/(L||u_t||²) · min{⟨u_t∘g_t, 1_{u_t∘g_t≥0}⟩, ⟨u_t∘g_t, 1_{u_t∘g_t<0}⟩}. The second term is a sum over coordinates with u_t,i g_t,i < 0, hence it is non-positive; if no such coordinates exist it is zero. Consequently the min is ≤ 0, and no positive step size can satisfy the bound. The proof's final sufficient condition is actually η_t ≤ 2/(L||u_t||²) Σ_{i: u_t,i g_t,i < 0} |u_t,i||g_t,i|, which is non-negative and needs a separate argument when the set is empty (then Grams and Cautious make identical updates). The printed theorem therefore does not follow from its proof. Since the global convergence proof (Theorem D.1, informal 4.15) explicitly invokes Theorem B.3 to compare Grams iterates with Adam iterates, the O(1/T) guarantee inherits the same unsupported step. The comparison to Adam is also not what B.3 proves (it compares Grams to Cautious optimizers), so D.1's chain is doubly unjustified. Because the headline theoretical guarantees are asserted through these theorems, the central claims of faster descent and global convergence are not established as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Grams, an optimizer whose update is w_{t+1} = w_t - eta_t sign(g_t) \\circ |u_t|, where u_t is the Adam-style normalized momentum. The claimed contributions are (i) a per-step loss-descent comparison showing Grams decreases loss at least as fast as cautious optimizers (Theorem 4.3, formalized as Theorem B.3), (ii) Hamiltonian-dynamics descent and comparison results (Theorems 4.6-4.7, formalized as C.3-C.4), and (iii) a global O(1/T) convergence guarantee under L-smoothness, bounded gradients, and the PL condition (Theorem 4.15, formalized as D.1). The paper also reports pre-training, fine-tuning, and PEFT experiments with code released.","tokens_in":20503,"tokens_out":7764,"duration_ms":70729,"significance":"If the theoretical claims were valid, Grams would be a simple, drop-in optimizer with per-step domination over cautious methods and a global convergence guarantee, which would be a useful contribution. The paper also contains positive elements: the algorithm is clearly specified, the code is released, and the empirical study covers several realistic settings. However, the central comparison theorem has an unsatisfiable hypothesis as printed, and the global convergence proof relies on a comparison to Adam that no stated theorem provides. The headline theoretical guarantees are therefore not established; the empirical results alone are limited by single-run reporting and would not carry the paper's main claims.","major_comments":[{"comment":"Theorem 4.3 and its formal version Theorem B.3 state the sufficient condition eta_t <= 2/(L||u_t||^2) * min{<u_t o g_t, 1_{u_t o g_t >= 0}>, <u_t o g_t, 1_{u_t o g_t < 0}>}. The second term is a sum over coordinates with u_{t,i} g_{t,i} < 0, so it is non-positive, and it is zero when no such coordinate exists. Hence the min is always non-positive, and no positive learning rate can satisfy the inequality unless the bound is vacuous in a different way. The claimed comparison Delta_L^{Grams} <= Delta_L^{Cautious} <= 0 therefore does not follow from the stated theorem in the generic case.","section":"Section 4.2 / Theorem B.3"},{"comment":"The proof actually derives a different condition from the one printed in the theorem. The derivation shows that Delta_L^{Grams} - Delta_L^{Cautious} <= 0 follows from eta_t <= 2/(L||u_t||^2) * sum_{i in I^-} |u_{t,i}| |g_{t,i}|, a nonnegative quantity. This is not the min expression stated in Theorem B.3. Moreover, when I^- is empty, the proof's upper bound becomes L eta_t^2 ||u_t||^2 / 2, which does not establish the desired inequality; the cases I^- empty and g_{t,i}=0 require a separate argument. As printed, the theorem is not a consequence of its proof.","section":"Appendix B, proof of Theorem B.3"},{"comment":"The global convergence proof invokes Theorem B.3 in Eq. (17) to replace per-step loss changes of Adam iterates w'_t with those of Grams iterates w_t. However, Theorem B.3 compares Grams with Cautious optimizers, not with Adam. No result in the paper bounds the loss descent of Adam relative to Grams, so this step is unjustified. Additionally, the argument implicitly requires Grams' loss values to be non-increasing so that the final iterate can be bounded by the averaged gradient bound; no such monotonicity is established under the stated assumptions. Thus Theorem 4.15 does not follow from the proof as written.","section":"Section 4.4 / Appendix D, Theorem D.1"}],"minor_comments":[{"comment":"In the proof of Lemma B.1 Part 2, and similarly in Lemma B.2 Part 2, the smoothness lower bound is written with <g_t, w_{t+1} + w_t>; it should be <g_t, w_{t+1} - w_t>.","section":"Appendix B"},{"comment":"In the proof of Theorem C.3, the term <K(s_t), ...> should be <\\nabla K(s_t), ...>. In the proof of Theorem C.4, the second term should be -<|\\nabla L(w_t)|, |\\nabla K(s_t)|> rather than -<|\\nabla L(w_t)|, |\\nabla L(w_t)|>.","section":"Appendix C"},{"comment":"There are several presentation errors: 'CIF AR-10' should be 'CIFAR-10'; the abstract says 'Grams descents faster' and should say 'descends'; the PEFT paragraph refers to Table 7 when it should refer to Table 8; and the wording in Section 5.1 comparing C-Adam and Lion to 'their respective base optimizers, Adam and C-Lion' is confusing because C-Lion is not the base version of Lion.","section":"Sections 4-5 and Appendix E"},{"comment":"The empirical evaluation would be substantially stronger with multiple seeds, standard deviations, and training curves. As reported, most tables give single final numbers without variance or significance information, which makes it hard to assess whether the observed gaps are reliable.","section":"Section 5 / Appendix E"}],"recommendation":"reject","confidential_remarks":"I concur with the stress-test analysis. The unsatisfiable step-size condition is a decisive issue in the main theorem, and the global convergence proof compounds it by invoking a comparison that is not established. Fixing these would require reworking the theoretical core of the paper, not a local correction. The empirical section is too limited to independently support the abstract's stronger claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the Grams update rule is a real and simple variant of the cautious mechanism — flip the momentum coordinate to the gradient's sign instead of masking it — and the per-step descent lemma (B.2) is correct. But the paper's headline theory doesn't hold. Theorem 4.3/B.3 states a step-size condition that is unsatisfiable for any positive learning rate whenever any coordinate of u_t and g_t disagree, because the min includes a sum over disagreeing coordinates that is non-positive. The proof actually needs a different, non-negative condition (sum of |u_i||g_i| over the disagreeing set), so the printed theorem does not follow from its proof. Since the global convergence proof (D.1) invokes B.3 to compare Grams with Adam — and B.3 compares Grams with Cautious, not Adam — the O(1/T) guarantee inherits the same unsupported step. The empirical section is thin: single runs, no error bars, 60M-3B models, 1,000 steps, and one PEFT result is a tie with Adam. The Hamilton dynamics section is largely a transcription of the cautious framework with the same kind of sign-flip replacement; it does not repair the discrete-time problem. What the paper does well: the update rule is simple, clearly specified, and the loss-descent bound is elementary and correct. The code is linked. If the comparison theorem were restated with the correct sufficient condition and the Adam comparison reworked or dropped, the paper could be made honest. As is, the advertised 'faster than state-of-the-art' and 'global convergence' claims are not supported. Who is this for? People who track optimizer variants might try Grams on their own benchmarks; the theory needs a rewrite. Recommendation: a serious referee should see this because the algorithm is a legitimate variant and the flaws are identifiable and fixable, but the current version should not be accepted. I would not cite the theoretical claims; I might mention the update rule in passing.","headline":"A real but small optimizer variant with a correct per-step lemma; the headline comparison and global convergence theorems are vacuous as stated due to an unsatisfiable step-size condition.","tokens_in":21040,"tokens_out":2853,"would_cite":false,"duration_ms":25310,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","90C26","65K05"],"pacs":[],"model":"deepseek-v4-flash","headline":"A new optimizer, Grams, claims to make each training step reduce loss at least as much as cautious Adam while guaranteeing global convergence.","keywords":["Grams optimizer","adaptive momentum scaling","Cautious optimizers","Adam","Lion","global convergence","Hamiltonian descent","large language model training"],"falsifier":"Take any training step where the momentum and gradient disagree in sign in at least one coordinate and compute the two inner products in Theorem 4.3's condition; the second inner product is negative, so the minimum is negative and the required step size is negative, making the theorem's inequality inapplicable — a direct calculation that settles whether the stated proof supports the faster-descent claim.","tokens_in":19994,"feed_emoji":"⚡","tokens_out":8252,"duration_ms":70409,"temperature":0.7,"pith_summary":"The paper introduces Grams, an optimizer that separates update direction from update magnitude: the direction is set by the current gradient's sign, while an Adam-style momentum term is used only to scale how far each coordinate moves. The central claim is that this decoupling makes Grams descend the loss at least as fast as Cautious Adam and Cautious Lion on every step, under a stated step-size condition, and that Grams converges globally at an O(1/T) rate under standard smoothness, bounded-gradient, and PL assumptions. If correct, Grams is a drop-in optimizer that could speed up pre-training and fine-tuning of large language models without extra memory. The paper supports the claim with experiments on a 60M-parameter Llama pre-training run, WideResNet on CIFAR-10, and fine-tuning of Llama 3.2 1B/3B, where Grams matches or beats Adam, Lion, and their cautious variants.","feed_headline":"Grams optimizer claims faster loss descent than Adam or Lion","feed_subtitle":"Momentum only scales step size; gradient sets direction, bringing faster LLM and vision convergence.","key_machinery":"The load-bearing object is the Grams update rule itself, $w_{t+1}=w_t-\\eta_t\\,\\mathrm{sign}(g_t)\\circ|u_t|$: the gradient's sign fixes each coordinate's direction and Adam's normalized momentum $u_t$ supplies only the magnitude, so the first-order loss change is always $-\\eta_t\\langle|g_t|,|u_t|\\rangle$, a sum of non-positive terms. The proof machinery is the L-smoothness bound of Fact 3.2 applied to this rule (Lemma B.2), the comparison to the Cautious mask via a sign-aware index split (Theorem B.3), and a Hamiltonian continuous-time formulation (Definition 4.5) in which the same sign-magnitude decoupling makes the Hamiltonian and loss decrease rates dominate the Cautious system. These pieces together carry the faster-descent and global-convergence conclusions.","core_discovery":"The paper's claim is that Grams' update rule, $w_{t+1}=w_t-\\eta_t\\,\\mathrm{sign}(g_t)\\circ|u_t|$, where $g_t$ is the gradient and $u_t$ is Adam's normalized momentum estimate, dominates cautious optimizers in loss descent. Lemma 4.2 bounds the one-step loss change by $-\\eta_t\\langle|g_t|,|u_t|\\rangle + \\frac{L\\eta_t^2}{2}\\|u_t\\|_2^2$, and Theorem 4.3 asserts that whenever the step size satisfies $\\eta_t \\le \\frac{2}{L\\|u_t\\|_2^2}\\min\\{\\langle u_t\\circ g_t, 1_{u_t\\circ g_t\\ge 0}\\rangle, \\langle u_t\\circ g_t, 1_{u_t\\circ g_t<0}\\rangle\\}$, Grams' loss decrease is at least that of the Cautious update and both are non-positive. Theorem 4.15 then states that under the PL condition Grams satisfies $L(w_T)-L^* \\le \\frac{4G}{\\mu\\eta T}(L(w_1)-L^*)$, the same O(1/T) global convergence rate as Adam. Empirically, Grams reports the lowest pretraining perplexity (38.60 vs 43.21 for C-Adam) on a 60M Llama run and the highest CIFAR-10 accuracy (90.55%) among the compared optimizers.","pith_inferences":["A condition implicit in Theorem 4.3 is that the step-size bound be positive; because the minimum in the bound is non-positive whenever any coordinate of momentum and gradient disagree, the theorem as stated covers only steps with no sign disagreement.","Since Grams' direction is the sign of the current gradient, its update differs from signSGD only by per-coordinate magnitudes; one testable extension is to analyze Grams in the nonconvex first-order-stationary-point framework used for signSGD.","The empirical comparisons are short-horizon (1,000 pre-training steps, 10 epochs on CIFAR-10); a natural extension is to verify whether Grams' margin persists at full-scale token budgets where AdamW's schedule and warm-up dominate.","The Hamiltonian formulation suggests that replacing $|u_t|$ by other magnitude statistics, such as $|u_t|^\\alpha$ or a clipped norm, yields a family of algorithms with the same descent identity and tunable noise."],"forward_implications":["Grams can be dropped into existing Adam-based training loops with the same two moment buffers and default hyperparameters, since its update is a cheap sign and absolute-value modification of the Adam step.","Under L-smoothness and the theorem's step-size condition, Grams inherits monotone loss decrease on every step, matching the guarantee that makes Cautious optimizers attractive.","Under the PL condition, Grams converges from any finite initial gap at the O(1/T) rate of Adam, so its practical speed advantage is not bought by sacrificing a worst-case convergence guarantee.","On the reported settings, Grams reaches the lowest perplexity among compared optimizers on a 60M Llama pretraining run and the highest accuracy on WideResNet/CIFAR-10, suggesting gains compound in early-stage and full fine-tuning.","The continuous-time Hamiltonian analysis implies the decoupled update has a Lyapunov structure, so stability arguments used for Adam-class methods carry over in the limit."],"supporting_citations":[{"why":"Defines the Cautious mechanism and its loss-descent bound that Grams is compared against, and supplies the Hamiltonian framework used in Section 4.3.","marker":"[LCLL24]"},{"why":"Supplies the Adam convergence bound (Lemma 4.14) that the Grams global convergence proof extends.","marker":"[LRJ23]"},{"why":"Defines Adam, whose first and second moment updates Grams reuses for its magnitude scaling.","marker":"[KB14]"},{"why":"Defines Lion, the sign-based optimizer whose direction convention Grams resembles, and sets the 1/10 learning-rate convention used in the experiments.","marker":"[CLH+24]"}],"fun_headline_variants":["Grams: momentum scales magnitude, gradient sets direction","Grams claims better loss descent than Adam and Lion","Grams: fast convergence and better generalization than Adam","Grams: decoupled momentum scaling for faster deep learning","Grams: O(1/T) convergence with lower perplexity than Adam"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof of Grams' faster-descent guarantee rests on a step-size condition that, because one of the two terms inside the minimum is non-positive, can only be met by a positive step size when the momentum and the gradient point in the same direction in every coordinate; in the typical case where they disagree in some coordinate, the theorem's premise does not hold.","fun_headline_variants_meta":{"raw":{"variants":["Grams: momentum scales magnitude, gradient sets direction","Grams claims better loss descent than Adam and Lion","Grams: fast convergence and better generalization than Adam","Grams: decoupled momentum scaling for faster deep learning","Grams: O(1/T) convergence with lower perplexity than Adam"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000621,"raw_usage":{"total_tokens":2925,"prompt_tokens":1038,"completion_tokens":1887,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":654,"completion_tokens_details":{"reasoning_tokens":1805}},"tokens_in":654,"tokens_out":1887,"duration_ms":12839,"temperature":1.0,"reasoning_tokens":1805,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:47:44.642165+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any training step where the momentum and gradient disagree in sign in at least one coordinate and compute the two inner products in Theorem 4.3's condition; the second inner product is negative, so the minimum is negative and the required step size is negative, making the theorem's inequality inapplicable — a direct calculation that settles whether the stated proof supports the faster-descent claim.","supporting_citations":[],"review_version":1}