{"id":"d2993a2f-c33b-40e6-967a-530b6c5214ef","arxiv_id":"1908.00700","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"The authors introduce softplus-calibrated adaptive learning rates (Sadam and SAMSGrad) and argue, with flawed proofs, that these converge faster and generalize better than Adam.","lead":"A machine learning paper proposes two new optimizer variants, Sadam and SAMSGrad, that replace Adam's numerical stabilization term with a softplus function to tame wildly varying per-dimension learning rates. The paper claims, and tries to prove, that these variants converge faster than Adam and generalize better on image and language benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unstated `v_t >= v_{t-1}` premise in Lemma 23 is false for Adam/Sadam as defined; telescoping proof fails, so claimed rates are unsupported.","rationale":"The paper's central claim is a theoretical comparison of convergence rates. The proof as written makes the hidden assumption `v_t >= v_{t-1}` in Lemma 23 and in the opening lines of the appendix proofs. This is not a stylistic issue: the inequality `(a-b)^2 <= a^2-b^2` is the mechanism by which the sums over inverse step sizes telescope and disappear from the final bound. For an exponential moving average, `v_t` can decrease; a single coordinate with a large accumulated `v` and a small current gradient violates the premise. Since Theorems 2 and 9 are stated unconditionally, the advertised rates are unproved. The released code and the empirical tables are good-faith evidence for the engineering claim, and SAMSGrad's max-based update does satisfy monotonicity, so that variant is less affected by this particular gap. But the headline contribution, namely the epsilon-dependent Adam analysis and the better-rate proof for Sadam, rests on this unsupported step. A rejection of the theoretical claim is therefore warranted even though the experiments may be useful.","tokens_in":31284,"tokens_out":11461,"duration_ms":106570,"concrete_test":"Independently re-derive Lemma 23 without the condition `v_t>=v_{t-1}`. Concretely, for one coordinate with `v_0=0`, `beta_2=0.999`, `epsilon=1e-8`, and an adversarial bounded gradient sequence `g_t` alternating between 10 and 0.1, compute `D_T = sum_{t=1}^T [(a_t-b_t)^2 - (a_t^2-b_t^2)]`, where `a_t=1/(sqrt(v_{t-1})+epsilon)` and `b_t=1/(sqrt(v_t)+epsilon)` with `v_t=beta_2 v_{t-1}+(1-beta_2) g_t^2`. If `D_T` is positive and grows with `T`, the inequality `(a-b)^2 <= a^2-b^2` fails exactly where the proof needs it, demonstrating that the stated rates rely on a condition the algorithm does not guarantee.","verdict_should_be":"REJECT","load_bearing_attack":"All the advertised rates (Theorems 2, 3, 4, 9, 12, 14 and their corollaries) depend on a coordinate-wise monotonicity premise `v_t >= v_{t-1}` that appears only inside the appendix proofs and Lemma 23, not in any theorem statement. The appendix is explicit: 'All the analyses hold true under the condition: vt>=vt-1.' This premise is false for the algorithms as defined: both Adam and Sadam (Algorithm 1) update `v_t = beta_2 v_{t-1} + (1-beta_2) g_t^2`, an EMA that decreases whenever recent squared gradients are small. The proof's telescoping step uses `(a-b)^2 <= a^2 - b^2` with `a=1/sqrt(v_{t-1})+epsilon` and `b=1/sqrt(v_t)+epsilon`, which requires `a>=b`, i.e. `v_t>=v_{t-1}`. Without that condition, the sums over `1/sqrt(v_t)` and `1/v_t` do not telescope, so the claimed `O(1/(epsilon^2 sqrt T)+...)` and `O(beta^2/sqrt T+...)` rates are unsupported. Only SAMSGrad (Algorithm 2) uses `v_t = max(v_{t-1}, vtilde_t)` and satisfies monotonicity; the Sadam theorems concern Algorithm 1. A separate local step in the same proofs lower-bounds `E[<grad f(x_t), c_t g_t>]` by sign-splitting on `grad_i g_i` and replacing `grad_i g_i` with `grad_i^2`; that inequality is not valid as written, although it could be repaired under an independence assumption. The structural gap is the unstated monotonicity condition.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper revisits adaptive gradient methods from the perspective of the adaptive learning rate (A-LR), argues that the A-LR is anisotropic, and proposes to calibrate it with a softplus function, giving the Sadam and SAMSGrad algorithms. The main theoretical claims are that Adam's convergence rate depends on epsilon as O(1/(epsilon^2 sqrt(T)) + d/(epsilon T) + d/(epsilon^2 T sqrt(T))) and that Sadam achieves O(beta^2/sqrt(T) + d beta/T + d beta^2/(T sqrt(T))), with similar improvements in non-strongly convex and Polyak-Lojasiewicz settings. Empirical comparisons on MNIST, CIFAR, and LSTM benchmarks report that the proposed methods improve test accuracy over Adam, AMSGrad, and several other optimizers.","tokens_in":31642,"tokens_out":8592,"duration_ms":80407,"significance":"The empirical study is broad and the code is released, and the observation that Adam's A-LR ranges widely across coordinates (Section 3.1) is clearly illustrated. If the claimed epsilon/beta dependence were rigorously established, the paper would make a useful contribution. However, the theoretical comparison is not currently supportable: the proofs rely on a coordinate-wise monotonicity condition that is false for the algorithms as defined, and another key inequality is invalid as written. The advertised advantage of Sadam over Adam therefore rests on unproven rates; the paper's empirical results do not compensate for this gap because the central claim is the convergence comparison.","major_comments":[{"comment":"The appendix begins the Adam and Sadam nonconvex proofs with the statement that all analyses hold under the condition vt >= vt-1, and Lemma 23's third inequality uses (a-b)^2 <= a^2 - b^2, which requires a >= b, i.e., v_t >= v_{t-1}. This condition is not stated in Theorems 2, 3, 4, 9, 12, or 14, and it is false for Algorithm 1: with v_t = beta_2 v_{t-1} + (1-beta_2) g_t^2, the sequence can decrease whenever recent squared gradients are small. Consequently, the telescoping sums over 1/sqrt(v_t) and 1/v_t do not go through, and the claimed rates O(1/(epsilon^2 sqrt T) + d/(epsilon T) + d/(epsilon^2 T sqrt T)) for Adam and O(beta^2/sqrt T + d beta/T + d beta^2/(T sqrt T)) for Sadam are unsupported. Only SAMSGrad (Algorithm 2, which uses the max update) satisfies the monotonicity condition. This is the central derivation gap and invalidates the abstract and Section 5 claims.","section":"Appendix 11.2, Lemma 23; Algorithms 1-2"},{"comment":"The proof lower-bounds E[<grad f(x_t), (1/(sqrt v_t + epsilon)) g_t>] by splitting coordinates according to the sign of grad_i g_i and then replacing grad_i g_i by grad_i^2. This is not valid pointwise: on the event grad_i > 0 and g_i > 0, one has grad_i g_i < grad_i^2 whenever g_i < grad_i, which is not ruled out by Assumption 1; the analogous failure occurs on the negative-sign branch. The resulting lower bound eta mu_1 ||grad f(x_t)||^2 is therefore not established. The same defect appears in the Sadam proof. A repair would require an additional conditional independence or moment assumption that is not stated.","section":"Appendix 11.2 and 11.3, lower-bound step for the inner product"},{"comment":"The claimed 'better convergence speed' of Sadam over Adam is obtained by comparing beta = 50 with epsilon = 1e-8 inside loose upper bounds; this is a comparison of two hyperparameter values, not a parameter-free property of the algorithms. With epsilon = 1e-3, as used in the Adam and Yogi rows of Table 1, the constants O(1/epsilon^2) and O(1/epsilon) would shrink by orders of magnitude, eliminating the stated advantage. Thus, even setting aside the proof gap in the first major comment, the comparison does not support the paper's central claim of superior convergence.","section":"Section 5, Remark 10; Corollary 11"}],"minor_comments":[{"comment":"The sentence 'The second inequality is based on the fact that, when iteration t reaches the maximum number T, xt is the optimal solution, zT = x*' is false; the inequality is valid by simply dropping the nonpositive term -E[||z_T - x*||^2], so the proof should be reworded.","section":"Appendix 11.4, non-strongly convex proof"},{"comment":"The main text states eta = O(1/T^2) for the P-L condition, while Theorem 30 in the appendix states eta = O(1/T) and the proof then sets eta = 1/T^2; these should be aligned.","section":"Theorems 4 and 14 vs. Theorem 30"},{"comment":"The contribution statement claims 'theoretical guarantees of both optimization and generalization error analysis,' but no generalization bound is proved anywhere in the manuscript; either provide such a result or remove the claim.","section":"Introduction and Section 3.1"},{"comment":"Typos and inconsistent notation remain: 'funciton' (Section 4.1), 'diffentiable' (Lemma 29), 'perfoms' (Remark 28), duplicate '(d)' in the Figure 1 caption, and 'exsiting' (Appendix 9).","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The theoretical core is not salvageable by minor edits: the monotonicity condition contradicts Algorithm 1's update, so the proofs would need to be substantially reworked or the algorithm changed. The empirical part is solid enough to be a separate contribution, but as submitted the paper's main claims are unsupported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"At bottom, this is a paper whose advertised theory does not hold together, but the softplus calibration itself is a real empirical kernel. Replacing 1/(sqrt(v)+epsilon) with 1/softplus(sqrt(v)) is a simple, sensible way to cap the anisotropic A-LR, and the experiments are broad: MNIST, CIFAR-10/100, LSTM language modeling, with multiple baselines. The reported test-accuracy gains over Adam are real, and the proposed methods sometimes edge out S-Momentum. Code is released. That part is legitimate and citable as an empirical technique.\n\nThe soft spots, in order. The convergence theorems for Adam and Sadam are proved under vt >= vt-1. That condition appears only inside appendix proofs, not in any theorem statement, and it is false for both algorithms as defined: both use an exponential moving average that can decrease. The telescoping step (a-b)^2 <= a^2 - b^2 requires a >= b, meaning vt >= vt-1, so the advertised rates O(1/(epsilon^2 sqrt T) + d/(epsilon T) + ...) and O(beta^2/sqrt T + ...) are unsupported. SAMSGrad and AMSGrad do satisfy monotonicity via the max, but the Sadam theorems concern Algorithm 1, which does not. This is load-bearing, not a cosmetic gap. The appendix is candid that “all the analyses hold true under the condition: vt>=vt-1,” but that honesty is buried in proofs and contradicts the algorithm definitions.\n\nThe comparison of Sadam's recommended beta=50 against Adam's customary epsilon=1e-8 inside loose upper bounds is also weaker than it looks. It reduces to comparing hyperparameter values, not to a parameter-free derivation. One more technical detail: the lower bound on E[<grad f(x_t), c_t g_t>] sign-splits on grad_i g_i and replaces grad_i g_i with grad_i^2; as written that is not valid without an independence-type assumption. Probably repairable, but it is another sign the proofs were not polished.\n\nWho is this for? Empirical users of Adam who want a simple tweak and are willing to ignore the convergence theory. The theory section as written should not be relied on; I would not cite the rates, though I might cite the softplus calibration as an empirical trick.\n\nRecommendation: send to peer review. The empirical content and the claimed theoretical novelty deserve referee time, but expect major revision. The theorems need to be restricted to monotone v_t or the proofs need a different argument, and the Sadam claims should not be stated as they are. As is, a careful referee should reject the theory.","headline":"Softplus calibration is a plausible empirical trick, but the advertised convergence theory rests on a monotonicity condition the algorithms do not satisfy.","tokens_in":32159,"tokens_out":2302,"would_cite":false,"duration_ms":23685,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","90C26","90C15","65K05"],"pacs":[],"model":"deepseek-v4-flash","headline":"Adam's convergence rate depends on epsilon through a 1/epsilon^2 term, and replacing epsilon with a softplus-calibrated step size provably removes that penalty.","keywords":["adaptive gradient methods","Adam","softplus","adaptive learning rate","epsilon sensitivity","nonconvex convergence","generalization gap","Polyak-Lojasiewicz condition"],"falsifier":"Run Adam or Sadam exactly as defined in Algorithms 1 and 2 on a small smooth problem whose coordinate-wise v_t is strictly decreasing at some iterations, then check whether the claimed gradient-norm bound of Theorem 2 or Theorem 9 holds; a single instance where the bound is violated while all assumptions of Assumption 1 hold would show the theorems are not valid without the monotonicity condition. More directly, compute the term sum_j [(1/($\\sqrt$(v_{t-1,j})+epsilon))^2 - (1/($\\sqrt$(v_{t,j})+epsilon))^2]; whenever v_t < v_{t-1} this difference is negative, and the paper's Lemma 23 step (a-b)^2 <= $a^{2}$ - $b^{2}$ no longer applies.","tokens_in":31069,"feed_emoji":"📉","tokens_out":6846,"duration_ms":58978,"temperature":0.7,"pith_summary":"This paper argues that Adam's convergence rate carries a hidden dependence on the denominator constant epsilon, of order O(1/($epsilon^{2}$ $\\sqrt$(T)) + d/(epsilon T) + d/($epsilon^{2}$ T $\\sqrt$(T))) in the nonconvex case, and that this dependence is the reason small epsilon values slow down late-stage training. To fix it, the authors propose replacing the epsilon floor in the adaptive learning rate with a softplus calibration softplus($\\sqrt$(v_t)) = (1/$\\beta$) log(1 + exp($\\beta$ $\\sqrt$(v_t))), giving two new algorithms, Sadam and SAMSGrad. They prove that these methods converge at O($beta^{2}$/$\\sqrt$(T) + d $\\beta$/T + d $beta^{2}$/(T $\\sqrt$(T))), which is faster for the recommended $\\beta$=50 than Adam's epsilon=1e-8, and that they recover SGD's 1/$\\sqrt$(T) rate when $\\beta$ is small. If correct, this gives a principled, provable way to close Adam's generalization gap while keeping adaptivity. The cost is a standing assumption, stated only in the appendix, that the coordinate-wise second moment v_t never decreases.","feed_headline":"Softplus tweak removes Adam's hidden 1/epsilon^2 penalty","feed_subtitle":"Replacing epsilon with a beta-scaled softplus cap yields faster proven rates and better generalization.","key_machinery":"The load-bearing object is the adaptive learning rate itself, written as 1/($\\sqrt$(v_t)+epsilon) in Adam and 1/softplus($\\sqrt$(v_t)) in Sadam, where v_t is the exponential moving average of squared gradients. The proof machinery is a bounded-A-LR lemma: under L-smoothness, bounded gradients, and bounded variance, every coordinate of the A-LR lies in [mu_l, mu_u], with mu_l = 1/($\\sqrt$($sigma^{2}$+$G^{2}$)+epsilon) and mu_u = 1/epsilon for Adam, and analogous bounds (mu_3, mu_4) involving $\\beta$ for Sadam. These bounds are then fed into a telescoping sum over an auxiliary sequence z_t, which turns the per-iteration decrease of f(z_t) into a bound on the average squared gradient norm. The telescoping step uses the inequality (a-b)^2 <= $a^{2}$ - $b^{2}$, which is valid only when the coordinate-wise second moment is nondecreasing, i.e., v_t >= v_{t-1}. That monotonicity is stated in the appendix as a standing condition for all the analyses, even though the Adam and Sadam updates as defined in Algorithms 1 and 2 do not enforce it.","core_discovery":"The central claim is that Adam's convergence behavior is governed by the bounds of its adaptive learning rate, and that those bounds are set by epsilon: Adam has 1/($\\sqrt$(v_{t,j}) + epsilon) between mu_1 = 1/($\\sqrt$($sigma^{2}$+$G^{2}$)+epsilon) and mu_2 = 1/epsilon, so all rate constants scale as 1/$epsilon^{2}$, 1/epsilon, and d/$epsilon^{2}$. The paper further claims that calibrating the adaptive learning rate with softplus, which is lower bounded by (1/$\\beta$) log 2 and grows linearly for large arguments, replaces the 1/epsilon factors by $\\beta$ factors and yields strictly better dependence under nonconvex, non-strongly convex, and Polyak-Lojasiewicz settings. Concretely, Sadam's nonconvex rate is O($beta^{2}$/$\\sqrt$(T) + d $\\beta$/T + d $beta^{2}$/(T $\\sqrt$(T))), versus Adam's O(1/($epsilon^{2}$ $\\sqrt$(T)) + d/(epsilon T) + d/($epsilon^{2}$ T $\\sqrt$(T))). With the recommended $\\beta$=50 and standard epsilon=1e-8, the difference is six orders of magnitude in the leading constant. The authors' experiments on MNIST, CIFAR-10, CIFAR-100, and LSTM language models are offered as evidence that Sadam and SAMSGrad close the generalization gap between Adam and S-Momentum.","pith_inferences":["The monotonicity gap suggests a clean testable repair: if Adam and Sadam were run with the AMSGrad-style max operation v_t = max(v_{t-1}, tilde v_t), the theorems would hold as stated; the paper's SAMSGrad already does this, which may be why its theory is cleanest.","One could extend the calibration to other monotone activation functions; the bounded-A-LR lemma only needs a positive monotone map, so sigmoid, ELU, or tanh variants would likely enjoy the same rate form with different beta-like constants.","The anisotropic A-LR plots hint that the per-coordinate spread, not just its max, may predict generalization; a direct test would be measuring the interquartile range of 1/softplus(sqrt v_t) against final test accuracy across beta values.","If epsilon-dependence transfers to other adaptive methods such as Yogi or AdaBound, then their reported improvements may be partly explained by how they constrain the A-LR range rather than by their specific update rules."],"forward_implications":["Adam's convergence guarantees are incomplete without epsilon: the rate degrades as 1/epsilon^2, so the common choice epsilon = 1e-8 carries a huge hidden constant.","The softplus calibration gives Sadam and SAMSGrad rates that replace 1/epsilon with beta, and with beta=50 they are provably faster than Adam under the same assumptions.","When beta is small, Sadam recovers the SGD convergence rate O(1/sqrt(T)) in nonconvex and non-strongly convex settings, matching S-Momentum in T-dependence.","Under the Polyak-Lojasiewicz condition, both Adam and Sadam converge linearly at rate (1 - 2 lambda mu/T^2)^T, with mu depending on epsilon or beta respectively.","Because the calibration applies to any adaptive gradient method, combining softplus with AMSGrad (SAMSGrad) yields the same improved rates and better empirical test accuracy."],"supporting_citations":[{"why":"Introduces AMSGrad and the v_t >= v_{t-1} monotone second-moment condition that the paper's proofs explicitly assume.","marker":"(Reddi et al., 2018)"},{"why":"Prior nonconvex convergence analysis of Adam-type methods with O(log T / sqrt(T)), the rate the paper claims to improve on.","marker":"(Chen et al., 2018)"},{"why":"Yogi; documents empirical sensitivity of adaptive methods to epsilon and provides nonconvex adaptive-method analysis that the paper extends.","marker":"(Zaheer et al., 2018)"},{"why":"Defines Adam, the base algorithm whose adaptive learning rate and epsilon dependence are studied.","marker":"(Kingma and Ba, 2014)"},{"why":"Introduces Adagrad, the first adaptive gradient method whose A-LR framework the paper generalizes.","marker":"(Duchi et al., 2011)"},{"why":"Proves SGD's O(1/sqrt(T)) nonconvex rate, which the paper claims to recover.","marker":"(Ghadimi and Lan, 2013)"},{"why":"AdaBound; prior effort to close the generalization gap by constraining the A-LR, which the paper reframes as calibration.","marker":"(Luo et al., 2019)"},{"why":"PAdam/PAMSGrad; proposes partial adaptive learning rate and serves as a comparison baseline.","marker":"(Chen and Gu, 2018)"},{"why":"Extends nonconvex analysis to adaptive methods with element-wise bounds, a proof strategy the paper positions against.","marker":"(Zhou et al., 2018)"}],"fun_headline_variants":["Softplus calibration removes Adam's epsilon rate penalty","Adam's convergence gets a softplus boost over epsilon","Softplus adaptive rate: Adam's epsilon dependence broken","Replace epsilon with softplus for faster Adam training","Softplus-based Adam converges faster than classic Adam"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every convergence proof in the paper assumes the coordinate-wise second moment never decreases, v_t >= v_{t-1}, a condition stated only in the appendix and not enforced by Adam or Sadam as defined in Algorithms 1 and 2.","fun_headline_variants_meta":{"raw":{"variants":["Softplus calibration removes Adam's epsilon rate penalty","Adam's convergence gets a softplus boost over epsilon","Softplus adaptive rate: Adam's epsilon dependence broken","Replace epsilon with softplus for faster Adam training","Softplus-based Adam converges faster than classic Adam"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000261,"raw_usage":{"total_tokens":1670,"prompt_tokens":1096,"completion_tokens":574,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":712,"completion_tokens_details":{"reasoning_tokens":501}},"tokens_in":712,"tokens_out":574,"duration_ms":6111,"temperature":1.0,"reasoning_tokens":501,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:37:22.706834+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Adam or Sadam exactly as defined in Algorithms 1 and 2 on a small smooth problem whose coordinate-wise v_t is strictly decreasing at some iterations, then check whether the claimed gradient-norm bound of Theorem 2 or Theorem 9 holds; a single instance where the bound is violated while all assumptions of Assumption 1 hold would show the theorems are not valid without the monotonicity condition. More directly, compute the term sum_j [(1/($\\sqrt$(v_{t-1,j})+epsilon))^2 - (1/($\\sqrt$(v_{t,j})+epsilon))^2]; whenever v_t < v_{t-1} this difference is negative, and the paper's Lemma 23 step (a-b)^2 <= $a^{2}$ - $b^{2}$ no longer applies.","supporting_citations":[],"review_version":1}