Pith. sign in

REVIEW 2 major objections 2 minor 7 references

Under an assumption where smoothness is an affine function of the objective value, steepest descent, RMSProp and Adam achieve linear convergence for logistic regression, policy gradients and some neural networks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-29 08:17 UTC pith:PFIBFCSZ

load-bearing objection The affine-curvature smoothness assumption is the real novelty, but its verification on two-layer nets is the part that needs the closest look before the Adam linear rates can be taken as settled. the 2 major comments →

arxiv 2605.30648 v1 pith:PFIBFCSZ submitted 2026-05-28 cs.LG math.OC

Convergence of Steepest Descent and Adam under Non-Uniform Smoothness

Sharan Vaswani , Yifan Sun , Reza Babanezhad This is my paper
classification cs.LG math.OC
keywords non-uniform smoothnesslinear convergenceAdamRMSProplogistic regressionpolicy gradientgradient dominationseparable data
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes convergence guarantees for first-order methods by introducing a generalized non-uniform smoothness condition under which the gradient's Lipschitz constant varies linearly with the current objective value. This condition holds for logistic regression on separable data, softmax policy gradient objectives, and certain two-layer neural networks, and is paired with gradient domination to yield linear rates. The results show that sign-based gradient descent converges faster than standard gradient descent on logistic regression and policy gradient problems, while constant-step deterministic diagonal RMSProp and Adam also converge linearly on the neural network class. A matching lower bound further separates these methods from AdaGrad, AMSGrad, gradient descent and heavy-ball momentum.

Core claim

Under the assumption that the smoothness parameter is an affine function of the objective value together with gradient domination, the steepest descent method attains a general linear convergence rate. Deterministic diagonal RMSProp and Adam with constant step size and momentum parameter converge linearly on a class of two-layer neural networks trained on separable data. Sign gradient descent converges linearly and at a strictly faster rate than gradient descent for logistic regression on separable data and for the softmax policy gradient objective.

What carries the argument

The generalized non-uniform smoothness assumption stating that the gradient Lipschitz constant is an affine function of the objective value itself.

Load-bearing premise

The objective's curvature must be an affine function of its own value.

What would settle it

Measure the gradient Lipschitz constant at multiple points along the optimization trajectory for logistic regression on separable data and test whether those values lie on a straight line when plotted against the objective value.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Sign GD converges linearly and strictly faster than GD on logistic regression with separable data and on softmax policy gradient.
  • Deterministic diagonal RMSProp and Adam with fixed step size and momentum converge linearly on two-layer neural networks with separable data.
  • Steepest descent obtains a general linear rate under the affine smoothness assumption and gradient domination.
  • RMSProp and Adam are provably faster than AdaGrad, AMSGrad, GD and heavy-ball momentum under the same assumption.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The affine smoothness condition may guide the design of step-size schedules that remain constant rather than decaying for a wider range of losses.
  • Similar curvature-objective relations could be checked empirically on other separable classification problems to predict when adaptive methods will outperform plain gradient descent.
  • The lower bound separating RMSProp/Adam from other first-order methods suggests examining whether the same separation appears in non-separable or multi-layer settings.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper generalizes non-uniform smoothness to the assumption that the smoothness parameter L is an affine function of the objective value itself (L(f) = a + b f). It claims this holds for logistic regression, GLMs with logistic link, softmax policy gradient, and a class of two-layer neural networks on separable data. Under this assumption plus gradient domination, the paper proves linear convergence for steepest descent and for deterministic diagonal RMSProp/Adam (constant step-size and momentum). It further claims sign GD is provably faster than GD for logistic regression and softmax PG, and provides a lower bound showing RMSProp/Adam are faster than AdaGrad, AMSGrad, GD, and heavy-ball under the assumption.

Significance. If the central claims hold, the work would be significant for the analysis of adaptive first-order methods on ML objectives. It supplies linear rates with fixed parameters for diagonal Adam/RMSProp on two-layer networks and a lower bound that separates these methods from non-adaptive and other adaptive baselines; the lower bound and the explicit comparison of sign GD vs. GD are concrete strengths. The results would help explain empirical behavior of Adam on separable or policy-gradient problems under a curvature condition that is more realistic than uniform smoothness.

major comments (2)
  1. [section deriving the assumption for two-layer NNs (following the statement of the generalized non-uniform smoothness)] The generalized assumption (smoothness affine in objective value) is invoked for every linear rate and the lower bound. The verification that this relation holds for the stated class of two-layer neural networks on separable data is load-bearing; the manuscript must supply the explicit derivation (Hessian bound or smoothness-parameter calculation) showing the affine dependence, as any gap here collapses the applicability of all subsequent theorems.
  2. [lower-bound section] The lower bound (final section) separating RMSProp/Adam from AdaGrad, AMSGrad, GD, and heavy-ball is presented under the new assumption. The precise conditions, including whether gradient domination is required and the exact parameter regime, must be stated explicitly so that the lower bound is consistent with the upper bounds derived earlier.
minor comments (2)
  1. [Assumption statement] Notation for the affine coefficients a and b should be introduced once and used consistently when restating the assumption in each theorem.
  2. [main convergence theorems for NNs] The abstract claims the property holds for 'a class of neural networks'; the precise architectural restrictions (width, activation, separability condition) should be restated in the main theorem statements for clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their careful reading of the manuscript and their constructive comments. We appreciate the positive assessment of the significance of the work. Below we address each of the major comments point by point.

read point-by-point responses

  1. Referee: [section deriving the assumption for two-layer NNs (following the statement of the generalized non-uniform smoothness)] The generalized assumption (smoothness affine in objective value) is invoked for every linear rate and the lower bound. The verification that this relation holds for the stated class of two-layer neural networks on separable data is load-bearing; the manuscript must supply the explicit derivation (Hessian bound or smoothness-parameter calculation) showing the affine dependence, as any gap here collapses the applicability of all subsequent theorems.

    Authors: We agree that the explicit derivation for the two-layer neural networks is essential. We will add the detailed derivation showing the affine dependence in the revised manuscript. revision: yes

  2. Referee: [lower-bound section] The lower bound (final section) separating RMSProp/Adam from AdaGrad, AMSGrad, GD, and heavy-ball is presented under the new assumption. The precise conditions, including whether gradient domination is required and the exact parameter regime, must be stated explicitly so that the lower bound is consistent with the upper bounds derived earlier.

    Authors: We will revise the lower bound section to explicitly state the precise conditions, including the requirement of gradient domination and the exact parameter regime, to ensure consistency with the upper bounds. revision: yes

Circularity Check

0 steps flagged

No significant circularity; rates derived from explicitly stated assumptions

full rationale

The paper introduces the affine non-uniform smoothness condition L(f) = a + b f as a modeling assumption, verifies it holds for logistic regression, softmax PG and certain two-layer NNs on separable data, and then derives linear convergence rates for steepest descent, diagonal RMSProp and Adam under that condition plus gradient domination. All central claims are obtained by direct mathematical manipulation of the stated premises; no parameter is fitted to data and then relabeled a prediction, no result is obtained solely by self-citation, and no step reduces by construction to its own input. The derivation chain is therefore self-contained against the given axioms.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claims rest on the new smoothness assumption being valid for the listed problem classes and on gradient-domination conditions; no free parameters or invented entities are introduced in the abstract.

axioms (2)
  • domain assumption Curvature of the objective is an affine function of the objective value
    Generalized non-uniform smoothness assumption used to derive all stated rates
  • domain assumption Gradient domination condition holds
    Required alongside the smoothness assumption for linear convergence

pith-pipeline@v0.9.1-grok · 5724 in / 1213 out tokens · 24812 ms · 2026-06-29T08:17:08.372470+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Convergence of Steepest Descent and Adam under Non-Uniform Smoothness." pith.science (2026). https://pith.science/paper/PFIBFCSZ

@misc{pith2026260530648,
  author       = {Pith},
  title        = {Pith review of: Convergence of Steepest Descent and Adam under Non-Uniform Smoothness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PFIBFCSZ}},
  note         = {Machine review of arXiv:2605.30648}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recent work has analyzed the convergence of first-order methods under non-uniform smoothness assumptions that better model the loss landscape in machine learning tasks. We generalize this assumption to objectives whose curvature is an affine function of the objective value. This property is satisfied by a broad class of problems, including logistic regression, generalized linear models with a logistic link function, softmax policy gradient in reinforcement learning, and a class of neural networks. Under this assumption and gradient domination conditions, we establish a general convergence rate for the steepest descent method, and deterministic, diagonal variants of RMSProp and Adam. Our results imply that for logistic regression on separable data and the softmax policy gradient objective, sign GD converges linearly and is provably faster than GD. Furthermore, we show that for a class of two-layer neural networks on separable data, RMSProp and Adam can converge at a linear rate with a constant step-size and momentum parameter. Finally, we present a lower bound demonstrating that, under our assumption, RMSProp and Adam are provably faster than AdaGrad, AMSGrad, gradient descent, and heavy-ball momentum.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

7 extracted references · 1 canonical work pages

  1. [1]

    Bernstein, J., Wang, Y .-X., Azizzadenesheli, K., and Anand- kumar, A

    PMLR, 2023. Bernstein, J., Wang, Y .-X., Azizzadenesheli, K., and Anand- kumar, A. signsgd: Compressed optimisation for non- convex problems. InInternational conference on machine learning, pp. 560–569. PMLR, 2018. Boyd, S. and Vandenberghe, L.Convex optimization. Cam- bridge university press, 2004. Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Pham...

    2023

  2. [2]

    Kingma, D

    Springer, 2016. Kingma, D. P. and Ba, J. Adam: A method for stochastic op- timization. In3rd International Conference on Learning Representations, ICLR, 2015. Li, H., Qian, J., Tian, Y ., Rakhlin, A., and Jadbabaie, A. Convex and non-convex optimization under generalized smoothness.Advances in Neural Information Processing Systems, 36:40238–40271, 2023a. ...

    work page arXiv 2016

  3. [3]

    However, unlike Thm

    and convex functions. However, unlike Thm. 1, these papers do not derive dimension-free guarantees for general normalized steepest descent. Furthermore, under this assumption, Li et al. (2023b); Wang et al. (2024a;b) analyze the scalar or norm version of Adam and derive a O(1/ϵ2) stationary-point convergence for general non-convex functions. In Thm. 5, we...

  4. [4]

    1 n * − nX i=1 exp(−yi⟨xi, θ⟩)yi xi, u +# ≥

    For example, Mei et al. (2021) use normalized gradient descent for the softmax policy gradient objective, Taheri & Thrampoulidis (2023) use it for 2 layer neural networks and Axiotis & Sviridenko (2023) use greedy coordinate descent (corresponding top= 1, q=∞in our Theorem 1) and analyze its convergence on logistic regression. B. Examples Proposition 1.Co...

    2021

  5. [5]

    iff:R→Rsmooth andg:R n →Rconvex but not smooth, then ∂h(x) =f ′(g(x))·∂g(x)

  6. [6]

    Iff:R n →Rnonsmooth but continuous, andg:R p →R n smooth then ∂h(x) =J(x) T y, u∈∂f(u), u=g(x) whereJ(x) = [∇g 1(x), ...,∇g n(x)]T the Jacobian of the mappingg. Proof. The first statement is given exactly in (Clarke et al., 1998). Both are the result of the following statement, which is follows from the standard definition of lim sup: ConsiderA(t)→aandlim...

    1998

  7. [7]

    and simplifying Eq. (80) in this case, f(θ t+1)≤f(θ t)− η C2 2 ∥∇t∥1 + (2L1 ∥∇t∥1)η 2 (C1 +C 2 3) ≤f(θ t)− η C2 4 ∥∇t∥1 (Sinceη≤¯η 2) =⇒ η C2 4 ∥∇t∥1 ≤f(θ t)−f(θ t+1)(83) Note that from the above we havef(θ t+1)≤f(θ t) for all t≤T which indicates f(θ T )≤f(θ 1). Recursing for T iterations, we have η C2 4 T∥∇ T ∥ ≤ η C2 4 TX t=1 ∥∇t∥1 (Since∥∇ T ∥ ≤ L0 L1 ...

    1998