Pith. sign in

REVIEW 1 major objections 5 minor 36 references

Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothness

T0 review · 1 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Gradient descent provably reaches a stationary point on feedforward neural networks of any width and depth, under only mild smoothness assumptions on the activation and loss functions.

desk verdict Genuinely new global smoothness framework for GD on deep nets, but the descent lemma as stated only works for cJ≤2; the fix is simple. read the letter →

arxiv 2608.11479 v1 pith:MS2YTS53 submitted 2026-08-11 cs.LG

Siqiao Mu , Diego Klabjan This is my paper · ORCID
classification cs.LG MSC 68T0790C2665K05
keywords gradientdescentconvergenceguaranteesfeedforwardneuralnetworksgeneralizedLipschitzsmoothnessdoublepolynomialnonconvexoptimizationstationarypointparameternormgrowth
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proves convergence guarantees for vanilla gradient descent on feedforward neural networks of arbitrary width and depth, with no requirements on initialization, dataset, or bounded iterates. The assumptions are that the activation function is Lipschitz smooth, Lipschitz continuous, and linearly bounded (true for linear, tanh, softplus, and sigmoid), and that the loss is Lipschitz smooth in the model outputs (true for mean-squared error). The key new tool is a 'double polynomial smoothness' condition: the difference in loss gradients at two parameter points is bounded by the parameter difference times polynomials of the norms of both points. From this, the authors prove a descent lemma and control the sublinear growth of parameter norms, yielding a convergence rate of O(1/$T^{{1/L}}$) for the minimum squared gradient norm after T steps.

What carries the argument

The central object is the double polynomial smoothness condition and the descent lemma built from it. For any two points W1 and W2, Lemma 4.5 bounds the gradient difference ||∇L(W2)-∇L(W1)|| by 8 c_J d_max $L^{3}$ ||W2-W1|| (1+L(W1)^{1/2}) times the product of polynomial sums in ||W1|| and ||W2||. Lemma 4.6 integrates this bound along the segment and turns the polynomial in ||W2|| into higher powers of ||W2-W1||, so that a single gradient descent step with learning rate (12) decreases the loss by at least η_t/(2L) ||∇L(W_t)||^2. The remaining work is showing that the norms ||W_t|| grow only sublinearly, at rate Θ($T^{{(L-1)/L}}$), which makes the cumulative learning rate sum diverge as Θ($T^{{1/L}}$).

What would settle it

To test the descent lemma, choose a two-layer network with a linear activation and a loss J(u) = c $u^{2}$ with c > 2, initialize at a point with L(W0) large, and compute η_0 ||∇L(W0)|| under the learning rate (12); if that product exceeds 1/C, inequality (17) fails and the loss can increase at the first step, contradicting the claimed descent for all c_J ≥ 1.

Watch

Extended reading notes

Core claim

The paper's central claim is Theorem 4.8: after T steps of gradient descent with the step size given by equation (12), the minimum squared gradient norm of an L-layer neural network satisfies min_{t<T} ||∇L(W_t)||^2 = O(1/$T^{{1/L}}$), where the hidden constant is polynomial in c_J, d_max, L(W0), and ||W0|| and exponential in L. The discovery is that the Lipschitz smoothness of the activation functions is partially preserved under repeated composition, giving a global double polynomial smoothness bound for the model function and the loss (Lemmas 4.3–4.5). This replaces the global Lipschitz constant that neural networks lack, and the resulting descent lemma (Lemma 4.7) guarantees a loss decrease at every step provided the learning rate is small relative to the current parameter norm and loss value.

Load-bearing premise

The proof of the descent step requires the loss's smoothness constant c_J to be no larger than 2 for the stated learning-rate schedule to guarantee a decrease, whereas the theorem claims convergence for all c_J ≥ 1; this gap is the load-bearing assumption.

Editorial extensions

If this is right

  • For any L-layer network with tanh, sigmoid, softplus, or linear activations and squared-error loss, gradient descent with the stated step size reaches a stationary point without width, initialization, or dataset assumptions.
  • The squared-gradient-norm convergence rate is O(1/T^{1/L}), so deeper networks converge more slowly; a two-layer network achieves O(1/T^{1/2}) and an L-layer network O(1/T^{1/L}).
  • The analysis covers the feature-learning regime: parameters may move far from initialization, since the proof does not require iterates to remain in a bounded set.
  • The result holds for any loss that is c_J-Lipschitz smooth in the model outputs, which includes mean-squared error and other smooth losses under the assumptions of the theorem.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because double polynomial smoothness holds globally for any two points, the descent lemma should adapt to stochastic gradient descent or coordinate descent, with the sublinear norm-growth argument controlling the extra randomness.
  • The O(1/T^{1/L}) rate suggests a natural lower-bound conjecture: if tight, the depth of a network is an intrinsic hardness parameter for first-order optimization, which would give a theoretical reason to prefer adaptive or second-order methods for very deep architectures.
  • A direct empirical check of the framework is to run the step-size rule (12) on a small tanh or sigmoid network and verify both the predicted power-law decay of the squared gradient norm and the Θ(T^{(L-1)/L}) growth of parameter norms.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper presents a theoretical analysis of vanilla gradient descent on feedforward neural networks of arbitrary width and depth. Under assumptions that the activation is Lipschitz continuous, Lipschitz smooth, and linearly bounded (satisfied by tanh, sigmoid, softplus, and linear activations) and that the per-sample loss is nonnegative and Lipschitz smooth in the model output (satisfied by mean-squared error), the authors derive a generalized smoothness condition they call double polynomial smoothness. This condition bounds the change in the loss gradient by the parameter change multiplied by polynomial terms in the parameter norms at both endpoints. Using this condition, they prove a descent lemma (Lemma 4.7) and then bound the growth of the parameter norms to show that the minimum squared gradient norm converges to zero at rate O(1/T^{1/L}) for an L-layer network (Theorem 4.8). The claimed contribution is that convergence to a stationary point is established without overparameterization, special initialization, bounded iterates, or restrictive datasets, and the analysis covers the feature-learning regime.

Significance. If the main theorem is correct, this is a significant theoretical contribution: it provides the first convergence guarantee for vanilla gradient descent on general feedforward networks that does not rely on overparameterization, special initialization, or staying in a bounded region, and it explicitly covers the feature-learning regime where iterates move far from initialization. The proof strategy is self-contained and does not fit constants or assume the conclusion; the recursive Lipschitz bounds (Lemmas 4.3-4.6) are carefully derived and give explicit polynomial dependence on the network width and exponential dependence on depth. The framework of double polynomial smoothness is a new and plausible way to characterize neural-network loss landscapes. However, the stated scope of Theorem 4.8 is not fully justified because the descent lemma is only proven for losses with cJ ≤ 2, while Assumption 3.4 allows any cJ ≥ 1; this gap is repairable but currently blocks the main claim as stated.

major comments (1)
  1. [Appendix A.7, inequality (17)] The descent proof of Lemma 4.7 requires the bound η_t ∥∇L(W_t)∥ ≤ 1/C to control the higher-order terms in the descent inequality. Combining the authors' bound on ∥∇L(W)∥ (which follows from (14) and (16)) with the learning rate (12), the best uniform bound on this product is √(2c_J)/(2C) · sup_{L≥0} √L/(1+√L) = √(2c_J)/(2C). This is at most 1/C only when c_J ≤ 2. Since Assumption 3.4 only states c_J ≥ 1, the learning rate (12) is not proven to produce a descent step for losses with c_J > 2, e.g., J_i(u) = (c_J/2)(u−y_i)^2 with c_J = 3. Consequently, Theorem 4.8, which is stated for all losses satisfying Assumption 3.4, is not established for its full scope. This is repairable by multiplying ρ in (12) by a factor of √(c_J/2) for c_J > 2, or by restricting Assumption 3.4 to c_J ≤ 2 (which still covers mean-squared error with c_J = 2). The proof and the theorem statement must be adjusted accordingly.
minor comments (5)
  1. [Lemma 4.1] The statement of Lemma 4.1 has the second term as |F(x_i;W_1) − y_i|, but the proof in Appendix A.1 actually bounds this term by |J'_i(F(x_i;W_1))|. These are not equal for general losses satisfying Assumption 3.4; the proof's version is the correct one. Please correct the lemma statement to use |J'_i(F(x_i;W_1))|, or add a remark explaining the discrepancy (e.g., for mean-squared error, |J'| = 2|F−y|).
  2. [Proof of Lemma 4.7] The display for the γ = 2 case is garbled in the typesetting; the intended bound is η_t^2 ∥∇L∥^2 · C/2 · (1+L^{1/2}) Σ ∥W_t∥^i ≤ η_t · (C/(2ρ)) ∥∇L∥^2 ≤ η_t/(2L) ∥∇L∥^2. Please rewrite this step for clarity.
  3. [Theorem 4.8] The notation min_{t=0,...,T−1} ∥∇L(W_t)∥^2 is slightly ambiguous; please use min_{0≤t≤T−1} or an equivalent notation.
  4. [Introduction] The phrase 'fully characterizes feedforward neural networks' is too strong; the paper establishes an upper bound on the gradient norm under specific smoothness assumptions, not a full characterization of the loss landscape. Consider softening this wording.
  5. [Assumption 3.4] The assumption c_J ≥ 1 is stated 'for convenience', but the proof of Lemma 4.5 uses the inequality |J'(u)|^2 ≤ 2 c_J J(u) from [16, Lemma 2.28], which holds for any c_J > 0. Either remove the lower bound or note that the results extend to c_J > 0 with minor constant changes.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the convergence proof is self-contained, and the sole self-citation is used only as an inspiration, not as a load-bearing black box.

full rationale

The paper's derivation chain is self-contained. Assumptions 3.1-3.4 state regularity properties of the activation and loss functions, and every subsequent bound is proved from these assumptions: Lemmas 4.3 and 4.4 are obtained by explicit recursions on z_l, h_l, and delta_l in Appendix A.9, Lemma 4.5 combines them with Lemma 4.1, and Lemma 4.7 substitutes the learning rate (12) into the descent Lemma 4.6 and controls the higher-order terms. No parameter is fitted to data, no target convergence rate is assumed, and no external uniqueness or existence theorem is imported to force the chosen framework. The only self-citation, [23], appears in Related Work as 'we build on the convergence proof structure of the Low-Rank Adaptation (LoRA) algorithm'; the actual LoRA result is not used as a black box, and all lemmas used here are proved in the appendix, so this citation is not load-bearing. The noted issue in Appendix A.7 that inequality (17) yields descent only when cJ <= 2 is a correctness or scoping gap in the proof of Lemma 4.7, not a circularity: Assumption 3.4 does not contain the conclusion, and the gap is repairable by rescaling the learning rate by a cJ-dependent factor. Thus the central claim does not reduce to its own inputs by definition.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

No data-fitting constants appear; the analysis is parameter-free in the sense of optimization theory except for the hand-chosen step-size denominator rho. The proof also relies on an unstated cJ <= 2 condition, listed above as ad hoc. No new physical or model entities are introduced.

free parameters (1)
  • Step-size denominator rho = 2^(2L+3) cJ dmax^(3/2) L^5
    Hand-chosen to make the descent terms in Lemma 4.7 small; not fitted to data, but the proof's validity depends on this choice and on the cJ <= 2 condition.
assumptions (5)
  • domain assumption Activation sigma is C^1, linearly bounded, Lipschitz continuous with constant c2 <= 1, and Lipschitz smooth with constant c3 <= 1 (Assumption 3.1).
    Core premise; all recursive norm bounds in Lemmas A.1-A.6 use these constants, and the proof fails for activations such as ReLU that are not smooth.
  • domain assumption Data vectors satisfy ||x_i|| <= cx d^(1/2) with cx <= 1 (Assumptions 3.2 and 3.3, normalized for convenience).
    Used throughout the norm-bounding lemmas; the paper notes normalization is a convenience and not essential, so this is a mild domain assumption.
  • domain assumption Individual losses J_i are nonnegative and cJ-smooth in the model outputs (Assumption 3.4).
    Needed for Lemma 4.1 and for the gradient norm bound; mean-squared error satisfies it with cJ = 2.
  • standard math Standard calculus tools: fundamental theorem of calculus, binomial theorem, Cauchy-Schwarz, Lp norm monotonicity, and the inequality |J'|^2 <= 2 cJ J cited from Lemma 2.28 of [16].
    Unproved background used in the descent lemma and gradient bounds; standard and acceptable.
  • ad hoc to paper The step-size proof implicitly assumes cJ <= 2 (Appendix A.7, inequality (17)).
    Unstated and load-bearing; Theorem 4.8's generality over cJ is not supported by the given proof unless cJ is restricted or the step size is modified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothness." pith.science (2026). https://pith.science/paper/MS2YTS53

@misc{pith2026260811479,
  author       = {Pith},
  title        = {Pith review of: Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MS2YTS53}},
  note         = {Machine review of arXiv:2608.11479}
}
abstract

We establish convergence guarantees of gradient descent for general feedforward neural networks of arbitrary width or depth, with no special requirements on the initialization or dataset. We only assume that the activation functions are Lipschitz smooth, Lipschitz continuous, and linearly bounded--- properties that hold for linear, tanh, softplus, and sigmoid activation functions. For the loss function, we require that it is Lipschitz smooth in the model outputs, which is true for mean-squared error. The key theoretical insight is that the Lipschitz properties of the activation functions are partially preserved even through repeated compositions, leading to a novel generalized Lipschitz smoothness condition where the change in gradient is upper bounded by the change in the parameter space, multiplied by polynomial terms of the parameter norms at both endpoints. This type of condition holds for both the model function and the loss function, enabling a descent lemma where the loss decreases as long as the learning rate is small enough with respect to the parameter norms. By ensuring that the parameter norms do not grow too quickly to infinity, we prove that the minimum squared gradient norm converges to zero in $T$ iterations at rate $O(1/T^{1/L})$ for an $L$-layer neural network.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 28 canonical work pages

  1. [23]

    Siqiao Mu and Diego Klabjan.On the Convergence Rate of LoRA Gradient Descent. 2026. arXiv: 2512.18248 [cs.LG].url:https://arxiv.org/abs/2512.18248

    arXiv 2026

  2. [1]

    Foivos Alimisis, Rustem Islamov, and Aurelien Lucchi.Why Do We Need Warm-up? A Theoreti- cal Perspective. 2026. arXiv:2510.03164 [cs.LG].url:https://arxiv.org/abs/2510.03164

    arXiv 2026

  3. [2]

    A Convergence Theory for Deep Learning via Over-Parameterization

    Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. “A Convergence Theory for Deep Learning via Over-Parameterization”. In:Proceedings of the 36th International Conference on Machine Learning. Ed. by Kamalika Chaudhuri and Ruslan Salakhutdinov. Vol. 97. Proceedings of Ma- chine Learning Research. PMLR, 2019, pp. 242–252.url:https://proceedings.mlr.press/ v97/all...

    work page 2019

  4. [3]

    A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks

    Sanjeev Arora et al. “A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks”. In:International Conference on Learning Representations. 2019.url:https : / / openreview.net/forum?id=SkMQg3C5K7. 7

    work page 2019

  5. [4]

    On exact computation with an infinitely wide neural net

    Sanjeev Arora et al. “On exact computation with an infinitely wide neural net”. In:Proceedings of the 33rd International Conference on Neural Information Processing Systems. Red Hook, NY, USA: Curran Associates Inc., 2019

    work page 2019

  6. [5]

    Training Infinitely Deep and Wide Transformers

    Rapha¨ el Barboni et al. “Training Infinitely Deep and Wide Transformers”. In: (2026). arXiv: 2605.17660 [math.OC].url:https://arxiv.org/abs/2605.17660

    work page Pith review arXiv 2026

  7. [6]

    A Descent Lemma Beyond Lipschitz Gradient Continuity: First-Order Methods Revisited and Applications

    Heinz H. Bauschke, J´ erˆ ome Bolte, and Marc Teboulle. “A Descent Lemma Beyond Lipschitz Gradient Continuity: First-Order Methods Revisited and Applications”. In:Math. Oper. Res. 42.2 (May 2017), 330–348.issn: 0364-765X.doi:10 . 1287 / moor . 2016 . 0817.url:https : //doi.org/10.1287/moor.2016.0817

    arXiv 2017

  8. [7]

    Non-Uniform Smoothness for Gradient Descent

    Albert S. Berahas, Lindon Roberts, and Fred Roosta. “Non-Uniform Smoothness for Gradient Descent”. In:Transactions on Machine Learning Research(2024).issn: 2835-8856.url:https: //openreview.net/forum?id=17ESEjETbP

    work page 2024

Show all 36 references
  1. [8]

    Sourav Chatterjee.Convergence of gradient descent for deep neural networks. 2026. arXiv:2203. 16462 [cs.LG].url:https://arxiv.org/abs/2203.16462

    2026

  2. [9]

    A Non-local Convergence Analysis of Gradient Flow for Deep Linear Networks

    Kun Chen, Dachao Lin, and Zhihua Zhang. “A Non-local Convergence Analysis of Gradient Flow for Deep Linear Networks”. In:IEEE Transactions on Pattern Analysis and Machine Intelligence (2026), pp. 1–12.doi:10.1109/TPAMI.2026.3704447

    2026

  3. [10]

    On lazy training in differentiable program- ming

    L´ ena¨ ıc Chizat, Edouard Oyallon, and Francis Bach. “On lazy training in differentiable program- ming”. In:Proceedings of the 33rd International Conference on Neural Information Processing Systems. Red Hook, NY, USA: Curran Associates Inc., 2019

    2019

  4. [11]

    Infinite-width limit of deep linear neural networks

    L´ ena¨ ıc Chizat et al. “Infinite-width limit of deep linear neural networks”. In:Communications on Pure and Applied Mathematics77.10 (2024), pp. 3958–4007.doi:https://doi.org/10. 1002/cpa.22200. eprint:https://onlinelibrary.wiley.com/doi/pdf/10.1002/cpa.22200. url:https://on...

    2024 doi

  5. [12]

    Convergence of the Gradient Flow for Shallow ReLU Networks on Weakly Interacting Data

    L´ eo Dana, Loucas Pillaud-Vivien, and Francis Bach. “Convergence of the Gradient Flow for Shallow ReLU Networks on Weakly Interacting Data”. In:The Thirty-ninth Annual Conference on Neural Information Processing Systems. 2025.url:https://openreview.net/forum?id= dkXDyM66ce

    2025

  6. [13]

    Gradient Descent Provably Optimizes Over-parameterized Neural Networks

    Simon S. Du et al. “Gradient Descent Provably Optimizes Over-parameterized Neural Networks”. In:International Conference on Learning Representations. 2019.url:https://openreview. net/forum?id=S1eK3i09YQ

    2019

  7. [14]

    Taming Nonconvex Stochastic Mirror Descent with General Breg- man Divergence

    Ilyas Fatkhullin and Niao He. “Taming Nonconvex Stochastic Mirror Descent with General Breg- man Divergence”. In:Proceedings of The 27th International Conference on Artificial Intelligence and Statistics. Ed. by Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li. Vol. 238. Procee...

    2024

  8. [15]

    Glocal Smoothness: Line search and adaptive sizes can help in theory too!

    Curtis Fox et al. “Glocal Smoothness: Line search and adaptive sizes can help in theory too!” In:Transactions on Machine Learning Research(2026). Expert Certification.issn: 2835-8856. url:https://openreview.net/forum?id=be9PdukwEL

    2026

  9. [16]

    Gower.Handbook of Convergence Theorems for (Stochastic) Gradient Methods

    Guillaume Garrigos and Robert M. Gower.Handbook of Convergence Theorems for (Stochastic) Gradient Methods. 2024. arXiv:2301.11235 [math.OC].url:https://arxiv.org/abs/2301. 11235

    2024 arXiv

  10. [17]

    Neural tangent kernel: convergence and generalization in neural networks

    Arthur Jacot, Franck Gabriel, and Cl´ ement Hongler. “Neural tangent kernel: convergence and generalization in neural networks”. In:Proceedings of the 32nd International Conference on Neural Information Processing Systems. NIPS’18. Montr´ eal, Canada: Curran Associates Inc., 2...

    2018

  11. [18]

    Convex and Non-convex Optimization Under Generalized Smoothness

    Haochuan Li et al. “Convex and Non-convex Optimization Under Generalized Smoothness”. In:Thirty-seventh Conference on Neural Information Processing Systems. 2023.url:https : //openreview.net/forum?id=8aunGrXdkl

    2023

  12. [19]

    Learning overparameterized neural networks via stochastic gra- dient descent on structured data

    Yuanzhi Li and Yingyu Liang. “Learning overparameterized neural networks via stochastic gra- dient descent on structured data”. In:Proceedings of the 32nd International Conference on Neu- ral Information Processing Systems. NIPS’18. Montr´ eal, Canada: Curran Associates Inc., ...

    2018

  13. [20]

    Relatively Smooth Convex Optimization by First-Order Methods, and Applications

    Haihao Lu, Robert M. Freund, and Yurii Nesterov. “Relatively Smooth Convex Optimization by First-Order Methods, and Applications”. In:SIAM Journal on Optimization28.1 (2018), pp. 333–354.doi:10.1137/16M1099546. eprint:https://doi.org/10.1137/16M1099546.url: https://doi.org/10....

    2018 doi

  14. [21]

    Leveraging Non-uniformity in First-order Non-convex Optimization

    Jincheng Mei et al. “Leveraging Non-uniformity in First-order Non-convex Optimization”. In: Proceedings of the 38th International Conference on Machine Learning. Ed. by Marina Meila and Tong Zhang. Vol. 139. Proceedings of Machine Learning Research. PMLR, 2021, pp. 7555–7564. ...

    2021

  15. [22]

    Directional Smoothness and Gradient Methods: Convergence and Adaptiv- ity

    Aaron Mishkin et al. “Directional Smoothness and Gradient Methods: Convergence and Adaptiv- ity”. In:Advances in Neural Information Processing Systems. Ed. by A. Globerson et al. Vol. 37. Curran Associates, Inc., 2024, pp. 14810–14848.doi:10.52202/079017-0473.url:https:// proc...

    2024 doi

  16. [24]

    Dang Qua Nguyen and Taejoon Kim.On the Stability of Approximate Message Passing with Independent Measurement Ensembles. 2023. arXiv:2305.08754 [cs.IT].url:https://arxiv. org/abs/2305.08754

    2023 arXiv

  17. [25]

    Ruoyu Sun.Optimization for deep learning: theory and algorithms. 2019. arXiv:1912 . 08957 [cs.LG].url:https://arxiv.org/abs/1912.08957

    2019 arXiv

  18. [26]

    Hossein Taheri and Christos Thrampoulidis.Fast Convergence in Learning Two-Layer Neural Networks with Separable Data. 2023. arXiv:2305.13471 [cs.LG].url:https://arxiv.org/ abs/2305.13471

    2023 arXiv

  19. [27]

    Sharan Vaswani, Yifan Sun, and Reza Babanezhad.Convergence of Steepest Descent and Adam under Non-Uniform Smoothness. 2026. arXiv:2605.30648 [cs.LG].url:https://arxiv.org/ abs/2605.30648

    2026 arXiv

  20. [28]

    Empirical Limitations of the NTK for Understanding Scaling Laws in Deep Learning

    Nikhil Vyas, Yamini Bansal, and Preetum Nakkiran. “Empirical Limitations of the NTK for Understanding Scaling Laws in Deep Learning”. In:Transactions on Machine Learning Research (2023).issn: 2835-8856.url:https://openreview.net/forum?id=Y3saBb7mCE

    2023

  21. [29]

    Accelerating Rescaled Gradient Descent: Fast Optimization of Smooth Functions

    Ashia Wilson, Lester Mackey, and Andre Wibisono. “Accelerating Rescaled Gradient Descent: Fast Optimization of Smooth Functions”. In:Advances in Neural Information Processing Sys- tems. Ed. by H. Wallach et al. Vol. 32. Curran Associates, Inc., 2019.url:https://proceedings. ne...

    2019

  22. [30]

    Kernel and Rich Regimes in Overparametrized Models

    Blake Woodworth et al. “Kernel and Rich Regimes in Overparametrized Models”. In:Proceedings of Thirty Third Conference on Learning Theory. Ed. by Jacob Abernethy and Shivani Agarwal. Vol. 125. Proceedings of Machine Learning Research. PMLR, 2020, pp. 3635–3673.url:https: //pro...

    2020

  23. [31]

    How Over-Parameterization Slows Down Gradient Descent in Matrix Sensing: The Curses of Symmetry and Initialization

    Nuoya Xiong, Lijun Ding, and Simon Shaolei Du. “How Over-Parameterization Slows Down Gradient Descent in Matrix Sensing: The Curses of Symmetry and Initialization”. In:The Twelfth International Conference on Learning Representations. 2024.url:https://openreview.net/ forum?id=x...

    2024

  24. [32]

    Linear Convergence of Gradient Descent For Finite Width Over-parametrized Linear Networks With General Initialization

    Ziqing Xu et al. “Linear Convergence of Gradient Descent For Finite Width Over-parametrized Linear Networks With General Initialization”. In:Proceedings of The 26th International Confer- ence on Artificial Intelligence and Statistics. Ed. by Francisco Ruiz, Jennifer Dy, and Ja...

    2023

  25. [33]

    Hu.Feature Learning in Infinite-Width Neural Networks

    Greg Yang and Edward J. Hu.Feature Learning in Infinite-Width Neural Networks. 2022. arXiv: 2011.14522 [cs.LG].url:https://arxiv.org/abs/2011.14522

    2022 arXiv

  26. [34]

    Adaptive Gradient Normalization and Independent Sampling for (Stochas- tic) Generalized-Smooth Optimization

    Yufeng Yang et al. “Adaptive Gradient Normalization and Independent Sampling for (Stochas- tic) Generalized-Smooth Optimization”. In:Transactions on Machine Learning Research(2025). issn: 2835-8856.url:https://openreview.net/forum?id=KKSQQMlEfw. 9

    2025

  27. [35]

    On the Power and Limitations of Random Features for Un- derstanding Neural Networks

    Gilad Yehudai and Ohad Shamir. “On the Power and Limitations of Random Features for Un- derstanding Neural Networks”. In:Advances in Neural Information Processing Systems. Ed. by H. Wallach et al. Vol. 32. Curran Associates, Inc., 2019.url:https://proceedings.neurips. cc/paper...

    2019

  28. [36]

    Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity

    Jingzhao Zhang et al. “Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity”. In:International Conference on Learning Representations. 2020.url:https: //openreview.net/forum?id=BJgnXpVYwS. 10 A Proofs A.1 Proof of Lemma 4.1 Proof.We have∇L(W) ...

    2020

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.