Pith. sign in

REVIEW 2 major objections 4 minor 50 references

This paper proves a complete classification of all critical points of the squared-loss landscape for shallow neural networks with cubic activation and affine target functions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 23:54 UTC pith:QCIR7JUY

load-bearing objection A genuine extension of the prior landscape classification to cubic activations with a clean width/degree threshold, but the 'complete' classification rests on a few compressed or Mathematica-delegated algebraic steps that should be certified before the paper is fully settled. the 2 major comments →

arxiv 2607.15173 v1 pith:QCIR7JUY submitted 2026-07-16 math.OC

Landscape analysis for shallow neural networks: Complete classification of critical points for cubic activation and affine target functions

Shokhrukh Ibragimov , Ilkhom Mukhammadiev , Diyora Salimova This is my paper
classification math.OC MSC 68T0790C26
keywords landscape analysiscritical pointspolynomial neural networkscubic activationloss landscapenon-global local minimasaddle pointsmonomial activations
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper is trying to establish a complete catalogue of every stationary point of the squared-L2 training loss for one-hidden-layer networks whose activation is x^3 and whose target is a nonconstant affine function ax+b. It claims that the landscape has no local maxima, that global minimizers exist exactly when the hidden layer is at least as wide as the activation degree (for monomial activations generally), and that in the cubic case every critical point is either a global minimizer, a saddle, or a non-global local minimum with one explicitly known cubic realization. The significance is that for this analytically tractable family, the optimization obstructions are finite and explicit: the only bad local minima form a single family, and every other spurious stationary point is a saddle. The paper also proves a sharp width-versus-degree threshold: when the width is smaller than the degree, the infimum is zero but no minimizer exists, and any minimizing sequence must diverge to infinity.

Core claim

On its own terms, the paper's central discovery is Theorem 5.19: for a shallow network with h hidden neurons, activation x^3, and loss L = integral over [a,b] of (N_theta(x) - (mx+c))^2 dx, the zero set of the gradient partitions exactly into global minima (at least three active and visible neurons with pairwise distinct pivots, realization equal to the target), non-global local minima (all neurons active and visible, all pivots equal to the midpoint, exactly one neuron whose slope sign matches the target, realization f((a+b)/2) + (28 f'(a) / (5(b-a)^2)) (x-(a+b)/2)^3), and saddles (all remaining critical points: the constant midpoint realization, the two side cubic realizations with pivot a

What carries the argument

The machinery is an explicit polynomial-system analysis of the stationarity equations. The gradient formulas express each partial derivative as an integral against N_theta - f; after an affine rescaling that maps any affine target and interval to the identity function on [0,1], criticality forces the realization N_theta to lie in a tiny set. The decisive step solves the resulting algebraic system for the case of one common pivot and obtains exactly three cubic realizations with coefficients involving sqrt(7); a following lemma lists all critical realizations. The classification of saddle versus minimum is completed by explicit ascent and descent perturbations, a reduction of arbitrary pertur

Load-bearing premise

The classification collapses if the exhaustive algebraic solution of the one-pivot stationarity system (which asserts exactly three cubic realizations) is incomplete or mis-solved, or if the omitted proof of the perturbation identity fails; that system is the load-bearing step.

What would settle it

Run a polynomial-system solver on the stationarity equations for h=2 or h=3 with activation x^3 on [0,1] and identity target, and look for a real solution not listed as one of the five forms in Lemma 4.8: for example, any critical point with exactly two distinct active pivots, or a realization different from the constant 1/2, the three cubics, and the identity. Finding one would refute the complete classification; a verified exhaustive computation with no other real solutions would support it.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • No local maxima exist, so no trajectory of gradient-based training can converge to a local maximum.
  • Non-global local minima exist only in the all-active, all-pivot-at-midpoint configuration with a single slope matching the target; a network with one hidden neuron that realizes this shape is necessarily a non-global local minimum.
  • Critical points with exactly two distinct pivots do not exist; global minimizers require at least three active, visible, pairwise-distinct pivots.
  • For any monomial degree d, exact representability of a nonconstant affine target requires h at least d; when h is smaller, the infimum is zero but is not attained and minimizing sequences diverge.
  • Saddle points include all critical points with no active neurons or with the constant midpoint realization; such points are never minima.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If this classification extends to higher odd monomial activations, the same pattern would predict that spurious local minima are entirely absent except for finitely many one-parameter families tied to special realizations; this could be tested by solving the analogous stationarity system for degree 5.
  • The explicit saddle realizations have Hessians with both signs of curvature, which suggests that random first-order methods with noise or saddle-escape steps should avoid them; one could verify numerically that gradient descent with isotropic noise near these saddles escapes in the expected directions.
  • The h-at-least-d threshold is reminiscent of an interpolation threshold: if overparameterization is understood as width at least activation degree, then wide-enough cubic networks have no bad local minima at all, since the only non-global minima require the restrictive all-active, pivot-at-midpoint structure.
  • The appearance of sqrt(7) suggests a connection to orthogonal polynomials on the interval: the side realizations are likely the unique cubics that are L2-orthogonal to the affine target in a two-dimensional subspace, a link that could be made explicit by projecting the residual onto the span of (x-p)^k.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper studies the true squared-L2 loss for one-hidden-layer polynomial neural networks with monomial activation x^d, one-dimensional input/output, and a nonconstant affine target f. Theorem 1.1 gives a sharp criterion: inf L = 0, the loss is exactly zero for some parameter vector if and only if h >= d, and in the narrow regime h < d any minimizing sequence diverges. Theorem 1.2, for cubic activation, classifies all critical points: no local maxima; no critical points with exactly two distinct pivots; global minima are exactly the zero-loss critical points with at least three active, visible, pairwise-distinct-pivot neurons; non-global local minima are exactly the parameter vectors with all neurons active and visible, eta = 1, and realization (1.10); all other critical points are saddles described by (1.13). The proof proceeds via explicit gradient/Hessian formulas, Vandermonde determinant arguments, an algebraic resolution of the one-pivot case, and an affine rescaling from the canonical unit-interval/identity-target setting.

Significance. If the classification is complete, the paper is a substantial contribution: it provides one of the first complete, closed-form descriptions of a nonlinear shallow-network loss landscape, including explicit non-global local minima and saddle realizations, together with a sharp width threshold for global minimization. The proof strategy is largely elementary and transparent: explicit derivatives in Section 2, Vandermonde-based representability in Section 3, a careful affine normalization in Lemma 5.14, and explicit perturbation constructions in Section 5. The paper also makes falsifiable structural predictions (e.g., absence of two-pivot critical points, realization formulas (1.10)) that can be checked numerically. The main weakness is that two load-bearing algebraic steps are not fully demonstrated: the exhaustive branch resolution in Lemma 4.6 and the identity in Lemma 5.11, which is stated without proof and delegated to Mathematica. No CAS notebooks or certificates are shipped, so the claimed completeness is only as strong as these unverified symbolic reductions.

major comments (2)
  1. [Section 4.4, Lemma 4.6, Eqs. (4.13)-(4.15)] The transition from the system (4.13)-(4.14) to the three branches in (4.15) is the single most load-bearing algebraic step in the paper. It is compressed into 'Hence, we obtain that' with no demonstration that the system has no other solutions. This lemma feeds directly into Lemma 4.8, Proposition 5.17, and Proposition 5.18, so a missing branch would invalidate the 'complete' classification in Theorem 1.2/5.19. My own substitution verifies that the three listed triples satisfy the system, so this is not an observed error, but completeness requires a certified exhaustive resolution. Please provide the full elimination (e.g., resultants or a Groebner basis computation with exact arithmetic) or ship a machine-readable CAS notebook with certificates.
  2. [Section 5.3, Lemma 5.11] Identity (5.53) is not merely decorative: it is the key used in Lemma 5.12 to prove the absence of descent directions around the central cubic realization, and hence to establish that the eta = 1 branch consists of non-global local minima. The proof is omitted with the statement 'elementary and therefore omitted' and a reference to Mathematica. Since the manuscript ships no CAS output, the local-minimum characterization rests on an unverified computational assertion. The identity is in fact correct by direct expansion, but in a journal submission this is not a proof. Please include the expansion or a certified notebook.
minor comments (4)
  1. [Section 1.1, Eq. (1.1)-(1.3)] The symbol d is used both for the activation degree and for the total parameter dimension (d = 3h+1). This is confusing, especially in Theorem 1.1 where the target is affine but d also denotes the degree. Consider renaming the activation degree, e.g., m, or writing d_act.
  2. [Theorems 1.1, 3.8, 5.19] The notation '#(...) - d >= 0 = (nabla L)(theta)' is nonstandard and can be misread as an equality chain. Please state the inequality and the gradient-vanishing condition as separate assertions.
  3. [Theorem 1.2(iv) and Theorem 5.19(iv), Eq. (1.13)/(5.85)] The product formula in (1.13) is very hard to parse, especially with the indicator 1_{0}(j) and the quantities f_j defined via a supremum involving N_theta(x). Since Proposition 5.17 gives a much clearer structural description, the theorem statement should either use that form or include an explicit explanation of how the product condition unpacks into the three saddle cases.
  4. [References] Reference [50] should include the access date for the web resource, and it would be helpful to specify the Mathematica version in the main text as well as in the bibliography.

Circularity Check

0 steps flagged

No circularity: the classification is derived from the definition of the loss via explicit algebra; the only load-bearing computational steps are unverified in the text but not circular.

full rationale

The paper's derivation chain starts from the definition of the squared L2 loss (1.3) and computes explicit gradient formulas (Lemma 2.2), Hessian entries (Lemma 2.3), and then solves the critical-point equations by direct polynomial algebra. The central results, Theorem 1.1 and Theorem 1.2 / Theorem 5.19, reduce to internal lemmas such as Lemma 3.2, Lemma 3.3, Lemma 4.6, Lemma 4.7, and Lemma 5.14. No parameter is fitted to data and then renamed a prediction; no critical-point conclusion is assumed as an input. The constants in the classification (28/5, 14/5, p0 = (a+b)/2, etc.) are obtained from solving explicit equations, e.g., system (4.13)-(4.15) in Lemma 4.6. The paper's self-citations (e.g., [20], [21], [23]) are background or standard technical references (Leibniz rule, convergence phenomena) and are not load-bearing for the classification. The omitted proof of Lemma 5.11 and the Mathematica-verified symbolic identities are a completeness/verifiability limitation, not circularity: the identity (5.53) is a claimed algebraic identity used to prove absence of descent directions, but it is not derived from the conclusion. Therefore no circular step is exhibited, and the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

This is a pure-mathematics derivation. There are no data-fitted parameters and no invented entities; the constants in the classification solve explicit polynomial equations. The load-bearing background is standard analysis plus one explicitly unproved algebraic identity (Lemma 5.11), and CAS-verified reductions are used in Sections 4 and 5.

axioms (4)
  • standard math Vandermonde determinant formula (Lemma 3.1)
    Used to prove representability and non-representability of affine targets by sums of shifted monomials, which underlies Theorem 1.1.
  • standard math Leibniz integral rule for differentiating parameter-dependent integrals
    Used to derive the explicit gradient and Hessian formulas in Lemmas 2.2 and 2.3.
  • domain assumption Affine rescaling invariance for nonconstant affine targets with f'(a) != 0
    Lemma 5.14 maps the canonical [0,1]/identity-target landscape to the general [a,b]/affine-target landscape; the assumption f'(a) != 0 excludes constant targets.
  • ad hoc to paper The algebraic identity in Lemma 5.11 is correct
    Lemma 5.11 is load-bearing for proving absence of descent directions around non-global local minima, but its proof is explicitly omitted and delegated to Wolfram Mathematica.

pith-pipeline@v1.3.0-alltime-deepseek · 39761 in / 12981 out tokens · 100769 ms · 2026-08-01T23:54:30.199719+00:00 · methodology

0 comments
read the original abstract

In this paper, we study the optimization landscape induced by the true loss for shallow polynomial neural networks (PNNs) with $\mathfrak{h} \in \mathbb{N}$ neurons on the hidden layer, one-dimensional input and output layers, and a monomial activation of degree $d \in \mathbb{N}$, trained against a non-constant affine linear target function. Our first main result provides for arbitrary activation degree $d$ a sharp existence/non-existence criterion for \emph{global minimizers} with necessary structural conditions. We show that the infimum of the loss is always zero and achievable with at least $d$ active and visible hidden neurons -- that is, hidden neurons with non-zero inner and outer weights -- with pairwise distinct pivots. In contrast, if $\mathfrak{h} < d$, then the infimum cannot be attained and any minimizing sequence of parameters necessarily diverges to infinity. In the second main result, we provide a complete classification of all critical points of the loss function for the cubic activation. We show that the loss landscape admits no \emph{local maximizers}, critical points cannot have exactly two distinct pivots, global minimizers require at least three distinct pivots, critical points with no active hidden neurons correspond to \emph{saddle points} only, and consequently, \emph{non-global local minimizers} and non-trivial saddle points arise only in networks where all pivots coincide. Moreover, non-global local minimizers require all hidden neurons to be active and visible with exactly one hidden neuron having a slope sign matching that of the target function. Our second main result also guarantees that each hidden neuron of a critical point that is not a global minimizer has either input-dependent or zero contribution, but has no nonzero input-independent contribution, to its corresponding realization function.

Figures

Figures reproduced from arXiv: 2607.15173 by Diyora Salimova, Ilkhom Mukhammadiev, Shokhrukh Ibragimov.

Figure 1.1
Figure 1.1. Figure 1.1: Numerical illustrations associated with Theorem 1.2 in the canonical setting, where we consider a unit integration domain for the loss function L and an identity target function: 𝒶 = 0, 𝒷 = 1, and f = idR. We plot the realization functions of critical points of the loss function with their corresponding loss values: The loss value 1/75 of non-global local minimizers corresponds to 𝒻0 = 0 and 7/300 of sad… view at source ↗
Figure 1.2
Figure 1.2. Figure 1.2: Regions for activity types of hid￾den neurons as seen in the (w θ i , bθ i )-plane. We note that under these notations the realization function in (1.15) can also be expressed as R ∋ x 7→ N θ (x) = c θ + Ph ℓ=1 v θ ℓA(b θ ℓ + w θ ℓx) ∈ R. (1.16) The classification of hidden neurons of a shallow PNN parameter vector θ ∈ R d into activity and visibility types provides an intuitive framework for understandi… view at source ↗
Figure 1.3
Figure 1.3. Figure 1.3: Graphical illustration of the three-layer (shallow) [PITH_FULL_IMAGE:figures/full_fig_p010_1_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 21 linked inside Pith

  1. [1]

    Geometry and optimization of shallow polynomial networks

    Arjevani, Y., Bruna, J., Kileel, J., Polak, E., and Trager, M. Geometry and optimization of shallow polynomial networks. SIAM J. Appl. Algebra Geom. 10 , 2 (2026), 174--209

    2026

  2. [2]

    Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)

    Bach, F., and Moulines, E. Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n) . arXiv:1306.2119 https://arxiv.org/abs/1306.2119\/ (2013), 42 pages

    Pith/arXiv arXiv 2013

  3. [3]

    Neural networks and principal component analysis: Learning from examples without local minima

    Baldi, P., and Hornik, K. Neural networks and principal component analysis: Learning from examples without local minima . Neural Networks 2 , 1 (1989), 53--58

    1989

  4. [4]

    Landscape analysis for shallow neural networks: complete classification of critical points for affine target functions

    Cheridito, P., Jentzen, A., and Rossmannek, F. Landscape analysis for shallow neural networks: complete classification of critical points for affine target functions. J. Nonlinear Sci. 32 , 5 (2022), Paper No. 64, 45 pp

    2022

  5. [5]

    Gradient descent provably escapes saddle points in the training of shallow R e LU networks

    Cheridito, P., Jentzen, A., and Rossmannek, F. Gradient descent provably escapes saddle points in the training of shallow R e LU networks. J. Optim. Theory Appl. 203 , 3 (2024), 2617--2648

    2024

  6. [6]

    B., and LeCun, Y

    Choromanska, A., Henaff, M., Mathieu, M., Arous, G. B., and LeCun, Y. The Loss Surfaces of Multilayer Networks . arXiv:1412.0233 https://arxiv.org/abs/1412.0233\/ (2014), 13 pages

    Pith/arXiv arXiv 2014

  7. [7]

    Open problem: The landscape of the loss surfaces of multilayer networks

    Choromanska, A., LeCun, Y., and Ben Arous, G. Open problem: The landscape of the loss surfaces of multilayer networks. In Proceedings of The 28th Conference on Learning Theory\/ (Paris, France, 03--06 Jul 2015), P. Grünwald, E. Hazan, and S. Kale, Eds., vol. 40 of Proceedings of Machine Learning Research , PMLR, pp. 1756--1760

    2015

  8. [8]

    On the omnipresence of spurious local minima in certain neural network training problems

    Christof, C., and Kowalczyk, J. On the omnipresence of spurious local minima in certain neural network training problems . Constr. Approx. 60 , 2 (2024), 197--224

    2024

  9. [9]

    Approximation by superpositions of a sigmoidal function

    Cybenko, G. Approximation by superpositions of a sigmoidal function. Math. Control Signals Systems 2 , 4 (1989), 303--314

    1989

  10. [10]

    Identifying and attacking the saddle point problem in high-dimensional non-convex optimization

    Dauphin, Y., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., and Bengio, Y. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization . arXiv:1406.2572 https://arxiv.org/abs/1406.2572\/ (2014), 14 pages

    Pith/arXiv arXiv 2014

  11. [11]

    On the existence of minimizers in shallow residual R e LU neural network optimization landscapes

    Dereich, S., Jentzen, A., and Kassing, S. On the existence of minimizers in shallow residual R e LU neural network optimization landscapes. SIAM J. Numer. Anal. 62 , 6 (2024), 2640--2666

    2024

  12. [12]

    Do, T., Hannibal, S., and Jentzen, A. Non-convergence to global minimizers in data driven supervised deep learning: Adam and stochastic gradient descent optimization provably fail to converge to global minimizers in the training of deep neural networks with ReLU activation . arXiv:2410.10533 https://arxiv.org/abs/2410.10533\/ (2024), 91 pages

    Pith/arXiv arXiv 2024

  13. [13]

    S., and Lee, J

    Du, S. S., and Lee, J. D. On the Power of Over-parametrization in Neural Networks with Quadratic Activation . arXiv:1803.01206 https://arxiv.org/abs/1803.01206\/ (2018), 17 pages

    Pith/arXiv arXiv 2018

  14. [14]

    I., Wu, C., and Yahl, T

    Finkel, B., Rodriguez, J. I., Wu, C., and Yahl, T. Activation degree thresholds and expressiveness of polynomial neural networks. Algebr. Stat. 16 , 2 (2025), 113--130

    2025

  15. [15]

    Local minima and plateaus in hierarchical structures of multilayer perceptrons

    Fukumizu, K., and Amari, S. Local minima and plateaus in hierarchical structures of multilayer perceptrons. Neural Networks 13 , 3 (2000), 317--327

    2000

  16. [16]

    Blow up phenomena for gradient descent optimization methods in the training of artificial neural networks

    Gallon, D., Jentzen, A., and Lindner, F. Blow up phenomena for gradient descent optimization methods in the training of artificial neural networks . arXiv:2211.15641 https://arxiv.org/abs/2211.15641\/ (2022), 84 pages

    Pith/arXiv arXiv 2022

  17. [17]

    C., and Zadik, I

    Gamarnik, D., Kizildağ, E. C., and Zadik, I. Stationary points of a shallow neural network with quadratic activations and the global optimality of the gradient descent algorithm. Math. Oper. Res. 50 , 1 (2025), 209--251

    2025

  18. [18]

    Approximation capabilities of multilayer feedforward networks

    Hornik, K. Approximation capabilities of multilayer feedforward networks. Neural Networks 4 , 2 (1991), 251--257

    1991

  19. [19]

    Multilayer feedforward networks are universal approximators

    Hornik, K., Stinchcombe, M., and White, H. Multilayer feedforward networks are universal approximators. Neural Networks 2 , 5 (1989), 359--366

    1989

  20. [20]

    On the existence of infinitely many realization functions of non-global local minima in the training of artificial neural networks with ReLU activation

    Ibragimov, S., Jentzen, A., Kröger, T., and Riekert, A. On the existence of infinitely many realization functions of non-global local minima in the training of artificial neural networks with ReLU activation . arXiv:2202.11481 https://arxiv.org/abs/2202.11481\/ (2022), 49 pages

    Pith/arXiv arXiv 2022

  21. [21]

    Ibragimov, S., Jentzen, A., and Riekert, A. Convergence to good non-optimal critical points in the training of neural networks: Gradient descent optimization with one random initialization overcomes all bad non-global local minima with high probability . arXiv:2212.13111 https://arxiv.org/abs/2212.13111\/ (2022), 98 pages

    Pith/arXiv arXiv 2022

  22. [22]

    Strong error analysis for stochastic gradient descent optimization algorithms

    Jentzen, A., Kuckuck, B., Neufeld, A., and von Wurstemberger, P. Strong error analysis for stochastic gradient descent optimization algorithms . IMA J. Numer. Anal. 41 , 1 (2021), 455--492

    2021

  23. [23]

    On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks

    Jentzen, A., and Riekert, A. On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks. J. Mach. Learn. 1 , 2 (2022), 141--246

    2022

  24. [24]

    Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networks

    Jentzen, A., and Riekert, A. Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networks . arXiv:2402.05155 https://arxiv.org/abs/2402.05155\/ (2024), 36 pages

    Pith/arXiv arXiv 2024

  25. [25]

    Best approximation by Heaviside perceptron networks

    Kainen, P., Kůrková, V., and Vogt, A. Best approximation by Heaviside perceptron networks . Neural Networks 13 , 7 (2000), 695--697

    2000

  26. [26]

    Deep Learning without Poor Local Minima

    Kawaguchi, K. Deep Learning without Poor Local Minima . arXiv:1605.07110 https://arxiv.org/abs/1605.07110\/ (2016), 24 pages

    Pith/arXiv arXiv 2016

  27. [27]

    On the Expressive Power of Deep Polynomial Neural Networks

    Kileel, J., Trager, M., and Bruna, J. On the Expressive Power of Deep Polynomial Neural Networks . arXiv:1905.12207 https://arxiv.org/abs/1905.12207\/ (2019), 13 pages

    Pith/arXiv arXiv 1905

  28. [28]

    Geometry of polynomial neural networks

    Kubjas, K., Li, J., and Wiesmann, M. Geometry of polynomial neural networks. Algebr. Stat. 15 , 2 (2024), 295--328

    2024

  29. [29]

    Some useful LaTeX commands

    Kuckuck, B. Some useful LaTeX commands . https://latex.bennokuckuck.de https://latex.bennokuckuck.de [Accessed March 2026]\/ (2025)

    2026

  30. [30]

    D., Panageas, I., Piliouras, G., Simchowitz, M., Jordan, M

    Lee, J. D., Panageas, I., Piliouras, G., Simchowitz, M., Jordan, M. I., and Recht, B. First-order methods almost always avoid strict saddle points. Math. Program. 176 , 1-2 (2019), 311--337

    2019

  31. [31]

    D., Simchowitz, M., Jordan, M

    Lee, J. D., Simchowitz, M., Jordan, M. I., and Recht, B. Gradient Descent Converges to Minimizers . arXiv:1602.04915 https://arxiv.org/abs/1602.04915\/ (2016), 11 pages

    Pith/arXiv arXiv 2016

  32. [32]

    Y., Pinkus, A., and Schocken, S

    Leshno, M., Lin, V. Y., Pinkus, A., and Schocken, S. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Networks 6 , 6 (1993), 861--867

    1993

  33. [33]

    PowerNet: Efficient Representations of Polynomials and Smooth Functions by Deep Neural Networks with Rectified Power Units

    Li, B., Tang, S., and Yu, H. PowerNet: Efficient Representations of Polynomials and Smooth Functions by Deep Neural Networks with Rectified Power Units . arXiv:1909.05136 https://arxiv.org/abs/1909.05136\/ (2019), 23 pages

    Pith/arXiv arXiv 1909

  34. [34]

    Visualizing the Loss Landscape of Neural Nets

    Li, H., Xu, Z., Taylor, G., Studer, C., and Goldstein, T. Visualizing the Loss Landscape of Neural Nets . arXiv:1712.09913 https://arxiv.org/abs/1712.09913\/ (2017), 17 pages

    Pith/arXiv arXiv 2017

  35. [35]

    On the Computational Efficiency of Training Neural Networks

    Livni, R., Shalev-Shwartz, S., and Shamir, O. On the Computational Efficiency of Training Neural Networks . arXiv:1410.1141 https://arxiv.org/abs/1410.1141\/ (2014), 17 pages

    Pith/arXiv arXiv 2014

  36. [36]

    S., Vanden-Eijnden, E., and Zdeborová, L

    Mannelli, S. S., Vanden-Eijnden, E., and Zdeborová, L. Optimization and Ge\-ne\-ra\-li\-za\-tion of Shallow Neural Networks with Quadratic Activation Functions . arXiv:2006.15459 https://arxiv.org/abs/2006.15459\/ (2020), 26 pages

    Pith/arXiv arXiv 2006

  37. [37]

    Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning

    Moulines, E., and Bach, F. Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning . In Advances in Neural Information Processing Systems\/ (2011), J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger, Eds., vol. 24, Curran Associates, Inc

    2011

  38. [38]

    Universal gradient methods for convex optimization problems

    Nesterov, Y. Universal gradient methods for convex optimization problems . Mathematical Programming 152\/ (2014), 381 -- 404

    2014

  39. [39]

    Lectures on convex optimization , second ed., vol

    Nesterov, Y. Lectures on convex optimization , second ed., vol. 137 of Springer Optimization and Its Applications . Springer, Cham, 2018

    2018

  40. [40]

    Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regions

    Panageas, I., and Piliouras, G. Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regions . arXiv:1605.00405 https://arxiv.org/abs/1605.00405\/ (2016), 11 pages

    Pith/arXiv arXiv 2016

  41. [41]

    First-order methods almost always avoid saddle points: the case of vanishing step-sizes

    Panageas, I., Piliouras, G., and Wang, X. First-order methods almost always avoid saddle points: the case of vanishing step-sizes . arXiv:1906.07772 https://arxiv.org/abs/1906.07772\/ (2019), 27 pages

    arXiv 1906

  42. [42]

    Topological properties of the set of functions generated by neural networks of fixed size

    Petersen, P., Raslan, M., and Voigtlaender, F. Topological properties of the set of functions generated by neural networks of fixed size. Found. Comput. Math. 21 , 2 (2021), 375--444

    2021

  43. [43]

    Spurious Local Minima are Common in Two-Layer ReLU Neural Networks

    Safran, I., and Shamir, O. Spurious Local Minima are Common in Two-Layer ReLU Neural Networks . arXiv:1712.08968 https://arxiv.org/abs/1712.08968\/ (2017), 29 pages

    Pith/arXiv arXiv 2017

  44. [44]

    Soltanolkotabi, M., Javanmard, A., and Lee, J. D. Theoretical insights into the optimization landscape of over-parameterized shallow neural networks. IEEE Trans. Inform. Theory 65 , 2 (2019), 742--769

    2019

  45. [45]

    No bad local minima: Data independent training error guarantees for multilayer neural networks

    Soudry, D., and Carmon, Y. No bad local minima: Data independent training error guarantees for multilayer neural networks . arXiv:1605.08361 https://arxiv.org/abs/1605.08361\/ (2016), 12 pages

    Pith/arXiv arXiv 2016

  46. [46]

    Exponentially vanishing sub-optimal local minima in multilayer neural networks

    Soudry, D., and Hoffer, E. Exponentially vanishing sub-optimal local minima in multilayer neural networks . arXiv:1702.05777 https://arxiv.org/abs/1702.05777\/ (2017), 40 pages

    Pith/arXiv arXiv 2017

  47. [47]

    M., and Pascanu, R

    Swirszcz, G., Czarnecki, W. M., and Pascanu, R. Local minima in training of neural networks . arXiv:1611.06310 https://arxiv.org/abs/1611.06310\/ (2016), 12 pages

    Pith/arXiv arXiv 2016

  48. [48]

    S., and Bruna, J

    Venturi, L., Bandeira, A. S., and Bruna, J. Spurious valleys in one-hidden-layer neural network optimization landscapes. J. Mach. Learn. Res. 20\/ (2019), Paper No. 133, 34 pp

    2019

  49. [49]

    Vidal, R., Zhu, Z., and Haeffele, B. D. Optimization landscape of neural networks . In Mathematical aspects of deep learning . Cambridge Univ. Press, Cambridge, 2023, pp. 200--228

    2023

  50. [50]

    Mathematica, Version 14.3, Champaign, IL (2025)

    Wolfram Research, Inc. Mathematica, Version 14.3, Champaign, IL (2025) . https://www.wolfram.com/mathematica

    2025