Pith. sign in

REVIEW 3 major objections 5 minor 60 references

Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Weight decay yields a benign landscape for two-layer ReLU networks only when the width is at least about min(n^d, 2^n), and only when initialization is large.

desk verdict Solid sufficient-condition theorem for weight-decay landscapes, but the necessity claim has a square-root gap and the small-init proof leans on an unverified import. read the letter →

arxiv 2505.22578 v2 pith:J2EDCTEC submitted 2025-05-28 cs.LG

Etienne Boursier , Matthew Bowditch , Matthias Englert , Ranko Lazic This is my paper
classification cs.LG
keywords weightdecaylosslandscapeoverparameterizationReLUnetworksactivationconesspuriouslocalminimaorthogonaldatasmallinitialization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks when adding weight decay ($\ell_2$ regularization) to a two-layer ReLU network makes the training loss easy to optimize. The authors prove that the regularized loss landscape becomes benign — almost every constant-activation region contains a global minimum and no spurious local minimum — once the width $m$ is at least on the order of $\min(n^d, 2^n)\log(n/\varepsilon)$, where $n$ is the number of training points and $d$ their dimension. They show this threshold is not loose: for orthogonal data, widths below it leave nearly all activation regions free of global minima, and gradient flow from a rotation-invariant initialization provably converges to a non-global solution. They further show that even at arbitrarily large width, a sufficiently small initialization scale drives subgradient flow into a spurious stationary point for certain data sets, so the benign geometry is relevant only in the large-initialization regime. If correct, this establishes that weight decay buys optimization benefit only at the price of much larger overparametrization than the unregularized case.

What carries the argument

The central object is the activation cone $C_A$: the region of parameter space where every neuron's ReLU pattern — which inputs it fires on, and the sign of its output weight — is fixed by a binary matrix $A$. Inside a cone the network is linear in the parameters, so the regularized loss is convex there, and the paper transfers global landscape questions to statements about cones. Two tools carry the argument: the convex reformulation of the regularized and min-norm problems, which implies that a global minimum exists using only $n+1$ non-zero neurons, and a coupon-collector count showing that a uniformly random cone with $m \gtrsim \min(n^d, 2^n)$ contains the activation patterns of those $n+1$ neurons with high probability. For the dynamics results the machinery is a three-phase analysis of subgradient flow — an early alignment phase taken from prior work, a growth phase in which active neurons track a composite teacher direction, and a final phase ruled by a local Polyak–Łojasiewicz inequality — which pins the limit to a rank-1 manifold of non-global minimizers. For orthogonal data, the key simplification is that neuron activation patterns are invariant along the flow, which turns the question into another coupon-collector bound.

What would settle it

Simulate subgradient flow on the Theorem 2 family of data sets (points near the specified centers $\hat{x}_k$, labels from the teacher $v_\star$, dimension $d \ge 3$, width $m = 2$) with initialization scale $\alpha$ very small and $\lambda \le \mu_{\min}\alpha^\varepsilon$, and check whether any trajectory ever leaves the manifold of rank-1 aligned networks toward the rank-2 configuration proven to have strictly smaller $L_\lambda$. Theorem 2 predicts the flow cannot; a single observed escape falsifies it. A second check targets the borrowed phase: at time $T_1 = \varepsilon \ln(1/\alpha)/\|\gamma\|$, every initially active neuron with positive output weight should satisfy $\hat{w}_i(T_1)^\top \gamma \ge 1-\alpha^\varepsilon$; a violation would falsify Lemma 3.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a tripartite characterization of the $\ell_2$-regularized square loss $L_\lambda$ for two-layer ReLU networks. Theorem 1 states that whenever $m = \Omega(\min(n^d, 2^n)\log(n/\varepsilon))$, in all but an $\varepsilon$-fraction of the non-empty activation cones the restricted loss has a global minimum and no bad local minimum, for every $\lambda > 0$, and the same holds for the minimal-norm interpolation problem. Theorem 3 shows the width bound is sharp on orthogonal data: below that scale only a fraction $m\,2^{-\max(\mathrm{card}\{k:y_k>0\},\mathrm{card}\{k:y_k<0\})}$ of cones contain a global minimum, and typical gradient flows converge to points that are not global minima. Theorem 2 shows that landscape guarantees do not transfer to small initialization: for a family of data sets in any dimension $d \ge 3$ and any width $m \ge 2$, subgradient flow with initialization scale $\alpha$ below a threshold converges to an interpolating stationary point that is not a global minimum of $L_\lambda$. Together the theorems delineate exactly when weight decay produces a benign optimization problem: only with both large overparametrization and large initialization.

Load-bearing premise

The proof that small initialization gets stuck borrows its early-alignment phase from earlier work without proving that weight decay leaves that phase unchanged; if regularization perturbs the early dynamics, the convergence-to-a-bad-minimum conclusion could fail.

Editorial extensions

If this is right

  • With weight decay, the width needed for a benign landscape jumps from $m \gtrsim n/d$ (unregularized) to $m \gtrsim \min(n^d, 2^n)\log(n/\varepsilon)$.
  • On orthogonal data the regularized landscape is the worst case: below the width threshold almost every activation cone contains a bad local minimum and no global one, and gradient flow with rotation-invariant initialization lands on a non-global solution with high probability.
  • Landscape benignity does not imply optimization success: in the small-initialization (feature learning) regime, gradient flow converges to spurious stationary points even when the landscape is overwhelmingly benign, for every width $m \ge 2$.
  • In the large-initialization regime, Theorem 1 gives a formal justification for random large-width interpolating networks: with high probability their cone contains a global minimum of the min-norm problem, so the obtained estimator is small-norm and should generalize.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The sharp transition predicted on orthogonal data — the fraction of benign cones collapses below $m \approx 2^{\max(n_+,n_-)}$ — is quantitatively testable at larger $n$ than the experiments in the paper reach; observing a gradual rather than sharp transition would suggest that benignity depends on more than the coupon-collector count.
  • Theorem 2's limit points are usually saddles rather than local minima, because zero neurons allow norm reduction; this suggests that the practical failure mode of small initialization under weight decay is not entrapment but convergence to a high-norm interpolator, which the min-norm view says should generalize worse — a claim the paper does not itself make.
  • The necessity result is proved only for orthogonal data and rotation-invariant initializations; extending it to Gaussian data is open, and the paper's own experiments hint that the bad-cone fraction there is lower, so the exponential threshold may be an upper bound on what generic data require.
  • Theorems 1 and 3 together imply that the common practice of training with weight decay and then measuring norm is most faithfully described by the min-norm interpolation problem only in the large-width, large-init regime; outside it, the effective objective has spurious minima that no amount of width alone removes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies the loss landscape of two-layer ReLU networks under square loss with ℓ2 weight decay. It establishes a sufficient overparametrization threshold m = Ω(min(n^d, 2^n) log(n/ε)) for a benign landscape: with high probability over non-empty activation cones, the cone contains a global minimum and no bad local minimum (Theorem 1). It then studies dynamics: with small balanced random initialization, gradient flow can converge to a stationary point that is nearly interpolating but not a global minimum, despite the benign landscape (Theorem 2). Finally, it considers orthogonal data and claims that the large overparametrization is necessary for benignity and for convergence to global minima (Theorem 3). Experiments on synthetic data validate the transition predicted by Theorem 1, and additional experiments illustrate the small-initialization phenomenon of Theorem 2.

Significance. If the technical claims are fully established, the paper makes a valuable contribution: it quantifies how weight decay changes the landscape threshold relative to the unregularized case, and it shows that even a benign landscape does not guarantee good optimization under small initialization. The coupon-collector proof of Theorem 1 is clean and the experiments match the predicted threshold. The necessity result in Theorem 3, however, is the paper's headline and is currently overstated; the proof as written supports only a weaker lower bound for balanced orthogonal data, and there is a quantitative error in Lemma 8. The paper also relies on an unproved carry-over of a phase of the dynamics from prior work. These issues affect the central advertised claims, so the paper needs revision before its results can be accepted as stated.

major comments (3)
  1. [§6, Theorem 3; abstract and conclusion] The advertised necessity claim is stronger than the theorem proves. Theorem 3 bounds the fraction of activation cones whose closure contains a global minimum by m·2^{-max(N_+,N_-)}, where N_+ and N_- are the numbers of positive and negative labels. For balanced orthogonal data, max(N_+,N_-) ≈ n/2, so the bound is informative only when m is much smaller than 2^{n/2}. It does not rule out benignity for m between 2^{n/2} and 2^n, which is the regime relevant to the sufficiency threshold min(n^d, 2^n). The abstract and conclusion state that m ≳ min(n^d, 2^n) is "not only sufficient but also necessary", and Section 6 repeats this claim; this is not what the theorem establishes. The rigorous statement is a sufficient threshold of min(n^d, 2^n) and a separate necessary threshold of 2^{max(N_+,N_-)} on orthogonal data. The claims and the surrounding discussion must be revised to match the proof.
  2. [Appendix D.2, proof of Lemma 8] The perturbation argument in Lemma 8 contains a factor-of-n mismatch. In the second case, the loss change is computed as L_λ(θ_ε) = L_λ(θ*) − (2/n)ε²∥D_+∥³ + 2λε²∥D_+∥² + O(ε⁴). For this to be strictly negative, one needs λ < ∥D_+∥/n, but the stated assumption is only 0 < λ ≤ ∥D_+∥ (i.e., λ ≤ sqrt(∑_{y_k>0} y_k²∥x_k∥²)). The stated λ range is therefore insufficient for the claimed contradiction. Since Lemma 8 is used to prove Lemma 6 and hence Theorem 3, either the λ range in Theorem 3 must be strengthened or the perturbation argument must be corrected.
  3. [Appendix C, Lemma 3 and proof of Theorem 2] The early-alignment phase, which is load-bearing for Theorem 2, is not proved in the manuscript. Lemma 3 states that the phase concludes as claimed, and its proof asserts that the results "follow by the proofs of Chistikov et al. [2023, Lemmas 3 and 5, and Proposition 18]" once Chistikov et al.'s Lemma 19 is replaced by Claims 1 and 2. This is a non-trivial transfer: the regularized dynamics change the evolution of d ln∥w_i(t)∥/dt by the additional −2λ term, and the replacement claims are exactly what must control this term. The current text does not demonstrate that the rest of Chistikov et al.'s argument is unaffected by the regularization. Please provide the actual transfer argument or state clearly which parts of the earlier proof are being reused and why they remain valid.
minor comments (5)
  1. [§6, Theorem 3 statement] The theorem statement says "if inf_{t→∞} θ(t) = θ_∞ exists"; this should be "if lim_{t→∞} θ(t) = θ_∞ exists", since the limit, not the infimum, is meant.
  2. [Figure 1 caption] The caption says the plot shows cones containing "a global minimum or local minimum"; the text and legend indicate these are bad local minima, so the caption should read "bad local minimum" for clarity.
  3. [Throughout the paper] The sufficient threshold is sometimes written as min(n^d, 2^n) and sometimes as min(nd, 2n) or min(2n, nd); these notations are inconsistent and should be standardized.
  4. [§5, Assumption 2(b)] Assumption 2(b), which forbids reactivation of fully deactivated neurons, is strong and is asserted to exclude "unrealistic flows". It would be helpful to add a remark on whether this property holds with probability one under the random initialization of Assumption 2(a), or to give evidence that dropping it does not affect the conclusion.
  5. [Appendix A.2.1] There is a typo, "initalization" for "initialization", in the description of Figure 6.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the sufficient and necessity results are derived from external convex reformulations, coupon-collector bounds, and self-contained dynamical proofs; the imported early-alignment step in Lemma 3 is a proof-completeness gap, not a circular reduction.

full rationale

The paper's central sufficient-threshold claim (Theorem 1) is derived from the convex reformulations of Wang et al. (2022) and Ergen & Pilanci (2021), the existence of an n+1-neuron global minimizer, a merging/balancing argument (Lemma 1), and the coupon-collector bound of Karhadkar et al. (2024)/Feller (1991); none of these inputs contain the conclusion, and no fitted parameter is renamed as a prediction. The necessity result (Theorem 3) is proved from the orthogonal-data structure via a KKT/merging analysis (Appendix D) and a rotation-invariance argument on weight cones (Lemma 7); its stated bound m * 2^{-max(N_+,N_-)} is exactly what is proved, and the text's extrapolation to a larger necessity threshold is an evidentiary overclaim, not a circular reduction. The only self-citations (Boursier et al. 2022 for the balancedness convention and for the unregularized two-neuron characterization; Chistikov et al. 2023 for the early-alignment phase) refer to externally checkable prior results whose assumptions do not include the target regularized statement. The one flagged caveat is Lemma 3: the paper asserts that 'these follow by the proofs of Chistikov et al. [2023, Lemmas 3 and 5, and Proposition 18], which carry over to our setting without significant modifications.' Because two present authors (Englert, Lazic) overlap with Chistikov et al., this is a load-bearing self-citation, but it is a proof-completeness gap and not a circle: the transferred claims concern unregularized early dynamics, and the paper separately proves the regularization-specific Claims 1 and 2 and the late-phase Polyak-Lojasiewicz inequality. I flag this missing carry-over proof explicitly, but I do not count it as circularity.

Assumptions & free parameters 0 free parameters · 7 assumptions · 0 invented entities

The derivation relies on published convex reformulations, the coupon collector lemma, and Cover's count of activation regions; these are standard and not free parameters. No constant is fitted to data. The main non-standard dependency is the asserted carry-over of a self-authored proof into Lemma 3 of Appendix C, which is a proof-completeness gap rather than a fitted quantity. Assumption 2(b) is an explicit modeling assumption that excludes reactivation of dead neurons.

assumptions (7)
  • domain assumption Existence of a global minimizer with n+1 non-zero neurons with distinct activation patterns via convex reformulation (Wang et al. 2022, Ergen and Pilanci 2021).
    Used in Appendix B to argue that any cone containing the n+1 winning neuron patterns contains a global minimum and no bad local minimum.
  • standard math Coupon collector tail bound (Karhadkar et al. 2024, Lemma 11).
    Used in the proof of Theorem 1 to lower-bound the probability that m random neuron cones include the n+1 winning patterns.
  • standard math Cover 1965 bound: the number of non-empty neuron cones is O(min(2^n, n^d)).
    Converts the coupon-collector probability into the width threshold m = Ω(min(n^d, 2^n) log(n/ε)).
  • standard math Balancedness of any local minimum of the ℓ2-regularized loss; rescaling a neuron (w, a) to (cw, c^{-1}a) preserves the output function.
    Used in the proof of Lemma 1 to reduce to a convex problem in β_i = a_i w_i.
  • domain assumption Assumption 2(b): a neuron deactivated on all training points remains deactivated for the rest of training.
    Explicitly imposed in the statement of Theorem 2 to exclude some subdifferential flows that the authors call unrealistic. The proof relies on this invariance.
  • ad hoc to paper The early-alignment phase of Chistikov et al. 2023 (Lemmas 3, 5, Proposition 18) carries over to the regularized setting with only Lemma 19 replaced by Claims 1 and 2.
    This is the principal unresolved dependency in the paper: Lemma 3 in Appendix C imports a substantial proof from a self-authored prior work without fully reproducing the modifications.
  • standard math Subgradient flow chain rule and monotonicity of the loss along the flow (Davis et al. 2020).
    Used throughout the proof of Theorem 2 to derive the differential equations for hidden neurons and output weights.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization." pith.science (2026). https://pith.science/paper/J2EDCTEC

@misc{pith2026250522578,
  author       = {Pith},
  title        = {Pith review of: Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J2EDCTEC}},
  note         = {Machine review of arXiv:2505.22578}
}
abstract

The optimization of neural networks under weight decay remains poorly understood from a theoretical standpoint. While weight decay is standard practice in modern training procedures, most theoretical analyses focus on unregularized settings. In this work, we investigate the loss landscape of the $\ell_2$-regularized training loss for two-layer ReLU networks. We show that the landscape becomes benign -- i.e., free of spurious local minima -- under large overparametrization, specifically when the network width $m$ satisfies $m \gtrsim \min(n^d, 2^n)$, where $n$ is the number of data points and $d$ the input dimension. More precisely in this regime, almost all constant activation regions contain a global minimum and no spurious local minima. We further show that this level of overparametrization is not only sufficient but also necessary via the example of orthogonal data. Finally, we demonstrate that such loss landscape results primarily hold relevance in the large initialization regime. In contrast, for small initializations -- corresponding to the feature learning regime -- optimization can still converge to spurious local minima, despite the global benignity of the landscape.

Figures

Figures reproduced from arXiv: 2505.22578 by the authors.

Figure 1
Figure 1. Proportion of activation cones containing global minima (blue) and bad local minima [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. We sample a nonempty activation cone by generating a random network and observing the [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗
Figure 3
Figure 3. Proportion of activation cones containing global minima (blue) and local minima (orange) [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The square of the Euclidean norm of all network weights after training has finished, starting [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]
Figure 5
Figure 5. Figure 5: The difference between the final regularized loss of the trained network and the optimal [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: This figure shows the evolution of the network during training for different initialization [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

60 extracted references · 45 canonical work pages

  1. [1]

    Du, Wei Hu, Zhiyuan Li, and Ruosong Wang

    Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks http://proceedings.mlr.press/v97/arora19a.html. In ICML, pages 322--332, 2019

    work page 2019

  2. [2]

    Francis R. Bach. Breaking the Curse of Dimensionality with Convex Neural Networks https://jmlr.org/papers/v18/14-546.html. J. Mach. Learn. Res., 18: 0 19:1--19:53, 2017

    work page 2017

  3. [3]

    Penalising the biases in norm regularisation enforces sparsity https://papers.neurips.cc/paper_files/paper/2023/hash/b444ad72520a5f5c467343be88e352ed-Abstract-Conference.html

    Etienne Boursier and Nicolas Flammarion. Penalising the biases in norm regularisation enforces sparsity https://papers.neurips.cc/paper_files/paper/2023/hash/b444ad72520a5f5c467343be88e352ed-Abstract-Conference.html. In NeurIPS, 2023

    work page 2023

  4. [4]

    Early alignment in two-layer networks training is a two-edged sword 10.48550/arxiv.2401.10791

    Etienne Boursier and Nicolas Flammarion. Early alignment in two-layer networks training is a two-edged sword 10.48550/arxiv.2401.10791. CoRR, abs/2401.10791, 2024

    work page doi:10.48550/arxiv.2401.10791 2024

  5. [5]

    Simplicity bias and optimization threshold in two-layer ReLU networks https://openreview.net/forum?id=qAarsvflTa

    Etienne Boursier and Nicolas Flammarion. Simplicity bias and optimization threshold in two-layer ReLU networks https://openreview.net/forum?id=qAarsvflTa. In ICML, 2025

    work page 2025

  6. [6]

    Etienne Boursier, Loucas Pillaud - Vivien, and Nicolas Flammarion. Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs https://proceedings.neurips.cc/paper_files/paper/2022/hash/7eeb9af3eb1f48e29c05e8dd3342b286-Abstract-Conference.html. In NeurIPS, 2022

    work page 2022

  7. [7]

    How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teachers https://openreview.net/forum?id=3eHNvPHL9Z

    Gon Buzaglo, Itamar Harel, Mor Shpigel Nacson, Alon Brutzkus, Nathan Srebro, and Daniel Soudry. How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teachers https://openreview.net/forum?id=3eHNvPHL9Z. In ICML, 2024

    work page 2024

  8. [8]

    Convergence of gradient descent for deep neural networks 10.48550/arxiv.2203.16462

    Sourav Chatterjee. Convergence of gradient descent for deep neural networks 10.48550/arxiv.2203.16462. CoRR, abs/2203.16462, 2022

    work page doi:10.48550/arxiv.2203.16462 2022

Show all 60 references
  1. [9]

    Loss Landscapes are All You Need: Neural Network Generalization Can Be Explained Without the Implicit Bias of Gradient Descent https://openreview.net/forum?id=QC10RmRbZy9

    Ping - yeh Chiang, Renkun Ni, David Yu Miller, Arpit Bansal, Jonas Geiping, Micah Goldblum, and Tom Goldstein. Loss Landscapes are All You Need: Neural Network Generalization Can Be Explained Without the Implicit Bias of Gradient Descent https://openreview.net/forum?id=QC10RmR...

    2023

  2. [10]

    Dmitry Chistikov, Matthias Englert, and Ranko Lazi\'c. Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs http://papers.neurips.cc/paper\_files/paper/2023/hash/4af24e6ce753c181e703f3f0be3b5e20-Abstract-Conference.html. In NeurIPS, 2023

    2023

  3. [11]

    L \' e na \" c Chizat and Francis R. Bach. On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport https://proceedings.neurips.cc/paper/2018/hash/a1afc58c6ca9540d057299ec3016d726-Abstract.html. In NeurIPS, pages 3040--3050, 2018

    2018

  4. [12]

    L \' e na \" c Chizat, Edouard Oyallon, and Francis R. Bach. On Lazy Training in Differentiable Programming https://proceedings.neurips.cc/paper/2019/hash/ae614c557843b1df326cb29c57225459-Abstract.html. In NeurIPS, pages 2933--2943, 2019

    2019

  5. [13]

    Thomas M. Cover. Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition 10.1109/pgec.1965.264137. IEEE Trans. Electron. Comput., EC-14 0 (3): 0 326--334, 1965

    1965

  6. [14]

    Bach, and Loucas Pillaud - Vivien

    L \' e o Dana, Francis R. Bach, and Loucas Pillaud - Vivien. Convergence of Shallow ReLU Networks on Weakly Interacting Data 10.48550/arxiv.2502.16977. CoRR, abs/2502.16977, 2025

    2025 doi

  7. [15]

    Francesco D'Angelo, Maksym Andriushchenko, Aditya Vardhan Varre, and Nicolas Flammarion. Why Do We Need Weight Decay in Modern Deep Learning? http://papers.neurips.cc/paper\_files/paper/2024/hash/29496c942ed6e08ecc469f4521ebfff0-Abstract-Conference.html In NeurIPS, 2024

    2024

  8. [16]

    Kakade, and Jason D

    Damek Davis, Dmitriy Drusvyatskiy, Sham M. Kakade, and Jason D. Lee. Stochastic Subgradient Method Converges on Tame Functions 10.1007/s10208-018-09409-5. Found. Comput. Math., 20 0 (1): 0 119--154, 2020

    2020 doi

  9. [17]

    https://www.jmlr.org/papers/v17/15-408.html CVXPY : A P ython-embedded modeling language for convex optimization

    Steven Diamond and Stephen Boyd. https://www.jmlr.org/papers/v17/15-408.html CVXPY : A P ython-embedded modeling language for convex optimization . J. Mach. Learn. Res., 17 0 (83): 0 1--5, 2016

    2016

  10. [18]

    Hamprecht

    Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred A. Hamprecht. Essentially No Barriers in Neural Network Energy Landscape http://proceedings.mlr.press/v80/draxler18a.html. In ICML, pages 1308--1317, 2018

    2018

  11. [19]

    Du, Xiyu Zhai, Barnab \' a s P \' o czos, and Aarti Singh

    Simon S. Du, Xiyu Zhai, Barnab \' a s P \' o czos, and Aarti Singh. Gradient Descent Provably Optimizes Over-parameterized Neural Networks https://openreview.net/forum?id=S1eK3i09YQ. In ICLR, 2019

    2019

  12. [20]

    Convex Geometry and Duality of Over-parameterized Neural Networks http://jmlr.org/papers/v22/20-1447.html

    Tolga Ergen and Mert Pilanci. Convex Geometry and Duality of Over-parameterized Neural Networks http://jmlr.org/papers/v22/20-1447.html. J. Mach. Learn. Res., 22 0 (212): 0 1--63, 2021

    2021

  13. [21]

    An introduction to probability theory and its applications, volume 2

    William Feller. An introduction to probability theory and its applications, volume 2. Wiley, 1991

    1991

  14. [22]

    Vetrov, and Andrew Gordon Wilson

    Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P. Vetrov, and Andrew Gordon Wilson. Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs https://proceedings.neurips.cc/paper/2018/hash/be3087e74e9100d4bc4c6268cdbe8456-Abstract.html. In NeurIPS, pages 8803--...

    2018

  15. [23]

    https://openreview.net/forum?id=HgOJlxzB16 SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problem

    Margalit Glasgow. https://openreview.net/forum?id=HgOJlxzB16 SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problem . In ICLR, 2024

    2024

  16. [24]

    Truth or backpropaganda? An empirical investigation of deep learning theory https://openreview.net/forum?id=HyxyIgHFvr

    Micah Goldblum, Jonas Geiping, Avi Schwarzschild, Michael Moeller, and Tom Goldstein. Truth or backpropaganda? An empirical investigation of deep learning theory https://openreview.net/forum?id=HyxyIgHFvr. In ICLR, 2020

    2020

  17. [25]

    Goulart and Yuwen Chen

    Paul J. Goulart and Yuwen Chen. Clarabel: An interior-point solver for conic programs with quadratic objectives http://arxiv.org/abs/2405.12762, 2024

    2024 arXiv

  18. [26]

    Haeffele and Ren \' e Vidal

    Benjamin D. Haeffele and Ren \' e Vidal. Global Optimality in Neural Network Training 10.1109/cvpr.2017.467. In CVPR, pages 4390--4398, 2017

    2017 doi

  19. [27]

    Piecewise linear activations substantially shape the loss surfaces of neural networks https://openreview.net/forum?id=B1x6BTEKwr

    Fengxiang He, Bohan Wang, and Dacheng Tao. Piecewise linear activations substantially shape the loss surfaces of neural networks https://openreview.net/forum?id=B1x6BTEKwr. In ICLR, 2020

    2020

  20. [28]

    Deep Residual Learning for Image Recognition 10.1109/cvpr.2016.90

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition 10.1109/cvpr.2016.90. In CVPR, pages 770--778, 2016

    2016 doi

  21. [29]

    Neural Tangent Kernel: Convergence and Generalization in Neural Networks https://proceedings.neurips.cc/paper/2018/hash/5a4be1fa34e62bb8a6ec6b91d2462f5a-Abstract.html

    Arthur Jacot, Cl \' e ment Hongler, and Franck Gabriel. Neural Tangent Kernel: Convergence and Generalization in Neural Networks https://proceedings.neurips.cc/paper/2018/hash/5a4be1fa34e62bb8a6ec6b91d2462f5a-Abstract.html. In NeurIPS, pages 8580--8589, 2018

    2018

  22. [30]

    John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Z \' i dek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera-Paredes, ...

  23. [31]

    Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape https://openreview.net/forum?id=10WARaIwFn

    Kedar Karhadkar, Michael Murray, Hanna Tseran, and Guido Mont \' u far. Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape https://openreview.net/forum?id=10WARaIwFn. Trans. Mach. Learn. Res., 2024, 2024

    2024

  24. [32]

    Deep Learning without Poor Local Minima https://proceedings.neurips.cc/paper/2016/hash/f2fc990265c712c49d51a18a32b39f0c-Abstract.html

    Kenji Kawaguchi. Deep Learning without Poor Local Minima https://proceedings.neurips.cc/paper/2016/hash/f2fc990265c712c49d51a18a32b39f0c-Abstract.html. In NeurIPS, pages 586--594, 2016

    2016

  25. [33]

    Exploring The Loss Landscape Of Regularized Neural Networks Via Convex Duality https://openreview.net/forum?id=4xWQS2z77v

    Sungyoon Kim, Aaron Mishkin, and Mert Pilanci. Exploring The Loss Landscape Of Regularized Neural Networks Via Convex Duality https://openreview.net/forum?id=4xWQS2z77v. In ICLR, 2025

    2025

  26. [34]

    Deep Linear Networks with Arbitrary Loss: All Local Minima Are Global http://proceedings.mlr.press/v80/laurent18a.html

    Thomas Laurent and James von Brecht. Deep Linear Networks with Arbitrary Loss: All Local Minima Are Global http://proceedings.mlr.press/v80/laurent18a.html. In ICML, pages 2908--2913, 2018

    2018

  27. [35]

    Michaud, and Max Tegmark

    Ziming Liu, Eric J. Michaud, and Max Tegmark. Omnigrok: Grokking Beyond Algorithmic Data https://openreview.net/forum?id=zDiHoIWa0q1. In ICLR, 2023

    2023

  28. [36]

    Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Bias https://proceedings.neurips.cc/paper/2021/hash/6c351da15b5e8a743a21ee96a86e25df-Abstract.html

    Kaifeng Lyu, Zhiyuan Li, Runzhe Wang, and Sanjeev Arora. Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Bias https://proceedings.neurips.cc/paper/2021/hash/6c351da15b5e8a743a21ee96a86e25df-Abstract.html. In NeurIPS, pages 12978--12991, 2021

    2021

  29. [37]

    Lee, and Wei Hu

    Kaifeng Lyu, Jikai Jin, Zhiyuan Li, Simon Shaolei Du, Jason D. Lee, and Wei Hu. Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking https://openreview.net/forum?id=XsHqr9dEGH. In ICLR, 2024

    2024

  30. [38]

    Gradient Descent Quantizes ReLU Network Features http://arxiv.org/abs/1803.08367

    Hartmut Maennel, Olivier Bousquet, and Sylvain Gelly. Gradient Descent Quantizes ReLU Network Features http://arxiv.org/abs/1803.08367. CoRR, abs/1803.08367, 2018

    2018 arXiv

  31. [39]

    A mean field view of the landscape of two-layer neural networks 10.1073/pnas.1806579115

    Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks 10.1073/pnas.1806579115. Proc. Natl. Acad. Sci. U.S.A., 115 0 (33): 0 E7665--E7671, 2018

    2018 doi

  32. [40]

    Early Neuron Alignment in Two-layer ReLU Networks with Small Initialization https://openreview.net/forum?id=QibPzdVrRu

    Hancheng Min, Enrique Mallada, and Ren \' e Vidal. Early Neuron Alignment in Two-layer ReLU Networks with Small Initialization https://openreview.net/forum?id=QibPzdVrRu. In ICLR, 2024

    2024

  33. [41]

    Optimal Sets and Solution Paths of ReLU Networks https://proceedings.mlr.press/v202/mishkin23a.html

    Aaron Mishkin and Mert Pilanci. Optimal Sets and Solution Paths of ReLU Networks https://proceedings.mlr.press/v202/mishkin23a.html. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, ICML, pages 24888--24924, 2023

    2023

  34. [42]

    In search of the real inductive bias: On the role of implicit regularization in deep learning 10.48550/arxiv.1412.6614

    Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning 10.48550/arxiv.1412.6614. In ICLR (Workshop), 2015

    2015 doi arXiv

  35. [43]

    On Connected Sublevel Sets in Deep Learning http://proceedings.mlr.press/v97/nguyen19a.html

    Quynh Nguyen. On Connected Sublevel Sets in Deep Learning http://proceedings.mlr.press/v97/nguyen19a.html. In ICML, pages 4790--4799, 2019

    2019

  36. [44]

    A Note on Connectivity of Sublevel Sets in Deep Learning https://arxiv.org/abs/2101.08576

    Quynh Nguyen. A Note on Connectivity of Sublevel Sets in Deep Learning https://arxiv.org/abs/2101.08576. CoRR, abs/2101.08576, 2021

    2021 arXiv

  37. [45]

    When Are Solutions Connected in Deep Networks? https://proceedings.neurips.cc/paper/2021/hash/af5baf594e9197b43c9f26f17b205e5b-Abstract.html In NeurIPS, pages 20956--20969, 2021

    Quynh Nguyen, Pierre Br \' e chet, and Marco Mondelli. When Are Solutions Connected in Deep Networks? https://proceedings.neurips.cc/paper/2021/hash/af5baf594e9197b43c9f26f17b205e5b-Abstract.html In NeurIPS, pages 20956--20969, 2021

    2021

  38. [46]

    Banach space representer theorems for neural networks and ridge splines https://dl.acm.org/doi/10.5555/3546258.3546301

    Rahul Parhi and Robert D Nowak. Banach space representer theorems for neural networks and ridge splines https://dl.acm.org/doi/10.5555/3546258.3546301. J. Mach. Learn. Res., 22 0 (43): 0 1--40, 2021

    2021

  39. [47]

    Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networks http://proceedings.mlr.press/v119/pilanci20a.html

    Mert Pilanci and Tolga Ergen. Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networks http://proceedings.mlr.press/v119/pilanci20a.html. In ICML, pages 7695--7705, 2020

    2020

  40. [48]

    Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets https://arxiv.org/abs/2201.02177

    Alethea Power, Yuri Burda, Harrison Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets https://arxiv.org/abs/2201.02177. CoRR, abs/2201.02177, 2022

    2022 arXiv

  41. [49]

    Trainability and accuracy of artificial neural networks: An interacting particle system approach 10.1002/cpa.22074

    Grant Rotskoff and Eric Vanden-Eijnden. Trainability and accuracy of artificial neural networks: An interacting particle system approach 10.1002/cpa.22074. Comm. Pure Appl. Math., 75 0 (9): 0 1889--1935, 2022

    1935 doi

  42. [50]

    Spurious Local Minima are Common in Two-Layer ReLU Neural Networks http://proceedings.mlr.press/v80/safran18a.html

    Itay Safran and Ohad Shamir. Spurious Local Minima are Common in Two-Layer ReLU Neural Networks http://proceedings.mlr.press/v80/safran18a.html. In ICML, pages 4430--4438, 2018

    2018

  43. [51]

    How do infinite width bounded norm networks look in function space? http://proceedings.mlr.press/v99/savarese19a.html In COLT, pages 2667--2690, 2019

    Pedro Savarese, Itay Evron, Daniel Soudry, and Nathan Srebro. How do infinite width bounded norm networks look in function space? http://proceedings.mlr.press/v99/savarese19a.html In COLT, pages 2667--2690, 2019

    2019

  44. [52]

    Understanding machine learning: From theory to algorithms 10.1017/CBO9781107298019

    Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms 10.1017/CBO9781107298019 . Cambridge University Press, 2014

    2014 doi

  45. [53]

    Jamaloddin Golestani

    Arsalan Sharif - Nassab, Saber Salehkaleybar, and S. Jamaloddin Golestani. Bounds on Over-Parameterization for Guaranteed Existence of Descent Paths in Shallow ReLU Networks https://openreview.net/forum?id=BkgXHTNtvS. In ICLR, 2020

    2020

  46. [54]

    Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances http://proceedings.mlr.press/v139/simsek21a.html

    Berfin Simsek, Fran c ois Ged, Arthur Jacot, Francesco Spadaro, Cl \' e ment Hongler, Wulfram Gerstner, and Johanni Brea. Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances http://proceedings.mlr.press/v139/simsek21a.html. In ICML, ...

    2021

  47. [55]

    The Global Landscape of Neural Networks: An Overview 10.1109/msp.2020.3004124

    Ruoyu Sun, Dawei Li, Shiyu Liang, Tian Ding, and Rayadurgam Srikant. The Global Landscape of Neural Networks: An Overview 10.1109/msp.2020.3004124. IEEE Signal Process. Mag. , 37 0 (5): 0 95--108, 2020

    2020

  48. [56]

    Bandeira, and Joan Bruna

    Luca Venturi, Afonso S. Bandeira, and Joan Bruna. Spurious Valleys in One-hidden-layer Neural Network Optimization Landscapes https://jmlr.org/papers/v20/18-674.html. J. Mach. Learn. Res., 20: 0 133:1--133:34, 2019

    2019

  49. [57]

    The Hidden Convex Optimization Landscape of Regularized Two-Layer ReLU Networks: an Exact Characterization of Optimal Solutions https://openreview.net/forum?id=Z7Lk2cQEG8a

    Yifei Wang, Jonathan Lacotte, and Mert Pilanci. The Hidden Convex Optimization Landscape of Regularized Two-Layer ReLU Networks: an Exact Characterization of Optimal Solutions https://openreview.net/forum?id=Z7Lk2cQEG8a. In ICLR, 2022

    2022

  50. [58]

    On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime https://arxiv.org/abs/2005.13530

    Stephan Wojtowytsch. On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime https://arxiv.org/abs/2005.13530. CoRR, abs/2005.13530, 2020

    2005 arXiv

  51. [59]

    Woodworth, Suriya Gunasekar, Jason D

    Blake E. Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and Rich Regimes in Overparametrized Models http://proceedings.mlr.press/v125/woodworth20a.html. In COLT, pages 3635--3673, 2020

    2020

  52. [60]

    Small nonlinearities in activation functions create bad local minima in neural networks https://openreview.net/forum?id=rke\_YiRct7

    Chulhee Yun, Suvrit Sra, and Ali Jadbabaie. Small nonlinearities in activation functions create bad local minima in neural networks https://openreview.net/forum?id=rke\_YiRct7. In ICLR, 2019

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.