Asymptotic convexity of wide and shallow neural networks

Parthe Pandit; Vivek Borkar

REVIEW 3 major objections 5 minor 17 references

Asymptotic convexity of wide and shallow neural networks

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For a shallow wide network whose output averages many parameterized component maps, the paper proves that as width grows the network objective's epigraph converges to that of a convex function, so the limiting training problem has no…

desk verdict The minimizer convergence claim doesn't follow from the epigraph result, but the Shapley-Folkman framing is a fresh heuristic that could become a publishable note after recasting to averaged parameters. read the letter →

arxiv 2507.01044 v2 pith:BXBQCME2 submitted 2025-06-23 stat.ML cs.LGmath.PR

Vivek Borkar , Parthe Pandit This is my paper · ORCID

classification stat.MLcs.LGmath.PR MSC 68T0752A4190C26

keywords shallowandwidenetworkstruncatedepigraphsMinkowskisumsconvexminorantstochasticgradientdescentasymptoticconvexityconvexificationnonconvexoptimization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks why wide neural networks train so well despite nonconvex objectives. For a simple architecture—a shallow network whose output is a weighted average of many copies of a parameterized map, with weights decaying geometrically away from the central copy—it proves an asymptotic convexity result: the epigraph of the finite-width error function converges in Hausdorff distance to the epigraph of the convex minorant of the single-component error. Consequently the minimizers of the finite-width objective converge, as width grows and then the averaging weight flattens, to minimizers of that convex minorant, whose argmin is the closed convex hull of the original problem's global minimizers. If correct, this gives a structural reason why infinitely wide shallow networks are effectively convex: all local minima are global, and stochastic gradient descent can find them.

What carries the argument

The load-bearing device is the Minkowski sum of truncated epigraphs. For a continuous function $q$ on a compact convex domain, its truncated epigraph is the compact set $\{(\beta,y): \beta\in D,\, q(\beta)\le y\le M\}$. A classical convexification theorem, presented as Theorem 3.1 in the paper and drawn from reference [4], says that $(1/n)$ times the $n$-fold Minkowski sum of a compact set converges in Hausdorff metric to its closed convex hull, at rate $O(1/n)$. Corollary 3.3 applies this to $q$: the normalized Minkowski sum of truncated epigraphs converges to the truncated epigraph of the convex minorant of $q$. The paper then observes that the network objective $\Phi^{\alpha,n}_N$ is exactly such a weighted average of component errors over the product parameter space, and uses equicontinuity plus a Tauberian theorem from reference [15] to replace the geometric kernel average by a Cesàro average, turning the geometric-mean structure into the Minkowski-averaged epigraph whose limit is convex.

What would settle it

Take a one-dimensional nonconvex error with two separated global minima, e.g. $q(\beta)=(\beta^2-1)^2$, form the finite-width objective (II.3), and enumerate all local minima for growing $N$ with $\alpha$ close to 1. If any sequence of local minima stays a positive distance away from $\operatorname{co}(\operatorname{Argmin}(q))=\{-1,1\}$ as $N\to\infty$ and $\alpha\to 1$, the conclusion of Theorem 4.5 is false.

Watch

Extended reading notes

Core claim

The central claim is Theorem 4.5: for the weighted error $\Phi^{\alpha,n}_N(B_N)=\sum_{i=-N}^N K^\alpha_N(i)\,g(X_n,Y_i^n;\beta_i)$ with $K^\alpha_N(i)\propto \alpha^{|i|}$, any sequence of minimizers $\hat{\beta}_N$ satisfies $\lim_{\alpha\uparrow 1}\lim_{N\to\infty}\hat{\beta}_N\in\operatorname{Argmin}(g^*)$, where $g^*$ is the convex minorant of the single-component error $g$ and $\operatorname{Argmin}(g^*)=\operatorname{co}(\operatorname{Argmin}(g))$. In words, minimizing the finite, randomly weighted network error over all component parameters drives the parameters into the closed convex hull of the global minimizers of one component's error. Because that limit is a convex minimization problem, the paper concludes that the infinite-width training landscape is effectively convex and that every local minimum is global. The same mechanism is applied to the least mean square criterion: conditioned on the input, the Cesàro-averaged error epigraphs converge to the epigraph of a convex function, and for the squared error the gradient of the asymptotic stochastic gradient descent tends to zero in mean square, with iterates converging in probability to the argmin set.

Load-bearing premise

The proof assumes that Minkowski-averaging the epigraphs of the individual component errors accurately represents the landscape of the actual combined network objective over the whole product parameter space; if that identification fails, the convex minorant $g^*$ would not describe the trained network's minimizers.

Editorial extensions

If this is right

The infinite-width limiting objective is convex: its minimizers form a convex set, so every local minimum is a global minimum.
Finite-width minimizers approach the closed convex hull of the original nonconvex problem's global minimizers, meaning descent methods that succeed on the limit are seeking global minima of the original problem.
For the least mean square criterion, the asymptotic stochastic gradient descent has gradient norm tending to zero in mean square and iterates converging in probability to the argmin set, with unstable local maxima avoided almost surely.
The $O(1/N)$ rate in the Minkowski-sum convergence suggests that finite-width minimizers deviate from the convex-minorant argmin by at most $O(1/N)$ in Hausdorff distance.
The convexification mechanism applies to any family of parameterized maps with a uniformly bounded modulus of continuity, not to a specific activation or architecture beyond the averaging structure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves implicit that the convexification is a property of ensembling many parameter copies, not of the network architecture itself; the same Minkowski-sum argument should apply to any averaged ensemble of uniformly continuous nonlinear maps with a common input, giving a general explanation for why ensembling smooths loss landscapes.
A natural extension is to deep networks: if layers or residual blocks are treated as the averaged components, the same reasoning would predict that sufficiently wide deep networks also develop convex-minorant landscapes, though the paper only treats shallow networks.
A testable consequence is that for finite width the set of local minima of the finite objective should cluster around $\operatorname{co}(\operatorname{Argmin}(q))$ with diameter $O(1/N)$; measuring this clustering for a simple two-well error would directly probe the mechanism.
The sign-indefinite case in the LMS section shows that convexity of the epigraph can be lost after squaring, yet all local minima remain global along the zero level set; this suggests that the robust signature of width may be the weaker property of having no spurious local minima rather than full convexity.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know one thing about this paper up front: the main theorem, Theorem 4.5, is not supported by the preceding material. The epigraph convexification result (Corollary 3.3 and Lemma 4.1) lives in a fixed parameter space R^{p+1} and controls averages of points drawn from a single truncated epigraph. The objective Φ_N is minimized over the product space D^{2N+1}, one parameter vector per network copy, and the proof never connects a minimizer tuple to an averaged parameter. As a result, "β̂_N → Argmin(g*)" is not well-typed in the natural topology: a tuple could sit in a product of individual local minima. That is a load-bearing gap, not a technicality.

The paper does deserve some credit. The idea of using Shapley-Folkman/Starr convexification of Minkowski sums as an explanation for the trainability of wide shallow networks is new in framing, and the writing is direct and honest—the abstract's 'plausible explanation' is the right register. The citations to Starr and to Fradelizi et al. are apt, and the Tauberian argument in Lemma 4.4 appears sound.

The soft spots, in proportion. The product-space gap is fatal to the stated theorem. A weaker, salvageable statement would be convergence of the averaged parameter (or the empirical distribution of the tuple) to Argmin(g*). The LMS section has a second, related overreach: the claim that all local minima are global for the squared error goes beyond what epigraph convergence can show, and the square of a sign-indefinite convex minorant is generally not convex. The local-minima issue is not rescued by the observation that unstable maxima are avoided.

If you want a quick geometric intuition for why wide networks might behave well, this note gives it. But as a rigorous proof, it doesn't hold up. I would send it to a referee who knows convex geometry and stochastic approximation, with the expectation of heavy revision or a recast conclusion. If the authors can prove the averaged-parameter version, that would be a publishable short note; as written, it is not there yet.

Referee Report

3 major / 5 minor

Summary. This paper considers a shallow-and-wide network whose output is a convex combination \sum_{i=-N}^N K_N^\alpha(i) f_{\beta_i}(x) of component outputs, with kernel weights K_N^\alpha(i) proportional to \alpha^{|i|}. For fixed data, the training objective (II.3) is a weighted sum of per-component errors over the product parameter space D^{2N+1}. The paper recalls the Shapley-Folkman-Starr result that Minkowski averages of truncated epigraphs converge to the epigraph of the convex minorant, and it claims that this implies asymptotic convexity of the neural-network map, culminating in Theorem 4.5, which asserts convergence of minimizers of Phi_N to the Argmin of the convex minorant. Section V applies the same idea to the LMS criterion and sketches an SGD convergence argument.

Significance. If true, the paper would provide a compact geometric explanation for the empirical success of wide shallow networks and would illustrate a known convexification mechanism in a new setting. It is concise, transparent about the geometric ingredients, and it correctly identifies the relevant literature (Fradelizi-Madiman-Marsiglietti-Zvavitch; Starr; Shapley-Folkman). The isolated epigraph statements in Section III are correct as far as they go. However, the central transfer from Minkowski averages of epigraphs to the actual product-space optimization problem is not made; Theorem 4.5 has a type mismatch between the averaged parameter and the trained parameter tuple, and Section V inherits this gap. I therefore view the main claim as unsupported rather than merely insufficiently detailed.

major comments (3)

[Section IV, Theorem 4.5 and its proof] Lemma 4.1 and Corollary 3.3 are statements about averages (1/n)\sum_{i=1}^n(\beta_i,y_i) of points chosen from a single truncated epigraph fepi(q) in R^{p+1}. The objective (II.3), however, is minimized over the product space D^{2N+1}, and its argument is the tuple B_N=(\beta_{-N},...,\beta_N), not the averaged parameter \bar\beta_N=(2N+1)^{-1}\sum_i\beta_i. The proof of Theorem 4.5 uses (IV.1) and (IV.2) only to compare scalar loss values; it then says that 'the claim follows by Lemma 4.1', but it never constructs a point of fepi(g_*) whose first coordinate is obtained from a minimizer \hat\beta_N by any averaging map. In addition, lim_{N\to\infty}\hat\beta_N is not well-typed, since \hat\beta_N lives in a different product space for each N; the natural limit object is a point of \prod_i Argmin(g_i), not of the convex set Argmin(g_*) in R^p. This is the load-bearing step of the paper, and it is missing.
[Section I and Section IV (local-minima and convexity claims)] Even under a charitable reading that replaces \hat\beta_N by its componentwise average, the conclusion would not establish the claimed convexity of the training landscape. For finite N, Phi_N^{\alpha,n} is coordinatewise separable, so if q has a non-global local minimum \beta_loc, the constant tuple (\beta_loc,...,\beta_loc) is a non-global local minimum of Phi_N^{\alpha,n}: each coordinate is locally minimal and the total value exceeds the value at the constant global-minimizer tuple. A convex function has no non-global local minima, but Phi_N^{\alpha,n} is not the convex function g_* on R^p, and no argument in the paper transfers the convexity of g_* to the function that is actually minimized. Consequently, the statement in Section I that all local minima are global minima is not supported by the preceding lemmas.
[Section V (LMS criterion and SGD)] The LMS section applies 'the theory developed above' to Cesaro averages of (f_{\beta_*}(x)-f_{\beta_i}(x))^2, but it does not identify the resulting convex function \tilde f_\beta(x) with the value function of the network output \sum_i K_N^\alpha(i)f_{\beta_i}(x); without this identification, the replacement of the trained objective by \tilde f is unjustified. The claim that the squared sign-indefinite convex function 'will not be convex, but it will nevertheless have all its local minima = global minima' is asserted without proof. Finally, the SGD convergence argument assumes that the limiting objective is the one being optimized, that gradient norms are controlled outside the minimizer set, and that unstable equilibria are almost surely avoided; these are nontrivial conditions that are not derived from the lemmas.

minor comments (5)

[Section II, Eq. (II.3)] The function g(x,y;\beta) is defined as the vector y-f_\beta(x), but Phi_N^{\alpha,n} is stated to take values in R_+; please state explicitly that g in (II.3) is the squared Euclidean norm or otherwise clarify the codomain.
[Section V, first display] The output is written as Y=\sum_{i=-N}^N K_N^\alpha(i)\alpha^i Y_i^n, which mixes the kernel K_N^\alpha(i) with an additional \alpha^i; this appears to be a typo and should be reconciled with (II.3).
[Section III, Corollary 3.3] The word 'Haussdorff' should be 'Hausdorff'.
[References] The citation to Radhakrishnan, Belkin and Uhler gives the year 2013, but the PNAS article cited is from 2023; please verify the year.
[Section IV, Lemma 4.4 and Theorem 4.5] Theorem 4.5 should state the assumption, used in (IV.2), that the Cesaro limit in Lemma 4.4 exists for the parameter sequences under consideration; otherwise the argument is conditional on an unstated hypothesis.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the convexification step is an external Shapley-Folkman theorem, no fitted parameter is renamed as a prediction, and the only self-citation is a non-load-bearing SGD remark.

full rationale

I walked the derivation chain. Section III takes Theorem 3.1 from Fradelizi et al. (external) about Minkowski averages of arbitrary compact sets converging to the convex hull. Corollary 3.3 identifies that limit with the epigraph of the convex minorant q*, using a separation proof; this is a standard convex-geometric fact, not an assumption of the network conclusion. The subsequent lemmas use external Tauberian/measure-convergence results (Serfozo; Sznajder-Filar) to relate the geometric kernel weights to Cesaro averages. No network parameter is fitted to data and then reported as a prediction: the random components q_i and the fixed kernel K_N^alpha are model inputs, not estimated quantities. The only self-citation is [1], cited for an unrelated remark that local maxima are unstable for SGD; it is not load-bearing for the convexity claim. The paper's phrase that the result 'leads to a plausible explanation' is explicitly heuristic, not a fitted prediction. The skeptic's objection that Theorem 4.5 does not follow from Lemma 4.1 because Phi_N is minimized over the product tuple rather than an averaged parameter is a substantive correctness/type question and would be a mathematical gap if sustained; it is not a circular reduction, because the theorem's conclusion is not built into the definition of any input quantity. Under the rule that only definitional equivalence or fitted-value renaming counts as circularity, I find no significant circularity.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The derivation relies on standard convex-geometry and Tauberian results, plus a highly stylized data model. The critical unstated input is that the network is analyzed in an averaged-parameter or measure space rather than in the actual product parameter space. No data are fitted, so the free-parameter ledger is limited to auxiliary choices like α and the truncation level M.

free parameters (2)

α (last-layer weight decay) = α↑1 (limit, no fitted value)
The weight kernel K_N(i)=C_N α^{|i|} is chosen by hand; the theorem is only stated in the iterated limit α→1, N→∞, so no finite value is fit.
M (epigraph truncation level) = M≫sup q(β)
Chosen to make truncated epigraphs compact; the Hausdorff limit is independent of M, so it is an auxiliary construction.

assumptions (6)

standard math Shapley-Folkman/Starr theorem on Minkowski sums of compact sets (Theorem 3.1, from [4]).
Used to claim 1/n A^n → co(A) and hence fepi(q*) = co(fepi(q)).
standard math Arzela-Ascoli theorem and uniform convergence of equicontinuous functions.
Used in Theorem 2.1 and Lemma 4.3 to pass from pointwise to uniform convergence and preserve minimizers.
standard math Hardy-Littlewood Tauberian theorem (Theorem 2 of [15]).
Used to equate weighted geometric averages to Cesàro averages as α↑1.
domain assumption Data model Y_i^n = f_{β*}(X_n)+ξ_i^n with common input and i.i.d. zero-mean noise.
The analysis is conditioned on a fixed input; the real data process is assumed to have this additive structure.
ad hoc to paper The Cesàro limit ℓ*(B∞)=lim(1/(2N+1))Σ q_i(β_i) exists for the parameter sequences under consideration.
Stated as a supposition in Lemma 4.4; no argument is given for training trajectories.
ad hoc to paper The network can be represented by epigraphs in a single parameter space R^{p+1}, so Minkowski summing across components convexifies the map.
Corollary 3.3 is applied to fepi(q_i) as if all β_i live in the same coordinate space, whereas the actual objective is a function of the product parameter space; this is the critical unstated identification.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Asymptotic convexity of wide and shallow neural networks." pith.science (2026). https://pith.science/paper/BXBQCME2

@misc{pith2026250701044,
  author       = {Pith},
  title        = {Pith review of: Asymptotic convexity of wide and shallow neural networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BXBQCME2}},
  note         = {Machine review of arXiv:2507.01044}
}

read the original abstract

For a simple model of shallow and wide neural networks, we show that the epigraph of its input-output map as a function of the network parameters approximates epigraph of a. convex function in a precise sense. This leads to a plausible explanation of their observed good performance.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 16 canonical work pages

[1]

Borkar, Stochastic Approximation: A Dynamical Systems Viewpoint (second edition), Hindustan Publishing Agency and Springer Nature, 2022/24

V . Borkar, Stochastic Approximation: A Dynamical Systems Viewpoint (second edition), Hindustan Publishing Agency and Springer Nature, 2022/24

work page 2022
[2]

Cagnetta, A

F. Cagnetta, A. Favero and M. Wyart, “What can be learnt with wide convolutional neural networks?’, in International Conference on Machine Learning, July 2023, pp. 3347-3379. PMLR

work page 2023
[3]

Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks

A. Canatar, B. Bordelon and C. Pehlevan, “Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks”, Nature Communications, 2021, 12(1), p. 2914

work page 2021
[4]

The convexification effect of Minkowski summation

M. Fradelizi, M. Madiman, A. Marsiglietti and A. Zvavitch, “The convexification effect of Minkowski summation”, EMS Surveys in Mathematical Sciences, 2018, 5(1), pp. 1-64

work page 2018
[5]

J. Lee, L. Xiao, S. Schoenholz, Y . Bahri, R. Novak, J. Sohl-Dickstein and J. Pennington, J., “Wide neural networks of any depth evolve as linear models under gradient descent. ¨, Advances in Neural Information Processing Systems, 2019, 32

work page 2019
[6]

The expressive power of neural networks: a view from the width

Z. Lu, H. Pu, F. Wang, Z. Hu and L. Wang, “The expressive power of neural networks: a view from the width”, Advances in Neural Information Processing Cystems, 2017, 30

work page 2017
[7]

Wide neural networks forget less catastrophically

S. I. Mirzadeh, A. Chaudhry, D. Yin, H., Hu, R. Pascanu, D. Gorur and M. Farajtabar, M., “Wide neural networks forget less catastrophically”, in International Conference on Machine Learning, June 2022, pp. 15699- 15717. PMLR

work page 2022
[8]

The loss surface of deep and wide neural networks

Q. Nguyen and M. Hein, “The loss surface of deep and wide neural networks”, in International Conference on Machine Learning”, July 2017, pp. 2603-2612. PMLR

work page 2017

Show all 17 references

[9]

Pilanci and T

M. Pilanci and T. Ergen, 2020. Neural networks are convex regularizers: Exact polynomial-time convex optimization formulations for two-layer networks. In International Conference on Machine Learning (pp. 7695- 7705). PMLR

2020
[10]

Wide and deep neural networks achieve consistency for classification

A. Radhakrishnan, M. Belkin and C. Uhler, C., “Wide and deep neural networks achieve consistency for classification”, Proceedings of the National Academy of Sciences, 2013, 120 (14), p. e2208779120

2013
[11]

Convergence of Lebesgue integrals with varying measures

R. Serfozo, “Convergence of Lebesgue integrals with varying measures”, Sankhy¯a: The Indian Journal of Statistics, Series A, 1982, pp. 380-402

1982
[12]

Starr’s problem

L. S. Shapley and J. H. Folkman, J. H., “Starr’s problem”, Unpublished private communication to R. M. Starr, 1966

1966
[13]

Quasi-equilibria in markets with non-convex preferences

R. M. Starr, “Quasi-equilibria in markets with non-convex preferences”, Econometrica: Journal of the Econometric Society, 1969, 25-38

1969
[14]

Approximation of points of the convex hull of a sum of sets by points of the sum: an elementary approach

R. M. Starr, “Approximation of points of the convex hull of a sum of sets by points of the sum: an elementary approach”, Journal of Economic Theory, 1981, 25 (2), 314-317

1981
[15]

Some comments on a theorem of Hardy and Littlewood

R. Sznajder and J. A. Filar, J. A., “Some comments on a theorem of Hardy and Littlewood”, Journal of Optimization Theory and Applica- tions, 1992, 75 (1), pp. 201-208

1992
[16]

Do wider neural networks really help adversarial robustness?

B. Wu, J. Chen, D. Cai, X. He and Q. Gu, “Do wider neural networks really help adversarial robustness?”, Advances in Neural Information Processing Systems 34, 2021, pp. 7054-7067

2021
[17]

Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation

G. Yang, “Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation”, arXiv preprint arXiv:1902.04760, 2019 [Online]

1902 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Borkar, Stochastic Approximation: A Dynamical Systems Viewpoint (second edition), Hindustan Publishing Agency and Springer Nature, 2022/24

V . Borkar, Stochastic Approximation: A Dynamical Systems Viewpoint (second edition), Hindustan Publishing Agency and Springer Nature, 2022/24

work page 2022

[2] [2]

Cagnetta, A

F. Cagnetta, A. Favero and M. Wyart, “What can be learnt with wide convolutional neural networks?’, in International Conference on Machine Learning, July 2023, pp. 3347-3379. PMLR

work page 2023

[3] [3]

Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks

A. Canatar, B. Bordelon and C. Pehlevan, “Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks”, Nature Communications, 2021, 12(1), p. 2914

work page 2021

[4] [4]

The convexification effect of Minkowski summation

M. Fradelizi, M. Madiman, A. Marsiglietti and A. Zvavitch, “The convexification effect of Minkowski summation”, EMS Surveys in Mathematical Sciences, 2018, 5(1), pp. 1-64

work page 2018

[5] [5]

J. Lee, L. Xiao, S. Schoenholz, Y . Bahri, R. Novak, J. Sohl-Dickstein and J. Pennington, J., “Wide neural networks of any depth evolve as linear models under gradient descent. ¨, Advances in Neural Information Processing Systems, 2019, 32

work page 2019

[6] [6]

The expressive power of neural networks: a view from the width

Z. Lu, H. Pu, F. Wang, Z. Hu and L. Wang, “The expressive power of neural networks: a view from the width”, Advances in Neural Information Processing Cystems, 2017, 30

work page 2017

[7] [7]

Wide neural networks forget less catastrophically

S. I. Mirzadeh, A. Chaudhry, D. Yin, H., Hu, R. Pascanu, D. Gorur and M. Farajtabar, M., “Wide neural networks forget less catastrophically”, in International Conference on Machine Learning, June 2022, pp. 15699- 15717. PMLR

work page 2022

[8] [8]

The loss surface of deep and wide neural networks

Q. Nguyen and M. Hein, “The loss surface of deep and wide neural networks”, in International Conference on Machine Learning”, July 2017, pp. 2603-2612. PMLR

work page 2017