Constructive Universal Approximation and Sure Convergence for Multi-Layer Neural Networks

Chien-Ming Chi

REVIEW 3 major objections 6 minor 1 cited by

Constructive Universal Approximation and Sure Convergence for Multi-Layer Neural Networks

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Sparse 2-input network approximates any function and provably trains

desk verdict A credible constructive approximation theorem for a sparse-indicator network with a real but modest sure-convergence result; patchable proof gaps, deserves review. read the letter →

arxiv 2507.04779 v2 pith:TYAMIY6P submitted 2025-07-07 stat.ML cs.LGstat.ME

Chien-Ming Chi This is my paper

classification stat.MLcs.LGstat.ME MSC 68T0762G08

keywords o1Neurosparseindicatoractivationsconstructiveuniversalapproximationgreedyboostingsureconvergencenonconvexoptimizationregression

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a multi-layer network built from sparse indicator threshold neurons—each neuron looks at at most two lower-layer signals and outputs 0 or 1—can simultaneously achieve what most neural-network theory treats separately. At the population level it proves a constructive universal approximation result: with $L \geq 2 + \lceil \log_2 k \rceil$ hidden layers and sufficiently many neurons per layer, the greedy-trained boosted estimator converges in $L^2$ to the regression function $E[Y \mid X]$ for any measurable function of $k$ of the $p$ features. At the sample level it proves sure convergence: the random-search greedy update eventually reaches a network satisfying the sample analogue of the optimality criterion with probability approaching one. The paper also shows that three hidden layers suffice for additive models with two-way interactions, and that the required number of update rounds is well controlled in an explicit linear example. If the theorems hold, this architecture offers a tractable training procedure with provable convergence together with approximation power over essentially arbitrary regression targets.

What carries the argument

The load-bearing object is the o1Neuro neuron: a threshold indicator $f_{l,h}(x)=1\{w_{l,h}^\top f_{l-1}(x)>c_{l,h}\}$ whose weight vector satisfies $\|w_{l,h}\|_2=1$ and $\|w_{l,h}\|_0 \leq 2$, so each neuron is a bounded-fanin indicator. The estimator is a boosting machine built from these neurons, and each output neuron is chosen greedily to maximize the criterion $W_h(f)=\sum_{l=0}^1 [E(e_{h-1} 1\{f(X)=l\})]^2 / P(f(X)=l)$. The argument runs on two mechanisms. First, because the width condition makes each output neuron's subnetwork independent of the previously fixed ones, the greedy sequence is a weak greedy approximation in Hilbert space, and Lemma 8 shows that with depth $2+\lceil \log_2 k \rceil$ a single neuron can realize any $k$-coordinate rectangle, so the network class is dense in $L(k,R_0)$. Second, in the sample version, the zero-gradient plateaus of indicator activations imply that every finite-sample subnetwork pattern has positive probability under the random refreshment of idle neurons, so repeated random search with $K$ candidates eventually finds an optimal subnetwork.

What would settle it

Reproduce Example 1 exactly—noiseless linear model on binary features, $p=8$, $R_0=8$, $L=3$, $K=15$, $\gamma=1$, $\epsilon_0=0$—and check whether after $b=4L\kappa R_0 p$ rounds the network satisfies (7) with probability at least $1-R_0 e^{-\kappa K}$ for a fixed $\kappa$; if the bound fails, the paper's quantitative convergence claim is false. Alternatively, test the density step of Lemma 8 by computing the $L^2$ error of the depth-$2+\lceil \log_2 k \rceil$ construction for a $k$-coordinate rectangle; if the error does not vanish as the number of output neurons grows, the universal approximation part collapses.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that activation sparsity can be bought off with depth and width without losing universality, and that the same sparse structure makes global optimization provable. Theorem 3 states that if the population-level optimal o1Neuro has $L \geq 2 + \lceil \log_2 k \rceil$ hidden layers and $p_l \geq 2^{L-l}M$ neurons at layer $l$, then the boosted estimator $e_m$ converges in $L^2$ to $E[Y \mid X]$ for every target in $L(k,R_0)$; in particular, any measurable function of $X$ is covered when $k=p$. Theorem 7 states that the sample-level random-search optimizer—$K$ candidate weight–bias pairs drawn per round, best kept—reaches a network satisfying condition (7), the sample analogue of the greedy optimality criterion, for every output neuron with probability approaching one as the number of update rounds $b$ grows. Together the theorems establish constructive approximation: the approximation is not merely an existence statement but is realized by the algorithm the paper runs.

Load-bearing premise

The width condition $p_l \geq 2^{L-l}M$ at the population level, or $p_l \geq 2^{L-l}(M+1)$ at the sample level, is what guarantees each output neuron's subnetwork can range over the whole dictionary of base learners while previously fixed neurons stay put; if the layers are too narrow, the greedy step cannot necessarily select the best base learner, and both the population approximation proof and the sample-level optimality argument collapse.

Editorial extensions

If this is right

Any measurable regression function of $p$ bounded-density features is consistently estimated by a deep o1Neuro with $L=2+\lceil \log_2 p \rceil$ hidden layers and widths $p_l \geq 2^{L-l}M$.
Three hidden layers suffice for additive models with two-way interactions, including XOR-type terms and univariate components.
With probability approaching one, the random-search update rounds eventually produce a network satisfying the sample optimality condition, tying the approximation guarantee to a concrete algorithm rather than an existence claim.
In the explicit linear binary-feature example, the number of update rounds needed grows at most linearly in $R_0$, the number of additive components.
On the tested regression benchmarks, shallow o1Neuro configurations rank at or near the top among boosted trees and a deep tabular baseline, especially when interaction features are present.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The width condition in Theorem 3 appears to be a proof device rather than a practical necessity: the text reports that deeper networks work without condition (6) in unreported experiments, suggesting a sharper width analysis may be possible.
The depth bound $2+\lceil \log_2 k \rceil$ comes from binary merges of coordinate-threshold indicators, so allowing each neuron to read $w_0>2$ inputs should lower the required depth to roughly $\log_{w_0} k$—a simple parametric trade-off the author does not state.
The sample-level sure-convergence argument works because indicator activations make the loss piecewise constant; this suggests that hybrid random-search-plus-local-optimization schemes could accelerate training without losing the guarantee.
Because the proof establishes consistency without convergence rates, a natural next question is whether the same greedy construction admits explicit rates and a cleaner statement about how the curse of dimensionality is mitigated.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

The one-sentence take: a genuinely constructive approximation theorem for a sparse-indicator network, paired with a sample-level optimization guarantee that is correct but less impressive than it first sounds. The rectangle construction in O(log k) depth is the real novelty; the rest is a fairly honest application of greedy approximation theory.

What is actually new and good: Lemma 8 shows that 2-sparse indicator activations can implement axis-aligned rectangles at depth 2+ceil(log2 k), and the boosted greedy machine over that dictionary then approximates any measurable function of k coordinates. That depth-sparsity tradeoff is worth having, and I do not think it appears in earlier work. The proof of Theorem 3 follows Temlyakov's line and is essentially correct, provided you keep the closure D distinct from the finite span D. The sure-convergence result is also correct for what it claims: the sample-level greedy criterion is optimized by random search because the function class is finite, and the idle neuron capacity condition gives the random search enough room to realize every split. The extra width in (6) is doing real work.

Where the soft spots are: the stress-test worry about (B.13)—that the restricted max over bG_h may not equal the max over bG—looks real at first, but it does not survive reading the proof: the idle-neuron argument in (B.14) is exactly the missing step, just stated too briefly. A referee should ask for that equality to be written out. The bigger caveat is that Theorem 7 is an optimization guarantee for the empirical greedy criterion, not a generalization guarantee. The paper says as much, but readers should not take 'sure convergence' as a promise of good prediction. Example 1's concentration details are omitted; the claim is plausible but not fully checked. The width-condition equivalence in Theorem 3's proof is asserted in one sentence and should be expanded.

Who it is for: people working on constructive approximation, greedy training, or tabular deep learning. The paper is honest and the code is public. It deserves a serious referee; I would support sending it to review, with requests for expanded proofs in B.2 and B.1. The flaws are patchable, and the core construction is genuinely worth publishing.

Recommendation: engage with it.

Referee Report

3 major / 6 minor

Summary. The paper introduces o1Neuro, a multi-layer neural network with indicator activations and at most two nonzero incoming weights per neuron. The population-level model is trained by a greedy boosting procedure that sequentially optimizes output neurons over a dictionary G; the sample-level model replaces the population oracle by random candidate sampling and idle-neuron refreshment. The main theoretical results are Theorem 3, claiming that with layer widths pl≥2^{L-l}M and depth L≥2+ceil(log2 k) the population estimator converges in L2 to E[Y|X] for targets in L(k,R0) (hence to any measurable function of X when k=p), and Theorem 7, claiming that the sample algorithm reaches a model satisfying the sample analogue of the greedy optimality inequality with probability approaching one as the number of update rounds b tends to infinity. Example 1 gives a bound on b for linear binary-feature models. The paper also reports experiments comparing o1Neuro with XGBoost, Random Forests, and TabNet on synthetic and real benchmark data.

Significance. If completed, the results would be a valuable contribution to constructive approximation: an explicit architecture and training procedure with both universal approximation and optimization-convergence guarantees, linking greedy approximation theory to deep sparse networks. Unlike NTK and mean-field analyses, the guarantee does not require infinite width and is tied to a concrete finite-network algorithm; the code is publicly available, and the empirical comparison is transparent, including clearly negative results for TabNet. However, the gap between the claims and the currently written proofs is nontrivial: two load-bearing steps in the supplementary material are asserted rather than proved, and a third independence step needs clarification. The paper is promising, but it is not ready in its present form.

major comments (3)

[Supplementary B.2, Eq. (B.13)] The equality between the maximum over the restricted class bG_h and the maximum over the full class bG is asserted with the justification 'by the 2-sparse activation assumption and pl ≥ 2^{L-l}M', but no proof is given that every element of bG, in particular the argmax over bG, can be realized as the subnetwork of f_{L,h} while the subnetworks of f_{L,1},...,f_{L,h-1} are frozen. Lemma 9 only shows that a random draw can realize one fixed function in bG with positive probability under sufficient idle capacity; it does not show that the maximizing function over the restricted class attains the unrestricted maximum. Since the recursive argument proving Theorem 7 uses (B.13) at every step to conclude that each output neuron reaches a global sample optimum, this is a load-bearing gap. Please provide an explicit construction of the embedding and a lower bound on the sampling probability that is uniform across rounds.
[Supplementary B.1, proof of Theorem 3] The proof conflates the finite span D with its closure. Lemma 8 establishes L(k,R0) ⊂ closure(D), and the L2 limit ν of the residuals lives in the closure; the sentence 'both m⋆ and each m_s ... lie in D' is therefore not justified, and the contradiction argument assumes ν ∈ D ('Such a result contradicts the fact that ν(X) ∈ D'). The orthogonality argument can be repaired by working in the closure: if ν is orthogonal to every dictionary element, then it is orthogonal to every finite combination, and density gives ‖ν‖^2 = 0. As written, however, the proof of the central convergence claim contains a genuine logical gap that needs to be fixed explicitly.
[Supplementary B.2, independence claim] The proof states that the events Q_{b1}, Q_{b1+1}, ... are independent. But if an earlier event succeeds, the residuals bR_{i,h-1} change and the target function g in (B.13) can change; if the event fails, the residuals are unchanged. The events are therefore not independent in the stated unconditional sense. This can likely be fixed by conditioning on the event that no earlier success occurred and using a uniform lower bound on the conditional success probability, but the argument should be written out, since the 'probability approaching one' conclusion of Theorem 7 depends on it.

minor comments (6)

[Section 3.1, Theorem 3 statement] The expression 'lim_{M→∞} E[em(X)-E[Y|X]]^2 = 0' should be written as E[(em(X)-E[Y|X])^2] → 0; as printed, the missing parentheses make the target of the expectation ambiguous.
[Section 2.1] The set eG is defined twice: once as 'eG := {All possible f_{L,1} satisfying (1)}' and again in the following sentence as 'for each h we have eG = {All possible f_{L,h} satisfying (1)}'; the duplicate definition should be removed.
[Lemma 8 statement] The notation surrounding the closure is inconsistent: the statement writes D for what appears to be the closure of the finite span D, but the surrounding proof refers to D without consistently distinguishing the two. Please align the notation throughout the lemma and its proof.
[Table 4, Ailerons row] The entry '-0.609≪0≪04' appears corrupted; it should be replaced with the intended mean, standard deviation, and rank values in the same format as the other rows.
[Section 5.2.1] The text contains a typo: 'se Figure 3' should read 'see Figure 3'.
[Section B.3, Example 1 proof] The proof invokes 'standard concentration inequalities' but omits the details, stating only that they are omitted for brevity. Since Example 1 is the sole quantitative illustration of the required number of update rounds, this omission should be filled in, or the example should be explicitly labeled as heuristic.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the derivation is self-contained and relies on external measure theory and greedy approximation lemmas rather than on fitted quantities or self-citations.

full rationale

I found no circular step. The population-level approximation argument (Theorem 3) proceeds by showing that the sparse network class can realize the greedy dictionary D and then invokes Temlyakov's weak-greedy lemmas; the dictionary density Lemma 8 is proved from the network's own rectangle construction plus standard measure theory, i.e., it is self-contained against external benchmarks rather than assumed. The 'population optimal' condition in Section 2.1.1 is the definition of the greedy update, not a fitted quantity renamed as a prediction. The sample-level Theorem 7 is a probabilistic statement about the random-search update: Lemma 9 gives a positive-probability event for realizing an optimal split, and the proof multiplies these probabilities over layers. Whether the equality in (B.13) is fully justified is a correctness or verification question, not a circularity one. There are no load-bearing self-citations, no imported uniqueness theorems from the author's prior work, and no empirical fit is relabeled as a theoretical conclusion. The empirical sections compare o1Neuro against XGBoost, Random Forests, and TabNet on external OpenML/UCI data, which is independent evidence. I therefore assign score 0.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The theoretical claims rest on standard measure-theoretic and greedy-approximation results, the bounded-density Condition 1, and the width conditions that let each output neuron span the full dictionary. The only ad hoc assumption is the exponentially decaying coefficient condition in Example 1. No numeric free parameters are fitted to data to obtain the main theorems; γ, K, and ε0 are algorithm hyperparameters that the theorems allow to be arbitrary.

assumptions (6)

domain assumption X has a bounded probability density (Condition 1)
Invoked in Lemma 8 to equate L2 convergence under P with Lebesgue measure approximation by step functions.
domain assumption Target function E[Y|X] belongs to L(k,R0) for some k≤p (Condition 1)
Defines the class of approximable functions; for universal approximation k=p.
domain assumption Width conditions pl≥2^{L-l}M (population) and pl≥2^{L-l}(M+1) (sample)
Ensures each output neuron's subnetwork can range over the full dictionary G independently, making the greedy step equivalent to unconstrained greedy selection.
standard math Temlyakov's weak greedy approximation lemmas (Lemmas 2.3, 2.4 in Temlyakov 2000)
Used in proof of Theorem 3 to show L2 convergence of residuals.
standard math Measure-theoretic approximation of measurable functions by finite linear combinations of rectangles (Stein and Shakarchi 2009; Cohn 2013)
Basis for Lemma 8.
ad hoc to paper Exponential coefficient decay condition in Example 1: 2^{-p+1}β_j^2 > Σ_{l>j}β_l^2
Introduced specifically to make the greedy selection path provable in the example; not needed for the main theorems.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Constructive Universal Approximation and Sure Convergence for Multi-Layer Neural Networks." pith.science (2026). https://pith.science/paper/TYAMIY6P

@misc{pith2026250704779,
  author       = {Pith},
  title        = {Pith review of: Constructive Universal Approximation and Sure Convergence for Multi-Layer Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TYAMIY6P}},
  note         = {Machine review of arXiv:2507.04779}
}

abstract

We propose o1Neuro, a new neural network model built on sparse indicator activation neurons, with two key statistical properties. (1) Constructive universal approximation: At the population level, a deep o1Neuro can approximate any measurable function of $\boldsymbol{X}$, while a shallow o1Neuro suffices for additive models with two-way interaction components, including XOR and univariate terms, assuming $\boldsymbol{X} \in [0,1]^p$ has bounded density. Combined with prior work showing that a single-hidden-layer non-sparse network is a universal approximator, this highlights a trade-off between activation sparsity and network depth in approximation capability. (2) Sure convergence: At the sample level, the optimization of o1Neuro reaches an optimal model with probability approaching one after sufficiently many update rounds, and we provide an example showing that the required number of updates is well bounded under linear data-generating models. Empirically, o1Neuro is compared with XGBoost, Random Forests, and TabNet for learning complex regression functions with interactions, demonstrating superior predictive performance on several benchmark datasets from OpenML and the UCI Machine Learning Repository with $n = 10000$, as well as on synthetic datasets with $100 \le n \le 20000$.

Figures

Figures reproduced from arXiv: 2507.04779 by the authors.

**Figure 1.** Illustration of a two-hidden-layer o1Neuro network with L = 2, p1 = 4, p2 = 2, and p0 = p = 4. Solid and dashed edges indicate nonzero weights to lower neurons; dashed edges denote upper neurons unused by any output neuron. In each update round, output neurons f2,1 and f2,2 are updated sequentially. After updating f2,1 and its subnetwork (neurons in light blue in the left panel), f2,2 is next. The updated f2,2 assig… view at source ↗

**Figure 2.** In contrast, o1Neuro allows each neuron fl,h to take inputs from any neuron in the (l − 1)th layer, enabling more efficient sample-level optimization. We will see in Theorem 7 that the sample optimal o1Neuro satisfies the sample analogue of (5). The second part of Theorem 3 shows that a properly tuned o1Neuro can approximate a broad class of data-generating functions in L(k, R0) with k ≤ p, including cases where E(Y… view at source ↗

**Figure 2.** In the second panel, both g2,1 and h2,1 are from Ge at L = 2. The networks in both panels are equivalent at the population level, but each f2,h of o1Neuro can take any of f1,1, . . . , f1,4 as input, enabling more efficient sample-level optimization. measurable function of X. Taken together, our results and theirs reveal a trade-off between activation sparsity and network depth in achieving strong approximation capa… view at source ↗

Figures from the paper (1 more)

**Figure 3.** Figure 3: R2 of o1Neuro across b ∈ {1, . . . , 20}; [PITH_FULL_IMAGE:figures/full_fig_p014_3.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Breaking the Curse with BAND: Nonparametric Distribution Estimation in High Dimensions
stat.ML 2026-07 conditional novelty 6.0 of 10

Sparse Bayesian-network factorization plus sparsity-aware regression yields polynomial TV rates for high-dimensional mixed-type distribution estimation, beating classical histogram rates under sparsity.

Reference graph

Works this paper leans on

55 extracted references · 51 canonical work pages · cited by 1 Pith paper

[1]

Adcock, B. and N. Dexter (2021). The gap between theory and practice in function approximation with deep neural networks. SIAM Journal on Mathematics of Data Science\/ 3\/ (2), 624--655

work page 2021
[2]

Arora, S., S. Du, W. Hu, Z. Li, and R. Wang (2019). Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International conference on machine learning , pp.\ 322--332. PMLR

work page 2019
[3]

Arora, S., S. S. Du, W. Hu, Z. Li, R. R. Salakhutdinov, and R. Wang (2019). On exact computation with an infinitely wide neural net. Advances in neural information processing systems\/ 32

work page 2019
[4]

Barron, A. R. (1993). Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory\/ 39\/ (3), 930--945

work page 1993
[5]

Barron, A. R., A. Cohen, W. Dahmen, and R. A. DeVore (2008). Approximation and learning by greedy algorithms . The Annals of Statistics\/ 36\/ (1), 64 -- 94

work page 2008
[6]

Bauer, B. and M. Kohler (2019). On deep learning as a remedy for the curse of dimensionality in nonparametric regression. Ann. Statist. 47 (4) 2261 - 2285, August 2019.\/

work page 2019
[7]

Biau, G. and B. Cadre (2021). Optimization by gradient boosting. In Advances in Contemporary Statistics and Econometrics: Festschrift in Honor of Christine Thomas-Agnan , pp.\ 23--44. Springer

work page 2021
[8]

Breiman, L. (2001). Random forests. Machine learning\/ 45 , 5--32

work page 2001

Show all 55 references

[9]

Brown, T., B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020). Language models are few-shot learners. Advances in neural information processing systems\/ 33 , 1877--1901

2020
[10]

Cao, Y. and Q. Gu (2019). Generalization bounds of stochastic gradient descent for wide and deep neural networks. Advances in neural information processing systems\/ 32

2019
[11]

Chen, T. and C. Guestrin (2016). Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining , pp.\ 785--794

2016
[12]

Henaff, M

Choromanska, A., M. Henaff, M. Mathieu, G. B. Arous, and Y. LeCun (2015). The loss surfaces of multilayer networks. In Artificial intelligence and statistics , pp.\ 192--204. PMLR

2015
[13]

Cohn, D. L. (2013). Measure theory , Volume 2. Springer

2013
[14]

Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems\/ 2\/ (4), 303--314

1989
[15]

Bottou, F

D \'e fossez, A., L. Bottou, F. Bach, and N. Usunier (2020). A simple convergence proof of adam and adagrad. arXiv preprint arXiv:2003.02395\/

2020 arXiv
[16]

Hazan, and Y

Duchi, J., E. Hazan, and Y. Singer (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research\/ 12\/ (7)

2011
[17]

Eldan, R. and O. Shamir (2016). The power of depth for feedforward neural networks. In Conference on learning theory , pp.\ 907--940. PMLR

2016
[18]

Ma, and Y

Fan, J., C. Ma, and Y. Zhong (2020). A selective overview of deep learning. Statistical science: a review journal of the Institute of Mathematical Statistics\/ 36\/ (2), 264

2020
[19]

Friedman, J. H. (2001). Greedy function approximation: a gradient boosting machine. Annals of statistics\/ , 1189--1232

2001
[20]

Gentile, R. and G. Welper (2024). Approximation results for gradient flow trained shallow neural networks in 1d. Constructive Approximation\/ 60\/ (3), 547--594

2024
[21]

Bengio, and A

Goodfellow, I., Y. Bengio, and A. Courville (2016). Deep Learning . MIT Press. http://www.deeplearningbook.org

2016
[22]

Gribonval, R. and M. Nielsen (2001). Approximate weak greedy algorithms. Advances in Computational Mathematics\/ 14 , 361--378

2001
[23]

Oyallon, and G

Grinsztajn, L., E. Oyallon, and G. Varoquaux (2022). Why do tree-based models still outperform deep learning on typical tabular data? Advances in neural information processing systems\/ 35 , 507--520

2022
[24]

Grohs, P. and F. Voigtlaender (2024). Proof of the theory-to-practice gap in deep learning via sampling complexity bounds for neural network approximation spaces. Foundations of Computational Mathematics\/ 24\/ (4), 1085--1143

2024
[25]

Tibshirani, J

Hastie, T., R. Tibshirani, J. H. Friedman, and J. H. Friedman (2009). The elements of statistical learning: data mining, inference, and prediction , Volume 2. Springer

2009
[26]

Zhang, S

He, K., X. Zhang, S. Ren, and J. Sun (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp.\ 770--778

2016
[27]

Herrmann, L., J. A. Opschoor, and C. Schwab (2022). Constructive deep relu neural network approximation. Journal of Scientific Computing\/ 90\/ (2), 75

2022
[28]

Hinton, G. (2012). Neural networks for machine learning — lecture 6a: Overview of mini-batch gradient descent. Coursera Course

2012
[29]

M \"u ller, K

Hollmann, N., S. M \"u ller, K. Eggensperger, and F. Hutter (2022). Tabpfn: A transformer that solves small tabular classification problems in a second. arXiv preprint arXiv:2207.01848\/

2022 arXiv
[30]

Stinchcombe, and H

Hornik, K., M. Stinchcombe, and H. White (1989). Multilayer feedforward networks are universal approximators. Neural networks\/ 2\/ (5), 359--366

1989
[31]

Gabriel, and C

Jacot, A., F. Gabriel, and C. Hongler (2018). Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems\/ 31

2018
[32]

Jentzen, A. and A. Riekert (2024). Non-convergence to global minimizers for adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networks. arXiv preprint arXiv:2402.05155\/

2024 arXiv
[33]

Evans, A

Jumper, J., R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Z \' dek, A. Potapenko, et al. (2021). Highly accurate protein structure prediction with alphafold. nature\/ 596\/ (7873), 583--589

2021
[34]

Kingma, D. P. and J. Ba (2015). Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations (ICLR)

2015
[35]

Bengio, and G

LeCun, Y., Y. Bengio, and G. Hinton (2015). Deep learning. nature\/ 521\/ (7553), 436--444

2015
[36]

Rakhlin, and A

Li, H., A. Rakhlin, and A. Jadbabaie (2023). Convergence of adam under relaxed assumptions. Advances in Neural Information Processing Systems\/ 36 , 52166--52196

2023
[37]

Li, Y., Y. Guo, S. Zhang, S. Deng, Y. Hai, and S. Gu (2021). Differentiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in neural information processing systems\/ 34 , 23426--23439

2021
[38]

Wettig, D

Malladi, S., A. Wettig, D. Yu, D. Chen, and S. Arora (2023). A kernel-based view of language model fine-tuning. In International Conference on Machine Learning , pp.\ 23610--23641. PMLR

2023
[39]

Markelle Kelly, Rachel Longjohn, K. N. (2017). The uci machine learning repository

2017
[40]

Khandagale, J

McElfresh, D., S. Khandagale, J. Valverde, V. Prasad C, G. Ramakrishnan, M. Goldblum, and C. White (2023). When do neural nets outperform boosted trees on tabular data? Advances in Neural Information Processing Systems\/ 36 , 76336--76369

2023
[41]

Misiakiewicz, and A

Mei, S., T. Misiakiewicz, and A. Montanari (2019). Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit. In Conference on learning theory , pp.\ 2388--2464. PMLR

2019
[42]

Montanari, and P.-M

Mei, S., A. Montanari, and P.-M. Nguyen (2018). A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences\/ 115\/ (33), E7665--E7671

2018
[43]

Gusev, A

Prokhorenkova, L., G. Gusev, A. Vorobev, A. V. Dorogush, and A. Gulin (2018). Catboost: unbiased boosting with categorical features. Advances in neural information processing systems\/ 31

2018
[44]

Robbins, H. and S. Monro (1951). A stochastic approximation method. The annals of mathematical statistics\/ , 400--407

1951
[45]

Schmidt-Hieber, J. (2020). Nonparametric regression using deep neural networks with relu activation function. The Annals of Statistics\/

2020
[46]

Shwartz-Ziv, R. and A. Armon (2022). Tabular data: Deep learning is not all you need. Information Fusion\/ 81 , 84--90

2022
[47]

Siegel, J. W. and J. Xu (2022). Optimal convergence rates for the orthogonal greedy algorithm. IEEE Transactions on Information Theory\/ 68\/ (5), 3354--3361

2022
[48]

Huang, C

Silver, D., A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. (2016). Mastering the game of go with deep neural networks and tree search. nature\/ 529\/ (7587), 484--489

2016
[49]

Goldblum, A

Somepalli, G., M. Goldblum, A. Schwarzschild, C. B. Bruss, and T. Goldstein (2021). Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342\/

2021 arXiv
[50]

Stein, E. M. and R. Shakarchi (2009). Real analysis: measure theory, integration, and Hilbert spaces . Princeton University Press

2009
[51]

Telgarsky, M. (2016). Benefits of depth in neural networks. In Conference on learning theory , pp.\ 1517--1539. PMLR

2016
[52]

Temlyakov, V. N. (2000). Weak greedy algorithms. Advances in Computational Mathematics\/ 12\/ (2), 213--227

2000
[53]

Vanschoren, J., J. N. van Rijn, B. Bischl, and L. Torgo (2013). Openml: Networked science in machine learning. SIGKDD Explorations\/ 15\/ (2), 49--60

2013
[54]

Xu, J. and H. Zhu (2024). Overparametrized multi-layer neural networks: Uniform concentration of neural tangent kernel and convergence of stochastic gradient descent. Journal of Machine Learning Research\/ 25\/ (94), 1--83

2024
[55]

Zhang, Y., C. Chen, N. Shi, R. Sun, and Z.-Q. Luo (2022). Adam can converge without any modification on update rules. Advances in neural information processing systems\/ 35 , 28386--28399

2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Adcock, B. and N. Dexter (2021). The gap between theory and practice in function approximation with deep neural networks. SIAM Journal on Mathematics of Data Science\/ 3\/ (2), 624--655

work page 2021

[2] [2]

Arora, S., S. Du, W. Hu, Z. Li, and R. Wang (2019). Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International conference on machine learning , pp.\ 322--332. PMLR

work page 2019

[3] [3]

Arora, S., S. S. Du, W. Hu, Z. Li, R. R. Salakhutdinov, and R. Wang (2019). On exact computation with an infinitely wide neural net. Advances in neural information processing systems\/ 32

work page 2019

[4] [4]

Barron, A. R. (1993). Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory\/ 39\/ (3), 930--945

work page 1993

[5] [5]

Barron, A. R., A. Cohen, W. Dahmen, and R. A. DeVore (2008). Approximation and learning by greedy algorithms . The Annals of Statistics\/ 36\/ (1), 64 -- 94

work page 2008

[6] [6]

Bauer, B. and M. Kohler (2019). On deep learning as a remedy for the curse of dimensionality in nonparametric regression. Ann. Statist. 47 (4) 2261 - 2285, August 2019.\/

work page 2019

[7] [7]

Biau, G. and B. Cadre (2021). Optimization by gradient boosting. In Advances in Contemporary Statistics and Econometrics: Festschrift in Honor of Christine Thomas-Agnan , pp.\ 23--44. Springer

work page 2021

[8] [8]

Breiman, L. (2001). Random forests. Machine learning\/ 45 , 5--32

work page 2001