Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Constructive Universal Approximation and Sure Convergence for Multi-Layer Neural Networks

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Sparse 2-input network approximates any function and provably trains

desk verdict A credible constructive approximation theorem for a sparse-indicator network with a real but modest sure-convergence result; patchable proof gaps, deserves review. read the letter →

arxiv 2507.04779 v2 pith:TYAMIY6P submitted 2025-07-07 stat.ML cs.LGstat.ME

Chien-Ming Chi This is my paper
classification stat.MLcs.LGstat.ME MSC 68T0762G08
keywords o1Neurosparseindicatoractivationsconstructiveuniversalapproximationgreedyboostingsureconvergencenonconvexoptimizationregression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a multi-layer network built from sparse indicator threshold neurons—each neuron looks at at most two lower-layer signals and outputs 0 or 1—can simultaneously achieve what most neural-network theory treats separately. At the population level it proves a constructive universal approximation result: with $L \geq 2 + \lceil \log_2 k \rceil$ hidden layers and sufficiently many neurons per layer, the greedy-trained boosted estimator converges in $L^2$ to the regression function $E[Y \mid X]$ for any measurable function of $k$ of the $p$ features. At the sample level it proves sure convergence: the random-search greedy update eventually reaches a network satisfying the sample analogue of the optimality criterion with probability approaching one. The paper also shows that three hidden layers suffice for additive models with two-way interactions, and that the required number of update rounds is well controlled in an explicit linear example. If the theorems hold, this architecture offers a tractable training procedure with provable convergence together with approximation power over essentially arbitrary regression targets.

What carries the argument

The load-bearing object is the o1Neuro neuron: a threshold indicator $f_{l,h}(x)=1\{w_{l,h}^\top f_{l-1}(x)>c_{l,h}\}$ whose weight vector satisfies $\|w_{l,h}\|_2=1$ and $\|w_{l,h}\|_0 \leq 2$, so each neuron is a bounded-fanin indicator. The estimator is a boosting machine built from these neurons, and each output neuron is chosen greedily to maximize the criterion $W_h(f)=\sum_{l=0}^1 [E(e_{h-1} 1\{f(X)=l\})]^2 / P(f(X)=l)$. The argument runs on two mechanisms. First, because the width condition makes each output neuron's subnetwork independent of the previously fixed ones, the greedy sequence is a weak greedy approximation in Hilbert space, and Lemma 8 shows that with depth $2+\lceil \log_2 k \rceil$ a single neuron can realize any $k$-coordinate rectangle, so the network class is dense in $L(k,R_0)$. Second, in the sample version, the zero-gradient plateaus of indicator activations imply that every finite-sample subnetwork pattern has positive probability under the random refreshment of idle neurons, so repeated random search with $K$ candidates eventually finds an optimal subnetwork.

What would settle it

Reproduce Example 1 exactly—noiseless linear model on binary features, $p=8$, $R_0=8$, $L=3$, $K=15$, $\gamma=1$, $\epsilon_0=0$—and check whether after $b=4L\kappa R_0 p$ rounds the network satisfies (7) with probability at least $1-R_0 e^{-\kappa K}$ for a fixed $\kappa$; if the bound fails, the paper's quantitative convergence claim is false. Alternatively, test the density step of Lemma 8 by computing the $L^2$ error of the depth-$2+\lceil \log_2 k \rceil$ construction for a $k$-coordinate rectangle; if the error does not vanish as the number of output neurons grows, the universal approximation part collapses.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that activation sparsity can be bought off with depth and width without losing universality, and that the same sparse structure makes global optimization provable. Theorem 3 states that if the population-level optimal o1Neuro has $L \geq 2 + \lceil \log_2 k \rceil$ hidden layers and $p_l \geq 2^{L-l}M$ neurons at layer $l$, then the boosted estimator $e_m$ converges in $L^2$ to $E[Y \mid X]$ for every target in $L(k,R_0)$; in particular, any measurable function of $X$ is covered when $k=p$. Theorem 7 states that the sample-level random-search optimizer—$K$ candidate weight–bias pairs drawn per round, best kept—reaches a network satisfying condition (7), the sample analogue of the greedy optimality criterion, for every output neuron with probability approaching one as the number of update rounds $b$ grows. Together the theorems establish constructive approximation: the approximation is not merely an existence statement but is realized by the algorithm the paper runs.

Load-bearing premise

The width condition $p_l \geq 2^{L-l}M$ at the population level, or $p_l \geq 2^{L-l}(M+1)$ at the sample level, is what guarantees each output neuron's subnetwork can range over the whole dictionary of base learners while previously fixed neurons stay put; if the layers are too narrow, the greedy step cannot necessarily select the best base learner, and both the population approximation proof and the sample-level optimality argument collapse.

Editorial extensions

If this is right

  • Any measurable regression function of $p$ bounded-density features is consistently estimated by a deep o1Neuro with $L=2+\lceil \log_2 p \rceil$ hidden layers and widths $p_l \geq 2^{L-l}M$.
  • Three hidden layers suffice for additive models with two-way interactions, including XOR-type terms and univariate components.
  • With probability approaching one, the random-search update rounds eventually produce a network satisfying the sample optimality condition, tying the approximation guarantee to a concrete algorithm rather than an existence claim.
  • In the explicit linear binary-feature example, the number of update rounds needed grows at most linearly in $R_0$, the number of additive components.
  • On the tested regression benchmarks, shallow o1Neuro configurations rank at or near the top among boosted trees and a deep tabular baseline, especially when interaction features are present.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The width condition in Theorem 3 appears to be a proof device rather than a practical necessity: the text reports that deeper networks work without condition (6) in unreported experiments, suggesting a sharper width analysis may be possible.
  • The depth bound $2+\lceil \log_2 k \rceil$ comes from binary merges of coordinate-threshold indicators, so allowing each neuron to read $w_0>2$ inputs should lower the required depth to roughly $\log_{w_0} k$—a simple parametric trade-off the author does not state.
  • The sample-level sure-convergence argument works because indicator activations make the loss piecewise constant; this suggests that hybrid random-search-plus-local-optimization schemes could accelerate training without losing the guarantee.
  • Because the proof establishes consistency without convergence rates, a natural next question is whether the same greedy construction admits explicit rates and a cleaner statement about how the curse of dimensionality is mitigated.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces o1Neuro, a multi-layer neural network with indicator activations and at most two nonzero incoming weights per neuron. The population-level model is trained by a greedy boosting procedure that sequentially optimizes output neurons over a dictionary G; the sample-level model replaces the population oracle by random candidate sampling and idle-neuron refreshment. The main theoretical results are Theorem 3, claiming that with layer widths pl≥2^{L-l}M and depth L≥2+ceil(log2 k) the population estimator converges in L2 to E[Y|X] for targets in L(k,R0) (hence to any measurable function of X when k=p), and Theorem 7, claiming that the sample algorithm reaches a model satisfying the sample analogue of the greedy optimality inequality with probability approaching one as the number of update rounds b tends to infinity. Example 1 gives a bound on b for linear binary-feature models. The paper also reports experiments comparing o1Neuro with XGBoost, Random Forests, and TabNet on synthetic and real benchmark data.

Significance. If completed, the results would be a valuable contribution to constructive approximation: an explicit architecture and training procedure with both universal approximation and optimization-convergence guarantees, linking greedy approximation theory to deep sparse networks. Unlike NTK and mean-field analyses, the guarantee does not require infinite width and is tied to a concrete finite-network algorithm; the code is publicly available, and the empirical comparison is transparent, including clearly negative results for TabNet. However, the gap between the claims and the currently written proofs is nontrivial: two load-bearing steps in the supplementary material are asserted rather than proved, and a third independence step needs clarification. The paper is promising, but it is not ready in its present form.

major comments (3)
  1. [Supplementary B.2, Eq. (B.13)] The equality between the maximum over the restricted class bG_h and the maximum over the full class bG is asserted with the justification 'by the 2-sparse activation assumption and pl ≥ 2^{L-l}M', but no proof is given that every element of bG, in particular the argmax over bG, can be realized as the subnetwork of f_{L,h} while the subnetworks of f_{L,1},...,f_{L,h-1} are frozen. Lemma 9 only shows that a random draw can realize one fixed function in bG with positive probability under sufficient idle capacity; it does not show that the maximizing function over the restricted class attains the unrestricted maximum. Since the recursive argument proving Theorem 7 uses (B.13) at every step to conclude that each output neuron reaches a global sample optimum, this is a load-bearing gap. Please provide an explicit construction of the embedding and a lower bound on the sampling probability that is uniform across rounds.
  2. [Supplementary B.1, proof of Theorem 3] The proof conflates the finite span D with its closure. Lemma 8 establishes L(k,R0) ⊂ closure(D), and the L2 limit ν of the residuals lives in the closure; the sentence 'both m⋆ and each m_s ... lie in D' is therefore not justified, and the contradiction argument assumes ν ∈ D ('Such a result contradicts the fact that ν(X) ∈ D'). The orthogonality argument can be repaired by working in the closure: if ν is orthogonal to every dictionary element, then it is orthogonal to every finite combination, and density gives ‖ν‖^2 = 0. As written, however, the proof of the central convergence claim contains a genuine logical gap that needs to be fixed explicitly.
  3. [Supplementary B.2, independence claim] The proof states that the events Q_{b1}, Q_{b1+1}, ... are independent. But if an earlier event succeeds, the residuals bR_{i,h-1} change and the target function g in (B.13) can change; if the event fails, the residuals are unchanged. The events are therefore not independent in the stated unconditional sense. This can likely be fixed by conditioning on the event that no earlier success occurred and using a uniform lower bound on the conditional success probability, but the argument should be written out, since the 'probability approaching one' conclusion of Theorem 7 depends on it.
minor comments (6)
  1. [Section 3.1, Theorem 3 statement] The expression 'lim_{M→∞} E[em(X)-E[Y|X]]^2 = 0' should be written as E[(em(X)-E[Y|X])^2] → 0; as printed, the missing parentheses make the target of the expectation ambiguous.
  2. [Section 2.1] The set eG is defined twice: once as 'eG := {All possible f_{L,1} satisfying (1)}' and again in the following sentence as 'for each h we have eG = {All possible f_{L,h} satisfying (1)}'; the duplicate definition should be removed.
  3. [Lemma 8 statement] The notation surrounding the closure is inconsistent: the statement writes D for what appears to be the closure of the finite span D, but the surrounding proof refers to D without consistently distinguishing the two. Please align the notation throughout the lemma and its proof.
  4. [Table 4, Ailerons row] The entry '-0.609≪0≪04' appears corrupted; it should be replaced with the intended mean, standard deviation, and rank values in the same format as the other rows.
  5. [Section 5.2.1] The text contains a typo: 'se Figure 3' should read 'see Figure 3'.
  6. [Section B.3, Example 1 proof] The proof invokes 'standard concentration inequalities' but omits the details, stating only that they are omitted for brevity. Since Example 1 is the sole quantitative illustration of the required number of update rounds, this omission should be filled in, or the example should be explicitly labeled as heuristic.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the derivation is self-contained and relies on external measure theory and greedy approximation lemmas rather than on fitted quantities or self-citations.

full rationale

I found no circular step. The population-level approximation argument (Theorem 3) proceeds by showing that the sparse network class can realize the greedy dictionary D and then invokes Temlyakov's weak-greedy lemmas; the dictionary density Lemma 8 is proved from the network's own rectangle construction plus standard measure theory, i.e., it is self-contained against external benchmarks rather than assumed. The 'population optimal' condition in Section 2.1.1 is the definition of the greedy update, not a fitted quantity renamed as a prediction. The sample-level Theorem 7 is a probabilistic statement about the random-search update: Lemma 9 gives a positive-probability event for realizing an optimal split, and the proof multiplies these probabilities over layers. Whether the equality in (B.13) is fully justified is a correctness or verification question, not a circularity one. There are no load-bearing self-citations, no imported uniqueness theorems from the author's prior work, and no empirical fit is relabeled as a theoretical conclusion. The empirical sections compare o1Neuro against XGBoost, Random Forests, and TabNet on external OpenML/UCI data, which is independent evidence. I therefore assign score 0.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The theoretical claims rest on standard measure-theoretic and greedy-approximation results, the bounded-density Condition 1, and the width conditions that let each output neuron span the full dictionary. The only ad hoc assumption is the exponentially decaying coefficient condition in Example 1. No numeric free parameters are fitted to data to obtain the main theorems; γ, K, and ε0 are algorithm hyperparameters that the theorems allow to be arbitrary.

assumptions (6)
  • domain assumption X has a bounded probability density (Condition 1)
    Invoked in Lemma 8 to equate L2 convergence under P with Lebesgue measure approximation by step functions.
  • domain assumption Target function E[Y|X] belongs to L(k,R0) for some k≤p (Condition 1)
    Defines the class of approximable functions; for universal approximation k=p.
  • domain assumption Width conditions pl≥2^{L-l}M (population) and pl≥2^{L-l}(M+1) (sample)
    Ensures each output neuron's subnetwork can range over the full dictionary G independently, making the greedy step equivalent to unconstrained greedy selection.
  • standard math Temlyakov's weak greedy approximation lemmas (Lemmas 2.3, 2.4 in Temlyakov 2000)
    Used in proof of Theorem 3 to show L2 convergence of residuals.
  • standard math Measure-theoretic approximation of measurable functions by finite linear combinations of rectangles (Stein and Shakarchi 2009; Cohn 2013)
    Basis for Lemma 8.
  • ad hoc to paper Exponential coefficient decay condition in Example 1: 2^{-p+1}β_j^2 > Σ_{l>j}β_l^2
    Introduced specifically to make the greedy selection path provable in the example; not needed for the main theorems.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Constructive Universal Approximation and Sure Convergence for Multi-Layer Neural Networks." pith.science (2026). https://pith.science/paper/TYAMIY6P

@misc{pith2026250704779,
  author       = {Pith},
  title        = {Pith review of: Constructive Universal Approximation and Sure Convergence for Multi-Layer Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TYAMIY6P}},
  note         = {Machine review of arXiv:2507.04779}
}
abstract

We propose o1Neuro, a new neural network model built on sparse indicator activation neurons, with two key statistical properties. (1) Constructive universal approximation: At the population level, a deep o1Neuro can approximate any measurable function of $\boldsymbol{X}$, while a shallow o1Neuro suffices for additive models with two-way interaction components, including XOR and univariate terms, assuming $\boldsymbol{X} \in [0,1]^p$ has bounded density. Combined with prior work showing that a single-hidden-layer non-sparse network is a universal approximator, this highlights a trade-off between activation sparsity and network depth in approximation capability. (2) Sure convergence: At the sample level, the optimization of o1Neuro reaches an optimal model with probability approaching one after sufficiently many update rounds, and we provide an example showing that the required number of updates is well bounded under linear data-generating models. Empirically, o1Neuro is compared with XGBoost, Random Forests, and TabNet for learning complex regression functions with interactions, demonstrating superior predictive performance on several benchmark datasets from OpenML and the UCI Machine Learning Repository with $n = 10000$, as well as on synthetic datasets with $100 \le n \le 20000$.

Figures

Figures reproduced from arXiv: 2507.04779 by the authors.

Figure 1
Figure 1. Illustration of a two-hidden-layer o1Neuro network with L = 2, p1 = 4, p2 = 2, and p0 = p = 4. Solid and dashed edges indicate nonzero weights to lower neurons; dashed edges denote upper neurons unused by any output neuron. In each update round, output neurons f2,1 and f2,2 are updated sequentially. After updating f2,1 and its subnetwork (neurons in light blue in the left panel), f2,2 is next. The updated f2,2 assig… view at source ↗
Figure 2
Figure 2. In contrast, o1Neuro allows each neuron fl,h to take inputs from any neuron in the (l − 1)th layer, enabling more efficient sample-level optimization. We will see in Theorem 7 that the sample optimal o1Neuro satisfies the sample analogue of (5). The second part of Theorem 3 shows that a properly tuned o1Neuro can approximate a broad class of data-generating functions in L(k, R0) with k ≤ p, including cases where E(Y… view at source ↗
Figure 2
Figure 2. In the second panel, both g2,1 and h2,1 are from Ge at L = 2. The networks in both panels are equivalent at the population level, but each f2,h of o1Neuro can take any of f1,1, . . . , f1,4 as input, enabling more efficient sample-level optimization. measurable function of X. Taken together, our results and theirs reveal a trade-off between activation sparsity and network depth in achieving strong approximation capa… view at source ↗
Figures from the paper (1 more)
Figure 3
Figure 3. Figure 3: R2 of o1Neuro across b ∈ {1, . . . , 20}; [PITH_FULL_IMAGE:figures/full_fig_p014_3.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Breaking the Curse with BAND: Nonparametric Distribution Estimation in High Dimensions

    stat.ML 2026-07 conditional novelty 6.0 of 10

    Sparse Bayesian-network factorization plus sparsity-aware regression yields polynomial TV rates for high-dimensional mixed-type distribution estimation, beating classical histogram rates under sparsity.

Reference graph

Works this paper leans on

55 extracted references · 51 canonical work pages · cited by 1 Pith paper

  1. [1]

    Adcock, B. and N. Dexter (2021). The gap between theory and practice in function approximation with deep neural networks. SIAM Journal on Mathematics of Data Science\/ 3\/ (2), 624--655

    work page 2021

  2. [2]

    Arora, S., S. Du, W. Hu, Z. Li, and R. Wang (2019). Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International conference on machine learning , pp.\ 322--332. PMLR

    work page 2019

  3. [3]

    Arora, S., S. S. Du, W. Hu, Z. Li, R. R. Salakhutdinov, and R. Wang (2019). On exact computation with an infinitely wide neural net. Advances in neural information processing systems\/ 32

    work page 2019

  4. [4]

    Barron, A. R. (1993). Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory\/ 39\/ (3), 930--945

    work page 1993

  5. [5]

    Barron, A. R., A. Cohen, W. Dahmen, and R. A. DeVore (2008). Approximation and learning by greedy algorithms . The Annals of Statistics\/ 36\/ (1), 64 -- 94

    work page 2008

  6. [6]

    Bauer, B. and M. Kohler (2019). On deep learning as a remedy for the curse of dimensionality in nonparametric regression. Ann. Statist. 47 (4) 2261 - 2285, August 2019.\/

    work page 2019

  7. [7]

    Biau, G. and B. Cadre (2021). Optimization by gradient boosting. In Advances in Contemporary Statistics and Econometrics: Festschrift in Honor of Christine Thomas-Agnan , pp.\ 23--44. Springer

    work page 2021

  8. [8]

    Breiman, L. (2001). Random forests. Machine learning\/ 45 , 5--32

    work page 2001

Show all 55 references
  1. [9]

    Brown, T., B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020). Language models are few-shot learners. Advances in neural information processing systems\/ 33 , 1877--1901

    2020

  2. [10]

    Cao, Y. and Q. Gu (2019). Generalization bounds of stochastic gradient descent for wide and deep neural networks. Advances in neural information processing systems\/ 32

    2019

  3. [11]

    Chen, T. and C. Guestrin (2016). Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining , pp.\ 785--794

    2016

  4. [12]

    Henaff, M

    Choromanska, A., M. Henaff, M. Mathieu, G. B. Arous, and Y. LeCun (2015). The loss surfaces of multilayer networks. In Artificial intelligence and statistics , pp.\ 192--204. PMLR

    2015

  5. [13]

    Cohn, D. L. (2013). Measure theory , Volume 2. Springer

    2013

  6. [14]

    Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems\/ 2\/ (4), 303--314

    1989

  7. [15]

    Bottou, F

    D \'e fossez, A., L. Bottou, F. Bach, and N. Usunier (2020). A simple convergence proof of adam and adagrad. arXiv preprint arXiv:2003.02395\/

    2020 arXiv

  8. [16]

    Hazan, and Y

    Duchi, J., E. Hazan, and Y. Singer (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research\/ 12\/ (7)

    2011

  9. [17]

    Eldan, R. and O. Shamir (2016). The power of depth for feedforward neural networks. In Conference on learning theory , pp.\ 907--940. PMLR

    2016

  10. [18]

    Ma, and Y

    Fan, J., C. Ma, and Y. Zhong (2020). A selective overview of deep learning. Statistical science: a review journal of the Institute of Mathematical Statistics\/ 36\/ (2), 264

    2020

  11. [19]

    Friedman, J. H. (2001). Greedy function approximation: a gradient boosting machine. Annals of statistics\/ , 1189--1232

    2001

  12. [20]

    Gentile, R. and G. Welper (2024). Approximation results for gradient flow trained shallow neural networks in 1d. Constructive Approximation\/ 60\/ (3), 547--594

    2024

  13. [21]

    Bengio, and A

    Goodfellow, I., Y. Bengio, and A. Courville (2016). Deep Learning . MIT Press. http://www.deeplearningbook.org

    2016

  14. [22]

    Gribonval, R. and M. Nielsen (2001). Approximate weak greedy algorithms. Advances in Computational Mathematics\/ 14 , 361--378

    2001

  15. [23]

    Oyallon, and G

    Grinsztajn, L., E. Oyallon, and G. Varoquaux (2022). Why do tree-based models still outperform deep learning on typical tabular data? Advances in neural information processing systems\/ 35 , 507--520

    2022

  16. [24]

    Grohs, P. and F. Voigtlaender (2024). Proof of the theory-to-practice gap in deep learning via sampling complexity bounds for neural network approximation spaces. Foundations of Computational Mathematics\/ 24\/ (4), 1085--1143

    2024

  17. [25]

    Tibshirani, J

    Hastie, T., R. Tibshirani, J. H. Friedman, and J. H. Friedman (2009). The elements of statistical learning: data mining, inference, and prediction , Volume 2. Springer

    2009

  18. [26]

    Zhang, S

    He, K., X. Zhang, S. Ren, and J. Sun (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp.\ 770--778

    2016

  19. [27]

    Herrmann, L., J. A. Opschoor, and C. Schwab (2022). Constructive deep relu neural network approximation. Journal of Scientific Computing\/ 90\/ (2), 75

    2022

  20. [28]

    Hinton, G. (2012). Neural networks for machine learning — lecture 6a: Overview of mini-batch gradient descent. Coursera Course

    2012

  21. [29]

    M \"u ller, K

    Hollmann, N., S. M \"u ller, K. Eggensperger, and F. Hutter (2022). Tabpfn: A transformer that solves small tabular classification problems in a second. arXiv preprint arXiv:2207.01848\/

    2022 arXiv

  22. [30]

    Stinchcombe, and H

    Hornik, K., M. Stinchcombe, and H. White (1989). Multilayer feedforward networks are universal approximators. Neural networks\/ 2\/ (5), 359--366

    1989

  23. [31]

    Gabriel, and C

    Jacot, A., F. Gabriel, and C. Hongler (2018). Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems\/ 31

    2018

  24. [32]

    Jentzen, A. and A. Riekert (2024). Non-convergence to global minimizers for adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networks. arXiv preprint arXiv:2402.05155\/

    2024 arXiv

  25. [33]

    Evans, A

    Jumper, J., R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Z \' dek, A. Potapenko, et al. (2021). Highly accurate protein structure prediction with alphafold. nature\/ 596\/ (7873), 583--589

    2021

  26. [34]

    Kingma, D. P. and J. Ba (2015). Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations (ICLR)

    2015

  27. [35]

    Bengio, and G

    LeCun, Y., Y. Bengio, and G. Hinton (2015). Deep learning. nature\/ 521\/ (7553), 436--444

    2015

  28. [36]

    Rakhlin, and A

    Li, H., A. Rakhlin, and A. Jadbabaie (2023). Convergence of adam under relaxed assumptions. Advances in Neural Information Processing Systems\/ 36 , 52166--52196

    2023

  29. [37]

    Li, Y., Y. Guo, S. Zhang, S. Deng, Y. Hai, and S. Gu (2021). Differentiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in neural information processing systems\/ 34 , 23426--23439

    2021

  30. [38]

    Wettig, D

    Malladi, S., A. Wettig, D. Yu, D. Chen, and S. Arora (2023). A kernel-based view of language model fine-tuning. In International Conference on Machine Learning , pp.\ 23610--23641. PMLR

    2023

  31. [39]

    Markelle Kelly, Rachel Longjohn, K. N. (2017). The uci machine learning repository

    2017

  32. [40]

    Khandagale, J

    McElfresh, D., S. Khandagale, J. Valverde, V. Prasad C, G. Ramakrishnan, M. Goldblum, and C. White (2023). When do neural nets outperform boosted trees on tabular data? Advances in Neural Information Processing Systems\/ 36 , 76336--76369

    2023

  33. [41]

    Misiakiewicz, and A

    Mei, S., T. Misiakiewicz, and A. Montanari (2019). Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit. In Conference on learning theory , pp.\ 2388--2464. PMLR

    2019

  34. [42]

    Montanari, and P.-M

    Mei, S., A. Montanari, and P.-M. Nguyen (2018). A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences\/ 115\/ (33), E7665--E7671

    2018

  35. [43]

    Gusev, A

    Prokhorenkova, L., G. Gusev, A. Vorobev, A. V. Dorogush, and A. Gulin (2018). Catboost: unbiased boosting with categorical features. Advances in neural information processing systems\/ 31

    2018

  36. [44]

    Robbins, H. and S. Monro (1951). A stochastic approximation method. The annals of mathematical statistics\/ , 400--407

    1951

  37. [45]

    Schmidt-Hieber, J. (2020). Nonparametric regression using deep neural networks with relu activation function. The Annals of Statistics\/

    2020

  38. [46]

    Shwartz-Ziv, R. and A. Armon (2022). Tabular data: Deep learning is not all you need. Information Fusion\/ 81 , 84--90

    2022

  39. [47]

    Siegel, J. W. and J. Xu (2022). Optimal convergence rates for the orthogonal greedy algorithm. IEEE Transactions on Information Theory\/ 68\/ (5), 3354--3361

    2022

  40. [48]

    Huang, C

    Silver, D., A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. (2016). Mastering the game of go with deep neural networks and tree search. nature\/ 529\/ (7587), 484--489

    2016

  41. [49]

    Goldblum, A

    Somepalli, G., M. Goldblum, A. Schwarzschild, C. B. Bruss, and T. Goldstein (2021). Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342\/

    2021 arXiv

  42. [50]

    Stein, E. M. and R. Shakarchi (2009). Real analysis: measure theory, integration, and Hilbert spaces . Princeton University Press

    2009

  43. [51]

    Telgarsky, M. (2016). Benefits of depth in neural networks. In Conference on learning theory , pp.\ 1517--1539. PMLR

    2016

  44. [52]

    Temlyakov, V. N. (2000). Weak greedy algorithms. Advances in Computational Mathematics\/ 12\/ (2), 213--227

    2000

  45. [53]

    Vanschoren, J., J. N. van Rijn, B. Bischl, and L. Torgo (2013). Openml: Networked science in machine learning. SIGKDD Explorations\/ 15\/ (2), 49--60

    2013

  46. [54]

    Xu, J. and H. Zhu (2024). Overparametrized multi-layer neural networks: Uniform concentration of neural tangent kernel and convergence of stochastic gradient descent. Journal of Machine Learning Research\/ 25\/ (94), 1--83

    2024

  47. [55]

    Zhang, Y., C. Chen, N. Shi, R. Sun, and Z.-Q. Luo (2022). Adam can converge without any modification on update rules. Advances in neural information processing systems\/ 35 , 28386--28399

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.