Pith. sign in

REVIEW 2 major objections 5 minor 59 references

Effect of Activation Functions on the Training of Overparametrized Neural Nets

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proves that activation functions with a derivative kink guarantee large minimum eigenvalues of the training Gram matrix, while smooth activations can force it exponentially small on low-dimensional data, and depth restores the…

desk verdict You should know two things before reading: this is the first paper I know that gives quantitative eigenvalue bounds for activations with a kink and explicit negative results for smooth ones, and I believe the headline ReLU bound is not proven as stated. read the letter →

arxiv 1908.05660 v4 pith:BGJ72746 submitted 2019-08-16 cs.LG stat.ML

Abhishek Panigrahi , Abhishek Shetty , Navin Goyal This is my paper · ORCID
classification cs.LGstat.ML MSC 68Q3268T07
keywords G-matrixminimumeigenvalueactivationfunctionoverparametrizationsmoothvsnon-smoothactivationsReLUtanhHermiteexpansion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks what makes one activation function train faster than another in overparametrized two-layer networks, where gradient descent provably reaches zero training error. Its answer is that a single quantity, the minimum eigenvalue of the Gradient Gram matrix at initialization, separates activations into two classes. Activations with a derivative kink have this eigenvalue polynomially large under mild separation of the data, which predicts fast convergence. Smooth activations have it zero or exponentially small when the data occupies a low-dimensional subspace, which predicts slow training, while enough depth or data smoothing restores the large-eigenvalue regime.

What carries the argument

The central object is the Gradient Gram matrix at initialization, with entries $g_{ij} = \frac{1}{m}\sum_{k=1}^{m} a_k^2 \varphi'(w_k^{\mathsf{T}} x_i)\varphi'(w_k^{\mathsf{T}} x_j)\langle x_i, x_j\rangle$, whose eigenvalue decomposition governs the loss dynamics. The proofs split a Gaussian weight vector into a large part $w'$ and a small part $w''$ to isolate the kink, then use Taylor expansion of $\sum_i \zeta_i \varphi'(w^{\mathsf{T}} x_i)$ to convert a jump in the $r$-th derivative into a lower bound on the norm over an $\epsilon$-net. For smooth activations the counterpoint is polynomial approximation: $\tanh'$ is approximated by Chebyshev or Hermite polynomials whose decaying coefficients give the exponentially small upper bound on $\lambda_{\min}(G(0))$, while the lower bounds for smoothed data go through Khatri-Rao powers of the data matrix and anticoncentration of polynomial maps.

What would settle it

Compute $\lambda_{\min}(G(0))$ at initialization for a two-layer tanh network on $n$ points whose span has dimension $d' = O(\log^{0.75} n)$: the theorem predicts $\lambda_{\min}(G(0)) \le \exp(-\Omega(n^{1/(2d')}))$, so a measurement showing it is instead polynomially large would refute the upper-bound claim.

Watch

Extended reading notes

Core claim

The central claim is that the smoothness of the activation determines the conditioning of the G-matrix at initialization. For activations whose $r$-th derivative has a jump discontinuity, $\lambda_{\min}(G(0))$ is large: at least $\Omega(\delta^3 n^{-8}(\log n)^{-1})$ for the $J_2$ class and $\Omega((\delta/\log n)^{1.5})$ for ReLU. For smooth activations such as tanh, if the span of the $n$ data points has dimension $d' = O(\log^{0.75} n)$, then $\lambda_{\min}(G(0)) \le \exp(-\Omega(n^{1/(2d')}))$, and for polynomial activations many eigenvalues are exactly zero when $d' = O(n^{1/p})$. Thus the same architecture trains fast or slow depending on whether the activation has a kink and on the intrinsic dimension of the data, and the paper shows these bounds are tight in a precise sense.

Load-bearing premise

The training-speed conclusions hold only in the small-perturbation (lazy) regime, where weights move little enough that the G-matrix stays close to its value at initialization; the paper guarantees this only when the learning rate satisfies $\eta \le O(n^{-2}\lambda_{\min}(G(0)))$.

Editorial extensions

If this is right

  • For activations in the $J_r$ classes (e.g. ReLU, SELU, ELU), a sufficiently wide two-layer network trained by gradient descent reaches near-zero training error in polynomially many steps, with the step count controlled by $\lambda_{\min}(G(0))$.
  • For smooth activations on low-dimensional data, the G-matrix has many small eigenvalues, so loss decays slowly even in the ideal perturbative training regime.
  • The low-dimensional limitation of smooth activations disappears with depth: for tanh with depth $L = \Theta(\log(1/\delta))$, $\lambda_{\min}(G(0)) \ge e^{-O(\sqrt{\log n})}$, which is much larger than any inverse polynomial in $n$.
  • For smoothed data whose span has dimension at least $n^{\gamma}$, tanh regains a polynomial lower bound on $\lambda_{\min}(G(0))$, so the earlier slow-training result is not robust to small data perturbation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One could use $\lambda_{\min}(G(0))$ as a cheap diagnostic for predicting whether a given activation will train slowly on a particular dataset, prior to actually running gradient descent.
  • The results suggest a design principle for new activations: introduce a derivative kink of some order if the target data is low-dimensional, or rely on depth or data smoothing to rescue smooth activations.
  • Because the training-speed conclusions hold only under learning rates bounded by $O(n^{-2}\lambda_{\min}(G(0)))$, an immediate extension is to test whether the gap between kinked and smooth activations persists at practical learning rates; the eigenvalue bounds alone do not decide this.
  • The Hermite-coefficient perspective links activation smoothness to training speed, implying that activations with slowly decaying Hermite coefficients (like the step function underlying ReLU) should train fast, which could guide automated activation search.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper studies training of highly overparametrized two-layer neural networks in the DZPS (Du et al.) random-initialization setting, focusing on the minimum eigenvalue at initialization of the Gradient Gram matrix G(0), which controls the convergence rate of gradient descent in the lazy/perturbative regime. The main results are: for activations with a kink (class Jr, including ReLU, SELU, ELU), all eigenvalues of G(0) are polynomially large under mild data separation; for smooth activations (tanh, swish, polynomial activations), the minimum eigenvalue can be zero or exponentially small when the data span has low dimension; positive results are given for smoothed high-dimensional data and for sufficiently deep networks. The paper contains detailed appendix proofs, applications to training bounds, and experiments on synthetic data and a CIFAR-10 subset.

Significance. If the results hold as stated, the paper would provide the first quantitative eigenvalue bounds distinguishing smooth from nonsmooth activations in the NTK/Gram-matrix analysis of overparametrized networks. The proofs are built from standard tools (Hermite and Chebyshev approximation, Carbery-Wright anticoncentration, Gershgorin's theorem) and no constants are fitted to data; the predicted smooth-vs-kink dichotomy is falsifiable and is qualitatively supported by the included experiments. However, one of the headline quantitative claims, the ReLU lower bound of Theorem 4.2, appears to rest on an incorrect translation of Assumption 2, and the training-speed conclusions are explicitly valid only in the small-learning-rate lazy regime acknowledged by the authors.

major comments (2)
  1. [Appendix L, proof of Theorem L.2; Theorem 4.2] The proof of Theorem L.2 states that Assumption 2 implies |x_i^T x_j| ≤ 1−δ, but Assumption 2 gives |x_i^T x_j| ≤ sqrt(1−δ^2) = 1−Θ(δ^2) for unit vectors. Consequently, the threshold r0 should be Θ(log n / δ^2) rather than Θ(log n / δ), and the r^{−3/2} decay of the squared Hermite coefficients of ReLU′ yields λmin(G∞) ≥ Ω(δ^3 / log^{3/2} n), not Ω((δ/log n)^{1.5}). The neuron requirement m = Ω(n^4 δ^{−3} log^4 n) also needs to be re-derived with the corrected eigenvalue. Therefore Theorem 4.2, advertised as the best known ReLU bound, is not proved as stated; the kink-versus-smooth dichotomy may survive with weaker exponents, but the quantitative claim needs revision.
  2. [Section 4.2, Theorem 4.6] The abstract and introduction phrase the results as "leading to slow training" for smooth activations on low-dimensional data, but the formal transfer from eigenvalue bounds to training speed is proved only in the small-learning-rate perturbative regime: Theorem 4.6 requires η ≤ O(n^{−2} λmin(G(0))) and m ≥ Ω(n^5 κ^{−1} λmin(G(0))^{−4} ε^{−2}), and the text explicitly says that analyzing higher learning rates remains open. The claim should be qualified accordingly; as it stands, the eigenvalue bounds alone do not establish slow training outside the lazy regime.
minor comments (5)
  1. [Theorem 4.2] The displayed bound is malformed: it reads "Ω((δ1.5 log−1.5n)" and should be Ω((δ/log n)^{1.5}) (or the corrected exponent after the scaling issue described in the major comment).
  2. [Section 6, Theorem 6.4] The informal Hermite-approximation statement for tanh′ is attributed to "Theorem G.2", but Theorem G.2 in the appendix is the corresponding statement for swish; the correct reference appears to be Corollary F.7.2.
  3. [Section 2, notation] The notation "‖u‖2 := ‖u‖2^2 := ∑ u_i^2" defines the norm as the square of the norm; please use distinct symbols, e.g., ‖u‖_2 for the Euclidean norm and ‖u‖_2^2 for its square.
  4. [Section 7, Figure 2(b)] The caption says "Semilog plot of sum of squares of top k singular values" while the text says "cumulative sums of eigenvalues"; please make the caption consistent with the quantity plotted.
  5. [Theorem 4.4] The expression exp(−Ω(n^{1/2d′})) is ambiguous; it should be written as exp(−Ω(n^{1/(2d′)})) to make clear that the exponent is 1/(2d′).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the eigenvalue bounds are derived from explicit spectral/approximation arguments, with no fitted input or self-citation chain carrying the central claims.

full rationale

The paper's central results are quantitative bounds on λmin(G(0)) for kinked and smooth activations. These are proved from the definition of the G-matrix (Eq. 2), Hermite/Chebyshev approximation theorems (Hille 1940; Boyd 1984), standard linear-algebra tools (Weyl, Gershgorin, Carbery-Wright), and concentration inequalities. No parameter is fitted to data and then relabeled as a prediction: the constants (δ, n, m) appear in theorem statements and are not tuned to match the experiments. The training-speed interpretation (Theorem 4.6) explicitly invokes the prior perturbative/lazy-training framework of Du et al. and Arora et al., and the paper openly acknowledges that higher learning rates remain outside the analysis; this is an honest reliance on published external frameworks, not a circular import of the target result. The only self-citation, Anderson et al. (2014) with coauthor Navin Goyal, supplies an anticoncentration argument for smoothed-analysis lower bounds; the mathematical content is the external Carbery-Wright inequality, and the citation is not load-bearing for the core kink-versus-smooth dichotomy. The skeptic's concern about the ReLU bound (using |x_i^T x_j| ≤ 1−δ instead of sqrt(1−δ^2) from Assumption 2) is a potential correctness/scaling error in a proof step, not a definitional equivalence or a fitted-input-as-prediction; even if true, it would weaken an exponent without making the derivation circular. Accordingly, the derivation chain is self-contained with respect to circularity.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The paper introduces no new free parameters fitted to data; the constants in the Jr conditions are normalized to 1 for simplicity. The main load-bearing external premises are the δ-separation data assumption and the lazy-training/NTK regime imported from prior work, which connects eigenvalue bounds to actual training speed.

assumptions (6)
  • domain assumption Data satisfies Assumption 1 and 2: unit norm and δ-separation
    All positive eigenvalue lower bounds scale polynomially in δ; without separation the bounds degrade or vanish. Stated in Section 2.
  • domain assumption The NTK/lazy training regime applies: G(t) stays close to G(0) and training speed is governed by λmin(G(0))
    Imported from Du et al. (2019a,b) and Arora et al. (2019c); connects eigenvalue bounds to convergence. The paper's Theorem 4.6 proves slow convergence only under a small learning rate constraint, and the paper notes higher learning rates remain open.
  • domain assumption Smoothed data assumption (Assumption 3): data points are Gaussian-perturbed within their span
    Used only for the positive results for smooth activations (Theorem 4.7); rules out non-robust low-dimensional spans.
  • standard math Lemma I.1 (from Oymak & Soltanolkotabi 2019) relating G∞ to Khatri-Rao powers and Hermite coefficients
    Cited lemma used to lower bound λmin via Hermite coefficient decay; not proven in the paper.
  • standard math Hille's theorem on Hermite series decay and Boyd's asymptotic for Hermite coefficients of tanh'
    Used to bound polynomial approximation error; classical analytic results.
  • domain assumption Activation function belongs to class Jr (derivative jump discontinuity of order r)
    Defines the scope of the positive results. The constants in the Jr conditions are normalized to 1 for simplicity, which is a modeling choice, not a fitted parameter.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Effect of Activation Functions on the Training of Overparametrized Neural Nets." pith.science (2026). https://pith.science/paper/BGJ72746

@misc{pith2026190805660,
  author       = {Pith},
  title        = {Pith review of: Effect of Activation Functions on the Training of Overparametrized Neural Nets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BGJ72746}},
  note         = {Machine review of arXiv:1908.05660}
}
read the original abstract

It is well-known that overparametrized neural networks trained using gradient-based methods quickly achieve small training error with appropriate hyperparameter settings. Recent papers have proved this statement theoretically for highly overparametrized networks under reasonable assumptions. These results either assume that the activation function is ReLU or they crucially depend on the minimum eigenvalue of a certain Gram matrix depending on the data, random initialization and the activation function. In the later case, existing works only prove that this minimum eigenvalue is non-zero and do not provide quantitative bounds. On the empirical side, a contemporary line of investigations has proposed a number of alternative activation functions which tend to perform better than ReLU at least in some settings but no clear understanding has emerged. This state of affairs underscores the importance of theoretically understanding the impact of activation functions on training. In the present paper, we provide theoretical results about the effect of activation function on the training of highly overparametrized 2-layer neural networks. A crucial property that governs the performance of an activation is whether or not it is smooth. For non-smooth activations such as ReLU, SELU and ELU, all eigenvalues of the associated Gram matrix are large under minimal assumptions on the data. For smooth activations such as tanh, swish and polynomials, the situation is more complex. If the subspace spanned by the data has small dimension then the minimum eigenvalue of the Gram matrix can be small leading to slow training. But if the dimension is large and the data satisfies another mild condition, then the eigenvalues are large. If we allow deep networks, then the small data dimension is not a limitation provided that the depth is sufficient. We discuss a number of extensions and applications of these results.

Figures

Figures reproduced from arXiv: 1908.05660 by the authors.

Figure 1
Figure 1. Experiments on synthetic dataset (From left to right) (a)Rate of convergence of 2-layer [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Experiments on a random subset of 104 images from CIFAR10 dataset: (a) L 2 -distances between all pairs of preprocessed images (b) Semilog plot of sum of squares of top k singular values of data matrix (c) Eigenvalue distribution of G-matrix at initialization (d) Convergence speed of 2 layer networks using different activation functions. 8 CONCLUSION In this paper we characterized the effect of activation function o… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 40 canonical work pages

  1. [1]

    Learning and generalization in overparameterized neural networks, going beyond two layers

    Zeyuan Allen - Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. CoRR, abs/1811.04918, 2018. URL http://arxiv.org/abs/1811.04918

    arXiv 2018

  2. [2]

    A convergence theory for deep learning via over-parameterization

    Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.\ 242--252, Long Beach, California, USA, 09--15 Jun 2019. PMLR....

    work page 2019

  3. [3]

    Joseph Anderson, Mikhail Belkin, Navin Goyal, Luis Rademacher, and James R. Voss. The more, the merrier: the blessing of dimensionality for learning large gaussian mixtures. In Proceedings of The 27th Conference on Learning Theory, COLT 2014, Barcelona, Spain, June 13-15, 2014 , pp.\ 1135--1164, 2014. URL http://proceedings.mlr.press/v35/anderson14.html

    work page 2014

  4. [4]

    A convergence analysis of gradient descent for deep linear neural networks

    Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. A convergence analysis of gradient descent for deep linear neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 , 2019 a . URL https://openreview.net/forum?id=SkMQg3C5K7

    work page 2019

  5. [5]

    On exact computation with an infinitely wide neural net

    Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, Russ Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In NeurIPS, 2019 b

    work page 2019

  6. [6]

    Du, Wei Hu, Zhi yuan Li, and Ruosong Wang

    Sanjeev Arora, Simon S. Du, Wei Hu, Zhi yuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. volume abs/1901.08584, 2019 c

    arXiv 1901

  7. [7]

    Concentration inequalities

    St\' e phane Boucheron, G\' a bor Lugosi, and Pascal Massart. Concentration inequalities. Oxford University Press, Oxford, 2013. ISBN 978-0-19-953525-5. doi:10.1093/acprof:oso/9780199535255.001.0001. URL https://doi.org/10.1093/acprof:oso/9780199535255.001.0001. A nonasymptotic theory of independence, With a foreword by Michel Ledoux

    arXiv 2013

  8. [8]

    Asymptotic coefficients of hermite function series

    John P Boyd. Asymptotic coefficients of hermite function series. Journal of Computational Physics, 54 0 (3): 0 382--410, 1984

    work page 1984

Show all 59 references
  1. [9]

    SGD learns over-parameterized networks that provably generalize on linearly separable data

    Alon Brutzkus, Amir Globerson, Eran Malach, and Shai Shalev - Shwartz. SGD learns over-parameterized networks that provably generalize on linearly separable data. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2...

    2018

  2. [10]

    Distributional and L^q norm inequalities for polynomials over convex bodies in R^n

    Anthony Carbery and James Wright. Distributional and L^q norm inequalities for polynomials over convex bodies in R^n . Math. Res. Lett., 8 0 (3): 0 233--248, 2001. ISSN 1073-2780. doi:10.4310/MRL.2001.v8.n3.a1. URL https://doi.org/10.4310/MRL.2001.v8.n3.a1

    2001 doi

  3. [11]

    On the global convergence of gradient descent for over-parameterized models using optimal transport

    L\' e na\" c Chizat and Francis Bach. On the global convergence of gradient descent for over-parameterized models using optimal transport. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Sys...

    2018

  4. [12]

    Fast and accurate deep network learning by exponential linear units (elus)

    Djork-Arn \'e Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). In ICLR, 2016

    2016

  5. [13]

    Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity

    Amit Daniely, Roy Frostig, and Yoram Singer. Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity. In Advances In Neural Information Processing Systems, pp.\ 2253--2261, 2016

    2016

  6. [14]

    Du and Jason D

    Simon S. Du and Jason D. Lee. On the power of over-parametrization in neural networks with quadratic activation. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm \" a ssan, Stockholm, Sweden, July 10-15, 2018 , pp.\ 1328--1337, 20...

    2018

  7. [15]

    Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh

    Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. In ICLR, 2019 a

    2019

  8. [16]

    Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai

    Simon Shaolei Du, Jason D. Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In ICML, 2019 b

    2019

  9. [17]

    Is it time to swish? comparing deep learning activation functions across nlp tasks

    Steffen Eger, Paul Youssef, and Iryna Gurevych. Is it time to swish? comparing deep learning activation functions across nlp tasks. In EMNLP, 2018

    2018

  10. [18]

    Sigmoid-weighted linear units for neural network function approximation in reinforcement learning

    Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. CoRR, abs/1702.03118, 2017. URL http://arxiv.org/abs/1702.03118

    2017 arXiv

  11. [19]

    Understanding the difficulty of training deep feedforward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings o...

    2010

  12. [20]

    Deep Learning

    Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org

    2016

  13. [21]

    Which neural net architectures give rise to exploding and vanishing gradients? In S

    Boris Hanin. Which neural net architectures give rise to exploding and vanishing gradients? In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp.\ 582--591. Curran Associates, Inc....

    2018

  14. [22]

    How to start training: The effect of initialization and architecture

    Boris Hanin and David Rolnick. How to start training: The effect of initialization and architecture. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp.\ 571--581. Curran Associa...

    2018

  15. [23]

    On the impact of the activation function on deep neural networks training

    Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. On the impact of the activation function on deep neural networks training. CoRR, abs/1902.06853, 2019. URL http://arxiv.org/abs/1902.06853

    1902 arXiv

  16. [24]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp.\ 1026--1034, 2015

    2015

  17. [25]

    Contributions to the theory of H ermitian series

    Einar Hille. Contributions to the theory of H ermitian series. II . T he representation problem. Trans. Amer. Math. Soc., 47: 0 80--94, 1940. ISSN 0002-9947. doi:10.2307/1990002. URL https://doi.org/10.2307/1990002

    1940 doi

  18. [26]

    Neural tangent kernel: Convergence and generalization in neural networks

    Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp.\ ...

    2018

  19. [27]

    Solutions to some functional equations and their applications to characterization of probability distributions

    CG Khatri and C Radhakrishna Rao. Solutions to some functional equations and their applications to characterization of probability distributions. Sankhy \=a : The Indian Journal of Statistics, Series A , pp.\ 167--180, 1968

    1968

  20. [28]

    On the expressive power of deep polynomial neural networks

    Joe Kileel, Matthew Trager, and Joan Bruna. On the expressive power of deep polynomial neural networks. In NeurIPS, 2019

    2019

  21. [29]

    Self-normalizing neural networks

    G \"u nter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In Advances in neural information processing systems, pp.\ 971--980, 2017

    2017

  22. [30]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009

    2009

  23. [31]

    Adaptive estimation of a quadratic functional by model selection

    Beatrice Laurent and Pascal Massart. Adaptive estimation of a quadratic functional by model selection. Annals of Statistics, pp.\ 1302--1338, 2000

    2000

  24. [32]

    N. N. Lebedev. Special functions and their applications. Dover Publications, Inc., New York, 1972. Revised edition, translated from the Russian and edited by Richard A. Silverman, Unabridged and corrected republication

    1972

  25. [33]

    Deep neural networks as gaussian processes

    Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint arXiv:1711.00165, 2017

    2017 arXiv

  26. [34]

    Lin, Allan Pinkus, and Shimon Schocken

    Moshe Leshno, Vladimir Ya. Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Networks, 6 0 (6): 0 861 -- 867, 1993. ISSN 0893-6080. doi:https://doi.org/10.1016/S0893-6080(05)801...

    1993 doi

  27. [35]

    Learning overparameterized neural networks via stochastic gradient descent on structured data

    Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp.\ 8...

    2018

  28. [36]

    A random matrix approach to neural networks

    Cosme Louart, Zhenyu Liao, and Romain Couillet. A random matrix approach to neural networks. Ann. Appl. Probab., 28 0 (2): 0 1190--1248, 04 2018. doi:10.1214/17-AAP1328. URL https://doi.org/10.1214/17-AAP1328

    2018 doi

  29. [37]

    Mason and D.C

    J.C. Mason and D.C. Handscomb. Chebyshev Polynomials. CRC Press, 2002. ISBN 9781420036114. URL https://books.google.co.in/books?id=8FHf0P3to0UC

    2002

  30. [38]

    A mean field view of the landscape of two-layer neural networks

    Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 115 0 (33): 0 E7665--E7671, 2018. ISSN 0027-8424. doi:10.1073/pnas.1806579115. URL https://www.pnas.org/content/1...

    2018 doi

  31. [39]

    In search of the real inductive bias: On the role of implicit regularization in deep learning

    Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedi...

    2015 arXiv

  32. [40]

    Activation functions: Comparison of trends in practice and research for deep learning

    Chigozie Nwankpa, Winifred Ijomah, Anthony Gachagan, and Stephen Marshall. Activation functions: Comparison of trends in practice and research for deep learning. CoRR, abs/1811.03378, 2018

    2018 arXiv

  33. [41]

    Analysis of boolean functions

    Ryan O'Donnell. Analysis of boolean functions. Cambridge University Press, 2014

    2014

  34. [42]

    Towards moderate overparameterization: global convergence guarantees for training shallow neural networks

    Samet Oymak and Mahdi Soltanolkotabi. Towards moderate overparameterization: global convergence guarantees for training shallow neural networks. arXiv preprint arXiv:1902.04674, 2019

    1902 arXiv

  35. [43]

    Nonlinear random matrix theory for deep learning

    Jeffrey Pennington and Pratik Worah. Nonlinear random matrix theory for deep learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp.\ 2637--2646. Curran Associates...

    2017

  36. [44]

    The spectrum of the fisher information matrix of a single-hidden-layer neural network

    Jeffrey Pennington and Pratik Worah. The spectrum of the fisher information matrix of a single-hidden-layer neural network. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp.\ 5...

    2018

  37. [45]

    Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice

    Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Informa...

    2017

  38. [46]

    Schoenholz, and Surya Ganguli

    Jeffrey Pennington, Samuel S. Schoenholz, and Surya Ganguli. The emergence of spectral universality in deep networks. In International Conference on Artificial Intelligence and Statistics, AISTATS 2018, 9-11 April 2018, Playa Blanca, Lanzarote, Canary Islands, Spain , pp.\ 192...

    2018

  39. [47]

    Approximation theory of the MLP model in neural networks

    Allan Pinkus. Approximation theory of the MLP model in neural networks. Acta Numerica, 8: 0 143–195, 1999. doi:10.1017/S0962492900002919

    1999 doi

  40. [48]

    Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions. In ICLR Workshop, 2018

    2018

  41. [49]

    Smallest singular value of a random rectangular matrix

    Mark Rudelson and Roman Vershynin. Smallest singular value of a random rectangular matrix. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences, 62 0 (12): 0 1707--1739, 2009

    2009

  42. [50]

    Learning kernel-based halfspaces with the 0-1 loss

    Shai Shalev-Shwartz, Ohad Shamir, and Karthik Sridharan. Learning kernel-based halfspaces with the 0-1 loss. SIAM J. Comput., 40 0 (6): 0 1623--1646, December 2011. ISSN 0097-5397. doi:10.1137/100806126. URL http://dx.doi.org/10.1137/100806126

    2011 doi

  43. [51]

    Neural network with unbounded activation functions is universal approximator

    Sho Sonoda and Noboru Murata. Neural network with unbounded activation functions is universal approximator. Applied and Computational Harmonic Analysis, 43 0 (2): 0 233 -- 268, 2017. ISSN 1063-5203. doi:https://doi.org/10.1016/j.acha.2015.12.005. URL http://www.sciencedirect.c...

    2017 doi

  44. [52]

    Spielman and Shang - Hua Teng

    Daniel A. Spielman and Shang - Hua Teng. Smoothed analysis of algorithms: Why the simplex algorithm usually takes polynomial time. J. ACM , 51 0 (3): 0 385--463, 2004. doi:10.1145/990308.990310. URL https://doi.org/10.1145/990308.990310

    2004

  45. [53]

    Orthogonal polynomials

    G\' a bor Szeg o . Orthogonal polynomials. American Mathematical Society, Providence, R.I., fourth edition, 1975. American Mathematical Society, Colloquium Publications, Vol. XXIII

    1975

  46. [54]

    Lectures on H ermite and L aguerre expansions , volume 42 of Mathematical Notes

    Sundaram Thangavelu. Lectures on H ermite and L aguerre expansions , volume 42 of Mathematical Notes. Princeton University Press, Princeton, NJ, 1993. ISBN 0-691-00048-4. With a preface by Robert S. Strichartz

    1993

  47. [55]

    Invariance of weight distributions in rectified mlps

    Russell Tsuchida, Farbod Roosta - Khorasani, and Marcus Gallagher. Invariance of weight distributions in rectified mlps. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm \" a ssan, Stockholm, Sweden, July 10-15, 2018 , pp.\ 5002--...

    2018

  48. [56]

    Ger s gorin and his circles , volume 36

    Richard S Varga. Ger s gorin and his circles , volume 36. Springer Science & Business Media, 2010

    2010

  49. [57]

    Das asymptotische V erteilungsgesetz der E igenwerte linearer partieller D ifferentialgleichungen (mit einer A nwendung auf die T heorie der H ohlraumstrahlung)

    Hermann Weyl. Das asymptotische V erteilungsgesetz der E igenwerte linearer partieller D ifferentialgleichungen (mit einer A nwendung auf die T heorie der H ohlraumstrahlung). Math. Ann., 71 0 (4): 0 441--479, 1912. ISSN 0025-5831. doi:10.1007/BF01456804. URL https://doi.org/1...

    1912 doi

  50. [58]

    Diverse neural network learns true target functions

    Bo Xie, Yingyu Liang, and Le Song. Diverse neural network learns true target functions. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL, USA , pp.\ 1216--1224, 2017. URL http://p...

    2017

  51. [59]

    Revise saturated activation functions

    Bing Xu, Ruitong Huang, and Mu Li. Revise saturated activation functions. In ICLR Workshop, 2016

    2016

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.