Pith. sign in

REVIEW 3 major objections 5 minor 33 references

A ZeNN architecture to avoid the Gaussian trap

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A modified MLP architecture, Zeta Neural Networks, gives wide networks a pointwise-convergent, non-Gaussian infinite-width limit with a kernel that still depends on the parameters.

desk verdict A genuinely new wide-network architecture with sound convergence and non-Gaussian limit results, but the feature-learning claim outruns the proofs and the experiments are too thin to carry the practical weight. read the letter →

arxiv 2505.20553 v2 pith:C3UDQT22 submitted 2025-05-26 cs.LG math.PR

Luís Carvalho , João L. Costa , José Mourão , Gonçalo Oliveira This is my paper
classification cs.LGmath.PR MSC 68T07
keywords ZetaNeuralNetworkswidetangentkernelfeaturelearningGaussianprocesslimitspectralbiashigh-frequencycumulants
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Zeta Neural Networks (ZeNNs), a one-line change to multi-layer perceptrons: each neuron is assigned an integer label $j$, its input is scaled by $j$, and its output is weighted by a non-learnable factor $j^{-\alpha}$. The paper argues that this restores three properties that wide MLPs lose: a well-defined pointwise limit as width goes to infinity, non-Gaussian limiting statistics, and a kernel that depends on the parameters. If correct, very wide ZeNNs remain genuine parametric models whose training can adapt features, rather than collapsing into a fixed Gaussian kernel regime. The paper also shows empirically that finite ZeNNs with sine-like activations learn high-frequency functions that defeat standard MLPs.

What carries the argument

The carrying object is the ZeNN sum itself: enumerated perceptrons $p_{\hat\theta_j}(jx)=W_j^{(2)}\sigma\left(j W_j^{(1)} x + b_j^{(1)}\right)+b_j^{(2)}$, a non-learnable convergence weight $j^{-\alpha}$, and a frequency factor $j$ inside the activation. The key identity is cumulant additivity: for iid perceptrons, $\lambda^{(r)}_N(x)=\sum_{j=1}^N \lambda^{(r)}(jx)/j^{r\alpha}$, which at $x=0$ evaluates to $\lambda^{(r)}(0)$ times the Riemann zeta value $\zeta(r\alpha)$. This identity is what turns a divergence risk into a convergent, non-Gaussian limit, and the same weighted-sum structure carries through to the ZeNTK $K^N_\theta(x,y)$, whose second derivative with respect to a single output weight is nonzero on a full-measure set of parameters.

What would settle it

Take a ReLU perceptron with weight and bias distributions having a nonzero third cumulant at $x=0$, build $f^N_\theta(0)$ for growing $N$, and measure the standardized third cumulant; Theorem 2.2 predicts it converges to the nonzero constant $\lambda^{(3)}(0)\zeta(3\alpha)$, so a limit of zero (Gaussian behavior) would falsify the escape-from-the-Gaussian-trap claim. For the feature-learning claim, compute $K^\infty_\theta(x,y)$ at two nearby parameter values with $\dot\sigma\neq 0$ on an interval; if the values coincide, Theorem 2.5 fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that replacing the averaged MLP sum $\frac{1}{N^\beta}\sum_j p_{\hat\theta_j}(x)$ by the weighted, frequency-scaled sum $f^N_\theta(x)=\sum_{j=0}^N j^{-\alpha}\left(W_j^{(2)}\sigma\left(W_j^{(1)}jx+b_j^{(1)}\right)+b_j^{(2)}\right)$ changes the infinite-width story completely. For continuous activations of at most polynomial growth $k$, and $\alpha>k+1$, the ZeNN sequence converges uniformly on compact sets and pointwise almost surely to an honest infinite series of perceptrons, so the limit is again a parametric network. Because the perceptrons are independent, the $r$-th cumulant of $f^N_\theta(x)$ is $\sum_j \lambda^{(r)}(jx)/j^{r\alpha}$; at $x=0$ this becomes $\lambda^{(r)}(0)\zeta(r\alpha)$, which does not vanish when a single perceptron has a nonzero cumulant of order at least three. The infinite-width distribution therefore retains non-Gaussian fingerprints of the initialization. Finally, the paper defines the ZeNTK and proves that, unlike the MLP neural tangent kernel, it is not a constant function of the parameters even in the $N\to\infty$ limit, which the paper equates with feature learning.

Load-bearing premise

That infinite-width ZeNNs perform feature learning is assumed to follow from the ZeNTK being a non-constant function of the parameters, an equivalence the paper does not derive; separately, the Schrödinger experiment's advantage depends on initializing the eigenvalue guess within about 1 of the true $k^2$.

Editorial extensions

If this is right

  • Infinite-width ZeNNs are well-defined parametric models whose pointwise limit is an infinite series of scaled perceptrons, not a Gaussian field or the zero function.
  • The limiting distribution records the initialization through all cumulants, so width does not erase architecture or initialization information.
  • The ZeNTK depends on parameters even at infinite width, so ZeNNs do not fall into the fixed kernel regime that disables feature learning in MLPs.
  • With sine-type activations, finite-width ZeNNs and their randoZeNN variant match or beat Fourier-feature baselines at fitting high-frequency functions of low-dimensional inputs.
  • The Schrödinger PINN experiments indicate ZeNNs can find high-energy oscillatory eigenfunctions that standard MLPs miss, given a rough prior on the eigenvalue.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The factor $j^{-\alpha}$ can be read as a tunable spectral prior: smaller $\alpha$ puts more weight on high-frequency neurons, and the paper's experiments use $\alpha=0$ or $\alpha=0.1$ when high frequencies matter, suggesting $\alpha$ controls a bias–variance trade-off across frequencies.
  • Because the ZeNTK is parameter-dependent, ZeNN training dynamics are not linear in function space; this may change generalization and optimization behavior compared with NTK analysis, a testable consequence.
  • The same enumerating-and-weighting trick could be applied to other basis-like architectures, as the paper's KAZeNN proposal hints, potentially taming their infinite-width limits too.
  • The cumulant formula suggests ZeNNs could serve as controlled generators of non-Gaussian stochastic processes, with tail behavior tuned by $\alpha$ and the single-perceptron distribution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Zeta Neural Networks (ZeNNs), defined as f_N^θ(x)=Σ_{j=1}^N j^{-α}(W_j^{(2)}σ(W_j^{(1)}jx+b_j^{(1)})+b_j^{(2)}), with α>0 a fixed convergence parameter. The main theoretical results are: (Theorem 2.1) for α>k+1 the finite-width ZeNN converges uniformly on compact sets and pointwise a.e. to an infinite series; (Theorem 2.2 and Remark 2.3) the limiting cumulants are Σ_j λ^{(r)}(jx)/j^{rα}, so the infinite-width limit is non-Gaussian when a single-perceptron cumulant of order ≥3 is nonzero; (Theorem 2.5) the ZeNTK is a non-constant function of parameters, which the paper interprets as feature learning in the infinite-width limit. The paper also proposes deep variants (oZeNN, radZeNN, randoZeNN, KAZeNN) and reports experiments on 1D regression, image regression, and PINN solutions of Schrödinger and sinc problems.

Significance. The construction is simple and the core theoretical computations are valuable: the pointwise-convergent series limit, the exact cumulant formula, and the closed-form ZeNTK are derived cleanly in Appendices A-C. If the feature-learning claim were established, the paper would offer a novel resolution of the Gaussian trap with a minimal architectural change. The paper is also honest in flagging its own limitations, notably the eigenvalue-initialization requirement in §4.3.1. However, the central feature-learning claim is currently supported only by parameter-dependence of the kernel, not by training dynamics, and the experimental evidence is not parameter-fair and lacks error bars or seeds. These gaps are load-bearing for the advertised contributions.

major comments (3)
  1. [§2, paragraph before Theorem 2.5; Theorem 2.5] The paper defines feature learning as 'the ability of changing the kernel through training' but Theorem 2.5 only proves that θ↦K^N_θ(x,y) and θ↦K^∞_θ(x,y) are non-constant as functions of θ. This is not sufficient: even finite-width MLPs have parameter-dependent NTKs, and the infinite-width MLP pathology is precisely that the kernel stops changing along the gradient-flow trajectory. The manuscript never computes d/dt K^N_{θ(t)} or its infinite-width limit; Appendix C derives the output dynamics (C.1) with K^N_θ treated as a fixed object. To support the abstract and §1.3 claim, the authors need to show that the kernel changes along the training trajectory (e.g., that the infinite-width limit of d/dt K^N_{θ(t)} is nonzero on a set of positive measure), or explicitly downgrade the claim to 'parameter-dependent infinite-width kernel' rather than 'feature learning.'
  2. [§4.3.1, Schrödinger experiment] The reported success at energy level n=10 relies on the initialization satisfying |k0^2 - k*^2| ≈ 1, i.e., a strong prior on the unknown eigenvalue; the text itself calls this 'clearly an undesirable feature.' As presented, this does not demonstrate an advantage of ZeNNs over MLPs under comparable prior knowledge, since the MLP baseline is not given the same information or a tuning procedure. At minimum, the authors should report the sensitivity of the method to k0 across a range of values and compare against MLP/FF baselines with similarly tuned initialization.
  3. [§4.1-4.2, experimental methodology] The empirical comparisons are not parameter-fair and no error bars, seeds, or run-to-run variability are reported. For instance, Figure 1 compares an MLP with 128 neurons trained for 50,000 epochs against a ZeNN with 64 neurons trained for 15,000 epochs, and §4.2 reports only the 'best performance we were able to achieve' after non-exhaustive hyperparameter search. Since the paper's practical claim ('ZeNNs excel at learning high-frequency features') is partly empirical, the authors should report mean and standard deviation over multiple seeds and match parameter counts or training budgets across architectures.
minor comments (5)
  1. [Definition 1.1] The sum runs from j=0 to N with factor 1/j^α, which is undefined at j=0; the theorems and proofs all start at j=1, so the definition should either start at j=1 or specify that the j=0 term is excluded.
  2. [Remark 2.3 vs Appendix B, Remark B.3] The main text says non-Gaussianity follows if a cumulant of order at least three is nonzero, but Remark B.3 says 'at least two.' The latter is inconsistent with the argument (the second cumulant is the variance and does not control Gaussianity).
  3. [Appendix D and §4.3] The first sentence of Appendix D says 'all such models' in a way that suggests it refers to Section 4.3, but the specifications are for the image-regression models of Section 4.2; this should be corrected for clarity.
  4. [References] Reference [10] lists the second author as 'G. Franck'; the correct name is Arthur Jacot, Franck Gabriel, and Clément Hongler. Also, the header of the paper contains 'A VOID' instead of 'AVOID' in the title.
  5. [Captions of Figures 6 and 7] The captions say 'sample points form the exact solution'; 'form' should be 'from.'

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central theorems are direct analytic consequences of the ZeNN definition; the only self-citation is non-load-bearing.

full rationale

ZeNNs are defined by f_N^theta(x) = sum_{j=0}^N j^{-alpha}(W_j^(2) sigma(W_j^(1) j x + b_j^(1)) + b_j^(2)). Theorem 2.1 is proved directly in Appendix A by bounding |sigma(jWx+b)| <= C(1 + j^k |W|^k |x|^k + |b|^k) and showing the j^{-alpha} j^k tail converges for alpha > k+1; no fitted quantity or external result is invoked. Theorem 2.2/B.1 derives the cumulant identity lambda_N^(r)(x) = sum_j lambda^(r)(jx) j^{-r alpha} from the iid structure and characteristic-function factorization; the non-Gaussian conclusion at x=0 follows because the cumulant series does not vanish, unlike the 1/sqrt(N) MLP scaling. Theorem 2.5/C.2 computes the second derivative of the ZeNTK with respect to W_j^(2) and shows the kernel and its N->infinity limit are non-constant in theta; this is a direct computation from the kernel definition. The inference from 'kernel non-constant' to 'feature learning' is the paper's explicitly adopted perspective ('it is this later perspective that we will adopt here', Section 1.1), not a hidden equivalence; the stronger claim that the kernel changes along the training trajectory is not established, but that is a correctness/evidential gap, not circularity. The only self-citation is reference [6] in the related-results survey of finite-width non-Gaussianity; the present proofs do not depend on it, so it is not load-bearing. Experimental alpha values are hyperparameters, not fitted parameters inside the derivations. The disclosed limitations in Section 4.3.1 (need |k0^2 - k*^2| approx 1) and Section 4.3.2 (sine-vs-sine criticism) are acknowledged weaknesses, not circular inferences. No equation reduces to its own input, and no fitted quantity is renamed as a prediction.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claims rely on standard probabilistic convergence theorems plus two domain assumptions about activations and parameter moments; the feature-learning claim rests on an interpretive equivalence that is not independently benchmarked.

free parameters (1)
  • α (convergence parameter) = 1.1 in 1D regressions, 0.1 in PINNs, 0 in image regression
    Chosen by hand; the theory requires α>k+1 for pointwise convergence (Theorem 2.1), but experiments use α=0.1 (below the threshold for bounded activations like sine) and α=0, indicating α is tuned for performance rather than derived.
assumptions (3)
  • domain assumption Activation functions have at most polynomial growth of order k
    Used in Theorem 2.1 and 2.2 to bound the tail series and cumulant convergence; sine activation (used in experiments) satisfies this with k=0.
  • domain assumption Parameter measures ν_j have uniformly bounded moments
    Assumed in Theorem 2.1 and Appendix A to guarantee the Chebyshev-based tail bound; required for the a.e. convergence argument.
  • ad hoc to paper Feature learning is equivalent to the ZeNTK being non-constant in parameters
    The paper defines feature learning via the existence of a parameter-dependent kernel (Section 2, before Theorem 2.5) and does not provide an independent measure of feature reuse or transfer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A ZeNN architecture to avoid the Gaussian trap." pith.science (2026). https://pith.science/paper/C3UDQT22

@misc{pith2026250520553,
  author       = {Pith},
  title        = {Pith review of: A ZeNN architecture to avoid the Gaussian trap},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C3UDQT22}},
  note         = {Machine review of arXiv:2505.20553}
}
read the original abstract

We propose a new simple architecture, Zeta Neural Networks (ZeNNs), in order to overcome several shortcomings of standard multi-layer perceptrons (MLPs). Namely, in the large width limit, MLPs are non-parametric, they do not have a well-defined pointwise limit, they lose non-Gaussian attributes and become unable to perform feature learning; moreover, finite width MLPs perform poorly in learning high frequencies. The new ZeNN architecture is inspired by three simple principles from harmonic analysis: i) Enumerate the perceptons and introduce a non-learnable weight to enforce convergence; ii) Introduce a scaling (or frequency) factor; iii) Choose activation functions that lead to near orthogonal systems. We will show that these ideas allow us to fix the referred shortcomings of MLPs. In fact, in the infinite width limit, ZeNNs converge pointwise, they exhibit a rich asymptotic structure beyond Gaussianity, and perform feature learning. Moreover, when appropriate activation functions are chosen, (finite width) ZeNNs excel at learning high-frequency features of functions with low dimensional domains.

Figures

Figures reproduced from arXiv: 2505.20553 by the authors.

Figure 1
Figure 1. Learning synthetic data generated by (4.1), using different MLPs and ZeNNs. The blue line is the graph of the generating function, the green balls correspond to the training data, the red crosses are the predictions of the trained model and the yellow line is the graph of the model function. (a) MLP with sine activation, 128 perceptrons, trained for 50.000 epochs. (b) MLP with ReLU activation, 1024 perceptrons, trai… view at source ↗
Figure 2
Figure 2. Curve fitting of the first 3000 temperature Jena data points, represented by green balls. Our fit (yellow line) uses a shallow ZeNN network with sine activation, 512 perceptrons and α = 1.1, trained for 250.000 epochs. 4. ZeNNs learn high frequencies - experimental results In this section we report on some experimental results that show that ZeNNs excel at learning high frequency features. 4.1. 1d problems with shal… view at source ↗
Figure 3
Figure 3. Image regression task. (a) Ground truth image (source [25]). (b) Image recon￾structed using MLP. (c) Image reconstructed using the randoZeNN model. track the success of the models in terms of the peak signal-to-noise ratio (PSNR) evaluated in the validation set; recall that in this case bigger is better. All the models we test have a similar global structure with 2d inputs, 3d outputs, with linear activation, and 4 … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: PSNR on validation set (bigger is better) along training in image regression task. Comparing 6 different architectures [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: PSNR on validation set (bigger is better) along training in image regression task. Comparing the 4 top models. 4.3.1. Schrödinger’s equation. First we consider Schrödinger’s equation, in 1 + 1-dimensions and an infinite squared well potential: iℏ∂tψ(t, x) = − ℏ 2 2m ∂ …
Figure 6
Figure 6. Figure 6: Solving Schrödinger’s equations using ZeNNs, energy level n = 10. The green line corresponds to the approximate solution generated by a shallow ZeNN, with 64 neurons and α = 0.1. The blue crosses are sample points form the exact solution with the same L 2 normalization…
Figure 7
Figure 7. Figure 7: Solving (4.7) with L = 100. The green line corresponds to the approximate solution generated by a shallow ZeNN, with 100 neurons and α = 0.1. The blue crosses are sample points form the exact solution. but are unable to find solutions once L starts increasing. On the o…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 21 canonical work pages

  1. [1]

    Why bigger is not always better: on finite and infinite neural networks

    Laurence Aitchison. Why bigger is not always better: on finite and infinite neural networks. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 156–164. PMLR, 13–18 Jul

    work page

  2. [2]

    Arora, S

    S. Arora, S. S. Du, W. Hu, Z. Li, R. Salakhutdinov, and R. Wang. On exact computation with an infinitely wide neural net. Advances in Neural Information Processing Systems, 32(731):8141–8150, 2018

    work page 2018

  3. [3]

    Arora, S

    S. Arora, S. S. Du, W. Hu, Z. Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. 36th International Conference on Machine Learning, pages 477–502, 2019

    work page 2019

  4. [4]

    Neural tangent kernel at initialization: linear width suffices

    Arindam Banerjee, Pedro Cisneros-Velarde, Libin Zhu, and Mikhail Belkin. Neural tangent kernel at initialization: linear width suffices. 216:110–118, 31 Jul–04 Aug 2023

    work page 2023

  5. [5]

    Towards understanding the spectral bias of deep learning

    Yuan Cao, Zhiying Fang, Yue Wu, Ding-Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 , pages 2205–2211. International Joint Conferences on Artificial Intelligence Organization, 8 2021. doi: 10.2496...

    doi:10.24963/ijcai.2021/304 2021

  6. [6]

    Wide neural networks: From non-gaussian random fields at initialization to the NTK geometry of training

    Luís Carvalho, João Lopes Costa, José Mourão, and Gonçalo Oliveira. Wide neural networks: From non-Gaussian random fields at initialization to the NTK geometry of training. 2023. URL https: //arxiv.org/abs/2304.03385

    work page Pith review arXiv 2023

  7. [7]

    Jena climate dataset

    Adam Erickson and Olaf Kolle. Jena climate dataset. URL www.bgc-jena.mpg.de/wetter

    work page

  8. [8]

    Mathematical aspects of deep learning

    Philipp Grohs and Gitta Kutyniok. Mathematical aspects of deep learning. Cambridge University Press, 2023

    work page 2023

Show all 33 references
  1. [9]

    Approximation capabilities of multilayer feedforward networks

    Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, 4 (2):251–257, 1991. ISSN 0893-6080. doi: https://doi.org/10.1016/0893-6080(91)90009-T. URL https://www.sciencedirect.com/science/article/pii/089360809190009T

    1991

  2. [10]

    Jacot, G

    A. Jacot, G. Franck, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems, page 8580–8589, 2018

    2018

  3. [11]

    Unsupervised neural networks for quantum eigenvalue problems

    Henry Jin, Marios Mattheakis, and Pavlos Protopapas. Unsupervised neural networks for quantum eigenvalue problems. 2020. URL https://arxiv.org/abs/2010.05075

    2020 arXiv

  4. [12]

    Physics-informed neural networks for quantum eigenvalue problems

    Henry Jin, Marios Mattheakis, and Pavlos Protopapas. Physics-informed neural networks for quantum eigenvalue problems. 2022. URL https://arxiv.org/abs/2203.00451

    2022 arXiv

  5. [13]

    Generalization ability of wide neural networks on R

    Jianfa Lai, Manyun Xu, Rui Chen, and Qian Lin. Generalization ability of wide neural networks on R

  6. [14]

    J. Lee, Y. Bahri, R. Novak, S. Schoenholz, J. Pennington, and J. Sohl-Dickstein. Deep neural networks as gaussian processes. International Conference on Learning Representations, 2018

    2018

  7. [15]

    Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein

    Jaehoon Lee, Samuel S. Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: an empirical study. 2020

    2020

  8. [16]

    Lin, Allan Pinkus, and Shimon Schocken

    Moshe Leshno, Vladimir Ya. Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Networks, 6(6):861– 867, 1993. ISSN 0893-6080. doi: https://doi.org/10.1016/S0893-6080(05)80131-5. U...

    1993 doi

  9. [17]

    Statistical optimality of deep wide neural networks

    Yicheng Li, Zixiong Yu, Guhan Chen, and Qian Lin. Statistical optimality of deep wide neural networks

  10. [18]

    Kan: Kolmogorov-arnold networks

    Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljačić, Thomas Y Hou, and Max Tegmark. Kan: Kolmogorov-arnold networks. arXiv preprint arXiv:2404.19756, 2024

    2024 arXiv

  11. [19]

    Multi-scale deep neural network (mscalednn) for solving poisson-boltzmann equation in complex domains

    Ziqi Liu, Wei Cai, and Zhi-Qin John Xu. Multi-scale deep neural network (mscalednn) for solving poisson-boltzmann equation in complex domains. Communications in Computational Physics , 28 (5):1970–2001, 2020. ISSN 1991-7120. doi: https://doi.org/10.4208/cicp.OA-2020-0179. URL ...

    1970 doi

  12. [20]

    URL http://arxiv.org/abs/2305.02657v3

    arXiv

  13. [21]

    N. Radford. Priors for infinite networks. tech. rep. no. crg-tr-94-1, University of Toronto, 1994

    1994

  14. [22]

    On the spectral bias of neural networks

    Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Mac...

  15. [23]

    The limitations of large width in neural networks: A deep gaussian process perspective

    Geoff Pleiss and John P Cunningham. The limitations of large width in neural networks: A deep gaussian process perspective. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P .S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, p...

    2021

  16. [24]

    Martel, Alexander W

    Vincent Sitzmann, Julien N.P . Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Proc. NeurIPS, 2020

    2020

  17. [25]

    fourier-feature-networks

    Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. fourier-feature-networks. URL https: //github.com/tancik/fourier-feature-networks?tab=readme-ov-file

  18. [26]

    D. A. Roberts and S. Yaida. The Principles of Deep Learning Theory: An E ffective Theory Approach to Understanding Neural Networks. Cambridge University Press, 2022

    2022

  19. [27]

    On the eigenvector bias of fourier feature net- works: From regression to solving multi-scale pdes with physics-informed neural networks

    Sifan Wang, Hanwen Wang, and Paris Perdikaris. On the eigenvector bias of fourier feature net- works: From regression to solving multi-scale pdes with physics-informed neural networks. Com- puter Methods in Applied Mechanics and Engineering, 384:113938, 2021. ISSN 0045-7825. d...

    2021

  20. [29]

    Fourier features let networks learn high frequency functions in low dimensional domains

    Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In H. Larochelle, M. Ranzato, R. Ha...

    2020

  21. [30]

    Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation

    Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. ArXiv, abs/1902.04760, 2019

    1902 arXiv

  22. [31]

    1 2Bit/jα eiBt/jα sin(xLt/jα−1) xLt/jα−1 − 1 ! + 1 2 # × NY j=⌈B/xL⌉

    Jiayu Yao, Yaniv Yacoby, Beau Coker, Weiwei Pan, and Finale Doshi-Velez. An empirical analysis of the advantages of finite- v.s. infinite-width bayesian neural networks. ArXiv, abs/2211.09184, 2022. URL https://api.semanticscholar.org/CorpusID:253581299. A ZeNN ARCHITECTURE TO...

    2022 arXiv

  23. [32]

    Yang and E

    G. Yang and E. J. Hu. Tensor programs iv: Feature learning in infinite-width neural networks. Interna- tional Conference on Machine Learning, 2021

    2021

  24. [2020]

    URL https://proceedings.mlr.press/v119/aitchison20a.html

  25. [2023]

    URL https://arxiv.org/abs/2302.05933

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.