Pith. sign in

REVIEW 2 major objections 9 minor 24 references

Even in wide nets, learning the first-layer weights beats a fixed random kernel, and singularities are why.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 23:15 UTC pith:T2AQTJAY

load-bearing objection Clean SLT comparison showing hierarchical Bayesian training beats fixed-kernel single-layer even at large width, with an explicit rate gap—but the large-H coefficient proof has a real rigor hole. the 2 major comments →

arxiv 2607.23397 v1 pith:T2AQTJAY submitted 2026-07-26 cs.LG math.STstat.MLstat.TH

A Statistical Difference between Single-Layer Learning and Hierarchical Learning in Wide Neural Networks

Sumio Watanabe This is my paper
classification cs.LG math.STstat.MLstat.TH MSC 62F1568T0714B05
keywords singular learning theoryreal log canonical thresholdwide neural networkskernel regimefeature learninggeneralization errorthree-layer networksBayesian free energy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper compares two ways of training a three-layer neural network with many hidden units: keep the input-to-hidden weights frozen at random initialization and train only the output weights, or train both layers. For learning a pure-noise target (the zero function plus Gaussian noise), it proves that the hierarchical scheme has a strictly smaller asymptotic generalization error than the fixed-kernel scheme once the width is large but still finite. The gap grows with width: fixed kernels pay a linear cost in the number of hidden units, while hierarchical learning pays only a sub-linear cost set by the real log canonical threshold of the singular parameter space. The result is offered as evidence that the two common infinite-width pictures—lazy kernels versus feature learning—are not equivalent, and that singularities remain essential even when networks are wide.

Core claim

Under the regime M, N ≪ H ≪ n, Bayesian generalization error for single-layer (fixed-b) learning of the zero map equals HN/(2n) + o(1/n), while hierarchical learning satisfies the tighter upper bound α(M) H^{M/(M+1)} N^{1/(M+1)} / n + o(1/n). Thus training the input-to-hidden weights yields a smaller statistical error than keeping them fixed, even though more parameters are optimized.

What carries the argument

The real log canonical threshold λ of the hierarchical model, bounded above via the growing index Δ(h) that counts the vanishing order of the tanh basis; the free-energy difference F1(n) − F2(n) then forces G1(n) ≥ G2(n) with the stated rates.

Load-bearing premise

The true map being learned is exactly the zero function plus isotropic Gaussian noise, so the comparison isolates only statistical error under perfect realizability and does not yet control approximation error for nonzero targets.

What would settle it

Compute or tightly bound the exact real log canonical threshold (or finite-n generalization error) for the same three-layer tanh network on a nonzero smooth target; if the hierarchical advantage disappears or reverses while M, N ≪ H ≪ n still holds, the claimed statistical gap fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Fixed-kernel (lazy) and feature-learning infinite-width limits remain statistically inequivalent even as width tends to infinity.
  • Generalization error of hierarchical nets scales sub-linearly in width, so wider networks improve the hierarchical advantage rather than erase it.
  • Phase transitions in the posterior support, driven by singularities, are predicted to persist in wide networks and to improve both approximation and estimation.
  • The same gap is expected for any odd analytic activation whose Taylor series begins with odd powers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same singularity-driven gap should appear in deeper architectures once the first-layer weights are allowed to move, suggesting that depth alone does not eliminate the need for singular learning theory.
  • Practical kernel methods that freeze early layers may systematically under-perform end-to-end training on pure statistical grounds, independent of optimization or approximation issues.
  • Measuring posterior mass near singular strata as width grows would give an empirical test of whether the real-log-canonical-threshold scaling is visible in finite networks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 9 minor

Summary. The paper compares two Bayesian learning schemes for a three-layer tanh network (M inputs, H hidden units, N outputs) when the data-generating conditional is the zero function plus isotropic Gaussian noise (eq. 23). In "single-layer learning" the input-to-hidden weights b are drawn from the prior and frozen (the kernel/lazy regime); in "hierarchical learning" both layers are trained (feature learning). Section 4 computes the single-layer generalization error exactly to leading order, G1(n) = HN/(2n) + o(1/n) (eq. 27), via the Gaussian integral over the last layer and an eigenvalue identity (eq. 35). Section 5 upper-bounds the hierarchical error through the real log canonical threshold, using the bound λ ≤ Λ from the author's earlier work and a new lower bound on the growing index Δ(k) ≥ C(k) k^{(M+1)/M} (eq. 49), obtaining G2(n) ≤ α(M) H^{M/(M+1)} N^{1/(M+1)}/n + o(1/n) with an explicit α(M) (Theorem 2, eq. 28). Since the exponent of H in the second bound is below 1, hierarchical learning is asymptotically superior for generic M, N in the regime M, N ≪ H ≪ n, and the paper concludes that singularities remain operative even at large width.

Significance. If correct, this is a clean, parameter-free asymptotic separation between the fixed-kernel (NNT/GP-like) and feature-learning regimes at large but finite width — a question of current interest given the lazy-vs-feature-learning literature. Strengths: the single-layer calculation is exact to leading order with no fitted constants; the hierarchical bound has a fully explicit coefficient α(M) (with a correct M→∞ expansion, α→e/2, and consistency with the known tight M=N=1 RLCT of [6]); the key inputs from prior singular learning theory (λ ≤ Λ, the Δ(k) machinery) enter as cited theorems with stated hypotheses, not as quantities tuned to the target; and the result yields concrete falsifiable numbers (eqs. 29-30: 500000/n vs ≲13737/n at M=N=100, H=10^4) that simulations could check. The qualitative message — that the infinite-width equivalence to kernel regression is limit-order dependent and that singularity-driven efficiency persists at width — is a genuinely useful theoretical data point.

major comments (2)
  1. [§5.2, eqs. (55)-(56)] The passage from eq. (55) to eq. (56) replaces C(s/t*^M) by C1(M), but the justification given is incorrect in two respects, and the step is load-bearing because α(M) — hence the quantitative comparison in eqs. (29)-(30) — depends on it. (i) Direction of the inequality: since C0(q) < 1 for finite q, C(h) < C1(M), so s − C(h)s^{(M+1)/M} ≥ s − C1(M)s^{(M+1)/M}; substituting C1(M) *lowers* the maximand, which is not a valid upper bound for max{0, s − C(h)s^{(M+1)/M}} unless a limit-interchange argument is supplied. (ii) The claimed uniform convergence of C(s/t*^M) to C1(M) on s ≥ M(t*)^M is false at the lower endpoint: at s = M(t*)^M one has h = M fixed as t*→0, so q(h) and C0(q(h)) do not approach their large-q limits. The step is likely repairable — C0 is monotone increasing in q (§5.1), so convergence is uniform on compact s-intervals bounded away from 0; for s ≥ C(M)^{-M} the maximand i
  2. [§3, Theorem 2; §4 eq. (35); §5 eqs. (36)-(37)] Theorem 2 is stated in the joint regime M, N ≪ H ≪ n, but both ingredient asymptotics are proved only as fixed-model limits. (a) Eq. (36)-(37), G2(n) = λ/n + o(1/n), is the standard RLCT asymptotic for a *fixed* model as n→∞; no uniformity of the o(1/n) remainder in H is established, yet the conclusion is applied with H growing with n. (b) On the single-layer side, eq. (35) requires Σ_h E_b[1/(nν_h(b)+1)] = o(1) for the leading term NH/(2n) to survive; for fixed H this follows from ν_h(b) > 0 a.s. (linear independence of the tanh features) and dominated convergence, but with H → ∞ the smallest eigenvalues of the H×H feature Gram matrix can drift toward 0, and the remainder is not controlled. Please state the precise order of limits (e.g., n→∞ first, then H, or H = H(n) with a rate condition) and bound the remainders accordingly; without this, the comparison G2 ≪ G1 in the stated regime i
minor comments (9)
  1. [§3, Theorem 2] Statement of Theorem 2: "G2(b)" should read "G2(n)".
  2. [§5.1] First line: "the growing index δ(k)" should be Δ(k); two lines below eq. (42), "If follows that" → "It follows that".
  3. [§5.2, eq. (53)] The maximization over 0 ≤ h in the definition of Λ is replaced by a maximum over M ≤ h in eq. (53) without comment. For h < M one has q = 0 and Δ(h) = 2h, so h − Δ(h)t = h(1−2t) ≤ M, contributing only an O(1) additive constant after the N/(2t^M) scaling in eq. (54) — negligible relative to the diverging main term. Please add this one-line justification.
  4. [§4, eqs. (31)-(35)] The symbol I(b) is used for both the empirical Gram matrix (defined after eq. (24)) and its population expectation ("Let I(b) be the expectation of I(b)"). Please use distinct symbols and state explicitly the concentration step I(b) → I(b) and the sense in which V1(X)² and V2(X) are lower order than V1(X).
  5. [§4, eq. (31)] Please double-check the constants in eq. (31), in particular the term tr(J(b)−1) and the prefactor NH/2; a one-line derivation of the Gaussian integral would help the reader verify them.
  6. [Abstract; §2, eqs. (21)-(22)] The abstract and §1 state the conclusion (training the first layer generalizes better) without noting that Theorem 2 is proved only for the noise-only realizable target q(y|x) ∝ exp(−∥y∥²/2). The decomposition (21)-(22) and the assertion G^n_1 ∼ G^n_2 are plausibility arguments, not part of the theorem; please say so explicitly and scope the abstract claim accordingly.
  7. [§5, Example after eq. (40)] Consistency check worth remarking on: for M = 1 the general bound gives α(1) = √2/2, i.e., Λ ≤ (√2/2)√(HN), whereas [19] and eq. (40) give the tight Λ = √(HN)/2. The general-M bound is therefore loose by a factor √2 in the one case where λ is known exactly; a sentence noting this would calibrate how sharp eq. (28) is likely to be.
  8. [§5.1, eq. (45)] The AM-GM step in eq. (45) is correct (the M−1 factors 2p,…,2p+M−2 have arithmetic mean 2p + M/2 − 1) but compressed; spelling out the mean computation would save the reader a verification.
  9. [§6, Discussion] It would strengthen the positioning to relate the H^{M/(M+1)}-vs-H scaling to known sample-complexity separations between kernel and feature-learning regimes (e.g., the works cited as [8, 10, 1]), since the present result is a Bayesian-average-case analog of those worst-case/gradient-based results.

Circularity Check

1 steps flagged

No definitional or fitted circularity; G2 bound rests on the author's prior RLCT/growing-index theorems, which are parameter-free mathematical results, not inputs fitted to the present claim.

specific steps
  1. self citation load bearing [Section 5, paragraph before 5.1; eqs. (36)–(39) and citation to Theorem 2 of [19]]
    "In the Theorem2 in [19], the formula of the upper bound of the real log canonical threshold λ≤Λ was proved, where Λ≡inf_{0<t<1}{HM t + N/2 max_{0≤h}(h−Δ(h)t)}, where Δ(h) is the growing index defined as follows."

    The upper envelope Λ that delivers eq. (28) is not re-derived; it is imported wholesale from the author's prior paper [19]. This is load-bearing for G2 but not circular in the definitional sense: [19] is a parameter-free theorem about RLCTs of three-layer nets, and the present work's new content is the large-H lower bound on Δ and the resulting α(M) coefficient. Recorded only as minor self-citation dependence.

full rationale

The paper's two main rates are obtained by direct asymptotic analysis, not by construction from the target. Section 4 derives G1(n)=HN/(2n)+o(1/n) from an explicit Gaussian integral over a, eigenvalue expansion of the Gram matrix, and elementary bounds on V1,V2; nothing is fitted and nothing is defined in terms of the claimed rate. Section 5 obtains the G2 upper bound by (i) invoking the general inequality λ≤Λ from the author's earlier Theorem 2 in [19], (ii) proving a new lower bound Δ(k)≥C(k)k^{(M+1)/M} for the growing index (eqs. 41–52), and (iii) optimizing the resulting expression over t and s to produce the explicit coefficient α(M). The self-citation to [19] (and the M=N=1 exact formula in [6]) is load-bearing for the existence of the Λ envelope, but those citations are parameter-free algebraic-geometry theorems whose assumptions do not include the wide-net comparison claimed here; they are independent support under the stated rules. There is no data fit renamed as a prediction, no uniqueness theorem used to forbid alternatives, and no ansatz smuggled in as a derived fact. The skeptic's concern about the C(h)→C1(M) exchange is a rigor gap in the large-H limit, not circularity. Score 1 reflects only the ordinary, non-circular dependence on the author's prior RLCT machinery.

Axiom & Free-Parameter Ledger

0 free parameters · 7 axioms · 0 invented entities

The central inequality rests on the Bayesian predictive framework, a highly specific realizable true distribution (zero map + Gaussian noise), Gaussian priors, tanh with odd Taylor series, the singular-learning identification G ∼ λ/n, and the prior upper-bound formula λ ≤ Λ built from the growing index Δ(k). No continuous parameters are fitted to data; the free choices are modeling assumptions that fix the setting rather than tuned constants.

axioms (7)
  • standard math Bayesian posterior and predictive distributions with product prior φ1(a)φ2(b); generalization error equals free-energy increment minus entropy (eqs. 18–19).
    Standard Bayesian decision theory; used throughout Sections 2–5.
  • domain assumption True conditional is exactly q(y|x) = (2π)^{-N/2} exp(−∥y∥²/2) with x ~ Uniform[−1,1]^M (eq. 23).
    Realizable zero-target setting that kills approximation error so only statistical/singularity effects remain; load-bearing for Theorem 2.
  • domain assumption Model is f(x,a,b) = Σ_h a_h tanh(b_h · x) with independent standard Gaussian priors on a and b.
    Fixes architecture and regularizer; Gaussian prior on a makes single-layer integrals closed form.
  • standard math For singular models, F2(n)−nS = λ log n + o(log n) and G2(n) = λ/n + o(1/n), with λ the real log canonical threshold.
    Core theorem of singular learning theory (Watanabe); invoked at eqs. 36–37.
  • standard math λ ≤ Λ where Λ = inf_{0<t<1}{H M t + (N/2) max_h (h − Δ(h) t)} with growing index Δ defined by binomial sums (eqs. 38–39), as proved in Watanabe 2001.
    Prior theorem used as black box to bound hierarchical free energy; correctness of Theorem 2 inherits this bound.
  • domain assumption Activation is odd with Taylor series Σ c_p x^{2p−1}, c_p ≠ 0 (discussion), so the same growing index applies.
    Needed to transfer Δ(k) from tanh to the stated class; different activations change Δ.
  • ad hoc to paper Regime M, N ≪ H ≪ n and large-H limit in which C(s/t*^M) → C1(M) uniformly on the relevant s-range.
    Technical hypothesis under which the optimized t* yields the clean α(M) expression; not proved with explicit error terms for finite H.

pith-pipeline@v1.2.0-grok45-kimik3 · 14198 in / 3971 out tokens · 71079 ms · 2026-07-30T23:15:33.527032+00:00 · methodology

0 comments
read the original abstract

Hierarchical neural networks are widely used in artificial intelligence, yet their mathematical properties remain incompletely understood. In the infinite-width limit, two different theoretical frameworks have been proposed. One reduces deep learning to kernel regression with a fixed kernel by assuming that the parameters remain close to their initialization, whereas the other allows the parameters to move away from their initialization, requiring the kernel itself to be optimized. In this paper, we study a three-layer neural network with a finite but large number of hidden units. We show that training the input-to-hidden weights yields a smaller generalization error than keeping them fixed. Furthermore, the latter setting exhibits singularities in the parameter space, whereas the former does not. These findings indicate that singularities play an essential role even in wide neural networks.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 3 linked inside Pith

  1. [1]

    Allen-Zhu and Y

    Z. Allen-Zhu and Y. Li. What can resnet learn efficiently, going beyond kernels? Advances in neural information processing systems, 32, 2019

    2019

  2. [2]

    Amari, N

    S. Amari, N. Fujita, and S. Shinomoto. Four types of leaning curves.Neural Computation, 4:605–618, 1992

    1992

  3. [3]

    Amari and N

    S. Amari and N. Murata. Statistical theory of learning curves under entropic loss criterion.Neural Computation, 5:140–153, 1993

    1993

  4. [4]

    M. Aoyagi. Consideration on the learning efficiency of multiple-layered neural networks with linear units.Neural Networks, 172:106132, 2024

    2024

  5. [5]

    Aoyagi and K

    M. Aoyagi and K. Nagata. Learning coefficient of generalization error in bayesian estimation and vandermonde matrix-type singularity.Neural Com- putation, 24(6):1569–1610, 2012. 13

    2012

  6. [6]

    Aoyagi and S

    M. Aoyagi and S. Watanabe. Resolution of singularities and the generaliza- tion error with bayesian estimation for layered neural network.IEICE Trans, 88(10):2112–2124, 2005

    2005

  7. [7]

    A.R. Barron. Universal approximation bounds for superpositions of a sigmoidal function.IEEE Transactions on Information theory, 39(3):930–945, 1993

    1993

  8. [8]

    Chizat, E

    L. Chizat, E. Oyallon, and F. Bach. On lazy training in differentiable program- ming.Advances in neural information processing systems, 32, 2019

    2019

  9. [9]

    Drton, E.Gross, , D

    M. Drton, E.Gross, , D. Kosta, A. Leykin, A. McCormack, S. Sullivant, and D. Windisch. Singular learning theory for factor analysis.arXiv preprint arXiv:2511.15419, 2025

    Pith/arXiv arXiv 2025

  10. [10]

    Ghorbani, S

    B. Ghorbani, S. Mei, T. Misiakiewicz, and A. Montanari. Limitations of lazy training of two-layers neural network.Advances in Neural Information Process- ing Systems, 32, 2019

    2019

  11. [11]

    N. Hayashi. The exact asymptotic form of bayesian generalization error in latent dirichlet allocation.Neural Networks, 137:127–137, 2021

    2021

  12. [12]

    Hoogland, G

    J. Hoogland, G. Wang, M. Farrugia-Roberts, L. Carroll, S. Wei, and D. Mur- fet. The developmental landscape of in-context learning.arXiv preprint, 2024(arXiv:2402.02364), 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Jacot, F

    A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks.Advances in neural information processing systems, 31, 2018

    2018

  14. [14]

    Kurumadani

    Y. Kurumadani. Upper bounds for local learning coefficients of three-layer neural networks.Neural Networks, page 10.1016/j.neunet.2026.109374, 2026

    arXiv 2026

  15. [15]

    J. Lee, Y. Bahri, R. Novak, S.S. Schoenholz, J. Pennington, and J. Sohl- Dickstein. Deep neural networks as gaussian processes.arXiv preprint arXiv:1711.00165, 2017

    Pith/arXiv arXiv 2017

  16. [16]

    Murata, S

    N. Murata, S. Yoshizawa, and S. Amari. Network information criterion- determining the number of hidden units for an artificial neural network model. IEEE transactions on neural networks, 5(6):865–872, 1994

    1994

  17. [17]

    Neal.Bayesian learning for neural networks, volume 118

    R.M. Neal.Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012

    2012

  18. [18]

    Watanabe

    S. Watanabe. Algebraic geometrical methods for hierarchical learning machines. Neural Networks, 14(8):1049–1060, 2001

    2001

  19. [19]

    Watanabe

    S. Watanabe. Learning efficiency of redundant neural networks in bayesian estimation.IEEE Transactions on Neural Networks, 12(6):1475–1486, 2001. 14

    2001

  20. [20]

    Watanabe.Algebraic geometry and statistical learning theory

    S. Watanabe.Algebraic geometry and statistical learning theory. Cambridge university press, Cambridge, 2009

    2009

  21. [21]

    S. Wei, D. Murfet, M. Gong, H. Li, J. Gell-Redman, and T. Quella. Deep learning is singular, and that’s good.IEEE Transactions on Neural Networks and Learning Systems, 34(12):10473–10486, 2022

    2022

  22. [22]

    Yamazaki and S

    K. Yamazaki and S. Watanabe. Singularities in mixture models and upper bounds of stochastic complexity.Neural networks, 16(7):1029–1038, 2003

    2003

  23. [23]

    Yamazaki and S

    K. Yamazaki and S. Watanabe. Algebraic geometry and stochastic complexity of hidden markov models.Neurocomputing, 69(1-3):62–84, 2005

    2005

  24. [24]

    Yamazaki and S

    K. Yamazaki and S. Watanabe. Singularities in complete bipartite graph-type boltzmann machines and upper bounds of stochastic complexities.IEEE trans- actions on neural networks, 16(2):312–324, 2005. 15

    2005