Singular leaning coefficients and efficiency in learning theory

Miki Aoyagi

REVIEW 3 major objections 5 minor 1 cited by

Singular leaning coefficients and efficiency in learning theory

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proves that a three-layer ReLU network's learning coefficient is a sum of linear-network coefficients over the input regions where the network is affine, after dead ReLU units are removed.

desk verdict A clean linear extension and a neat softmax lemma, but the ReLU theorem is asserted rather than proved—reject as is, though worth sending to review. read the letter →

arxiv 2501.12747 v2 pith:LF6ZQ6XV submitted 2025-01-22 stat.ML cs.LGmath.AGmath.STstat.TH

Miki Aoyagi This is my paper · ORCID

classification stat.MLcs.LGmath.AGmath.STstat.TH MSC 68T0762F1514B05

keywords learningcoefficientlogcanonicalthresholdReLUneuralnetworkdeeplinearsingulartheoryBayesiangeneralizationerrorSoftmaxfunctionresolutionofsingularities

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper works in singular Bayesian learning theory, where the learning coefficient $\lambda$ — the log canonical threshold of the average error function — controls the leading asymptotic of free energy and hence of generalization error. It aims to obtain exact values of $\lambda$ and its order $\theta$ for deep linear networks and for three-layer networks with ReLU activation. The central result is that a three-layer ReLU network, after segmenting the input space into regions where the network is an affine linear map and deleting dead ReLU units, has learning coefficient equal to a sum of the linear-network coefficients of the surviving affine pieces. The paper also proves a softmax reduction, showing that the singularity of a softmax output layer is governed only by pairwise differences of the underlying logits. If the main theorem is correct, exact asymptotic formulas for Bayesian generalization error become available for these architectures.

What carries the argument

The load-bearing object is the log canonical threshold $c_{w_0}(h,\varphi)$, the largest exponent $c$ for which $|h|^{-c}\varphi$ is locally integrable near $w_0$; Theorem 1 identifies it with the Bayesian learning coefficient $\lambda(w_0)$. Hironaka's resolution of singularities puts the average error function into normal-crossing monomial form $K(\pi(u))=u_1^{2k_1(u)}\cdots u_d^{2k_d(u)}$, from which the threshold and order are read as $\min_j (h_j+1)/(2k_j)$ and the count of coordinates attaining the minimum. Theorem 5 is the decomposition mechanism: for independent analytic pieces $(f^{(\alpha)}, g^{(\alpha)})$, the combined ideal $\langle f^{(1)},\dots,f^{(m)}\rangle$ has threshold $\sum_\alpha \lambda^{(\alpha)}$ and order $\sum_\alpha(\theta^{(\alpha)}-1)+1$. The ReLU argument uses Definition 5 to partition the input space into cells $V$ where each affine composition $F_{i(1)}^{(1)}\circ\cdots\circ F_{i(L)}^{(L)}$ has a fixed sign pattern; Lemma 2 removes rows and columns corresponding to dead units; and Theorem 3 replaces the $L^2$ average over $x$ with the squared norm of finitely many coefficient functions, so the analysis reduces to ideals in the parameter matrices.

What would settle it

Estimate the learning coefficient numerically from the free-energy asymptotics $-\log Z_n=\lambda\log n-(\theta-1)\log\log n+O_p(1)$ for a minimal three-layer ReLU network — one input unit, two hidden units, one output unit — at a true parameter where one hidden unit's pre-activation is exactly zero on the boundary of its active region. Compare the estimate with the Theorem 6 sum over the positive-volume regions; a mismatch would show the boundary contributes to the singularity and the segmentation assumption fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that ReLU nonlinearity creates no new singularities beyond linear ones: it only partitions the input space, and dead units can be pruned. Theorem 6 states that for the three-layer ReLU network $h_+(x,A,B)=A^{(1)}(A^{(2)}x+B^{(2)})_+$ with $H^{(3)}$ input units, $H^{(2)}$ hidden units and $H^{(1)}$ output units, after dividing the hidden layer into $k^{(2)}$ groups of sizes $H^{(2)}_i$, the learning coefficient is $\lambda = \sum_{i=1}^{k^{(2)}} \lambda(H'^{(1)}_i, H^{(2)}_i, H^{(3)}+1, r_i)$ and the order is $\theta = \sum_{i=1}^{k^{(2)}} (\theta(H'^{(1)}_i, H^{(2)}_i, H^{(3)}+1, r_i)-1)+1$, where $H^{(1)}-H'^{(1)}_i$ counts deleted output-side neurons for group $i$, $H^{(2)}-H'^{(2)}$ counts deleted hidden neurons, and $r_i$ is the rank of group $i$'s remaining linear map. The functions $\lambda$ and $\theta$ are the linear-network coefficients of Theorem 4, so the ReLU result is an additive reduction to the linear case. The additive form is powered by Theorem 5, which says that a product of independent analytic pieces has threshold equal to the sum of the pieces' thresholds and order equal to $\sum_\alpha(\theta^{(\alpha)}-1)+1$. The softmax section proves an ideal identity: the ideal generated by the softmax discrepancy equals the ideal generated by the logit differences $y_i(w)-y_1(w)-(y_i(w_0)-y_1(w_0))$.

Load-bearing premise

The central theorem rests on the unproved assertion that the log canonical threshold of the ReLU average error function is fully captured by summing the affine linear pieces over positive-volume regions, with no contribution from the boundaries where ReLU units switch, and that these pieces decompose additively across hidden-layer groups.

Editorial extensions

If this is right

For any three-layer ReLU network satisfying the theorem's hypotheses, the Bayesian free energy has the exact leading behaviour $-\lambda\log n+(\theta-1)\log\log n+O_p(1)$, so the asymptotic generalization error is known.
The theorem turns ReLU networks into the linear-unit case, so model-selection tools built on linear learning coefficients (WBIC, sBIC, MCMC benchmarks) apply to ReLU networks with the same values.
Dead ReLU units contribute nothing to the coefficient: pruning them before computing $\lambda$ and $\theta$ gives the same result, since the sum runs only over surviving affine regions.
The softmax reduction means any model with a softmax output can have its singularity analyzed through the ideal of logit differences rather than through the normalized exponential.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial: the same region-by-region decomposition appears extendable to deeper ReLU networks and to other piecewise-linear activations, because each layer only refines the sign-pattern partition; the paper proves the three-layer case only.
Editorial: the proof's reliance on positive-volume regions predicts that a hidden unit lying exactly on a switching boundary is removable rather than singular; this is a sharper, directly testable configuration than a generic parameter point.
Editorial: the softmax identity suggests a softmax layer reduces the effective output dimension by one without adding new singularities, which would let the ReLU/linear formulas carry over to classification models with normalized outputs.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this paper has a correct but modest linear-with-bias extension, a neat softmax ideal identity, and a ReLU theorem that is essentially unproved. I agree with the reader's verdict: reject as it stands, but the ReLU question is real and worth chasing.

What's new: Section 4 extends the prior linear result [7] to include bias terms. That reduction is fine—it follows from Theorem 3's norm equivalence and a change of variables. The softmax identity in Theorem 7 is a short algebraic observation that appears correct and could be useful to people working on softmax models. The ReLU claim in Theorem 6, if true, would give the first exact learning coefficients for three-layer ReLU networks, which matters for singular learning theory and Bayesian model selection.

The problem is the proof of Theorem 6. The step from K(w) = sum over positive-volume cells of integrals to c_w*(K(w)) = c_w*() is asserted in one line after Lemma 2. The stress-test note is correct: Theorem 3 only gives norm equivalence on a fixed region, and Lemma 2 is a matrix-element ideal identity. Neither justifies replacing a parameter-dependent partition integral by an ambient-parameter ideal. Region boundaries and cells that appear or disappear with parameters can change the log canonical threshold, and nothing here rules that out. Theorem 5, used to decompose the sum, is also only sketched: the proof assumes one monomial dominates the sum f^(1)^2+...+f^(m)^2 on each resolution chart, which is not guaranteed for general analytic functions.

The linear-with-bias result and the softmax lemma are fine, but they are not the main event. The main event needs a real proof, not a reference to 'by Theorem 3 and Lemma 2.' The citation pattern leans heavily on the author's own [7], which is acceptable if [7] is solid, but since the inheritance step is unproved, the reader cannot trust the conclusion.

Who is this for? People in singular learning theory and Bayesian model selection for neural networks. The paper has a useful core, but the central theorem is unsupported. I would send it to peer review rather than desk reject, because a referee could force the localization step to be addressed and the linear and softmax pieces are worth preserving. My recommendation is: reject in current form, but invite a major revision that actually proves the Section 5 equality.

Referee Report

3 major / 5 minor

Summary. The paper studies the log canonical threshold (learning coefficient) and its order for singular learning models, focusing on deep linear networks and three-layer networks with ReLU activation. Section 4 recovers the learning coefficient for linear networks by a change of variables and reduction to the author's previous theorem [7]. Section 5 claims that, after segmenting the input space by ReLU activation regions and removing dead neurons, the ReLU learning coefficient becomes a sum of linear-network learning coefficients, using Theorem 5 as the additivity tool. Section 6 sketches an analogous ideal reduction for the Softmax function. The main new quantitative claim is Theorem 6, which gives an explicit formula for the asymptotic Bayesian generalization error of three-layer ReLU networks.

Significance. If Theorem 6 is correct, it would be a genuinely useful result: it would give explicit learning coefficients for a canonical nonlinear activation and connect ReLU networks to the established linear-network theory. The paper is transparent that the linear-network part relies on [7], and the dead-neuron-removal viewpoint is natural. However, the central new theorem depends on a localization step that is asserted rather than proved, and Theorem 5, the additivity tool, has a proof that is at best a sketch with unjustified assumptions. The paper contains no independent verification, no numerical checks, and no machine-checked proofs, so the significance of the claimed result cannot yet be assessed.

major comments (3)

[Section 5, after Lemma 2] The displayed equality c_{w*}(K(w)) = c_{w*}(⟨F^{(1)}_{i(1)}∘⋯∘F^{(L)}_{i(L)}⟩), immediately after the partition of K(w) into cells V∈Ω^{(1)}, is the load-bearing step for Theorem 6, but it is only asserted, with a reference to Theorem 3 and Lemma 2. Theorem 3 gives a two-sided norm bound for a fixed polynomial on a fixed positive-volume set; it does not justify passing from an integral over a parameter-dependent partition of the input space to an ideal in the ambient parameter ring. The argument also does not address contributions from the boundaries of the cells, where the ReLU map is non-differentiable, nor cells whose volume or existence depends on w. Since no proof of this localization step is supplied, Theorem 6 is not supported as written.
[Section 5, Theorem 5] The proof of Theorem 5 assumes that, after taking a simultaneous resolution, each f^{(α)} is a monomial and that f^{(α0)2} ≥ f^{(α)2} for some α0 on each chart. Neither the existence of the simultaneous resolution with the stated matrix L and nonzero Jacobian nor the domination property is proved; for a sum of monomials, domination by a single term is not automatic on a chart. The later dimension count for the order θ is heuristic and does not by itself establish the claimed formula. Since Theorem 5 is the tool that converts the ReLU segmentation into an additive sum in Theorem 6, this gap independently blocks the central claim.
[Section 5, Theorem 6] Theorem 6 is stated with no derivation beyond the remark 'Using Theorem 5'. In particular, the formula λ = Σ_i λ(H'^{(1)}_i, H^{(2)}_i, H^{(3)}+1, r_i) requires showing that the parameter ideals for the k^{(2)} groups are independent, that removed neurons do not contribute, and that the input dimension enters as H^{(3)}+1 rather than H^{(3)}. These points are not demonstrated. As written, Theorem 6 is a conjecture supported only by the unproved localization step and the incomplete proof of Theorem 5.

minor comments (5)

[Title and Abstract] The title and abstract use 'leaning coefficients'; the intended term is 'learning coefficients'.
[Section 5, Definition 5] The paper does not state or prove that the cells V∈Ω^{(1)} with positive volume form a partition of X up to measure zero; this should be made explicit because the subsequent sum over V depends on it.
[Section 4, Proof of Theorem 4] The proof cites 'Lemma 1 (2)', but Lemma 1 has no numbered parts; please give the precise statement used and, ideally, restate the Main Theorem of [7] so that the reduction is self-contained.
[Section 5, Proof of Theorem 5] In the proof of Theorem 5, θ0 appears without definition and is later used as θ(α); the notation should be unified.
[Section 6, Proof of Theorem 7] The proof jumps from ideal equalities to LCT equalities; it should state explicitly that Lemma 1 is being applied to the ideals generated by the entries of the Softmax difference.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular reduction found; the ReLU theorem depends on an unproved localization step, which is a correctness gap rather than a circularity.

full rationale

I walked the derivation chain from K(w) through Theorem 3, Lemma 2, Theorem 5, and Theorem 6. No step equates the claimed prediction (the learning coefficient) to its own input by construction. The linear-network formula in Theorem 4 is taken from the author's prior paper [7] ('the Main Theorem in the paper [7]'), but this is a parameter-free theorem with an independent proof elsewhere, so it is legitimate self-citation, not circularity. The load-bearing line in Section 5, 'By Theorem 3 and by Lemma 2, we have c_w*(K(w)) = c_w*(<F_i(1)∘...∘F_i(L)>)', is an omitted proof: Theorem 3 is a norm equivalence on fixed positive-volume regions and Lemma 2 is an ideal identity; neither by itself establishes the localization of the log canonical threshold to the ideal of affine compositions, and boundary contributions are not addressed. This is a serious correctness gap, but it is not a circular reduction. Similarly, Theorem 5's assumption that a single monomial dominates f^(1)^2+...+f^(m)^2 is unjustified, but again is a proof gap. No fitted parameter is renamed as a prediction. The only circularity-adjacent feature is the repeated reliance on the author's own earlier results, but those results are independent support under the stated criteria, so the score stays low.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No free parameters are fitted to data; the formulas involve fixed model dimensions and ranks. The main burden is on unproved decomposition and resolution assumptions, plus the author's prior theorem [7].

assumptions (5)

standard math Hironaka's resolution of singularities applies to the average error function K(w) and yields normal crossing form.
Invoked in Section 2 to define λ and θ via equations (1)-(3), and in the proof of Theorem 1.
domain assumption The likelihood ratio function has relatively finite variance and the model is regular enough for Watanabe's asymptotic expansion.
Assumed in Section 2 to get the free energy expansion; standard in singular learning theory.
domain assumption The Main Theorem of Aoyagi [7] on learning coefficients of deep linear networks without thresholds is correct.
The proof of Theorem 4 reduces to this cited result; no derivation or independent verification is provided in this paper.
ad hoc to paper After segmenting the input space into regions V and removing dead neurons, the log canonical threshold of the ReLU model equals the LCT of the ideal generated by the affine maps F_i(1)∘...∘F_i(L) on each region, with boundary contributions vanishing.
This is the central unproved step in Section 5 used to go from K(w) to the ideal and then to Theorem 6.
ad hoc to paper The simultaneous resolution used in the proof of Theorem 5 exists, with the sum f^(1)^2+...+f^(m)^2 normal crossing and a nonzero Jacobian determinant on the relevant stratum.
The proof of Theorem 5 assumes these conditions to conclude the LCT is the sum of the λ(α); the existence is not proven.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Singular leaning coefficients and efficiency in learning theory." pith.science (2026). https://pith.science/paper/LF6ZQ6XV

@misc{pith2026250112747,
  author       = {Pith},
  title        = {Pith review of: Singular leaning coefficients and efficiency in learning theory},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LF6ZQ6XV}},
  note         = {Machine review of arXiv:2501.12747}
}

read the original abstract

Singular learning models with non-positive Fisher information matrices include neural networks, reduced-rank regression, Boltzmann machines, normal mixture models, and others. These models have been widely used in the development of learning machines. However, theoretical analysis is still in its early stages. In this paper, we examine learning coefficients, which indicate the general learning efficiency of deep linear learning models and three-layer neural network models with ReLU units. Finally, we extend the results to include the case of the Softmax function.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Upper Bounds for Local Learning Coefficients of Three-Layer Neural Networks
cs.LG 2026-03 conditional novelty 6.0 of 10

An upper-bound formula for local learning coefficients at singular points of three-layer networks is derived via blow-ups and matches known exact coefficients when the input dimension is one.

Reference graph

Works this paper leans on

24 extracted references · 22 canonical work pages · cited by 1 Pith paper

[7]

M. Aoyagi. Consideration on the learning efﬁciency of mu ltiple-layered neural networks with linear units. Neural Networks, 172-106132:1–11, 2024

work page 2024
[1]

M. Aoyagi. The zeta function of learning theory and gener alization error of three layered neural perceptron. RIMS Kokyuroku, Recent T opics on Real and Complex Singularities, 1501:153–167, 2006

work page 2006
[2]

M. Aoyagi. A Bayesian learning coefﬁcient of generaliza tion error and V andermonde matrix-type singulari- ties. Communications in Statistics - Theory and Methods , 39(15):2667–2687, 2010

work page 2010
[3]

M. Aoyagi. Consideration on singularities in learning t heory and the learning coefﬁcient. Entropy, 15(9):3714–3733, 2013

work page 2013
[4]

M. Aoyagi. Learning coefﬁcient in Bayesian estimation o f restricted Boltzmann machine. Journal of Alge- braic Statistics, 4(1):30–57, 2013

work page 2013
[5]

M. Aoyagi. Learning coefﬁcient of Vandermonde matrix-t ype singularities in model selection. Entropy (Information Theory, Probability and Statistics) , 21(6-561):1–12, 2019

work page 2019
[6]

M. Aoyagi. Learning coefﬁcients and information criter ia. Frontiers in Artiﬁcial Intelligence and Applica- tions, pages 351–362, 2019

work page 2019
[8]

Aoyagi and S

M. Aoyagi and S. Watanabe. Resolution of singularities a nd the generalization error with Bayesian estimation for layered neural network. IEICE Trans. J88-D-II, 10:2112–2124, 2005a

work page

Show all 24 references

[9]

M. Drton. Conference lecture: Bayesian information cri terion for singular models. Algebraic Statistics 2012 in the Alleghenies at The Pennsylvania State University, ht tp://jasonmorton.com/aspsu2012/, 2012

2012
[10]

Drton, S

M. Drton, S. Lin, L. Weihs, and P . Zwiernik. Marginal lik elihood and model selection for Gaussian latent tree and forest models. Bernoulli, 23(2):1202–1232, 2017

2017
[11]

W . Fulton. Introduction to toric varieties, Annals of Mathematics Studies. Princeton University Press, Prince- ton, NJ, USA, 1993

1993
[12]

Estimating the local learni ng coefﬁcient at scale

Zach Furman and Edmund Lau. Estimating the local learni ng coefﬁcient at scale. ArXiv, abs/2402.03698, 2024

2024 arXiv
[13]

Hironaka

H. Hironaka. Resolution of singularities of an algebra ic variety over a ﬁeld of characteristic zero. Annals of Math, 79:109–326, 1964

1964
[14]

Kashiwara

M. Kashiwara. B-functions and holonomic systems. Inventions Math., 38:33–53, 1976

1976
[15]

Koll´ ar

J. Koll´ ar. Singularities of pairs. Algebraic geometry-Santa Cruz 1995, Proc. Symp. Pure Math. , American Mathematical Society, Providence, RI,, 62:221–287, 1997

1995
[16]

M. Mustata. Singularities of pairs via jet schemes. J. Amer . Math. Soc., 15:599–615, 2002

2002
[17]

Nagata and S

K. Nagata and S. Watanabe. Exchange Monte Carlo samplin g from Bayesian posterior for singular learning machines. IEEE Transactions on Neural Networks , 19(7):1253–1266, 2008a. 12
[18]

Nagata and S

K. Nagata and S. Watanabe. Asymptotic behavior of excha nge ratio in exchange Monte Carlo method. International Journal of Neural Networks , 21(7):980–988, 2008b
[19]

Nakkiran, G

P . Nakkiran, G. Kaplun, Y . Bansal, T. Y ang, B. Barak, and I. Sutskever. Deep double descent: Where bigger models and more data hurt. ICLR2020, https://arxiv.org/pdf/1912.02292.pdf, 2020

1912 arXiv
[20]

Rusakov and D

D. Rusakov and D. Geiger. Asymptotic model selection fo r naive Bayesian networks. Proceedings of the Eighteenth Conference on Uncertainty in Artiﬁcial Intelli gence, pages 438–445, 2002

2002
[21]

Rusakov and D

D. Rusakov and D. Geiger. Asymptotic model selection fo r naive Bayesian networks. Journal of Machine Learning Research, 6:1–35, 2005

2005
[22]

Watanabe

S. Watanabe. Algebraic Geometry and Statistical Learning Theory , volume 25. Cambridge University Press, New Y ork, USA, 2009

2009
[23]

Watanabe

S. Watanabe. A widely applicable bayesian information criterion. Journal of Machine Learning Research , (14):867–897, 2013

2013
[24]

Zwiernik

P . Zwiernik. An asymptotic behavior of the marginal lik elihood for general Markov models. Journal of Machine Learning Research, 12:3283–3310, 2011. 13

2011

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [7]

M. Aoyagi. Consideration on the learning efﬁciency of mu ltiple-layered neural networks with linear units. Neural Networks, 172-106132:1–11, 2024

work page 2024

[2] [1]

M. Aoyagi. The zeta function of learning theory and gener alization error of three layered neural perceptron. RIMS Kokyuroku, Recent T opics on Real and Complex Singularities, 1501:153–167, 2006

work page 2006

[3] [2]

M. Aoyagi. A Bayesian learning coefﬁcient of generaliza tion error and V andermonde matrix-type singulari- ties. Communications in Statistics - Theory and Methods , 39(15):2667–2687, 2010

work page 2010

[4] [3]

M. Aoyagi. Consideration on singularities in learning t heory and the learning coefﬁcient. Entropy, 15(9):3714–3733, 2013

work page 2013

[5] [4]

M. Aoyagi. Learning coefﬁcient in Bayesian estimation o f restricted Boltzmann machine. Journal of Alge- braic Statistics, 4(1):30–57, 2013

work page 2013

[6] [5]

M. Aoyagi. Learning coefﬁcient of Vandermonde matrix-t ype singularities in model selection. Entropy (Information Theory, Probability and Statistics) , 21(6-561):1–12, 2019

work page 2019

[7] [6]

M. Aoyagi. Learning coefﬁcients and information criter ia. Frontiers in Artiﬁcial Intelligence and Applica- tions, pages 351–362, 2019

work page 2019

[8] [8]

Aoyagi and S

M. Aoyagi and S. Watanabe. Resolution of singularities a nd the generalization error with Bayesian estimation for layered neural network. IEICE Trans. J88-D-II, 10:2112–2124, 2005a

work page