REVIEW 2 major objections 3 minor 29 references
Non-identifiability distinguishes Neural Networks among Parametric Models
T0 review · 2 major / 3 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read This paper proves that non-identifiability lets neural networks always capture some signal, while identifiable parametric models can be forced to fit only the mean.
desk verdict Two clean population-level theorems separating neural nets from identifiable parametric models; Theorem 2.1 needs an infimum restatement because the optimum may not exist. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The core of the argument is two lemmas plus a Gram–Schmidt construction. Lemma 2.1 shows, via the monotone class theorem and Fourier inversion, that a nonconstant regression function $\mathbb{E}[Y\mid X]$ must have nonzero correlation with the indicator of some half-space $\{\alpha\cdot X<t\}$. Lemma 2.2 constructs network parameters that approximate, in $L^2$, any affine function $c_2\mathbf{1}_{\{\alpha\cdot X\le c_1\}}+c_0$ of such a half-space indicator, which lets the network convert the correlation into a strictly better-than-constant predictor. The converse uses Gram–Schmidt to build a function $h(X)$ orthogonal to the constant and to every partial derivative $\partial_i f_{\theta_0}(X)$; the invertible Fisher information controls the linearized error, while the strong identifiability condition plus a Taylor bound keeps distant parameters from competing, forcing $f_{\theta_0}$ to be the unique best fit to $g(X)=f_{\theta_0}(X)+h(X)$.
What would settle it
A direct numerical check of the logistic example in Proposition 1.1 would settle the converse: with $X$ standard Gaussian and $g_0(x)=\cos(x_1)-\mathbb{E}[\cos(X_1)]$, compute the infimum of the logistic risk for small $\varepsilon$; the proposition predicts it equals $\mathrm{Var}(Y)$, so a single $\varepsilon$ with risk strictly below $\mathrm{Var}(Y)$ would refute the claimed contrast.
Extended reading notes
Core claim
The paper establishes two population-level theorems. Theorem 2.1 states that for any square-integrable $Y$ and any $X$ with a density, if $\mathbb{E}[\mathrm{Var}(Y\mid X)] < \mathrm{Var}(Y)$, then for any feedforward network with at least one hidden layer, linear output, and a tanh-type activation, any mean-square-optimal network $\hat{f}$ satisfies $\mathbb{E}[(\hat{f}(X)-Y)^2] < \mathrm{Var}(Y)$; the same conclusion holds for ReLU networks when the last hidden layer has width at least two, without requiring $X$ to have a density. Theorem 2.2 states that for any $C^2$ parametric family $\{f_\theta\}$ with a constant $f_{\theta_0}$, invertible Fisher information at $\theta_0$, a strong identifiability condition, and support of $X$ larger than $d+1$ points, there is a nonconstant $g(X)$ such that the square-loss projection of $g(X)$ onto the family is exactly $f_{\theta_0}$. The pair is meant as a contrast: identifiable parametric models can sometimes be forced to predict only the mean, while neural networks, because they are non-identifiable, are immune to this failure.
Load-bearing premise
The load-bearing premise is that a mean-square-optimal network actually exists; the proof only establishes that the greatest lower bound of the squared error is below $\mathrm{Var}(Y)$, so if no minimizer exists the theorem as stated has nothing to apply to.
Editorial extensions
If this is right
- For any distribution with $X$ continuous and $\mathbb{E}[\mathrm{Var}(Y\mid X)]<\mathrm{Var}(Y)$, every standard feedforward network with at least one hidden layer and a tanh-type activation has population mean-squared error strictly below $\mathrm{Var}(Y)$.
- For ReLU networks the same guarantee holds even when $X$ has no density, provided the last hidden layer has width at least two.
- Every smooth parametric family satisfying local identifiability (invertible Fisher information) and strong identifiability has some nontrivial regression problem on which its best square-loss fit is constant.
- No such identifiable parametric family can guarantee weak learning on every nontrivial distribution, so the non-identifiability of neural networks is doing essential work.
- In the Gaussian logistic example, a one-layer network with two extra scale-and-shift parameters improves on the constant predictor for every nontrivial outcome, while the identifiable logistic model can be forced to fit only $\mathbb{E}[Y]$.
Reading between the lines
- The theorems are population-level statements about the best function in a class, not about what gradient descent or finite-sample training finds; the paper's distinction does not by itself say whether non-identifiability helps or hurts optimization in practice.
- The half-space lemma suggests a testable generalization: any parametric family that can approximate affine transformations of half-space indicators, not just standard neural networks, should inherit the weak-learning guarantee.
- Theorem 2.2 constructs a noiseless target $g(X)$; adding independent noise would presumably preserve the conclusion for small enough noise, but the paper gives no quantitative threshold, so deriving one would be a natural extension.
- The logistic-versus-neural-network example suggests a general recipe: take any identifiable model and add redundant scale-and-shift parameters; the singularity at the constant function created by those extra parameters may be enough to turn a model that can be forced to learn nothing into one that always learns a little.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies the population-level regression risk of feedforward neural networks and of classic identifiable parametric models. It proves two theorems: first, for any square-integrable Y with E[Var(Y|X)] < Var(Y) and X with a density, and for any feedforward network with linear output and tanh-type (or ReLU) activation, the best-fitting network in mean square has risk strictly below Var(Y). Second, for a smooth parametric family that is locally and strongly identifiable and whose parameter dimension is smaller than the cardinality of the support of X, it constructs a non-constant response g(X) such that the best-fitting model is the constant predictor. The authors interpret these results as showing that non-identifiability is the distinguishing feature of neural networks.
Significance. If the results are taken in the corrected 'infimum' form, they provide a crisp, self-contained explanation for a qualitative difference between neural networks and classical parametric models. The proofs use standard tools (monotone class theorem, Fourier inversion, dominated convergence, Taylor expansion) and are largely rigorous. The corollaries for linear and logistic regression are instructive. The main conceptual claim is thought-provoking and likely to stimulate further work, even though the paper does not address optimization or finite-sample issues.
major comments (2)
- [Section 2.2, Theorem 2.1] The theorem assumes the existence of a mean-square-optimal network f-hat, but the proof in Section 4 only establishes that the infimum of E[(Y - f_theta(X))^2] over the parameter space is strictly less than Var(Y). A minimizer need not exist. For example, let X be uniform on [0,1] and Y = 1_{X > 1/2}; then all hypotheses of Theorem 2.1 hold for a one-hidden-layer tanh network with one hidden unit, yet no finite parameter vector achieves the infimum of 0, so no mean-square-optimal network exists and the theorem's conclusion is vacuous. The proof supports the weaker statement 'inf_theta E[(Y - f_theta(X))^2] < Var(Y)' or the epsilon version 'for every epsilon > 0 there exists theta with E[(Y - f_theta(X))^2] < Var(Y)'. The authors should restate the theorem accordingly.
- [Section 4.1, proof of Theorem 2.2] The chain of inequalities in the proof contains an algebraic error: the equality C3/sqrt(2C1) = C3 sqrt((1 - 2/(4C1)*C1)/C1^2) is false; the right-hand side equals C3/(sqrt(2) C1), not C3/sqrt(2C1). The subsequent inequality requires C1 <= 1, which is not guaranteed. The gap is repairable by choosing the strong-identifiability threshold eta smaller, for instance eta = C3/(sqrt(2) C1), but as written the proof does not establish the needed bound on ||theta - theta0|| for general C1.
minor comments (3)
- [Section 2.2 and Section 4 (proof of Theorem 2.1)] Lemma 2.1 yields an event of the form {alpha . X < t}, but the proof of Theorem 2.1 uses A := {alpha . X + b <= 0}; the strict versus non-strict inequality should be reconciled, although it is immaterial when X has a density.
- [Lemma 2.2] The lemma states the tanh-form result without an explicit density assumption on X, but the proof relies on P(alpha^T X = c1) = 0. The density condition appears only in Theorem 2.1; it should be stated in Lemma 2.2 as well.
- [Introduction and Proposition 1.1] Proposition 1.1 is stated in terms of the infimum over parameters, while Theorem 2.1 later uses a minimizer f-hat. This inconsistency in notation should be harmonized to avoid confusion about what is being asserted.
Circularity Check
No significant circularity: the paper is a self-contained mathematical proof with no fitted inputs, no prediction-from-fit, and no load-bearing self-citations.
full rationale
The paper's central results are proven directly rather than assumed. Theorem 2.1 derives the strict MSE improvement from Lemma 2.1 (existence of a correlated half-space) and Lemma 2.2 (explicit construction of network parameters approximating an affine half-space indicator), and the proof of Lemma 2.2 is carried out inside the paper via dominated convergence, not imported by citation. Theorem 2.2 constructs a non-constant g orthogonal to the constant and the model gradients, then uses Taylor expansion, the positive-definite Fisher information, and strong identifiability to show that θ0 is the L2 projection; again, nothing in the proof presupposes the conclusion. The cited literature, including universal approximation and singular learning theory, is used only as context and discussion, not as a premise in either proof. The one substantive concern noted by the reader is that Theorem 2.1 is stated for a mean-square-optimal network whose existence is not established, since the proof only shows the infimum is below Var(Y); but this is a correctness or attainment gap in the theorem statement, not a circularity. No parameter is fitted to data, no prediction is generated from fitted values, and no claimed result reduces by construction to an input or to a self-citation.
Assumptions & free parameters
assumptions (4)
- standard math Functional monotone class theorem
- standard math Fourier inversion theorem
- standard math Dominated convergence theorem
- domain assumption Existence of a mean-square optimal neural network f-hat
Cite this review
Pith. "Pith review of Non-identifiability distinguishes Neural Networks among Parametric Models." pith.science (2026). https://pith.science/paper/YPBLM74Q
@misc{pith2026250418017,
author = {Pith},
title = {Pith review of: Non-identifiability distinguishes Neural Networks among Parametric Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/YPBLM74Q}},
note = {Machine review of arXiv:2504.18017}
}
abstract
One of the enduring problems surrounding neural networks is to identify the factors that differentiate them from traditional statistical models. We prove a pair of results which distinguish feedforward neural networks among parametric models at the population level, for regression tasks. Firstly, we prove that for any pair of random variables $(X,Y)$, neural networks always learn a nontrivial relationship between $X$ and $Y$, if one exists. Secondly, we prove that for reasonable smooth parametric models, under local and global identifiability conditions, there exists a nontrivial $(X,Y)$ pair for which the parametric model learns the constant predictor $\mathbb{E}[Y]$. Together, our results suggest that a lack of identifiability distinguishes neural networks among the class of smooth parametric models.
Reference graph
Works this paper leans on
-
[1]
Learning and inference in hierarchical models with singularities
Shun-ichi Amari, Tomoko Ozeki, and Hyeyoung Park. Learning and inference in hierarchical models with singularities. Systems and Computers in Japan , 34(7):34--42, 2003
work page 2003
-
[2]
Universal approximation bounds for superpositions of a sigmoidal function
Andrew R Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory , 39(3):930--945, 1993
work page 1993
-
[3]
Phuong Bui Thi Mai and Christoph Lampert. Functional vs. parametric equivalence of relu networks. In 8th International Conference on Learning Representations , 2020
work page 2020
-
[4]
Approximation by superpositions of a sigmoidal function
George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems , 2(4):303--314, 1989
work page 1989
-
[5]
Olivier Delalleau and Yoshua Bengio. Shallow vs. deep sum-product networks. Advances in neural information processing systems , 24, 2011
work page 2011
-
[6]
Sharp minima can generalize for deep nets
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning , pages 1019--1028. PMLR, 2017
work page 2017
-
[7]
The power of depth for feedforward neural networks
Ronen Eldan and Ohad Shamir. The power of depth for feedforward neural networks. In Conference on learning theory , pages 907--940. PMLR, 2016
work page 2016
-
[8]
Real analysis: modern techniques and their applications , volume 40
Gerald B Folland. Real analysis: modern techniques and their applications , volume 40. John Wiley & Sons, 1999
1999
Show all 29 references
-
[9]
Structural degeneracy in neural networks
Matthew Farrugia-Roberts. Structural degeneracy in neural networks. Master's thesis, The University of Melbourne, December 2022. Minor revision. Originally submitted October 2022
2022
-
[10]
A regularity condition of the information matrix of a multilayer perceptron network
Kenji Fukumizu. A regularity condition of the information matrix of a multilayer perceptron network. Neural networks , 9(5):871--879, 1996
1996
-
[11]
An investigation into neural net optimization via hessian eigenvalue density
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In International Conference on Machine Learning , pages 2232--2241. PMLR, 2019
2019
-
[12]
Universal function approximation by deep neural nets with bounded width and relu activations
Boris Hanin. Universal function approximation by deep neural nets with bounded width and relu activations. Mathematics , 7(10):992, 2019
2019
-
[13]
Approximating continuous functions by relu nets of minimal width
Boris Hanin and Mark Sellke. Approximating continuous functions by relu nets of minimal width. arXiv preprint arXiv:1710.11278 , 2017
2017 arXiv
-
[14]
Multilayer feedforward networks are universal approximators
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks , 2(5):359--366, 1989
1989
-
[15]
The normalization method for alleviating pathological sharpness in wide neural networks
Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari. The normalization method for alleviating pathological sharpness in wide neural networks. Advances in neural information processing systems , 32, 2019
2019
-
[16]
Pathological spectra of the fisher information metric and its variants in deep neural networks
Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari. Pathological spectra of the fisher information metric and its variants in deep neural networks. arXiv preprint arXiv:1910.05992 , 2019
1910 arXiv
-
[17]
Universal statistics of fisher information in deep neural networks: Mean field approach
Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari. Universal statistics of fisher information in deep neural networks: Mean field approach. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1032--1041. PMLR, 2019
2019
-
[18]
On the ability of neural nets to express distributions
Holden Lee, Rong Ge, Tengyu Ma, Andrej Risteski, and Sanjeev Arora. On the ability of neural nets to express distributions. In Conference on Learning Theory , pages 1271--1296. PMLR, 2017
2017
-
[19]
The connection between approximation, depth separation and learnability in neural networks
Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz, and Ohad Shamir. The connection between approximation, depth separation and learnability in neural networks. In Conference on Learning Theory , pages 3265--3295. PMLR, 2021
2021
-
[20]
Measurements of three-level hierarchical structure in the outliers in the spectrum of deepnet hessians
Vardan Papyan. Measurements of three-level hierarchical structure in the outliers in the spectrum of deepnet hessians. arXiv preprint arXiv:1901.08244 , 2019
1901 arXiv
-
[21]
Topological properties of the set of functions generated by neural networks of fixed size
Philipp Petersen, Mones Raslan, and Felix Voigtlaender. Topological properties of the set of functions generated by neural networks of fixed size. Foundations of computational mathematics , 21:375--444, 2021
2021
-
[22]
The spectrum of the fisher information matrix of a single-hidden-layer neural network
Jeffrey Pennington and Pratik Worah. The spectrum of the fisher information matrix of a single-hidden-layer neural network. Advances in neural information processing systems , 31, 2018
2018
-
[23]
Empirical analysis of the hessian of over-parametrized neural networks
Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of over-parametrized neural networks. arXiv preprint arXiv:1706.04454 , 2017
2017 arXiv
-
[24]
Understanding machine learning: From theory to algorithms
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014
2014
-
[25]
Representation benefits of deep feedforward networks
Matus Telgarsky. Representation benefits of deep feedforward networks. arXiv preprint arXiv:1509.08101 , 2015
2015 arXiv
-
[26]
Algebraic geometry and statistical learning theory , volume 25
Sumio Watanabe. Algebraic geometry and statistical learning theory , volume 25. Cambridge university press, 2009
2009
-
[27]
Mathematical theory of Bayesian statistics
Sumio Watanabe. Mathematical theory of Bayesian statistics . chapman and hall/cRc, 2018
2018
-
[28]
Deep learning is singular, and that’s good
Susan Wei, Daniel Murfet, Mingming Gong, Hui Li, Jesse Gell-Redman, and Thomas Quella. Deep learning is singular, and that’s good. IEEE Transactions on Neural Networks and Learning Systems , 34(12):10473--10486, 2022
2022
-
[29]
The lack of a priori distinctions between learning algorithms
David H Wolpert. The lack of a priori distinctions between learning algorithms. Neural computation , 8(7):1341--1390, 1996
1996
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.