{"id":"4313b0bc-9210-4eaa-aad4-56dc4281073b","arxiv_id":"2504.18017","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":8.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Neural networks always beat the constant predictor in population-level regression when a relationship exists, whereas any identifiable smooth parametric model can be reduced to predicting the mean for some distribution.","lead":"Mathematicians prove that neural networks, at the population level, always capture at least some of the signal in a regression task whenever any signal exists, while identifiable smooth parametric models can completely miss the signal for some data distributions. The result turns the usual complaint about neural networks, their non-identifiability, into a functional advantage that explains why they always weakly learn.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 2.1 assumes a mean-square-optimal network exists without proving attainment; the proof only shows the infimum is strictly below Var(Y), and for many distributions (e.g., Y a step) no optimum exists, so the theorem is vacuous as stated.","rationale":"The paper's goal is to show that lack of identifiability distinguishes neural networks from smooth parametric models at the population level. Theorem 2.1 is the first pillar, and its proof correctly shows that the infimum of the MSE over the neural-network parameter space is strictly below Var(Y). The theorem statement, however, quantifies over a mean-square-optimal network, an argmin that need not exist because the parameter space is non-compact and the function class is not closed in L2: the approximating step functions are only in the closure. The concrete example with Y = 1_{X > 1/2} and one tanh unit shows the loss can have a strict infimum that is not attained, making the theorem vacuous as stated. This is the same weakness the reader identified. The fix is straightforward (state the infimum or an approximation result), so the verdict remains CONDITIONAL rather than REJECT. I also noticed a secondary proof issue in Lemma 2.2: for tanh activations with D >= 2 hidden layers, the D-fold composition sigma^{(D)}(kZ) does not converge pointwise to 1{Z>0}; its limits are sigma^{(D-1)}(1) > 0 and sigma^{(D-1)}(0) < 1. However, the lemma's conclusion (approximation of an affine step) is still salvageable by adjusting the output-layer weights, so this is less load-bearing than the existence gap in Theorem 2.1. The paper merits acceptance conditional on correcting the theorem statement and tightening the proof of Lemma 2.2.","tokens_in":13548,"tokens_out":28143,"duration_ms":269224,"concrete_test":"Construct the distribution X ~ Uniform(0,1), Y = 1_{X > 1/2}, and the architecture with one hidden tanh unit: f_theta(x) = gamma + delta * sigma(alpha x + beta). Show analytically (or by plotting the loss) that as alpha -> infinity with beta = -alpha/2, gamma = 0, delta = 1, the MSE tends to 0, while for any finite alpha the sigmoid is not the exact step, so the MSE is strictly positive. Hence the infimum is 0 but is not attained; no mean-square-optimal network exists. This directly refutes the presupposition of Theorem 2.1 and settles whether the theorem as stated is vacuous.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central positive claim, Theorem 2.1, is stated for 'any mean-square optimal neural network' f-hat, defined as an argmin over the parameter space. The proof, however, establishes only that the infimum of E[(Y - f_theta(X))^2] over theta is strictly less than Var(Y): Lemma 2.1 finds a half-space A with Cov(Y,1_A) != 0, and Lemma 2.2 shows the network can approximate the best affine predictor c1 1_A + c0 arbitrarily well. Because the parameter space is non-compact and the limiting step function is generally not in the class (a finite-width sigmoid cannot exactly equal a step), the infimum need not be attained. For example, take X uniform on [0,1], Y = 1_{X > 1/2}, and a single hidden-layer tanh network with one hidden unit. Then E[Var(Y|X)] = 0 < Var(Y) = 1/4, all assumptions of Theorem 2.1 hold, and the MSE infimum is 0 (approached as the slope diverges), but no finite parameter vector achieves MSE 0. Thus no mean-square-optimal network exists, and the theorem's assertion is vacuous for this case. The correct statement would replace 'any mean-square optimal network' with 'the infimum over the parameter space' or 'for every epsilon > 0 there exists a network with MSE < Var(Y)'; the proof supports that weaker but sufficient claim. This does not destroy the paper's conceptual message but requires a corrected statement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the population-level regression risk of feedforward neural networks and of classic identifiable parametric models. It proves two theorems: first, for any square-integrable Y with E[Var(Y|X)] < Var(Y) and X with a density, and for any feedforward network with linear output and tanh-type (or ReLU) activation, the best-fitting network in mean square has risk strictly below Var(Y). Second, for a smooth parametric family that is locally and strongly identifiable and whose parameter dimension is smaller than the cardinality of the support of X, it constructs a non-constant response g(X) such that the best-fitting model is the constant predictor. The authors interpret these results as showing that non-identifiability is the distinguishing feature of neural networks.","tokens_in":13834,"tokens_out":15421,"duration_ms":127686,"significance":"If the results are taken in the corrected 'infimum' form, they provide a crisp, self-contained explanation for a qualitative difference between neural networks and classical parametric models. The proofs use standard tools (monotone class theorem, Fourier inversion, dominated convergence, Taylor expansion) and are largely rigorous. The corollaries for linear and logistic regression are instructive. The main conceptual claim is thought-provoking and likely to stimulate further work, even though the paper does not address optimization or finite-sample issues.","major_comments":[{"comment":"The theorem assumes the existence of a mean-square-optimal network f-hat, but the proof in Section 4 only establishes that the infimum of E[(Y - f_theta(X))^2] over the parameter space is strictly less than Var(Y). A minimizer need not exist. For example, let X be uniform on [0,1] and Y = 1_{X > 1/2}; then all hypotheses of Theorem 2.1 hold for a one-hidden-layer tanh network with one hidden unit, yet no finite parameter vector achieves the infimum of 0, so no mean-square-optimal network exists and the theorem's conclusion is vacuous. The proof supports the weaker statement 'inf_theta E[(Y - f_theta(X))^2] < Var(Y)' or the epsilon version 'for every epsilon > 0 there exists theta with E[(Y - f_theta(X))^2] < Var(Y)'. The authors should restate the theorem accordingly.","section":"Section 2.2, Theorem 2.1"},{"comment":"The chain of inequalities in the proof contains an algebraic error: the equality C3/sqrt(2C1) = C3 sqrt((1 - 2/(4C1)*C1)/C1^2) is false; the right-hand side equals C3/(sqrt(2) C1), not C3/sqrt(2C1). The subsequent inequality requires C1 <= 1, which is not guaranteed. The gap is repairable by choosing the strong-identifiability threshold eta smaller, for instance eta = C3/(sqrt(2) C1), but as written the proof does not establish the needed bound on ||theta - theta0|| for general C1.","section":"Section 4.1, proof of Theorem 2.2"}],"minor_comments":[{"comment":"Lemma 2.1 yields an event of the form {alpha . X < t}, but the proof of Theorem 2.1 uses A := {alpha . X + b <= 0}; the strict versus non-strict inequality should be reconciled, although it is immaterial when X has a density.","section":"Section 2.2 and Section 4 (proof of Theorem 2.1)"},{"comment":"The lemma states the tanh-form result without an explicit density assumption on X, but the proof relies on P(alpha^T X = c1) = 0. The density condition appears only in Theorem 2.1; it should be stated in Lemma 2.2 as well.","section":"Lemma 2.2"},{"comment":"Proposition 1.1 is stated in terms of the infimum over parameters, while Theorem 2.1 later uses a minimizer f-hat. This inconsistency in notation should be harmonized to avoid confusion about what is being asserted.","section":"Introduction and Proposition 1.1"}],"recommendation":"major_revision","confidential_remarks":"The paper contains a genuinely interesting pair of results and is likely to be accepted after a careful revision. The main issue is the incorrect existence assumption in Theorem 2.1, which is central to the paper's message. The proof of Theorem 2.2 also needs a small correction in the constant selection. Both are fixable without changing the core ideas. I would encourage the editor to request a revision rather than reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, clearly written theory paper. The new combination—Theorem 2.1’s half-space correlation lemma plus Theorem 2.2’s converse—makes a real point: at the population level, non-identifiability lets neural nets always find a weak signal, while identifiable smooth parametric families can be reduced to constants. The proof of Theorem 2.2 looks right: the Gram-Schmidt construction, Taylor bound, and strong identifiability are used carefully. The examples (linear, logistic) are nice.\n\nThe main soft spot is the one the stress-test flags. Theorem 2.1 quantifies over “any mean-square optimal neural network” but doesn’t prove one exists. The parameter space is non-compact, and the proof only shows the infimum is strictly below Var(Y). For something like a step function target, the optimum is not attained. So as written, the theorem is vacuous for those architectures. That’s a real presentation flaw, but not a conceptual one: the proof supports the infimum version, and the abstract’s “always learn” holds in the approximate sense. I’d advise the authors to restate Theorem 2.1 in terms of infimum or add a compactness/coercivity condition to guarantee attainment. Minor aside: the paper says ReLU works without density, but the lemma requires wD ≥ 2; that’s fine.\n\nThe citation pattern is appropriate—universal approximation and singular learning theory are context, not inputs. No data or code, but none needed.\n\nWho’s it for: statisticians and ML theorists who care about what makes neural nets different at the population level. It deserves peer review; a referee should ask for the restatement but shouldn’t block on it.","headline":"Two clean population-level theorems separating neural nets from identifiable parametric models; Theorem 2.1 needs an infimum restatement because the optimum may not exist.","tokens_in":14372,"tokens_out":2017,"would_cite":true,"duration_ms":20624,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proves that non-identifiability lets neural networks always capture some signal, while identifiable parametric models can be forced to fit only the mean.","keywords":["neural networks","identifiability","parametric models","regression","population risk","Fisher information","weak learning","non-identifiability"],"falsifier":"A direct numerical check of the logistic example in Proposition 1.1 would settle the converse: with $X$ standard Gaussian and $g_0(x)=\\cos(x_1)-\\mathbb{E}[\\cos(X_1)]$, compute the infimum of the logistic risk for small $\\varepsilon$; the proposition predicts it equals $\\mathrm{Var}(Y)$, so a single $\\varepsilon$ with risk strictly below $\\mathrm{Var}(Y)$ would refute the claimed contrast.","tokens_in":13331,"feed_emoji":"🧠","tokens_out":13068,"duration_ms":120086,"temperature":0.7,"pith_summary":"At the population level, this paper asks what makes feedforward neural networks different from classical parametric regression models. It proves that whenever $X$ carries any information about $Y$ — meaning $\\mathbb{E}[\\mathrm{Var}(Y\\mid X)] < \\mathrm{Var}(Y)$ — an optimally fitted neural network with standard activations and at least one hidden layer always beats the constant predictor. It also proves a partial converse: any smooth parametric family that is locally and strongly identifiable can be paired with some nontrivial distribution on which the best square-loss fit is exactly the constant mean. The explanation the paper offers is that neural networks are non-identifiable, and this lack of identifiability is precisely what lets them weakly learn in every nontrivial problem.","feed_headline":"Non-identifiability lets neural nets always beat the baseline","feed_subtitle":"Identifiable parametric models can be forced to fit only the mean; neural networks always find signal.","key_machinery":"The core of the argument is two lemmas plus a Gram–Schmidt construction. Lemma 2.1 shows, via the monotone class theorem and Fourier inversion, that a nonconstant regression function $\\mathbb{E}[Y\\mid X]$ must have nonzero correlation with the indicator of some half-space $\\{\\alpha\\cdot X<t\\}$. Lemma 2.2 constructs network parameters that approximate, in $L^2$, any affine function $c_2\\mathbf{1}_{\\{\\alpha\\cdot X\\le c_1\\}}+c_0$ of such a half-space indicator, which lets the network convert the correlation into a strictly better-than-constant predictor. The converse uses Gram–Schmidt to build a function $h(X)$ orthogonal to the constant and to every partial derivative $\\partial_i f_{\\theta_0}(X)$; the invertible Fisher information controls the linearized error, while the strong identifiability condition plus a Taylor bound keeps distant parameters from competing, forcing $f_{\\theta_0}$ to be the unique best fit to $g(X)=f_{\\theta_0}(X)+h(X)$.","core_discovery":"The paper establishes two population-level theorems. Theorem 2.1 states that for any square-integrable $Y$ and any $X$ with a density, if $\\mathbb{E}[\\mathrm{Var}(Y\\mid X)] < \\mathrm{Var}(Y)$, then for any feedforward network with at least one hidden layer, linear output, and a tanh-type activation, any mean-square-optimal network $\\hat{f}$ satisfies $\\mathbb{E}[(\\hat{f}(X)-Y)^2] < \\mathrm{Var}(Y)$; the same conclusion holds for ReLU networks when the last hidden layer has width at least two, without requiring $X$ to have a density. Theorem 2.2 states that for any $C^2$ parametric family $\\{f_\\theta\\}$ with a constant $f_{\\theta_0}$, invertible Fisher information at $\\theta_0$, a strong identifiability condition, and support of $X$ larger than $d+1$ points, there is a nonconstant $g(X)$ such that the square-loss projection of $g(X)$ onto the family is exactly $f_{\\theta_0}$. The pair is meant as a contrast: identifiable parametric models can sometimes be forced to predict only the mean, while neural networks, because they are non-identifiable, are immune to this failure.","pith_inferences":["The theorems are population-level statements about the best function in a class, not about what gradient descent or finite-sample training finds; the paper's distinction does not by itself say whether non-identifiability helps or hurts optimization in practice.","The half-space lemma suggests a testable generalization: any parametric family that can approximate affine transformations of half-space indicators, not just standard neural networks, should inherit the weak-learning guarantee.","Theorem 2.2 constructs a noiseless target $g(X)$; adding independent noise would presumably preserve the conclusion for small enough noise, but the paper gives no quantitative threshold, so deriving one would be a natural extension.","The logistic-versus-neural-network example suggests a general recipe: take any identifiable model and add redundant scale-and-shift parameters; the singularity at the constant function created by those extra parameters may be enough to turn a model that can be forced to learn nothing into one that always learns a little."],"forward_implications":["For any distribution with $X$ continuous and $\\mathbb{E}[\\mathrm{Var}(Y\\mid X)]<\\mathrm{Var}(Y)$, every standard feedforward network with at least one hidden layer and a tanh-type activation has population mean-squared error strictly below $\\mathrm{Var}(Y)$.","For ReLU networks the same guarantee holds even when $X$ has no density, provided the last hidden layer has width at least two.","Every smooth parametric family satisfying local identifiability (invertible Fisher information) and strong identifiability has some nontrivial regression problem on which its best square-loss fit is constant.","No such identifiable parametric family can guarantee weak learning on every nontrivial distribution, so the non-identifiability of neural networks is doing essential work.","In the Gaussian logistic example, a one-layer network with two extra scale-and-shift parameters improves on the constant predictor for every nontrivial outcome, while the identifiable logistic model can be forced to fit only $\\mathbb{E}[Y]$."],"supporting_citations":[{"why":"It supplies the Fourier-inversion and bump-function facts used in the proof of Lemma 2.1.","marker":"[Fol99]"},{"why":"It shows that for some ReLU architectures, permutation and scaling are the only function-preserving weight transformations, which supports the paper's claim that neural networks fail strong identifiability.","marker":"[BTML20]"},{"why":"It proves that functions generated by fixed-size networks can be close in uniform norm while no weight parametrizations are close, giving another sense in which neural network parametrizations are non-identifiable.","marker":"[PRV21]"},{"why":"It gives a sufficient condition, irreducibility, for the Fisher information of a multilayer perceptron to be positive definite, the local-identifiability contrast used in the discussion.","marker":"[Fuk96]"}],"fun_headline_variants":["Non-identifiability: why neural nets always beat parametric models","Identifiable models can fail; neural nets always find signal","Neural networks always detect signal; parametric models may not","Lack of identifiability guarantees neural nets find signal","Identifiability forces mean prediction; non-identifiability finds signal"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a mean-square-optimal network actually exists; the proof only establishes that the greatest lower bound of the squared error is below $\\mathrm{Var}(Y)$, so if no minimizer exists the theorem as stated has nothing to apply to.","fun_headline_variants_meta":{"raw":{"variants":["Non-identifiability: why neural nets always beat parametric models","Identifiable models can fail; neural nets always find signal","Neural networks always detect signal; parametric models may not","Lack of identifiability guarantees neural nets find signal","Identifiability forces mean prediction; non-identifiability finds signal"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00104,"raw_usage":{"total_tokens":4370,"prompt_tokens":936,"completion_tokens":3434,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":3348}},"tokens_in":552,"tokens_out":3434,"duration_ms":23836,"temperature":1.0,"reasoning_tokens":3348,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:27:54.701247+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct numerical check of the logistic example in Proposition 1.1 would settle the converse: with $X$ standard Gaussian and $g_0(x)=\\cos(x_1)-\\mathbb{E}[\\cos(X_1)]$, compute the infimum of the logistic risk for small $\\varepsilon$; the proposition predicts it equals $\\mathrm{Var}(Y)$, so a single $\\varepsilon$ with risk strictly below $\\mathrm{Var}(Y)$ would refute the claimed contrast.","supporting_citations":[],"review_version":1}