{"id":"7c31c6cf-f597-41a1-a097-bc9a06bc5c5d","arxiv_id":"2505.13582","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper introduces sample-independent and sample-dependent critical lifting between networks of different widths, shows known embeddings do not capture all sample-independent liftings, and proves sample-dependent lifted critical points and saddles exist for sufficiently large sample sizes.","lead":"This theoretical paper defines a way to tell whether a critical point of a neural network's loss is forced by the network architecture alone or depends on the training samples, by lifting critical points from narrower to wider networks. It shows known embedding operators miss some architecture-forced critical sets, and proves that sample-dependent lifted critical points, and their saddles, appear once the dataset is large enough.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Threshold inconsistency between Theorem 4.2.1 and its proof may understate sample size needed for sample-dependent saddles.","rationale":"The reader's weakest_assumption focused on the generic-position condition on inputs (xi nonzero, xi±xj nonzero), which is explicitly stated in the theorem and appears mathematically sound: the proof of Lemma A.1.1 handles that assumption, and the paper acknowledges it. I found a more direct internal inconsistency: the proof of Theorem 4.2.1 (Proposition A.2.3) claims to prove the saddle conclusion under n >= 2+(d+1)m, but the main-text theorem states the threshold as n > 2+(d+1)m. Since n and (d+1)m are integers, n > 2+(d+1)m is equivalent to n >= (d+1)m+3, whereas the proof's condition n >= 2+(d+1)m is equivalent to n >= (d+1)m+2. The proof gives a gap of one sample relative to the theorem statement. This is exactly the type of strict/non-strict inequality mismatch the reader noted as a minor issue, but it is not merely cosmetic: it concerns the advertised sample-size threshold for saddles, which is the central quantitative claim of the paper. If the proof's condition is the correct one, the main statement should be weakened; if the theorem's stronger statement is intended, the proof needs an additional argument to show that the extra sample buys a non-trivial kernel or a non-zero loss. Either way, the central existence claim's threshold is not fully settled as stated. The rest of the paper - the sample-independent lifting operator, the counterexample, and the saddle property conditional on existence - appears internally consistent and the generic-position assumptions are legitimate and clearly stated. Thus I recommend CONDITIONAL, not REJECT: the concern is about precision of the main threshold and the proof's gap, not a fundamental flaw in the construction. The concrete rank-count test would settle whether the proof's conditions actually imply the theorem's threshold or whether the theorem needs correction.","tokens_in":22142,"tokens_out":2185,"duration_ms":16543,"concrete_test":"Check the rank argument in A.2.3: compute the size and rank of the matrix M' constructed with rows [partial_{w_k} H(theta_narr,x_i)] for k=1..m and rows [sigma(w'·x_i)] and [sigma'(w'·x_i)x_i]. For a generic w', count the exact number of independent rows and determine the minimum n for which ker M' must be non-trivial. Verify whether n >= 2+(d+1)m (i.e., n >= (d+1)m+2) indeed guarantees ker M' nonempty, or whether n >= (d+1)m+3 is required. If the former, the theorem's strict inequality n > 2+(d+1)m is stronger than what the proof establishes; reconcile by changing the theorem to n > 1+(d+1)m or by adding an argument for the extra sample.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim: for n > 2 + (d+1)m, sample-dependent lifted saddles of form (2) exist. The proof in A.2.3 first builds a full-measure set of w' for which n >= 1+(d+1)m suffices for a sample-dependent critical point. But it then claims the saddle portion follows when n >= 2+(d+1)m by finding v' in ker M' with M' of size (d+1)m+1 rows and n columns. For a non-trivial kernel this needs n > (d+1)m+1 if rows are independent, i.e. n >= (d+1)m+2, not n >= (d+1)m+2 merely from n >= 2+(d+1)m. The latter is n >= (d+1)m+2, which is n > (d+1)m+1, so it actually coincides with the strict version: 'n > 1+(d+1)m' means n >= (d+1)m+2 when (d+1)m is an integer. So the condition n >= 2+(d+1)m in the proof matches n > 1+(d+1)m, not n > 2+(d+1)m as stated in the theorem. Because the theorem claims saddles when n > 2+(d+1)m, i.e. n >= (d+1)m+3, the proof establishes saddles for n >= (d+1)m+2, which is one sample less than the theorem's lower bound. This is a mismatch in the advertised threshold: the stronger claim n > 2+(d+1)m is not proven; only n > 1+(d+1)m is established. If the intended strict thresholds are load-bearing, the main-text statement should be corrected to n > 1+(d+1)m for saddles, or the proof should be re-examined for an additional independent condition requiring the extra sample.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a sample-independent critical lifting operator that maps a critical parameter of a narrower fully connected network to the set of parameters of a wider network that preserve both the output function and criticality for every sample for which the original parameter is critical (Definition 4.1). It then separates sample-independent from sample-dependent lifted critical points. The main results are: (i) an example showing that known embedding operators (splitting, null, and compatible embeddings) do not exhaust the sample-independent lifted critical points for three-hidden-layer networks; (ii) for one-hidden-layer networks, a zero-measure statement and a saddle statement for lifted critical points of the form (2); (iii) existence of sample-dependent lifted critical points and saddles once the sample size exceeds thresholds of the form n > 1 + (d+1)m and n > 2 + (d+1)m (Theorem 4.2.1); and (iv) analogous multi-layer statements with thresholds depending on the parameter count of the narrower network (Theorem 4.2.2). Proofs in the appendix use real analyticity, a linear-independence lemma for neurons, Mityagin's zero-measure lemma, and a kernel-membership construction to choose sample outputs. The paper is theoretical, with a small numerical illustration in Section 5.","tokens_in":22518,"tokens_out":28121,"duration_ms":271854,"significance":"If the results are correct, the paper provides a useful formal framework for separating architecture-driven, sample-independent structure from sample-dependent structure in neural network loss landscapes. The explicit sample-size thresholds are concrete and checkable, and the proof strategy is transparent: the linear-independence lemma for neurons is carefully stated, and the kernel arguments in the appendix are logically coherent. The paper is also honest about its limitations, including the generic-position condition on sample inputs and the restriction to analytic activations and width-wise lifting. A notable strength is that the main existence theorems are proved from the paper's own definitions, with no fitted parameters and no circular reliance on the target result. The counterexample to completeness of previously studied embeddings is simple and effective, although it needs a stated hypothesis on the activation (see major comments). Overall, the contribution is a solid incremental advance over the embedding-principle literature, and the open questions in the conclusion are appropriate.","major_comments":[{"comment":"The main-text example claims that all parameters in E_narr and E_wide yield the constant zero output function and are critical for arbitrary samples. This claim is true only when σ(0)=0, but that hypothesis appears only in the appendix Proposition A.2.1, not in the main-text statement of the example. Under the standing Assumption 3.1 alone, activations such as sigmoid or Gaussian satisfy the assumption yet have σ(0)≠0, and for those activations the parameters of the form (3) do not produce the zero output function and the derivative computations in Proposition A.2.1 fail. The hypothesis σ(0)=0 should be stated explicitly in Section 4.1 where the example is introduced, and in Remark 4.2 if the generalization is claimed for arbitrary L≥3.","section":"Section 4.1 and Proposition A.2.1"},{"comment":"There is a threshold mismatch between the statement of Proposition A.2.3 and its proof. The proposition states that sample-dependent lifted critical points exist for n ≥ 1 + (d+1)m, but the proof constructs the outputs y'_i by choosing a nonzero v' ∈ ker M', which is guaranteed only when n ≥ 2 + (d+1)m; for n = 1 + (d+1)m, the matrix M' is square and its kernel may be trivial. The main theorem's own critical-point condition n > 1 + (d+1)m is, for integer n, exactly n ≥ 2 + (d+1)m, so the theorem is not affected. However, the appendix statement should be aligned to n ≥ 2 + (d+1)m, or the proof should be completed by the alternative construction y'_i = H(θ_narr, x_i), which puts the zero vector in ker M' and still yields a sample-dependent lifted critical point at n = 1 + (d+1)m.","section":"Proposition A.2.3 and Theorem 4.2.1"}],"minor_comments":[{"comment":"The statement says 'for 1 ≤ i < j ≤ m' but the index should be n; the domain is x_1,...,x_n, so the condition should read 'for 1 ≤ i < j ≤ n'.","section":"Lemma A.1.1"},{"comment":"The sentence 'Since n≥..., M′ and M′′ have more rows than columns, so there is some v′∈ker M′′\\{0}' has the inequality backwards: a nontrivial kernel requires more columns than rows. This is a typo, but it should be corrected because the rank argument is inverted as printed.","section":"Appendix A.2.5(b)"},{"comment":"The text defines the plotted vector field with the factor 1/a_1 multiplying ∂R/∂w_1, but the caption of Figure 1 writes the factor as 3/a_1. The factor should be made consistent.","section":"Section 5, Figure 1"},{"comment":"The thresholds written as 'n≥ 1+N / D' and 'n≥ 1+D+Σ_{l=2}^L m_l(m'_{l-1}-m_{l-1})+N / D' are ambiguous in the rendered text. They should be typeset as fractions (1+N)/D and (1+D+Σ + N)/D to avoid being read as 1 + N/D.","section":"Theorem 4.2.2 and Proposition A.2.5"},{"comment":"The sentence 'Recall that we have shown that all sample-dependent critical points must be of the form (2)' overstates the scope: the form-(2) statement is established in Remark 4.5 for one-hidden-layer networks under parity or related assumptions, not for the general multi-layer case. The sentence should be qualified accordingly.","section":"Conclusion, Section 6"}],"recommendation":"minor_revision","confidential_remarks":"The paper is in good shape technically; the central existence theorems appear correct, and the issues I found are localized and fixable. The authors should be asked to state the σ(0)=0 assumption in the main-text example and to reconcile the threshold in Proposition A.2.3 with its proof. I do not see concerns about novelty or citation practice: the relationship to the embedding-principle literature is properly acknowledged, and the new operator and sample-dependence thresholds are sufficiently distinct to merit publication in a theoretical venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What is actually new here is the formal separation of sample-independent from sample-dependent lifted critical points, plus the counterexample showing that the previously studied embedding operators (splitting, null, compatible) do not exhaust the sample-independent structure. That is a genuine step beyond the embedding-principle literature, which mostly assumed sample-independent embeddings. The existence theorems for sample-dependent critical points and saddles, with explicit sample-size thresholds, are also new and useful.\n\nThe proofs are the real strength. They rest on the linear independence of neurons under the generic-position assumptions (xi nonzero, xi ± xj nonzero) and on standard zero-measure properties of real-analytic functions. I checked the main chain: the kernel-construction in Proposition A.2.3 is sound, and the saddle argument via Lemma A.1.2 is correct because output-preserving perturbations give points with the same loss and nonzero gradient. The multi-layer version is dense but follows the same logic.\n\nSoft spots are minor. First, the Section 4.1 counterexample silently requires sigma(0)=0; the appendix states it, the main text does not. That should be fixed. Second, the thresholds: the theorem statements use strict inequalities (n > 1+(d+1)m, n > 2+(d+1)m) while the proofs use non-strict ones (n >= ..., n >= ...). Since n is an integer, the proofs actually establish existence for one fewer sample than the theorems claim. So the theorems are conservative, not wrong. A stress-test note claimed this \"understates sample size needed,\" which is backwards; the only real issue is a minor mismatch in how the thresholds are advertised. Third, the multi-layer proof could use a worked example, but that is presentation.\n\nThe paper is honest about its open questions, especially whether embedding operators fully characterize sample-independent lifted points, and the dynamics near degenerate saddles. It is a theory paper for people working on loss landscapes and the embedding principle; the toy illustration is small but does support the claims.\n\nMy verdict: worth a serious referee. The central results appear correct, the proofs are reproducible, and the flaws are easily fixed. I would engage with it.","headline":"A careful existence theory for sample-dependent lifted critical points; the main theorems hold up, with minor clarifications needed on assumptions and threshold statements.","tokens_in":23001,"tokens_out":3994,"would_cite":true,"duration_ms":37548,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper establishes that, for sample sizes larger than a parameter-count threshold, widening a neural network creates sample-dependent critical points and saddles, and that previously studied critical embeddings fail to account for all…","keywords":["sample-dependent critical points","sample-independent critical lifting","loss landscape","neural network width","saddle points","critical embeddings","overparameterization","real analytic functions"],"falsifier":"Fix a one-hidden-layer tanh network with $m=1$, input dimension $d=1$, and four sample inputs containing a zero, say $x_1=0$ plus three generic nonzero inputs, so the threshold $n>3$ is met but the generic-position condition fails. Vary the outputs to search for parameters of the form $(a_1,w_1,0,w')$ that are critical for one output vector but not for another; if none exist, the generic-position assumption is essential to Theorem 4.2.1.","tokens_in":1875,"feed_emoji":"🧠","tokens_out":8396,"duration_ms":136329,"temperature":0.7,"pith_summary":"This paper studies how the critical points of a neural network's loss surface depend on the training sample. It introduces a sample-independent critical lifting operator, which maps a parameter of a narrower network to the parameters of a wider network that preserve both the output function and criticality for every sample for which the narrower parameter is critical. The central result is that once the number of samples exceeds a threshold proportional to the narrower network's parameter count, sample-dependent lifted critical points are unavoidable, and beyond a slightly larger threshold some of these are saddles. The message is that data dependence of the critical set is not a degenerate accident: for sufficiently large datasets it is the typical situation, and the extra critical points are not local minima.","feed_headline":"Large datasets create sample-dependent critical points in wider nets","feed_subtitle":"A new sample-independent lifting operator shows the extra critical points are saddles once n exceeds a threshold","key_machinery":"The engine is the sample-independent critical lifting operator, defined as the intersection over all samples for which the narrower parameter is critical, of the wider parameters that preserve both output and criticality. For one-hidden-layer networks, the decisive object is the analytic function $\\varphi(w') = \\sum_{i=1}^n \\partial_p \\ell(H(\\theta_{\\mathrm{narr}},x_i),y_i)\\,\\sigma(w'\\cdot x_i)$, whose zeros decide whether a dead output neuron with weight zero and input weight $w'$ is critical. Because the activation is non-polynomial analytic and the inputs satisfy $x_i\\neq 0$ and $x_i\\pm x_j\\neq 0$, the functions $w'\\mapsto \\sigma(w'\\cdot x_i)$ are linearly independent; hence $\\varphi$ is nonconstant analytic, and its zero set has measure zero, making criticality of newly added neurons exceptional for a fixed sample. The threshold $1+(d+1)m$ comes from counting equations versus unknowns in the matrix whose kernel contains the vector of loss derivatives: exceeding it guarantees a nonzero kernel direction, which allows the authors to choose output values that make $\\theta_{\\mathrm{wide}}$ critical, and the second threshold $2+(d+1)m$ keeps a kernel direction after adding an extra constraint, yielding a nonzero-loss critical point that a gradient-flow argument converts into a saddle.","core_discovery":"The paper claims that critical sets of deep neural networks have a sample-independent core and a sample-dependent shell, and that the two are separated by a lifting operator. Fixing a critical point of a narrower network, one can add zero-output neurons with freely chosen input weights; these parameters preserve the output function but, for a fixed sample, almost none of them are critical (Proposition 4.2.1). However, when the sample size is large enough, the situation reverses: Theorem 4.2.1 states that for $m$ neurons, input dimension $d$, and sample size $n>1+(d+1)m$, there exist sample-dependent lifted critical points of the form (2), and for $n>2+(d+1)m$ there exist sample-dependent lifted saddles. The multi-layer generalization, Theorem 4.2.2, gives the same phenomenon with thresholds expressed in terms of the narrower network's parameter count and the intermediate widths. The paper also proves by example that previously studied splitting, null, and compatible embeddings do not generate all sample-independent lifted critical points, so the full sample-independent structure is richer than earlier embedding operators captured.","pith_inferences":["The finite-sample thresholds suggest an interpolation between two regimes: with tiny sample size, the critical sets of wide networks are largely architecture-determined, while with large sample size, sample-specific saddles proliferate; this may be observable as a change in how gradient trajectories from random initializations behave as $n$ grows.","A testable extension would replace the analytic, non-polynomial activation by a polynomial or ReLU activation: the linear-independence lemma can fail there, so the sample-dependent saddles may disappear or become nongeneric, separating the analyticity assumption from the counting argument.","The same lifting argument should apply to other symmetric architectures, such as convolutional or residual networks, whenever zero-weight neurons preserve the output function and the relevant neuron maps are linearly independent; the threshold would then be parameter-count based.","The generic-position requirement (no zero or opposite inputs) holds for generic continuous data but is systematically violated by duplicate samples, mirrored samples, or constant features, so datasets with such structure may exhibit a different critical-set geometry."],"forward_implications":["For one-hidden-layer networks, every sample-independent lifted critical point obtained from a nonzero-loss critical point is a saddle, so gradient methods that avoid saddles are not trapped there.","The threshold $n>1+(d+1)m$ identifies a regime in which the loss landscape of a wider network has critical points that cannot be predicted from the narrower network alone; sample information matters exactly when the dataset is large relative to the narrow network's parameter count.","The multi-layer version gives explicit sample-size thresholds involving the widths of intermediate layers, so the phenomenon persists in depth rather than being special to one hidden layer.","Previously proposed splitting, null, and compatible embeddings are insufficient: there are sample-independent lifted critical points, such as parameters producing identical zero output functions, that no combination of those embeddings generates.","The sample-dependent saddles found are degenerate and may lack a negative eigenvalue, so standard saddle-avoidance results do not apply directly near them."],"supporting_citations":[{"why":"Defines critical embedding operators for networks of different widths and proves they preserve criticality; these are the operators the paper shows are incomplete.","marker":"Zhang et al. (2022)"},{"why":"States the embedding principle for networks of different widths and gives splitting and null embeddings that produce sample-independent lifted critical points.","marker":"Zhang et al. (2021)"},{"why":"Introduces critical lifting operators in depth, the direct template for the sample-independent critical lifting operator defined here.","marker":"Bai et al. (2024)"},{"why":"Provides the earlier observation that critical points embed across one-hidden-layer networks and produce saddles; used as precedent and comparison.","marker":"Fukumizu and ichi Amari (2000)"},{"why":"Supplies a linear-independence lemma for neurons used inside Lemma A.1.1, and documents global-minima geometry that this paper complements.","marker":"Simsek et al. (2021)"},{"why":"Provides the core zero-measure lemma: a nonconstant real analytic function has a zero set of measure zero, used repeatedly to show near-critical points produce saddles.","marker":"Mityagin (2015)"},{"why":"Supplies the real-analytic definitions and composition and derivative facts used to construct the analytic functions in the proofs.","marker":"Krantz and Parks (2002)"},{"why":"Gives the standard result that first-order methods avoid saddles; the paper notes this result does not directly apply to the degenerate saddles found here.","marker":"Lee et al. (2017)"}],"fun_headline_variants":["Data size threshold creates extra critical points in wider nets","Wider nets get sample-dependent saddles once data passes a bound","Lifting operator reveals sample-dependent saddles for large data","Bigger samples force extra critical points in wider networks","Critical set expands in wider nets once sample size is large enough"],"cache_read_input_tokens":25088,"weakest_assumption_plain":"The proofs rely on the sample inputs being in generic position: none is zero and no two are equal or opposite ($x_i\\pm x_j\\neq 0$); if a dataset contains a zero input or a pair like $x$ and $-x$, the neuron functions can become linearly dependent and the existence results for sample-dependent saddles are not proved.","fun_headline_variants_meta":{"raw":{"variants":["Data size threshold creates extra critical points in wider nets","Wider nets get sample-dependent saddles once data passes a bound","Lifting operator reveals sample-dependent saddles for large data","Bigger samples force extra critical points in wider networks","Critical set expands in wider nets once sample size is large enough"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001189,"raw_usage":{"total_tokens":4854,"prompt_tokens":838,"completion_tokens":4016,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":3934}},"tokens_in":454,"tokens_out":4016,"duration_ms":25692,"temperature":1.0,"reasoning_tokens":3934,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:17:09.624428+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fix a one-hidden-layer tanh network with $m=1$, input dimension $d=1$, and four sample inputs containing a zero, say $x_1=0$ plus three generic nonzero inputs, so the threshold $n>3$ is met but the generic-position condition fails. Vary the outputs to search for parameters of the form $(a_1,w_1,0,w')$ that are critical for one output vector but not for another; if none exist, the generic-position assumption is essential to Theorem 4.2.1.","supporting_citations":[],"review_version":1}