{"id":"3d498fa4-b7f7-4240-906c-c3d18655d5f6","arxiv_id":"2507.14177","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A two-layer network with smooth activations can be read as a smooth spline whose knots come from hidden unit thresholds, and this mechanism yields universal approximation in any dimension.","lead":"This paper proposes a constructive theory for how two-layer neural networks with smooth activation functions such as sigmoid fit data: they build smooth splines whose knots correspond to hidden unit thresholds. The authors claim this explains what trained networks are doing internally and proves universal approximation in any input dimension.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 3's divergent-coefficient step holds only on a shrinking interval; replacing x'_k by the fixed knot x_k makes Theorem 5 false for a fixed spline.","rationale":"The reader's REJECT verdict is supported, and the weakest assumption is indeed Lemma 3. However, the reader's stated concern is slightly different: it emphasizes the inference of polynomial identities from approximate equality and the divergent limit in equation 3.45. The sharper problem is that the divergent monomial approximation is proved only on an interval whose length tends to zero, and the proof then transfers it to a fixed neighboring interval. This makes Theorem 5 invalid as stated for a fixed spline, and it is the exact mechanism by which every local unit in Theorems 6, 7, 13, 16, 17, and 18 is supposed to realize a truncated power at a fixed knot. The experimental section does not repair this formal gap: the thresholds γ1, γ2, γ3 are manually chosen, and no error analysis is provided to show that trained units converge to the constructed spline solutions. The paper does cite and address prior universal approximation results, but those results prove existence of approximation, not the specific spline-knot mechanism claimed here. Thus the central constructive explanation of training solutions is not established by the arguments as written, and the REJECT verdict should stand unless the proof of Lemma 3 is corrected or Theorem 5 is weakened with an explicit shrinking-interval condition.","tokens_in":45327,"tokens_out":13044,"duration_ms":153254,"concrete_test":"Test Theorem 5 with σ(z)=1/(1+e^{-z}), m=2, x_k=0.5, x_{k+1}=0.6, S(x)=0 on [0,0.5], and S(x)=(x-0.5)^2 on (0.5,0.6]. For a sequence ρ_j → ∞, construct φ_{a,j} exactly as in Lemma 3 with a fixed ε' < ε, set c_j by the proof's endpoint rule (equation 3.44), and set λ_j = 1/c_j. Evaluate E_j^2 = ∫_0^{0.6} (λ_j φ_{a,j}(x) - S(x))^2 dx. Since λ_j φ_{a,j} → 0 pointwise on (0.5,0.6] and is uniformly bounded, dominated convergence gives E_j^2 → ∫_0^{0.1} u^4 du = 10^{-5}/5 > 0, contradicting Theorem 5's arbitrary-precision claim. As a stronger check, even with λ_j optimally chosen, the limiting step functions have L2 distance at least sqrt(4/45) from u^2 on the normalized interval.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is in Lemma 3 (Section 3.3), specifically equations 3.42-3.45 and the final paragraph. The proof establishes s'_k(x) ≈ c'_{k-1}(x-x'_k)^m_+ on the short interval (x'_k, x_k], whose length h = x_k - x'_k tends to 0 as ρ(γ) → +∞, and derives c'_{k-1} → +∞ from the fixed endpoint gap ε - ε'. It then deletes the knot x'_k, sets c_k = c'_{k-1}, and claims s_{k+1}(x) ≈ c_k(x-x_k)^m_+ on the fixed interval (x_k, x_{k+1}]. This replacement is not justified: a monomial approximation on a shrinking interval does not transfer to a neighboring fixed interval. For the logistic sigmoid, after the translation used in Lemma 3, φ_a(x_k) = ε' < ε but for every fixed x > x_k, φ_a(x) → 1 as ρ → ∞. With the output weight λ = d_k/c_k set in Theorem 5, one gets λφ_a(x) → 0 pointwise on (x_k, x_{k+1}], not d_k(x-x_k)^m. Consequently the L2 error in Theorem 5 does not tend to 0 for a fixed S(x) = d_k(x-x_k)^m_+ unless x_{k+1}-x_k also shrinks, which Theorem 5 does not assume. Since Theorem 6, and through it Theorems 7, 13, 16, 17, and 18, all attach local units to fixed knots using Theorem 5, the central spline-implementation and universal-approximation claims collapse if this gap cannot be repaired.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a constructive mechanism for two-layer networks with smooth activations, claiming that trained hidden units correspond to knots and polynomial pieces of a smooth spline. Section 2 builds local Taylor-series approximations using generalized Wronskian matrices; Section 3 introduces a 'zero-part error' lemma and uses it to implement univariate splines, culminating in a universal-approximation theorem for C^m functions; Sections 5-6 extend the construction to multivariate input via multivariate Wronskians and a smooth-continuity restriction; Section 7 gives qualitative experimental illustrations. The central claims are Theorem 18 (multivariate universal approximation) and the purported explanation of backpropagation-trained solutions.","tokens_in":45796,"tokens_out":4183,"duration_ms":48712,"significance":"If the construction were valid, it would give a deterministic, spline-based interpretation of two-layer sigmoidal networks, connecting local and global units to spline knots and providing a concrete mechanism for universal approximation. The local Wronskian-based construction is a plausible and interesting contribution, and the paper makes falsifiable experimental predictions about zero-error points and constant-producing tanh units. However, the global spline implementation rests on Lemma 3, whose proof has a load-bearing gap; without that lemma, Theorems 5-7 and 16-18 are not established. The experimental verification is suggestive but heuristic, relying on ad-hoc thresholds and not a quantitative validation of the theorems.","major_comments":[{"comment":"The proof obtains c'_k-1 -> +infinity from matching phi_t(x'_k)=epsilon' and phi_t(x_k)=epsilon on the interval (x'_k, x_k], whose length h=x_k-x'_k tends to 0 by eq. (3.41). It then deletes the knot x'_k, sets c_k=c'_k-1, and claims (3.19) on the fixed interval (x_k, x_{k+1}]. This transfer is not justified: a monomial approximation on a shrinking interval does not extend to a neighboring fixed interval. For the logistic sigmoid, the translated unit satisfies phi_a(x_k)=epsilon', but for every fixed x>x_k one has phi_a(x)->1 as rho->infinity; with output weight lambda=d_k/c_k set as in Theorem 5, lambda phi_a(x)->0 pointwise on (x_k, x_{k+1}], not d_k(x-x_k)^m. Hence the L2 error in Theorem 5 does not tend to 0 for the fixed spline S(x)=d_k(x-x_k)^m_+ unless the mesh interval also shrinks, which Theorem 5 does not assume. Since Theorem 6, and through it Theorems 7, 13, 16, 17, and 18, attach local units to fixed knots using Theorem 5, the central spline-implementation and universal-approximation claims collapse if this gap cannot be repaired.","section":"Section 3.3, Lemma 3, eqs. (3.42)-(3.45)"},{"comment":"The proof infers c_kappa approx 0 from s_{kappa+1}(x) approx 0 on intervals and from 'regarding' two polynomials as equal. Approximate equality on infinitely many points gives only a small L2 or pointwise difference; it does not imply exact identity of polynomial coefficients. In particular, eq. (3.31) cannot justify the conclusion s_1(x) approx -c_1(x-x_1)^m on [0,x_1] and hence c_1 approx 0. Even if eqs. (3.31)-(3.33) hold in the limit, the argument controls no rates and therefore cannot yield the monotone divergence of c_k claimed in part II of Lemma 3. This is the same load-bearing step as the previous major comment.","section":"Section 3.3, eqs. (3.30)-(3.34)"},{"comment":"Theorem 17 is proved only by analogy with Huang (2024)'s lemma 10, and Theorem 15's proof invokes a 'boundary-determination principle' from the same unpublished reference. Since Lemma 11 reduces the multivariate zero-part construction to the univariate Lemma 3, the multivariate construction inherits the error identified above. Moreover, the manuscript does not state or prove the required boundary-determination principle for smooth splines; it only refers to a proof in Huang (2024). Consequently Theorem 18 is not independently verifiable from the material provided in this paper. The explanatory claims about training solutions in Section 7 are also not derived from the theorems: the local/global classification uses thresholds gamma_1,...,gamma_4 and heuristic criteria with no sensitivity analysis or quantitative error control connecting gradient-descent solutions to the constructed spline solutions.","section":"Sections 6 and 7, Theorems 15-18"}],"minor_comments":[{"comment":"There are typos such as 'cure' for 'curve' in the text describing Figure 1a, and 'logistical' for 'logistic' in Section 7. The notation l0 and l+ is used in Sections 6-7 without a formal definition in this paper; it should be defined locally rather than only by reference to Huang (2024).","section":"Section 3.3, Lemma 3"},{"comment":"The sentence 'then equation 2.24 becomes' appears to refer to equation 2.18, and the displayed determinant in eq. (2.22) has an extra parenthesis. These are minor presentation issues, but they make the construction harder to follow.","section":"Section 2.3, Lemma 2"},{"comment":"The thresholds gamma_1,...,gamma_4 are free parameters of the experimental classification. The paper does not report sensitivity to these thresholds, and the discrete criteria in eqs. (7.3)-(7.14) are not shown to converge to the theoretical definitions in Definitions 5 and 15. A brief discussion of this gap would strengthen the experimental section.","section":"Section 7.1"}],"recommendation":"reject","confidential_remarks":"The manuscript's central constructive claim is not established because Lemma 3 transfers a divergent-coefficient approximation from a shrinking interval to a fixed interval without justification, and the same step is used in the multivariate theorems. This is a load-bearing error, not a presentation issue. The local Wronskian construction and the overall spline interpretation are interesting, and I would encourage a resubmission if the fixed-interval implementation can be repaired; however, within the current scope the theorem chain does not hold."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: the paper's core constructive claim — that a single sigmoidal unit can be squeezed into a truncated power on a fixed interval — does not go through as written. The stress-test note is right. In Lemma 3 the coefficient c'_{k-1} diverges on the shrinking interval (x'_k, x_k]; the proof then relabels the knot and asserts the same monomial on the fixed interval (x_k, x_{k+1}]. But the translated sigmoid either tends to 0 or to 1 on that fixed interval, depending on where the interval sits relative to the inflection point; it does not tend to (x-x_k)^m_+. Theorem 5, and everything that leans on it (Theorems 6, 7, 13, 16, 17, 18), collapses. The earlier coefficient-cancellation step (equations 3.30–3.34) is also hand-wavy: approximate equality on an interval doesn't imply the polynomials are coefficient-wise close, especially when the limit involves growing weights.\n\nWhat's genuinely good: the paper asks a real question — what do trained two-layer smooth networks actually compute? The proposed answer, that trained units act as spline knots and polynomial pieces, is concrete and testable. Section 2's local Taylor/Wronskian construction is clearly presented and honestly acknowledges prior work (Xu & Cao 2005). The multivariate generalized-Wronskian ordering in Lemma 6 is a neat combinatorial device. The smooth-continuity restriction and boundary-determination analogue to ReLU networks is a plausible idea worth exploring, and the experimental plots do show that trained units often look like local/global units in the paper's sense.\n\nBut the load-bearing Lemma 3 is not a minor gap; it is the hinge of the entire universal-approximation mechanism. The multivariate Theorem 17 also depends on unproved boundary-determination results from Huang 2024, and the experimental verification uses manually tuned thresholds gamma1–gamma4 with no error analysis or baselines. Those are real but secondary concerns; the Lemma 3 problem is primary.\n\nWho is this for? Researchers working on interpretability of sigmoidal networks or constructive approximation. They would find the framework suggestive, but they should not rely on the theorems as stated. With a repaired Lemma 3 (if one can be found) the paper could be a solid contribution; as it stands, the central claims are unsupported.\n\nRecommendation: send it to peer review anyway. A serious referee could either find a fix for Lemma 3 or force a much weaker, honest statement (e.g., approximation on intervals whose length shrinks with the scale parameter). Desk-rejecting would bury a genuinely interesting research program over what may be a repairable error.","headline":"The spline interpretation is attractive and the question is right, but Lemma 3's divergent-coefficient step doesn't transfer from a shrinking interval to a fixed one, so the main approximation theorems are unsupported as written.","tokens_in":46223,"tokens_out":3386,"would_cite":false,"duration_ms":37932,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["41A15","41A30","41A63","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Two-layer sigmoid networks can be read as smooth spline constructions, with hidden units marking the knots and output weights carrying the polynomial coefficients.","keywords":["two-layer neural networks","smooth activation functions","universal approximation","smooth splines","backpropagation explanation","generalized sigmoidal units","zero-error points","smooth-continuity restriction"],"falsifier":"Train a two-layer sigmoidal network on $f(x)=x^3+3$ over $[0,1]$ until the $L^2$ error is below $10^{-4}$, compute each unit's zero-error point by truncation, and fit the best smooth spline of order 3 with those knots to the network output; if the best such spline cannot reproduce the output to within the training error, the Lemma 3 mechanism is not what training found.","tokens_in":45123,"feed_emoji":"🧠","tokens_out":6156,"duration_ms":68741,"temperature":0.7,"pith_summary":"This paper claims that the 'black box' of a two-layer network with smooth sigmoidal or tanh hidden units is, at its core, a smooth-spline mechanism. The author proves that any sufficiently smooth function on the unit cube can be approximated to arbitrary accuracy by such a network, by first approximating the function with a multivariate smooth spline and then realizing each polynomial piece with hidden units whose weights and biases mark the spline's knots. In the resulting picture, the meaning of trained parameters is concrete: a weight vector and bias place a knot hyperplane, and the corresponding output weight sets the coefficient of the truncated-power term that joins two adjacent polynomial pieces. The paper further claims that backpropagation-trained solutions exhibit exactly this structure and verifies the claim on one- and two-dimensional examples. If correct, the result turns a heuristic fitting method into a deterministic, interpretable construction.","feed_headline":"Two-layer sigmoid solutions are smooth splines","feed_subtitle":"Backprop-trained sigmoid nets place knots in hidden units; a new proof reveals what the black box actually computes.","key_machinery":"The load-bearing mechanism is the truncated-power realization of a generalized sigmoidal unit (Lemma 3 in the univariate case, Lemma 11 in the multivariate case). By scaling the weight by $\\rho > 1$ and adjusting the bias, the unit's activation becomes negligible on the 'zero part' before a knot while on the next region it behaves like $c(w^T x+b)^m_+$, with $c$ growing to infinity; this supplies the recurrence $s_\\nu = s_{\\nu-1} + c_\\nu (w^T x+b)^m_+$ that connects spline pieces. Around it stand the generalized Wronskian matrix, which fixes the coefficients of the Taylor polynomial produced by global units, and the smooth-continuity restriction (Theorem 15), which propagates realized boundary polynomials into interior regions.","core_discovery":"The central result, Theorem 18, states that for any $f \\in C^m([0,1]^n)$, a two-layer network with generalized sigmoidal units can approximate $f$ with arbitrary precision by implementing a multivariate smooth spline $S(x) \\in S^m_n(H,R)$; the number of hidden units required is at least $(\\zeta^{1/n}-1)^n + \\binom{n+m}{m}$, where $\\zeta$ is the number of polynomial pieces in the spline. The construction has four parts: Taylor-series local approximation via a generalized Wronskian matrix; a strict partial order of knot hyperplanes that lets pieces be added one at a time; spline implementation in which a single unit's scaled, shifted activation mimics the truncated power $c(w^T x+b)^m_+$ on one side of a knot while vanishing on the other; and the smooth-continuity restriction, which says that in dimension $n \\ge 2$, once the boundary polynomials around a region are realized, the interior polynomial is automatically produced. The author interprets this as the explanation of training solutions: hidden units are either global units building the initial polynomial or local units carrying spline knots, with output weights carrying the spline coefficients.","pith_inferences":["If the mechanism is right, the same spline-and-knot reading should hold layer by layer in deep sigmoidal networks, with each layer implementing a hierarchical spline composition.","The zero-part scaling predicts a quantitative signature: in a trained network, each local unit's effective support should become more one-sided as training error decreases; this could be measured by truncation tests on held-out functions.","The theory suggests that the activation function's precise shape matters little beyond being smooth, monotone on the left tail, and tending to zero, so other saturating functions with similar tails should also produce spline-like solutions.","The smooth-continuity restriction implies a kind of boundary-value problem for network outputs, so some interior polynomial coefficients may be forced by boundary data; checking this in training could identify which parts of the solution are genuinely free."],"forward_implications":["Any $f \\in C^m([0,1]^n)$ is approximable by a two-layer sigmoidal network with the explicit unit count of Theorem 18.","Trained hidden units are interpretable: local units' weights and biases mark spline knots, and output weights are spline coefficients.","The backpropagation solution can in principle be constructed deterministically rather than found by gradient descent.","Negative units can be added freely at knots (two-sided bases) without changing the approximation, matching common experimental patterns.","For tanh units the same theory holds with one additional hidden unit, which experiments show is used to produce the constant term."],"supporting_citations":[{"why":"Establishes universal approximation by sigmoidal units, the baseline that this paper's spline construction extends.","marker":"(Cybenko, 1989)"},{"why":"Provides the general universal approximation result that the paper contrasts with its constructive spline explanation.","marker":"(Hornik, Stinchcombe, & White, 1989)"},{"why":"Characterizes non-polynomial activation functions for universal approximation, supporting the generalized-sigmoid setting.","marker":"(Leshno et al., 1993)"},{"why":"Supplies the spline theory and the generalized Wronskian terminology on which the Taylor and spline constructions rely.","marker":"(Schumaker, 2007)"},{"why":"Supplies the smooth multivariate spline piece relation that corresponds to the truncated-power recurrence used here.","marker":"(Chui & Wang, 1983)"},{"why":"The predecessor two-layer ReLU theory whose standard partitions, zero-error hyperplanes, negative units, and boundary-determination principles are adapted to smooth activations.","marker":"(Huang, 2024)"},{"why":"Defines the backpropagation algorithm that produces the training solutions the paper aims to explain.","marker":"(Rumelhart, Hinton, & Williams, 1986)"}],"fun_headline_variants":["Sigmoid nets turn out to be smooth spline machines","Backprop-trained sigmoid nets reveal hidden spline structure","Two-layer sigmoid nets implement smooth splines exactly","New proof: sigmoid networks are spline-based approximators","Hidden units in sigmoid nets are precisely spline knots"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on Lemma 3: a single sigmoidal unit can be pushed so that its value on the zero side of a knot tends to zero while the coefficient of the truncated power on the other side tends to infinity, and the limiting identities derived from approximate equality on subintervals are valid.","fun_headline_variants_meta":{"raw":{"variants":["Sigmoid nets turn out to be smooth spline machines","Backprop-trained sigmoid nets reveal hidden spline structure","Two-layer sigmoid nets implement smooth splines exactly","New proof: sigmoid networks are spline-based approximators","Hidden units in sigmoid nets are precisely spline knots"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000758,"raw_usage":{"total_tokens":3346,"prompt_tokens":898,"completion_tokens":2448,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":2366}},"tokens_in":514,"tokens_out":2448,"duration_ms":18351,"temperature":1.0,"reasoning_tokens":2366,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:23:19.655384+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a two-layer sigmoidal network on $f(x)=x^3+3$ over $[0,1]$ until the $L^2$ error is below $10^{-4}$, compute each unit's zero-error point by truncation, and fit the best smooth spline of order 3 with those knots to the network output; if the best such spline cannot reproduce the output to within the training error, the Lemma 3 mechanism is not what training found.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes universal approximation by sigmoidal units, the baseline that this paper's spline construction extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the general universal approximation result that the paper contrasts with its constructive spline explanation."},{"cited_title":"Y., Pinkus, A., & Schocken, S","cited_arxiv_id":null,"evidence_quote":"Characterizes non-polynomial activation functions for universal approximation, supporting the generalized-sigmoid setting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the spline theory and the generalized Wronskian terminology on which the Taylor and spline constructions rely."},{"cited_title":"K., & Wang, R","cited_arxiv_id":null,"evidence_quote":"Supplies the smooth multivariate spline piece relation that corresponds to the truncated-power recurrence used here."}],"review_version":1}