{"id":"6b18faa2-fa57-4dea-b7c6-821211d249f7","arxiv_id":"1908.05660","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":8.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In overparametrized 2-layer networks, non-smooth activations provably yield large NTK minimum eigenvalues, while smooth activations can have zero or exponentially small eigenvalues on low-dimensional data, predicting slow training.","lead":"This theory paper proves that training speed of overparametrized neural networks is governed by whether the activation function is smooth: activations with a kink such as ReLU give large Gram-matrix eigenvalues and fast convergence, while smooth activations like tanh can have exponentially small eigenvalues on low-dimensional data. The results provide quantitative eigenvalue bounds and explain when ReLU outperforms smooth activations.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ReLU lower bound in Theorem 4.2 appears to use |x_i^T x_j| ≤ 1−δ, while Assumption 2 only gives sqrt(1−δ^2); the stated δ^{1.5} bound is likely unproven.","rationale":"The reader’s weakest assumption concerns the lazy-training/small-learning-rate regime: the paper only proves that small λmin leads to slow training when η ≤ O(n^{−2}λmin(G(0))) and explicitly leaves larger learning rates open. That limitation is real but is disclosed in the paper and is inherited from the broader NTK literature. The ReLU bound issue is more concrete and more damaging: it is an internal inconsistency between Assumption 2 and a step in the proof of Theorem 4.2, not merely an acknowledged scope restriction. If the analysis above is correct, the published δ^{1.5} scaling for the ReLU G-matrix lower bound is unsupported; the proof yields at best δ^3. This affects a headline positive result but does not by itself destroy the paper’s main qualitative dichotomy, since the J1/J2 lower bounds and the smooth-activation upper bounds are separate and the core argument could be repaired. I therefore keep the reader’s CONDITIONAL verdict rather than escalating to REJECT, but the condition should now explicitly include re-deriving and correcting the ReLU exponent and the associated m dependence.","tokens_in":71880,"tokens_out":11288,"duration_ms":114730,"concrete_test":"Re-derive Theorem L.2 from Assumption 2 as written: for unit xi, xj, the hypothesis gives |x_i^T x_j| ≤ sqrt(1−δ^2), not 1−δ. Compute the required Hermite degree r0 ≈ 2 log(2n)/δ^2, then recompute λmin(G∞) ≥ (1/2)·c_{r0}^2 with c_r = Θ(r^{−3/4}) for ReLU′. If the result is Ω(δ^3/log^{3/2} n) rather than Ω((δ/log n)^{1.5}), Theorem 4.2’s exponent and m requirement need correction. Independently, check Lemma J.3’s contraction step using the same corrected correlation bound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is an internal scaling error in the ReLU eigenvalue bound, not the lazy-regime caveat (which the paper openly acknowledges). Assumption 2 states ||(I − x_i x_i^T)x_j|| ≥ δ. For unit vectors this gives |x_i^T x_j| ≤ sqrt(1−δ^2) = 1 − Θ(δ^2), not 1−δ. Appendix L.2 (proof of Theorem L.2) uses |x_i^T x_j| ≤ 1−δ to set r0 = log(2n)/δ and then concludes λmin(G∞) ≥ Ω((δ/log n)^{1.5}) from the r^{−3/2} decay of the squared Hermite coefficients of ReLU′. With the correct separation, r0 = Θ(log n/δ^2), giving only Ω(δ^3/log^{3/2} n), and the neuron requirement m = Ω(n^4δ^{−3}log^4 n) would need strengthening to match the smaller eigenvalue. The same incorrect translation appears in the depth argument: Lemma J.3 and Theorem J.5 assume x_i^T x_j ≤ 1−δ directly from Assumption 2. Thus Theorem 4.2’s advertised “best known” ReLU bound is unproven as stated. The kink-versus-smooth dichotomy may survive with weaker exponents, but the central quantitative claim for ReLU is not supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies training of highly overparametrized two-layer neural networks in the DZPS (Du et al.) random-initialization setting, focusing on the minimum eigenvalue at initialization of the Gradient Gram matrix G(0), which controls the convergence rate of gradient descent in the lazy/perturbative regime. The main results are: for activations with a kink (class Jr, including ReLU, SELU, ELU), all eigenvalues of G(0) are polynomially large under mild data separation; for smooth activations (tanh, swish, polynomial activations), the minimum eigenvalue can be zero or exponentially small when the data span has low dimension; positive results are given for smoothed high-dimensional data and for sufficiently deep networks. The paper contains detailed appendix proofs, applications to training bounds, and experiments on synthetic data and a CIFAR-10 subset.","tokens_in":72247,"tokens_out":7111,"duration_ms":63669,"significance":"If the results hold as stated, the paper would provide the first quantitative eigenvalue bounds distinguishing smooth from nonsmooth activations in the NTK/Gram-matrix analysis of overparametrized networks. The proofs are built from standard tools (Hermite and Chebyshev approximation, Carbery-Wright anticoncentration, Gershgorin's theorem) and no constants are fitted to data; the predicted smooth-vs-kink dichotomy is falsifiable and is qualitatively supported by the included experiments. However, one of the headline quantitative claims, the ReLU lower bound of Theorem 4.2, appears to rest on an incorrect translation of Assumption 2, and the training-speed conclusions are explicitly valid only in the small-learning-rate lazy regime acknowledged by the authors.","major_comments":[{"comment":"The proof of Theorem L.2 states that Assumption 2 implies |x_i^T x_j| ≤ 1−δ, but Assumption 2 gives |x_i^T x_j| ≤ sqrt(1−δ^2) = 1−Θ(δ^2) for unit vectors. Consequently, the threshold r0 should be Θ(log n / δ^2) rather than Θ(log n / δ), and the r^{−3/2} decay of the squared Hermite coefficients of ReLU′ yields λmin(G∞) ≥ Ω(δ^3 / log^{3/2} n), not Ω((δ/log n)^{1.5}). The neuron requirement m = Ω(n^4 δ^{−3} log^4 n) also needs to be re-derived with the corrected eigenvalue. Therefore Theorem 4.2, advertised as the best known ReLU bound, is not proved as stated; the kink-versus-smooth dichotomy may survive with weaker exponents, but the quantitative claim needs revision.","section":"Appendix L, proof of Theorem L.2; Theorem 4.2"},{"comment":"The abstract and introduction phrase the results as \"leading to slow training\" for smooth activations on low-dimensional data, but the formal transfer from eigenvalue bounds to training speed is proved only in the small-learning-rate perturbative regime: Theorem 4.6 requires η ≤ O(n^{−2} λmin(G(0))) and m ≥ Ω(n^5 κ^{−1} λmin(G(0))^{−4} ε^{−2}), and the text explicitly says that analyzing higher learning rates remains open. The claim should be qualified accordingly; as it stands, the eigenvalue bounds alone do not establish slow training outside the lazy regime.","section":"Section 4.2, Theorem 4.6"}],"minor_comments":[{"comment":"The displayed bound is malformed: it reads \"Ω((δ1.5 log−1.5n)\" and should be Ω((δ/log n)^{1.5}) (or the corrected exponent after the scaling issue described in the major comment).","section":"Theorem 4.2"},{"comment":"The informal Hermite-approximation statement for tanh′ is attributed to \"Theorem G.2\", but Theorem G.2 in the appendix is the corresponding statement for swish; the correct reference appears to be Corollary F.7.2.","section":"Section 6, Theorem 6.4"},{"comment":"The notation \"‖u‖2 := ‖u‖2^2 := ∑ u_i^2\" defines the norm as the square of the norm; please use distinct symbols, e.g., ‖u‖_2 for the Euclidean norm and ‖u‖_2^2 for its square.","section":"Section 2, notation"},{"comment":"The caption says \"Semilog plot of sum of squares of top k singular values\" while the text says \"cumulative sums of eigenvalues\"; please make the caption consistent with the quantity plotted.","section":"Section 7, Figure 2(b)"},{"comment":"The expression exp(−Ω(n^{1/2d′})) is ambiguous; it should be written as exp(−Ω(n^{1/(2d′)})) to make clear that the exponent is 1/(2d′).","section":"Theorem 4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong theory submission with an unusually detailed appendix, and the ReLU scaling error is likely localizable and fixable. The lazy-regime caveat is openly acknowledged and should not by itself block publication. If the corrected ReLU exponent turns out to be substantially weaker than advertised, the authors should explicitly restate what remains \"best known\" and adjust the corresponding training-time corollaries."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things before reading: this is the first paper I know that gives quantitative eigenvalue bounds for activations with a kink and explicit negative results for smooth ones, and I believe the headline ReLU bound is not proven as stated.\n\nWhat is actually new: for J2-type activations (ELU) they get a polynomial lower bound on the minimum eigenvalue of the G-matrix; for smooth activations they show that on low-dimensional data (d' = O(log^{0.75} n)) the minimum eigenvalue is exponentially small (or zero for polynomial activations), and that depth can fix this. These are real contributions. The proof machinery—Hermite/Chebyshev approximation, Carbery-Wright, Gershgorin—is used carefully, and the appendix is detailed. The paper also openly discloses a correction in the J2 bound (n^{-8} instead of n^{-7}), which is the right thing to do.\n\nThe soft spot is load-bearing. Assumption 2 is ||(I - x_i x_i^T)x_j|| >= delta. For unit vectors that means |x_i^T x_j| <= sqrt(1-delta^2) = 1-Theta(delta^2), not 1-delta. In the proof of Theorem L.2 (the ReLU bound) they use |x_i^T x_j| <= 1-delta to set r0 = log(2n)/delta, then conclude lambda_min >= Omega((delta/log n)^{1.5}) from the r^{-3/2} decay of the squared Hermite coefficients of ReLU'. With the correct separation, r0 = Theta(log n/delta^2), and the bound weakens to Omega(delta^3/log^{3/2} n). The neuron requirement m = Omega(n^4 delta^{-3} log^4 n) would also need strengthening. The same incorrect translation appears in the depth argument (Lemma J.3, Theorem J.5), where x_i^T x_j <= 1-delta is assumed directly from Assumption 2. So Theorem 4.2's advertised 'best known' ReLU bound is unproven as stated. I don't think the kink-versus-smooth dichotomy is dead: the J2 bound seems to use a different argument, and the negative results for smooth activations don't rely on that translation. But the quantitative ReLU claim needs a correction.\n\nThe lazy-regime caveat for translation to training speed is real but openly acknowledged in Section 4.2; I'd call that a limitation, not a hidden flaw. The illustrative experiments have no error bars, but they are not the main evidence.\n\nBottom line: this is a serious theory paper worth engaging with. It deserves a real referee, and I'd bring it to our reading group. If I cite it, I'll cite the smooth-activation negative results and the J2 bound, not the ReLU bound as stated.","headline":"You should know two things before reading: this is the first paper I know that gives quantitative eigenvalue bounds for activations with a kink and explicit negative results for smooth ones, and I believe the headline ReLU bound is not proven as stated.","tokens_in":72691,"tokens_out":3287,"would_cite":true,"duration_ms":31238,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68Q32","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proves that activation functions with a derivative kink guarantee large minimum eigenvalues of the training Gram matrix, while smooth activations can force it exponentially small on low-dimensional data, and depth restores the…","keywords":["G-matrix","minimum eigenvalue","activation function","overparametrization","smooth vs non-smooth activations","ReLU","tanh","Hermite expansion"],"falsifier":"Compute $\\lambda_{\\min}(G(0))$ at initialization for a two-layer tanh network on $n$ points whose span has dimension $d' = O(\\log^{0.75} n)$: the theorem predicts $\\lambda_{\\min}(G(0)) \\le \\exp(-\\Omega(n^{1/(2d')}))$, so a measurement showing it is instead polynomially large would refute the upper-bound claim.","tokens_in":1972,"feed_emoji":"⚡","tokens_out":1935,"duration_ms":76951,"temperature":0.7,"pith_summary":"The paper asks what makes one activation function train faster than another in overparametrized two-layer networks, where gradient descent provably reaches zero training error. Its answer is that a single quantity, the minimum eigenvalue of the Gradient Gram matrix at initialization, separates activations into two classes. Activations with a derivative kink have this eigenvalue polynomially large under mild separation of the data, which predicts fast convergence. Smooth activations have it zero or exponentially small when the data occupies a low-dimensional subspace, which predicts slow training, while enough depth or data smoothing restores the large-eigenvalue regime.","feed_headline":"Activation kinks make wide nets train fast; smooth ones can stall","feed_subtitle":"The G-matrix's smallest eigenvalue at initialization decides whether two-layer nets train fast or slow.","key_machinery":"The central object is the Gradient Gram matrix at initialization, with entries $g_{ij} = \\frac{1}{m}\\sum_{k=1}^{m} a_k^2 \\varphi'(w_k^{\\mathsf{T}} x_i)\\varphi'(w_k^{\\mathsf{T}} x_j)\\langle x_i, x_j\\rangle$, whose eigenvalue decomposition governs the loss dynamics. The proofs split a Gaussian weight vector into a large part $w'$ and a small part $w''$ to isolate the kink, then use Taylor expansion of $\\sum_i \\zeta_i \\varphi'(w^{\\mathsf{T}} x_i)$ to convert a jump in the $r$-th derivative into a lower bound on the norm over an $\\epsilon$-net. For smooth activations the counterpoint is polynomial approximation: $\\tanh'$ is approximated by Chebyshev or Hermite polynomials whose decaying coefficients give the exponentially small upper bound on $\\lambda_{\\min}(G(0))$, while the lower bounds for smoothed data go through Khatri-Rao powers of the data matrix and anticoncentration of polynomial maps.","core_discovery":"The central claim is that the smoothness of the activation determines the conditioning of the G-matrix at initialization. For activations whose $r$-th derivative has a jump discontinuity, $\\lambda_{\\min}(G(0))$ is large: at least $\\Omega(\\delta^3 n^{-8}(\\log n)^{-1})$ for the $J_2$ class and $\\Omega((\\delta/\\log n)^{1.5})$ for ReLU. For smooth activations such as tanh, if the span of the $n$ data points has dimension $d' = O(\\log^{0.75} n)$, then $\\lambda_{\\min}(G(0)) \\le \\exp(-\\Omega(n^{1/(2d')}))$, and for polynomial activations many eigenvalues are exactly zero when $d' = O(n^{1/p})$. Thus the same architecture trains fast or slow depending on whether the activation has a kink and on the intrinsic dimension of the data, and the paper shows these bounds are tight in a precise sense.","pith_inferences":["One could use $\\lambda_{\\min}(G(0))$ as a cheap diagnostic for predicting whether a given activation will train slowly on a particular dataset, prior to actually running gradient descent.","The results suggest a design principle for new activations: introduce a derivative kink of some order if the target data is low-dimensional, or rely on depth or data smoothing to rescue smooth activations.","Because the training-speed conclusions hold only under learning rates bounded by $O(n^{-2}\\lambda_{\\min}(G(0)))$, an immediate extension is to test whether the gap between kinked and smooth activations persists at practical learning rates; the eigenvalue bounds alone do not decide this.","The Hermite-coefficient perspective links activation smoothness to training speed, implying that activations with slowly decaying Hermite coefficients (like the step function underlying ReLU) should train fast, which could guide automated activation search."],"forward_implications":["For activations in the $J_r$ classes (e.g. ReLU, SELU, ELU), a sufficiently wide two-layer network trained by gradient descent reaches near-zero training error in polynomially many steps, with the step count controlled by $\\lambda_{\\min}(G(0))$.","For smooth activations on low-dimensional data, the G-matrix has many small eigenvalues, so loss decays slowly even in the ideal perturbative training regime.","The low-dimensional limitation of smooth activations disappears with depth: for tanh with depth $L = \\Theta(\\log(1/\\delta))$, $\\lambda_{\\min}(G(0)) \\ge e^{-O(\\sqrt{\\log n})}$, which is much larger than any inverse polynomial in $n$.","For smoothed data whose span has dimension at least $n^{\\gamma}$, tanh regains a polynomial lower bound on $\\lambda_{\\min}(G(0))$, so the earlier slow-training result is not robust to small data perturbation."],"supporting_citations":[{"why":"Supplies the convergence theorem where the step count depends inversely on $\\lambda_{\\min}(G)$, the template for the paper's fast-training implications.","marker":"Du et al. (2019a)"},{"why":"Defines the DZPS setting and the G-matrix, and proves $\\lambda_{\\min}(G_\\infty) > 0$ for analytic activations, the starting point the paper makes quantitative.","marker":"Du et al. (2019b)"},{"why":"Provides the $\\delta$-separation data assumptions and the gradient lower-bound technique that the kink lower bounds adapt.","marker":"Allen-Zhu et al. (2019)"},{"why":"Gives the fine-grained loss decomposition used in Theorem 4.6 to conclude slow training for smooth activations in the small-perturbation regime.","marker":"Arora et al. (2019c)"},{"why":"Establishes that the G-matrix stays close to its initial value as width grows, the lazy-regime premise linking $\\lambda_{\\min}(G(0))$ to convergence speed.","marker":"Jacot et al. (2018)"},{"why":"Provides the Hermite series convergence theorem giving exponential decay of coefficients for $\\tanh'$, the crux of the upper bound for smooth activations.","marker":"Hille (1940)"},{"why":"Shows the exponential decay of $\\tanh'$ Hermite coefficients is asymptotically tight, supporting the claim that the smooth-activation upper bound cannot be substantially improved.","marker":"Boyd (1984)"},{"why":"Supplies the anticoncentration inequality for polynomials used to lower-bound the minimum singular value of Khatri-Rao powers in the smoothed-data positive results.","marker":"Carbery & Wright (2001)"}],"fun_headline_variants":["Smooth activations can stall training; kinked ones speed it up","Data dimension decides if smooth activations train slowly","ReLU's kink is key: non-smooth activations train fast","G-matrix eigenvalue gap separates fast from slow training","Wide nets train fast only if activation has a kink"],"cache_read_input_tokens":74752,"weakest_assumption_plain":"The training-speed conclusions hold only in the small-perturbation (lazy) regime, where weights move little enough that the G-matrix stays close to its value at initialization; the paper guarantees this only when the learning rate satisfies $\\eta \\le O(n^{-2}\\lambda_{\\min}(G(0)))$.","fun_headline_variants_meta":{"raw":{"variants":["Smooth activations can stall training; kinked ones speed it up","Data dimension decides if smooth activations train slowly","ReLU's kink is key: non-smooth activations train fast","G-matrix eigenvalue gap separates fast from slow training","Wide nets train fast only if activation has a kink"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000633,"raw_usage":{"total_tokens":2983,"prompt_tokens":1066,"completion_tokens":1917,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":682,"completion_tokens_details":{"reasoning_tokens":1831}},"tokens_in":682,"tokens_out":1917,"duration_ms":15255,"temperature":1.0,"reasoning_tokens":1831,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:56:57.981062+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute $\\lambda_{\\min}(G(0))$ at initialization for a two-layer tanh network on $n$ points whose span has dimension $d' = O(\\log^{0.75} n)$: the theorem predicts $\\lambda_{\\min}(G(0)) \\le \\exp(-\\Omega(n^{1/(2d')}))$, so a measurement showing it is instead polynomially large would refute the upper-bound claim.","supporting_citations":[{"cited_title":"A convergence theory for deep learning via over-parameterization","cited_arxiv_id":null,"evidence_quote":"Provides the $\\delta$-separation data assumptions and the gradient lower-bound technique that the kink lower bounds adapt."},{"cited_title":"Neural tangent kernel: Convergence and generalization in neural networks","cited_arxiv_id":null,"evidence_quote":"Establishes that the G-matrix stays close to its initial value as width grows, the lazy-regime premise linking $\\lambda_{\\min}(G(0))$ to convergence speed."},{"cited_title":"Contributions to the theory of H ermitian series","cited_arxiv_id":null,"evidence_quote":"Provides the Hermite series convergence theorem giving exponential decay of coefficients for $\\tanh'$, the crux of the upper bound for smooth activations."},{"cited_title":"Asymptotic coefficients of hermite function series","cited_arxiv_id":null,"evidence_quote":"Shows the exponential decay of $\\tanh'$ Hermite coefficients is asymptotically tight, supporting the claim that the smooth-activation upper bound cannot be substantially improved."}],"review_version":1}