{"id":"b0b857a3-b87a-4cef-b3d6-c31e4bfdd7e8","arxiv_id":"2608.11479","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Gradient descent on any feedforward network with Lipschitz-smooth, linearly bounded activations drives the minimum squared gradient norm to zero at rate O(1/T^(1/L)) without boundedness or overparameterization assumptions.","lead":"Gradient descent on deep neural networks is proved to reach a point where the gradient is small, at rate O(1/T^(1/L)) for an L-layer network, assuming only smooth activations and a smooth loss such as squared error. The result removes the usual requirements of overparameterization, special initialization, or bounded iterates, so it is a broadly applicable theoretical landmark for optimization in deep learning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 4.7's descent condition (17) only holds for cJ≤2; for losses with cJ>2, the stated learning rate (12) is not proven to descend, so Theorem 4.8 overstates its scope.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing gap: inequality (17) in Lemma 4.7 is only valid for cJ≤2, while Assumption 3.4 and Theorem 4.8 allow arbitrary cJ≥1. This is a genuine correctness gap in the stated theorem, because the descent step can fail for large initial loss when cJ>2. However, the gap is localized and repairable: the learning rate ρ can be rescaled by √(cJ/2), or the assumption can be restricted to cJ≤2, which still covers the MSE example emphasized in the paper. The recursive double-polynomial-smoothness framework, the norm-growth induction in Theorem 4.8, and the final rate all appear consistent once the descent condition is repaired. I did not find a more serious flaw: the algebraic bounds in Lemmas 4.3–4.6 are plausible, the norm-growth induction closes, and the assumptions on activations are satisfied by the listed examples. Thus the appropriate outcome is the reader's conditional acceptance, unchanged by this pass.","tokens_in":24843,"tokens_out":19343,"duration_ms":149358,"concrete_test":"Analytic check: in Appendix A.7, evaluate the uniform bound in (17) for the schedule (12). Compute sup_{L≥0} √(2 cJ L)/(2 C (1+√L)) = √(2 cJ)/(2C). For cJ=4, this is 1/(√2 C) > 1/C, so the premise of the descent lemma fails for the admissible loss J_i(u)=2(u-y_i)^2 when L(W0) is large. This directly settles whether the cJ>2 restriction is needed. A complementary re-derivation of Lemma 4.7 with ρ multiplied by √(cJ/2) would confirm the theorem can be repaired without changing the rate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The proof of Lemma 4.7 (Appendix A.7) requires η_t∥∇L(W_t)∥ ≤ 1/C to control the higher-order terms k≥1. Combining the paper's bound (14) with (16), for the learning rate in (12) this product is at most √(2 cJ L(W_t)) / (2 C (1 + √L(W_t))). Since f(L)=√L/(1+√L) is increasing with sup_{L≥0} f(L)=1, the uniform sup is √(2 cJ)/(2C), which is ≤ 1/C only when cJ≤2. Assumption 3.4 only states cJ≥1, and L(W0) can be made large enough that f(L(W0))≈1, so the descent inequality fails for any cJ>2 (e.g., J_i(u)= (cJ/2)(u-y_i)^2). The proof of Theorem 4.8 therefore does not establish convergence for the full range of losses allowed by Assumption 3.4. This gap is repairable: multiply ρ in (12) by √(cJ/2), or restrict Assumption 3.4 to cJ≤2 (MSE has cJ=2). Since the theorem's hidden constant may depend polynomially on cJ, the repair preserves the O(1/T^{1/L}) rate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a theoretical analysis of vanilla gradient descent on feedforward neural networks of arbitrary width and depth. Under assumptions that the activation is Lipschitz continuous, Lipschitz smooth, and linearly bounded (satisfied by tanh, sigmoid, softplus, and linear activations) and that the per-sample loss is nonnegative and Lipschitz smooth in the model output (satisfied by mean-squared error), the authors derive a generalized smoothness condition they call double polynomial smoothness. This condition bounds the change in the loss gradient by the parameter change multiplied by polynomial terms in the parameter norms at both endpoints. Using this condition, they prove a descent lemma (Lemma 4.7) and then bound the growth of the parameter norms to show that the minimum squared gradient norm converges to zero at rate O(1/T^{1/L}) for an L-layer network (Theorem 4.8). The claimed contribution is that convergence to a stationary point is established without overparameterization, special initialization, bounded iterates, or restrictive datasets, and the analysis covers the feature-learning regime.","tokens_in":25170,"tokens_out":11629,"duration_ms":95121,"significance":"If the main theorem is correct, this is a significant theoretical contribution: it provides the first convergence guarantee for vanilla gradient descent on general feedforward networks that does not rely on overparameterization, special initialization, or staying in a bounded region, and it explicitly covers the feature-learning regime where iterates move far from initialization. The proof strategy is self-contained and does not fit constants or assume the conclusion; the recursive Lipschitz bounds (Lemmas 4.3-4.6) are carefully derived and give explicit polynomial dependence on the network width and exponential dependence on depth. The framework of double polynomial smoothness is a new and plausible way to characterize neural-network loss landscapes. However, the stated scope of Theorem 4.8 is not fully justified because the descent lemma is only proven for losses with cJ ≤ 2, while Assumption 3.4 allows any cJ ≥ 1; this gap is repairable but currently blocks the main claim as stated.","major_comments":[{"comment":"The descent proof of Lemma 4.7 requires the bound η_t ∥∇L(W_t)∥ ≤ 1/C to control the higher-order terms in the descent inequality. Combining the authors' bound on ∥∇L(W)∥ (which follows from (14) and (16)) with the learning rate (12), the best uniform bound on this product is √(2c_J)/(2C) · sup_{L≥0} √L/(1+√L) = √(2c_J)/(2C). This is at most 1/C only when c_J ≤ 2. Since Assumption 3.4 only states c_J ≥ 1, the learning rate (12) is not proven to produce a descent step for losses with c_J > 2, e.g., J_i(u) = (c_J/2)(u−y_i)^2 with c_J = 3. Consequently, Theorem 4.8, which is stated for all losses satisfying Assumption 3.4, is not established for its full scope. This is repairable by multiplying ρ in (12) by a factor of √(c_J/2) for c_J > 2, or by restricting Assumption 3.4 to c_J ≤ 2 (which still covers mean-squared error with c_J = 2). The proof and the theorem statement must be adjusted accordingly.","section":"Appendix A.7, inequality (17)"}],"minor_comments":[{"comment":"The statement of Lemma 4.1 has the second term as |F(x_i;W_1) − y_i|, but the proof in Appendix A.1 actually bounds this term by |J'_i(F(x_i;W_1))|. These are not equal for general losses satisfying Assumption 3.4; the proof's version is the correct one. Please correct the lemma statement to use |J'_i(F(x_i;W_1))|, or add a remark explaining the discrepancy (e.g., for mean-squared error, |J'| = 2|F−y|).","section":"Lemma 4.1"},{"comment":"The display for the γ = 2 case is garbled in the typesetting; the intended bound is η_t^2 ∥∇L∥^2 · C/2 · (1+L^{1/2}) Σ ∥W_t∥^i ≤ η_t · (C/(2ρ)) ∥∇L∥^2 ≤ η_t/(2L) ∥∇L∥^2. Please rewrite this step for clarity.","section":"Proof of Lemma 4.7"},{"comment":"The notation min_{t=0,...,T−1} ∥∇L(W_t)∥^2 is slightly ambiguous; please use min_{0≤t≤T−1} or an equivalent notation.","section":"Theorem 4.8"},{"comment":"The phrase 'fully characterizes feedforward neural networks' is too strong; the paper establishes an upper bound on the gradient norm under specific smoothness assumptions, not a full characterization of the loss landscape. Consider softening this wording.","section":"Introduction"},{"comment":"The assumption c_J ≥ 1 is stated 'for convenience', but the proof of Lemma 4.5 uses the inequality |J'(u)|^2 ≤ 2 c_J J(u) from [16, Lemma 2.28], which holds for any c_J > 0. Either remove the lower bound or note that the results extend to c_J > 0 with minor constant changes.","section":"Assumption 3.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of cs.LG and the related work is thorough. The main technical apparatus—the recursive double polynomial smoothness bounds—is sound in its derivation, and the proof does not appear circular. The single load-bearing issue is the cJ ≤ 2 gap in Lemma 4.7; because it is fixable by a simple rescaling of the learning rate or a restriction of Assumption 3.4, I do not see it as grounds for rejection. If the authors repair this gap and correct the statement of Lemma 4.1, the paper would be a solid contribution. I would also encourage the authors to double-check the typesetting of several displays in Appendix A.7 and A.8, which are currently difficult to parse."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"One thing to know: this is a serious theory paper with a real but narrow bug. The double polynomial smoothness idea is new, and the composition proof is the genuine contribution. The main convergence theorem is likely correct after a small constant fix.\n\nThe good parts. The paper proves an O(1/T^{1/L}) convergence rate to a stationary point for vanilla GD on L-layer feedforward networks with smooth activations (tanh, sigmoid, softplus, linear), without overparameterization, special initialization, or bounded iterates. The framework is self-contained: every Lipschitz bound is proven from Assumption 3.1, and the loop closes with an explicit growth bound on the parameter norms. No fitted constants, no black-box citations (the LoRA paper [23] is used as a proof skeleton, not as a theorem source). This is a real step beyond the line of (L0,L1)-smoothness and relative smoothness results, which only covered two-layer or linear/balanced settings.\n\nThe soft spot. Lemma 4.7's descent proof requires the bound η_t ||∇L(W_t)|| ≤ 1/C in inequality (17). Combining the paper's own bound (14) and (16), this product is at most sqrt(cJ/2)/C, uniformly. That is ≤ 1/C only when cJ≤2. Assumption 3.4 only states cJ≥1, so as written the learning rate (12) is not proven to descend for losses with cJ>2 (e.g., J_i(u)=(cJ/2)(u-y_i)^2). This is a genuine gap in Theorem 4.8 as stated. It is easily repairable: multiply ρ in (12) by sqrt(cJ/2), or restrict Assumption 3.4 to cJ≤2, which already covers MSE with equality. The O(1/T^{1/L}) rate is preserved since the hidden constant may depend polynomially on cJ. So the main idea holds; the paper simply overstates its scope by a constant.\n\nMinor notes: The proof of Theorem 4.8 has a lot of moving parts in the p-series argument; I didn't audit every summation index, but the structure is plausible. The rate is slow in L, but that's consistent with the difficulty. The discussion section is honest about what remains (ReLU, lower bounds, width dependence).\n\nWho it's for: researchers in nonconvex optimization and deep learning theory. It deserves a serious referee. I'd send it to peer review with a request to fix the cJ condition. After that fix, it's a solid, citable contribution.","headline":"Genuinely new global smoothness framework for GD on deep nets, but the descent lemma as stated only works for cJ≤2; the fix is simple.","tokens_in":25643,"tokens_out":5686,"would_cite":true,"duration_ms":48961,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","90C26","65K05"],"pacs":[],"model":"deepseek-v4-flash","headline":"Gradient descent provably reaches a stationary point on feedforward neural networks of any width and depth, under only mild smoothness assumptions on the activation and loss functions.","keywords":["gradient descent","convergence guarantees","feedforward neural networks","generalized Lipschitz smoothness","double polynomial smoothness","nonconvex optimization","stationary point","parameter norm growth"],"falsifier":"To test the descent lemma, choose a two-layer network with a linear activation and a loss J(u) = c $u^{2}$ with c > 2, initialize at a point with L(W0) large, and compute η_0 ||∇L(W0)|| under the learning rate (12); if that product exceeds 1/C, inequality (17) fails and the loss can increase at the first step, contradicting the claimed descent for all c_J ≥ 1.","tokens_in":24644,"feed_emoji":"📉","tokens_out":13167,"duration_ms":93912,"temperature":0.7,"pith_summary":"This paper proves convergence guarantees for vanilla gradient descent on feedforward neural networks of arbitrary width and depth, with no requirements on initialization, dataset, or bounded iterates. The assumptions are that the activation function is Lipschitz smooth, Lipschitz continuous, and linearly bounded (true for linear, tanh, softplus, and sigmoid), and that the loss is Lipschitz smooth in the model outputs (true for mean-squared error). The key new tool is a 'double polynomial smoothness' condition: the difference in loss gradients at two parameter points is bounded by the parameter difference times polynomials of the norms of both points. From this, the authors prove a descent lemma and control the sublinear growth of parameter norms, yielding a convergence rate of O(1/$T^{{1/L}}$) for the minimum squared gradient norm after T steps.","feed_headline":"Converges: gradient descent finds stationary points in deep nets","feed_subtitle":"No width or initialization tricks: plain gradient descent provably finds a zero-gradient point on any smooth network.","key_machinery":"The central object is the double polynomial smoothness condition and the descent lemma built from it. For any two points W1 and W2, Lemma 4.5 bounds the gradient difference ||∇L(W2)-∇L(W1)|| by 8 c_J d_max $L^{3}$ ||W2-W1|| (1+L(W1)^{1/2}) times the product of polynomial sums in ||W1|| and ||W2||. Lemma 4.6 integrates this bound along the segment and turns the polynomial in ||W2|| into higher powers of ||W2-W1||, so that a single gradient descent step with learning rate (12) decreases the loss by at least η_t/(2L) ||∇L(W_t)||^2. The remaining work is showing that the norms ||W_t|| grow only sublinearly, at rate Θ($T^{{(L-1)/L}}$), which makes the cumulative learning rate sum diverge as Θ($T^{{1/L}}$).","core_discovery":"The paper's central claim is Theorem 4.8: after T steps of gradient descent with the step size given by equation (12), the minimum squared gradient norm of an L-layer neural network satisfies min_{t<T} ||∇L(W_t)||^2 = O(1/$T^{{1/L}}$), where the hidden constant is polynomial in c_J, d_max, L(W0), and ||W0|| and exponential in L. The discovery is that the Lipschitz smoothness of the activation functions is partially preserved under repeated composition, giving a global double polynomial smoothness bound for the model function and the loss (Lemmas 4.3–4.5). This replaces the global Lipschitz constant that neural networks lack, and the resulting descent lemma (Lemma 4.7) guarantees a loss decrease at every step provided the learning rate is small relative to the current parameter norm and loss value.","pith_inferences":["Because double polynomial smoothness holds globally for any two points, the descent lemma should adapt to stochastic gradient descent or coordinate descent, with the sublinear norm-growth argument controlling the extra randomness.","The O(1/T^{1/L}) rate suggests a natural lower-bound conjecture: if tight, the depth of a network is an intrinsic hardness parameter for first-order optimization, which would give a theoretical reason to prefer adaptive or second-order methods for very deep architectures.","A direct empirical check of the framework is to run the step-size rule (12) on a small tanh or sigmoid network and verify both the predicted power-law decay of the squared gradient norm and the Θ(T^{(L-1)/L}) growth of parameter norms."],"forward_implications":["For any L-layer network with tanh, sigmoid, softplus, or linear activations and squared-error loss, gradient descent with the stated step size reaches a stationary point without width, initialization, or dataset assumptions.","The squared-gradient-norm convergence rate is O(1/T^{1/L}), so deeper networks converge more slowly; a two-layer network achieves O(1/T^{1/2}) and an L-layer network O(1/T^{1/L}).","The analysis covers the feature-learning regime: parameters may move far from initialization, since the proof does not require iterates to remain in a bounded set.","The result holds for any loss that is c_J-Lipschitz smooth in the model outputs, which includes mean-squared error and other smooth losses under the assumptions of the theorem."],"supporting_citations":[{"why":"Supplies Lemma 2.28 used to bound |J'_i(F(x_i;W))| by (2 c_J J_i(F(x_i;W)))^{1/2} in the proof of Lemma 4.5, which enters the constant of the double polynomial smoothness bound.","marker":"[16]"},{"why":"Provides the convergence-proof structure for LoRA gradient descent that the paper adapts to general feedforward networks, including the recursion for parameter-norm growth.","marker":"[23]"},{"why":"Defines relative smoothness, the closest prior condition, which the paper contrasts with double polynomial smoothness and which applies only to mirror descent on two-layer linear networks.","marker":"[20]"},{"why":"Shows two-layer linear networks satisfy a relative smoothness condition, serving as the nearest precedent that the paper extends to nonlinear networks under vanilla gradient descent.","marker":"[14]"},{"why":"Introduces (L0,L1)-smoothness, the generalized smoothness condition whose line of research this paper continues.","marker":"[36]"},{"why":"Extends (L0,L1)-smoothness to general nondecreasing functions of the gradient norm, establishing the toolkit of generalized smoothness conditions that double polynomial smoothness refines.","marker":"[18]"}],"fun_headline_variants":["Partial smoothness survives layers: GD converges on any network","No init tricks: plain GD converges to stationary points in deep nets","New proof: gradient descent always finds stationary points","Layered Lipschitz: why GD works on any neural network","Smoothness under composition: GD provably converges on nets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof of the descent step requires the loss's smoothness constant c_J to be no larger than 2 for the stated learning-rate schedule to guarantee a decrease, whereas the theorem claims convergence for all c_J ≥ 1; this gap is the load-bearing assumption.","fun_headline_variants_meta":{"raw":{"variants":["Partial smoothness survives layers: GD converges on any network","No init tricks: plain GD converges to stationary points in deep nets","New proof: gradient descent always finds stationary points","Layered Lipschitz: why GD works on any neural network","Smoothness under composition: GD provably converges on nets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001411,"raw_usage":{"total_tokens":5704,"prompt_tokens":950,"completion_tokens":4754,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":4670}},"tokens_in":566,"tokens_out":4754,"duration_ms":32814,"temperature":1.0,"reasoning_tokens":4670,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:13:54.178413+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"To test the descent lemma, choose a two-layer network with a linear activation and a loss J(u) = c $u^{2}$ with c > 2, initialize at a point with L(W0) large, and compute η_0 ||∇L(W0)|| under the learning rate (12); if that product exceeds 1/C, inequality (17) fails and the loss can increase at the first step, contradicting the claimed descent for all c_J ≥ 1.","supporting_citations":[{"cited_title":"Taming Nonconvex Stochastic Mirror Descent with General Breg- man Divergence","cited_arxiv_id":null,"evidence_quote":"Shows two-layer linear networks satisfy a relative smoothness condition, serving as the nearest precedent that the paper extends to nonlinear networks under vanilla gradient descent."},{"cited_title":"Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity","cited_arxiv_id":null,"evidence_quote":"Introduces (L0,L1)-smoothness, the generalized smoothness condition whose line of research this paper continues."},{"cited_title":"Convex and Non-convex Optimization Under Generalized Smoothness","cited_arxiv_id":null,"evidence_quote":"Extends (L0,L1)-smoothness to general nondecreasing functions of the gradient norm, establishing the toolkit of generalized smoothness conditions that double polynomial smoothness refines."}],"review_version":1}