{"id":"0e002707-3ede-4a28-81d0-e33e73a6dedc","arxiv_id":"2602.05600","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"SGD noise covariance is claimed to follow the second moment of per-sample Hessians, giving a superlinear power law C_ii ∝ H_ii^γ with 1 ≤ γ ≤ 2.","lead":"This paper claims that the noise from stochastic gradient descent scales with the square of loss curvature, not the curvature itself, producing a power-law relation with an exponent between 1 and 2. If right, it would replace a common approximation used in theories of why SGD prefers flat minima.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.4 rests on Eq. (14) isotropy that contradicts Lemma 3.2: Δw is rank-1, so Mp has rank ≤ d_out and cannot be σ_w²I; Eq. (15) and the 1≤γ≤2 bounds do not follow.","rationale":"The reader correctly identifies Eq. (14) as the linchpin. I traced the derivation: Lemma 3.3 → Eq. (13) → Theorem 3.4. Equation (13) itself is fine as a leading-order approximation, but the move from Eq. (13) to Eq. (15) requires an isotropic, sample-independent M_p. Lemma 3.2's explicit rank-1 solution makes this impossible in a D=2500 layer with d_out=50: the covariance of any random vector living in a 50-dimensional subspace has at most 50 nonzero eigenvalues in every basis. The figures cited as support (Figures 4–6(h), Figures 10–12) show only diagonal magnitudes in the global Hessian basis, not the full local-basis isotropy; a low-rank covariance can look diagonal in a favorable global basis while still violating Eq. (14). Therefore the central C ∝ E[h_p^2] formula and the universal bounds are not established. The empirical data may still show γ≈1–1.5, but that is a separate phenomenological claim. The correct disposition remains rejection of the paper's central theoretical claim; the reader's REJECT is appropriate.","tokens_in":58685,"tokens_out":4206,"duration_ms":46091,"concrete_test":"From a trained checkpoint of the 50×50 MLP layer (D=2500, d_out=50), compute M_p = E_{p'∈B_μ}[vec(ΔW_{p',p}) vec(ΔW_{p',p})^T] for several samples p using Eq. (9), and diagonalize it in each h_p's eigenbasis. If rank(M_p) ≤ 50 and/or the diagonal entries are far from a common σ_w² for all m, Eq. (14) is refuted. Then re-derive C_ii using the actual rank-structured M_p and compare with Eq. (15); if the resulting C_ii vs H_ii log-log slope leaves [1,2] or differs materially, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central relation C ∝ E_p[h_p^2] (abstract; Eqs. 15, 17) and Theorem 5.1's bounds derive from Theorem 3.4, whose assumption (14) is Mp,mn = E[(u_m^T Δw)(Δw^T u_n)] ≈ σ_w²δ_mn in the local per-sample Hessian eigenbasis. But Lemma 3.2 gives ΔW = (WΔa)a^T/||a||^2, a rank-1 matrix. For a fixed sample p, vectorized Δw spans at most the d_out-dimensional subspace {(Wδ)⊗a : δ∈R^{d_out}}, so M_p is at most rank d_out (50 in the D=2500 experiments), not full-rank isotropic. Consequently at least D−d_out diagonal entries in any basis are zero, and Eq. (14) cannot hold. Without Eq. (14), Eq. (15) is not derived; the paper's figures showing 'isotropy' compare only diagonal magnitudes in the global Hessian basis and cannot establish Eq. (14). The proof of γ∈[1,2] then has no object, and the claimed universal bounds are unsupported. This is an internal inconsistency, not a disagreement with prior consensus, so it undercuts the paper's central claim as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a general relationship between SGD noise covariance C and the loss Hessian H, claiming that under an Activity-Weight Duality (AWD) analysis C is proportional to the second moment of per-sample Hessians, E_p[h_p^2], rather than to H itself. From this it derives approximate commutativity of C and H and a universal superlinear power-law exponent 1≤γ≤2 for diagonal entries in the Hessian eigenbasis. The claims are tested on MLP and CNN models with cross-entropy (CE) and mean-squared-error (MSE) losses, finding γ>1 for CE and γ≈1 for MSE, and are supplemented by a suppression experiment isolating the role of the leading per-sample eigenvalues.","tokens_in":59016,"tokens_out":8489,"duration_ms":93024,"significance":"If the central derivation were sound, the paper would make a substantial contribution: it would replace the restrictive Fisher-information equivalence with a loss-agnostic formula and explain the empirical CE/MSE distinction in a unified way. The dimensional argument in Section 2.2 and the suppression experiment are interesting, and the paper includes an anonymized code release and fairly extensive experiments. However, the theoretical core is invalid: the isotropy assumption in Theorem 3.4 is structurally incompatible with the paper's own closed-form AWD perturbation from Lemma 3.2. Because the main quantitative claims (Eq. (15), Eq. (17), Theorem 5.1, and the abstract's C∝E_p[h_p^2]) all rest on that assumption, the significance of the paper is not realized as written.","major_comments":[{"comment":"The isotropy assumption M_{p,mn} ≈ σ_w² δ_mn in Eq. (14) contradicts Lemma 3.2. Lemma 3.2 gives ΔW = (WΔa)a^T/||a||^2, a rank-1 matrix. For fixed input activity a, the vectorized perturbation Δw = vec(ΔW) lies in the d_out-dimensional subspace {a⊗u : u∈R^{d_out}}. Therefore M_p = E[Δw Δw^T] has rank at most d_out (50 in the D=2500 experiments), so in any basis it has at most d_out nonzero eigenvalues and cannot equal σ_w² I. Consequently, Eq. (15), the abstract's C∝E_p[h_p^2], and the derived bounds in Theorem 5.1 are not established. The paper explicitly acknowledges this assumption in the Conclusion as 'insensitive to detailed architectural choices', but the rank argument shows it is not merely a choice of prefactor; it is structurally impossible.","section":"Theorem 3.4 / Eq. (14)"},{"comment":"The numerical evidence cited for Eq. (14) is incommensurate with the claim. Figures 4-6(h) compare diagonal magnitudes of C_hh, C_hh,SD, C_hh,SD,WD in the global Hessian basis, and Figures 10-12 show log-log diagonal scalings of C and H. None of these displays the matrix M_p in the per-sample Hessian eigenbasis, nor its off-diagonal entries. Thus the statement that 'M_{p,mn} ≈ σ_w² δ_mn' is not empirically supported by the presented figures, and cannot rescue the internal contradiction.","section":"Appendix E.2.3, Figures 4-6 and 10-12"},{"comment":"The paper describes γ_AWD as a 'prediction' and γ_emp as an independent measurement, but γ_AWD is computed from Eq. (15) using the same per-sample Hessians that appear in the theoretical relation, and γ_emp is computed from the same model and dataset. The agreement is therefore a consistency check of Eq. (15), not an out-of-sample or independent validation. While no free parameter is fit to force the slope, the reuse of the same spectral information substantially weakens the evidential value. Given that Eq. (15) relies on the invalid assumption (14), the agreement in Table 1 cannot by itself support the proposed C∝E_p[h_p^2] law.","section":"Section 4 / Table 1"}],"minor_comments":[{"comment":"The claim that the projections (u_m^{(p)}·v_i) and (u_m^{(p)}·v_j) are 'statistically independent' is not justified. For a random unit vector, these dot products with orthogonal directions are uncorrelated, but not independent in general. To conclude that off-diagonal elements C_ij vanish, one needs a variance or concentration argument, not merely zero mean.","section":"Remark 3.5"},{"comment":"The derivation of 1≤γ≤2 assumes an exact power law C_ii = c H_ii^γ over a range where H_ii can be arbitrarily small. The experiments only fit the top 1000 eigenvalues, so the logical step from Eq. (19) to Eq. (20) is more delicate than stated, even apart from the failure of Eq. (15).","section":"Theorem 5.1 / Eq. (20)"},{"comment":"The p-value is reported as 'p≪10^{-50}' after a Z-score of about -3378. This is reasonable, but the exact tail probability is machine zero; the notation is informal. Also, the effective number of spikes M≈20 is estimated from the same data used for the baseline, so the test is partly circular.","section":"Appendix D"}],"recommendation":"reject","confidential_remarks":"The central theorem of the paper is internally inconsistent with its own Lemma 3.2. The rank-1 structure of the AWD weight perturbation makes the isotropy assumption Eq. (14) impossible, so the main theoretical claims cannot be repaired by a small correction. The empirical findings—superlinear scaling for CE and near-linear for MSE—might be publishable on their own as a phenomenological study, but the current manuscript's framing and derivation would need to be substantially rewritten, and the central quantitative claim C∝E_p[h_p^2] would have to be abandoned or replaced with a different, supported formula."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: the paper's headline result doesn't survive contact with its own lemma. Theorem 3.4 assumes Mp ≈ σ_w² I in the per-sample Hessian eigenbasis, but Lemma 3.2 gives ΔW = (WΔa)a^T/||a||², which is rank 1. That means Mp has rank at most d_out (50 in their D=2500 experiments), so it cannot be isotropic. Eq. (15) does not follow, and neither do the abstract's C ∝ E[h_p²] or the 1≤γ≤2 bound. This is a load-bearing flaw, not a minor gap. The reader's stress-test is right, and I don't see a way around it.\n\nWhat the paper does well: the experiments are extensive, the code is released, and the CE-vs-MSE distinction via the suppression experiment is a genuine empirical contribution. The dimensional-consistency argument against strict proportionality is also clean and worth keeping. They're honest about prior power-law observations, citing Xie et al. and Tang et al.\n\nThe soft spots beyond the central theorem: the 'isotropy validation' in the figures compares diagonal magnitudes in the global Hessian basis, which cannot establish Eq. (14). Also, γ_AWD is computed from the same per-sample Hessians used to fit γ_emp, so it's a consistency check, not an independent prediction. The X–Y correlation story for CE vs MSE is plausible but not proven, and the paper says so itself.\n\nIf you read this as an empirical study of scaling exponents, there's material here. As a theory paper, the central derivation needs repair. I'd still send it to peer review — the question is important, the experimental work is substantial, and a careful referee might extract the useful empirical core — but I'd expect a major revision or a rejection unless the theorem is fixed.","headline":"The central derivation assumes an isotropy that its own rank-1 AWD lemma rules out; the empirical CE/MSE comparison still has value.","tokens_in":59589,"tokens_out":3033,"would_cite":false,"duration_ms":36231,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that SGD noise covariance is proportional to the expected squared per-sample Hessian, so noise and curvature connect superlinearly with a power-law exponent between 1 and 2.","keywords":["SGD noise covariance","Hessian","per-sample Hessian","Activity-Weight Duality","power-law scaling","Fisher information approximation","flat minima","cross-entropy loss"],"falsifier":"Compute M_p = E[Δw_p Δw_pᵀ] for a single fully connected layer at convergence; because ΔW* is a rank-1 outer product, M_p has rank at most d_out, which in the paper's 50×50 layer is only 50 out of 2500 dimensions, so it cannot be a scalar multiple of the identity. If the measured M_p violates isotropy, the spectral formula in Theorem 3.4 loses its foundation, and with it the universal exponent bound.","tokens_in":58476,"feed_emoji":"📊","tokens_out":5319,"duration_ms":53702,"temperature":0.7,"pith_summary":"The paper sets out to replace the common assumption that SGD noise covariance C is proportional to the loss Hessian H, an equivalence inherited from the Fisher information matrix. Using Activity–Weight Duality, it derives a loss-agnostic formula: C ∝ E_p[h_p²], the second moment of per-sample Hessians. Since H is the first moment, C and H share an approximate eigensystem but do not coincide, and their diagonals follow C_ii ∝ H_ii^γ with provable bounds 1 ≤ γ ≤ 2. This explains why cross-entropy training shows superlinear scaling (γ ≈ 1.3–1.5) while mean-squared error stays near γ ≈ 1, offering a more faithful picture of the geometry that biases SGD toward flat minima.","feed_headline":"SGD noise scales as curvature squared, new theory shows","feed_subtitle":"Cross-entropy training gives superlinear noise–Hessian scaling (γ up to ~1.5), overturning the Fisher equivalence.","key_machinery":"Activity–Weight Duality (AWD): a mapping from minibatch activity fluctuations to minimal-norm weight perturbations, whose explicit form is a rank-1 outer product ΔW* = (WΔa)aᵀ/‖a‖². The argument's engine is a spectral decomposition of the resulting covariance in the global Hessian eigenbasis, which assumes that the weight perturbations are isotropic in each sample's local Hessian eigenbasis (M_p ≈ σ_w²I). That assumption turns the noise into E_p[Σ_m (κ_m^(p))² (u_m^(p)·v_i)²], isolating curvature magnitudes from their alignment with global directions and yielding the 1–2 exponent bounds.","core_discovery":"The central discovery is that the SGD noise covariance is governed by the squared per-sample curvature: C ≈ (σ_w²/2B) E_p[h_p²], derived without assuming the model distribution equals the data distribution. Because the Hessian H = E_p[h_p], the noise amplifies high-curvature directions more than the Hessian itself, producing a power-law C_ii ∝ H_ii^γ with 1 ≤ γ ≤ 2. The paper proves the bound from Cauchy–Schwarz and bounded per-sample eigenvalues, and confirms it empirically on MLPs and CNNs across MNIST and CIFAR-10 with both cross-entropy and mean-squared-error losses.","pith_inferences":["One testable extension is to verify C ∝ E_p[h_p²] directly in Transformer blocks, where per-sample Hessians are approximated; the paper only analyzes fully connected layers.","The local isotropy assumption is contradicted by the paper's own rank-1 perturbation formula; replacing it with a realistic rank-deficient covariance model may change the derived exponent bound, so the 1–2 interval should be tested against that weaker premise.","A practical consequence the authors leave implicit: if γ is tuned by decoupling curvature magnitude from eigendirection alignment (the X–Y correlation), it could offer a new knob for controlling flat-minima selection without changing the loss landscape."],"forward_implications":["The Fisher-approximation identity C ∝ H is generically false for cross-entropy losses; algorithms and theories built on it inherit a misleading equivalence.","C and H approximately commute, so SGD noise analysis can treat them as simultaneously diagonalizable in the Hessian eigenbasis.","The exponent γ measures how strongly per-sample curvature fluctuations align with global Hessian directions, giving a quantitative diagnostic of landscape geometry.","Because γ exceeds 1 for cross-entropy, noise anisotropically weights high-curvature directions more than H alone, reinforcing curvature-driven escape from sharp minima."],"fun_headline_variants":["SGD noise scales superlinearly with curvature, theory shows","Squared per-sample Hessian sets SGD noise, bound proven","Noise–curvature relation in SGD: power law, not Fisher","Curvature squared drives SGD noise, bounds verified","SGD noise follows squared curvature, exponent 1–2"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that per-sample weight perturbations from Activity–Weight Duality have isotropic covariance in each sample's local Hessian eigenbasis (M_p ≈ σ_w²I, uniform across samples); since the paper's own solution ΔW* is rank 1, this isotropy cannot hold exactly, and if it fails the derived formula for C and the 1–2 exponent bound collapse.","fun_headline_variants_meta":{"raw":{"variants":["SGD noise scales superlinearly with curvature, theory shows","Squared per-sample Hessian sets SGD noise, bound proven","Noise–curvature relation in SGD: power law, not Fisher","Curvature squared drives SGD noise, bounds verified","SGD noise follows squared curvature, exponent 1–2"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000249,"raw_usage":{"total_tokens":1409,"prompt_tokens":786,"completion_tokens":623,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":536}},"tokens_in":530,"tokens_out":623,"duration_ms":6528,"temperature":1.0,"reasoning_tokens":536,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T04:11:50.559747+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute M_p = E[Δw_p Δw_pᵀ] for a single fully connected layer at convergence; because ΔW* is a rank-1 outer product, M_p has rank at most d_out, which in the paper's 50×50 layer is only 50 out of 2500 dimensions, so it cannot be a scalar multiple of the identity. If the measured M_p violates isotropy, the spectral formula in Theorem 3.4 loses its foundation, and with it the universal exponent bound.","supporting_citations":[],"review_version":1}