{"id":"54c5b973-4d02-462c-b8a8-e632fbc7859b","arxiv_id":"2505.18069","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"L2 weight decay generically makes many learning rules look Hebbian near stationarity, and added noise can make them look anti-Hebbian.","lead":"Near a training endpoint, any learning rule with weight decay produces updates that look like the classic Hebbian rule, even if no Hebbian computation is happening. This paper warns that observing Hebbian or anti-Hebbian patterns in brain data does not prove the brain uses Hebbian learning.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline monotonic-in-γ claim (Eq. 8 and Figure 3/Table 1) is not actually derived: C(W★)=γ c_a ‖W★‖²_F depends on the γ-dependent stationary point, so larger γ need not increase alignment unless ‖W★‖ is controlled.","rationale":"The reader correctly identified Assumption 2 as the weakest link for the stronger per-stimulus claim and flagged that the monotonic-in-γ statement is not proven, which aligns with my concern. However, I would elevate the γ-dependence of W★ as the most load-bearing issue because Eq. 8, the headline quantitative claim, is what the experiments are designed to corroborate, and it is not derivable from Lemma 1 without controlling ‖W★‖²_F as a function of γ. The reader's framing treats the universal average alignment (Eq. 12) as the central claim and the monotonicity as a secondary gap; I would argue the monotonicity is equally central because the abstract, introduction, and Table 1 all advertise 'increases monotonically with decay strength.' The paper does give credit-worthy empirical support: Table 1 sweeps four learning rules and multiple architectures, and the random-NN control is a useful demonstration that the average alignment phenomenon is not specific to gradient-based learning. But the theory as written does not establish the γ-trend, and the experiments do not report weight norms, so the monotone claim rests on an unverified and theory-unsupported assumption. A conditional verdict is appropriate: the paper's core identifiability caveat is plausibly correct and worth publishing, but the quantitative monotonicity claim should either be proven with a bound on ‖W★‖ or softened. I do not see grounds to reject, because the average alignment result (Eq. 12) is a clean algebraic identity and the empirical trend is suggestive even if the theory is incomplete.","tokens_in":22779,"tokens_out":4134,"duration_ms":29467,"concrete_test":"Re-derive the γ-dependence explicitly in the minimal linear regression model of Section 4: solve the stationarity equation E_x[-x(w★ᵀx - y)] = γ w★ for w★(γ), compute C(w★(γ)) = γ ‖w★(γ)‖², and check whether C is monotonically increasing in γ across the relevant range. If C initially increases but later decreases as γ grows large (because ‖w★‖→0), the monotone claim in Eq. 8 and Figure 3 needs a bound on ‖W★‖ or a different statement. Also recompute Table 1 with a fixed, matched γ grid and record ‖W★‖_F at convergence to test whether alignment is high only when the weight norm stays bounded away from zero.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central quantitative claim is that Hebbian alignment 'increases monotonically as we use a larger weight decay' (Section 1, point 1; Abstract; Eq. 8). The proof in Section 3 and Appendix C.5.2 (Lemma 1) establishes only that at a given non-zero stationary point W★ of the penalized loss, C(W★)=γ c_a ‖W★‖²_F. But W★ itself depends on γ: the stationarity condition is E_x[g(x;W★)] = γ W★. Nothing in the proof controls how ‖W★‖²_F scales with γ, so the monotonicity claim does not follow from the mathematics as written. In the linear regression model of Section 4, for example, w★(γ) ≈ c/(λ+γ), so C(γ) = γ ‖w★‖² first increases and then decreases as γ grows; the claimed monotone trend is therefore not a theorem, even in the toy model. The phase-boundary statement γ∝σ² in Eq. 18 is likewise approximate, with c_0 and c_1 asserted to be constants without derivation in the nonlinear case. Empirically, Table 1 shows the trend for several optimizers, but it also shows entries where the weights collapsed to zero (marked '–') at the largest γ, exactly the regime where ‖W★‖ shrinks and the monotone prediction should be qualified. The proof also relies on Assumption 2 (‖h_a(x)‖² = c_a a.s.) for the per-stimulus guarantee (Eq. 8), a norm-decoupling condition that fails for ReLU and heterogeneous inputs, and the experiment section explicitly notes weaker or anti-Hebbian alignment in larger networks and non-last layers, which the theory does not predict. Since the weaker average claim (Eq. 12), any regularized learning rule looks Hebbian at stationarity, is a clean algebraic identity and remains robust, the core identifiability caveat survives; but the monotone-in-γ and per-stimulus correlation claims, which the paper presents as its key quantitative support, are not supported by the derivation as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that Hebbian and anti-Hebbian plasticity signatures can arise as emergent epiphenomena of L2-regularized learning near stationarity, so their observation in neural data does not by itself imply an underlying mechanistic Hebbian computation. For a layer h_b = W h_a with learning signal g(x,W) and full update g - γW, stationarity gives E[g] = γW, from which the paper derives the averaged alignment Tr(E[g] E[h_a h_b^T]) = γ E[||h_b||^2] > 0 (Eq. 12). Under a stronger norm-decoupling assumption, the per-stimulus alignment is C(W*) = γ c_a ||W*||^2_F (Eq. 8 and Lemma 1 in Appendix C.5.2). The paper also analyzes additive noise in a linear regression model and claims an anti-Hebbian phase with a phase boundary γ ∝ σ^2 (Eq. 18), supported by simulations on MLPs and transformers with SGD, Adam, direct feedback alignment, and a random-network learning signal.","tokens_in":23076,"tokens_out":6089,"duration_ms":40329,"significance":"If the central claim survives, the paper delivers a useful identifiability caveat for neuroscience: Hebbian-looking update statistics can be equilibrium artifacts of regularization rather than evidence of Hebbian computation. The main strengths are the clear formal setup in Appendix C.5, the parameter-free nature of the universal averaged alignment result (Eq. 12), and the breadth of empirical controls, including a RandomNN that cannot learn yet still exhibits the alignment. The paper also states explicit limitations about larger networks and non-last layers. However, the headline monotonic-in-γ claim is not actually proven, the stronger per-stimulus result relies on an unverified and often violated norm-decoupling assumption, and the anti-Hebbian phase boundary is asserted rather than derived. These gaps affect the paper's main quantitative predictions and must be addressed.","major_comments":[{"comment":"The claim that Hebbian alignment increases monotonically with weight-decay strength is not established by the proof as written. Lemma 1 gives C(W*) = γ c_a ||W*||^2_F, but W* itself depends on γ through the stationarity condition E[g(x;W*)] = γW*, and nothing in the proof controls how ||W*||^2_F scales with γ. The paper's own linear regression example illustrates the problem: if w*(γ) ≈ c/(λ+γ), then γ||w*(γ)||^2 is non-monotone in γ. Table 1 also contains entries marked '–' where the weights collapsed to zero at the largest γ, exactly the regime where ||W*|| shrinks. The abstract's 'increasing monotonically with decay strength' and Section 1 point 1 therefore go beyond what the mathematics proves. Please either prove monotonicity under explicit conditions on the stationary path or rephrase these statements as empirical observations with appropriate qualifications.","section":"Section 3, Eq. (8), Appendix C.5.2 Lemma 1"},{"comment":"The per-stimulus correlation result and Eq. (8) require Assumption 2, that ||h_a(x)||^2 = c_a almost surely. This condition is not verified in any experiment, and it is violated by ReLU activations and heterogeneous inputs, both of which appear in the paper's own experiments. The main text's informal 'weak decoupling condition' is also stated differently from Assumption 2, which adds ambiguity. The averaged universal result in Eq. (12) survives without Assumption 2, but the stronger claim that the learning signal and the Hebbian rule are statistically correlated per stimulus is not supported by the reported experiments. Please either verify the norm-condition empirically (for example, by reporting the distribution of ||h_a(x)||^2 during training) or restrict the per-stimulus claims to settings where the assumption plausibly holds, such as normalized activations or neural collapse.","section":"Appendix C.5.2, Assumption 2; Section 3, Eqs. (7)-(8)"},{"comment":"The phase-boundary statement γ ∝ σ^2 is asserted through Eq. (18), where c_0 and c_1 are said to be 'positive coefficients' that can be treated as constants with respect to weight decay and noise. No derivation is given for this approximation, and the preceding exact computation in Eqs. (16)-(17) is a single-step calculation in which the stationary weight v itself generally depends on γ and σ. In nonlinear networks this is a heuristic, not a proven scaling law. Figure 4 provides qualitative support, but the boundary is not fitted or tested statistically. Please either derive the boundary from a solvable stationary distribution or clearly mark Eq. (18) as an approximation and report how the boundary was estimated from the simulations.","section":"Section 4, Eq. (18)"},{"comment":"The paper's own limitations section states that strong average anti-Hebbian alignment was often seen in a subset or all layers even at high weight decay, and Section 4 notes difficulty reproducing the effect outside the last few layers of larger networks. These observations sit in tension with the universal monotonic-in-γ claim and with Theorem 1's neighborhood statement, which predicts positive alignment near a nondegenerate stationary point for any layer satisfying the assumptions. The paper attributes the discrepancy to stationarity not holding, but offers no diagnostic test of this explanation. Please reconcile the theoretical scope with the empirical scope, for example by stating explicitly which layers and training regimes the theory is expected to cover and by reporting stationarity-gap measurements alongside the alignment measurements.","section":"Section 6; Section 3, Figure 3"}],"minor_comments":[{"comment":"Equations (4) and (7) contain stray markup (“⌏⟨⟨➪ rl➫ l⟩⟩...”) that will render incorrectly; please replace these with the intended labels 'learning signal' and 'Hebbian update'.","section":"Eqs. (4) and (7)"},{"comment":"Several typos and spacing issues should be fixed: 'CIF AR-10' should be 'CIFAR-10', 'F rozen Paramaters' in Appendix C.1.5 should be 'Frozen Parameters', and 'Full W eight Update' in Appendix C.3.1 should be 'Full Weight Update'.","section":"Throughout"},{"comment":"The main text's statement of the decoupling condition, “‖h_a‖^2 = E[‖h_a‖^2]”, is ambiguous and does not match the almost-sure constant-norm condition of Assumption 2 in Appendix C.5.2; please state the exact assumption used in the main text.","section":"Section 3, informal decoupling statement"},{"comment":"The '–' entries in Table 1 are only explained in the table title as indicating weight collapse; please define this in the caption and discuss how these points are treated when claiming a monotone trend, since they occur at the largest γ.","section":"Table 1"},{"comment":"The sentence 'Although we still detect this trend in larger MLPS, we occasionally observe some layers behaving in an anti-Hebbian direction as the weight increases' appears to mean 'as the weight decay increases'; please correct the wording.","section":"Section 3, Classification paragraph"}],"recommendation":"major_revision","confidential_remarks":"The paper's central mathematical observation is an elementary near-stationarity identity rather than a deep theorem; its value lies in the identifiability interpretation and the empirical breadth. The monotonic-in-γ gap and the unverified norm-decoupling assumption should be fixed before publication, and the phase-boundary claim should be either derived or explicitly heuristics. With those changes the paper would be acceptable; as it stands the headline claims exceed the proofs."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe punchline: this paper's core observation is real and, once stated, almost embarrassingly simple. Near a non-zero stationary point of an L2-regularized loss, any learning rule satisfies E[g] = γW, and the Hebbian correlation has positive inner product with W, so the expected learning signal aligns with the Hebbian update. That's a valid algebraic fact and a genuinely useful identifiability caveat: Hebbian or anti-Hebbian signatures in synaptic measurements are not, by themselves, evidence of Hebbian computation. The empirical sweep—SGD, Adam, direct feedback alignment, and a random network whose updates are random matrices—is a good stress test of the generic claim, and the RandomNN construction is genuinely clever. The authors also deserve credit for flagging their own limits: they note the anti-Hebbian phase boundary is less robust in larger networks and classification, and they admit they lack a theory for why some layers go anti-Hebbian.\n\nWhere it gets soft. The headline claim that alignment increases monotonically with γ is not proven by the math as written. Lemma 1 gives C(W*) = γ c_a ‖W*‖²_F, but W* itself depends on γ, and nothing controls how ‖W*‖ scales with γ. In their own linear regression toy model, C(γ) is non-monotonic. The monotone trend is visible in the experiments, but it is not a theorem, and Table 1 shows weight-collapse entries at the largest γ. The per-stimulus correlation result (Eq. 8) additionally relies on Assumption 2, that presynaptic activation norms are constant almost surely; that's strong, unverified, and fails for ReLU or heterogeneous inputs. The universal average result (Eq. 12) survives without Assumption 2, so the central caveat stands. The anti-Hebbian phase boundary is derived only in a linear regression with coefficients c0 and c1 asserted rather than derived; it's a plausible heuristic, not a robust prediction. I'd also note the machine-readable text has equation rendering corruption that impeded verification, and no code is provided. Neither is fatal, but both are avoidable.\n\nWho this is for: theoretical neuroscientists and ML researchers who work on learning rules. It deserves a serious referee. My recommendation: send it to review, but require the authors to either prove the monotonicity under explicit conditions or weaken the claim to match the proof, and state Assumption 2's role honestly.","headline":"A clean stationarity identity makes Hebbian signatures a generic artifact of weight decay, but the monotonicity and per-stimulus claims outrun the proof.","tokens_in":23757,"tokens_out":3163,"would_cite":true,"duration_ms":23294,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Near any nonzero stationary point of an L2-regularized loss, every learning rule's expected update aligns with the Hebbian direction, so Hebbian signatures are generic artifacts of weight decay.","keywords":["Hebbian learning","anti-Hebbian plasticity","weight decay","regularization","stationarity","emergent dynamics","identifiability","synaptic plasticity"],"falsifier":"Train a ReLU network on heterogeneous inputs to a nonzero stationary point with at least two weight-decay strengths, and compute per-example alignment $\\mathbb{E}_x[\\langle -\\nabla_W\\ell(x;W), h_b(x)h_a(x)^T\\rangle]$ and its average. If the per-example quantity is not positive or does not increase with $\\gamma$, the stronger version of the claim fails. Separately, in a noisy regression setting, vary $\\sigma^2$ and $\\gamma$ independently and check whether the zero-alignment contour in the $(\\sigma^2,\\gamma)$ plane follows a line $\\gamma\\propto\\sigma^2$; a different boundary shape would falsify the anti-Hebbian tradeoff prediction.","tokens_in":22466,"feed_emoji":"🧠","tokens_out":6613,"duration_ms":61458,"temperature":0.7,"pith_summary":"This paper tries to establish an identifiability result: observing Hebbian or anti-Hebbian structure in synaptic updates does not uniquely identify an underlying Hebbian computation. Near a nonzero stationary point of an L2-regularized loss, the expected learning signal of essentially any update rule aligns with the Hebbian direction, with alignment growing with weight-decay strength; this is a generic equilibrium artifact of regularization. In the presence of noise the same balance mechanism produces anti-Hebbian alignment, with a predicted phase transition at weight decay $\\gamma \\propto \\sigma^2$. A sympathetic reader should care because neuroscience measurements of LTP/LTD and synaptic plasticity are often interpreted mechanistically, and this paper shows those signatures can arise without any Hebbian computation, complicating such interpretations.","feed_headline":"Weight decay alone can mimic Hebbian learning","feed_subtitle":"Near stationarity, L2 regularization makes any learning rule Hebbian; noise flips it anti-Hebbian.","key_machinery":"The central object is the decomposition of the total weight update into a learning signal $g(x,\\theta)=-\\nabla_W\\ell(x;W)$ and a decay term $-\\gamma W$, with the Hebbian update defined as $H(x;W)=h_b(x)h_a(x)^T$ for a layer $h_b=Wh_a$. The identity that carries the argument is the stationarity balance $\\mathbb{E}_x[g]=\\gamma W$, which converts the contractive force of weight decay into a positive correlation between the learning signal and the expansive Hebbian direction. For the noise result, the machinery is the same balance applied to a linear regression loss $(\\ell(w)=(w^Tx-y)^2)$ with injected Gaussian noise $\\epsilon\\sim\\mathcal{N}(0,\\sigma I)$, where the $\\sigma^2\\|x\\|^2$ term makes the alignment negative.","core_discovery":"At stationarity of the penalized loss $L(W)=\\mathbb{E}_x[\\ell(x;W)]+\\frac{\\gamma}{2}\\|W\\|_F^2$, stationarity forces the expected learning signal to satisfy $\\mathbb{E}_x[g(x,\\theta)]=\\gamma W$. Taking the Frobenius inner product with the expected Hebbian update $\\mathbb{E}_x[h_a h_b^T]$ gives $\\mathrm{Tr}(\\mathbb{E}_x[g]\\,\\mathbb{E}_x[h_a h_b^T])=\\gamma\\,\\mathbb{E}[\\|h_b\\|^2]>0$ for any $W\\neq 0$, so any rule with weight decay looks Hebbian on average near stationarity. Under the stronger norm-decoupling assumption $\\|h_a(x)\\|^2=c_a$ almost surely, the per-stimulus alignment is $C(W^\\star)=\\gamma c_a \\|W^\\star\\|_F^2>0$ at a stationary point. Adding noise to the weights in linear regression makes the alignment $\\mathbb{E}[(\\Delta_{\\mathrm{SGD}}w)^T(\\Delta_{\\mathrm{Hebb}}w)]\\approx -\\sigma^2 c_0+\\gamma c_1$, so strong noise flips the signature anti-Hebbian and the two effects trade off across a phase boundary $\\gamma\\propto\\sigma^2$.","pith_inferences":["If the universal alignment result holds, then any homeostatic mechanism that acts like decay—synaptic normalization, spine shrinkage, slow biochemical turnover—should imprint Hebbian structure onto whatever learning rule a circuit uses, so the presence of Hebbian structure carries no information about the local plasticity rule.","The $\\gamma\\propto\\sigma^2$ phase boundary suggests a quantitative experimental design: in a circuit where intrinsic noise and decay-like homeostasis can be manipulated independently, anti-Hebbian plasticity should dominate in the high-noise, low-decay corner and Hebbian plasticity in the opposite corner.","The authors' own observation that larger models and some layers show anti-Hebbian alignment even at high $\\gamma$ hints that the stationarity condition is the fragile part; a theory of transient, out-of-stationary Hebbian phases would need to replace Assumption 2 with a condition that survives ReLU and heterogeneous inputs."],"forward_implications":["Observed Hebbian alignment at convergence in a regularized system is not evidence that the system is implementing a Hebbian rule.","The alignment strength is predicted to grow monotonically with weight-decay strength for any learning rule whose expected update matches the stationarity balance.","Noise and weight decay compete: in regression settings, the sign of Hebbian alignment is governed by the ratio $\\gamma/\\sigma^2$, predicting anti-Hebbian signatures in high-noise regimes.","Because the effect appears long before convergence and for random update rules, time-course or rule-identity checks cannot by themselves distinguish emergent from mechanistic Hebbian dynamics."],"supporting_citations":[{"why":"Supplies the canonical Hebbian learning rule and the observation that Hebbian updates need weight decay or normalization to converge; the object the paper compares against.","marker":"Oja, 1982b"},{"why":"Provides the weak decoupling/norm-constancy condition used in the proof and the expansion-contraction view of representation learning.","marker":"Ziyin et al. (2024)"},{"why":"Gives neural collapse as a setting where representation norms become nearly constant, used to justify Assumption 2.","marker":"Papyan et al., 2020"},{"why":"Motivates the large batch sizes used in experiments so that dynamics are near stationarity despite SGD noise.","marker":"Xu et al., 2023"},{"why":"Defines Direct Feedback Alignment, one of the alternative learning rules whose Hebbian alignment is measured in Table 1.","marker":"Nøkland, 2016"},{"why":"Supplies the CIFAR-10 dataset used in the classification experiments.","marker":"Krizhevsky, 2009"}],"fun_headline_variants":["Weight decay alone mimics Hebbian","L2 weight decay creates Hebbian illusion","Noise flips Hebbian to anti-Hebbian","Regularized learning mimics Hebbian dynamics","Weight decay mimics Hebbian, noise flips it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's sharper per-stimulus and monotonic-in-$\\gamma$ predictions rest on Assumption 2: the presynaptic activation norm $\\|h_a(x)\\|^2$ is the same constant $c_a$ for every input, almost surely. Real ReLU networks and heterogeneous inputs violate this, and without it the expectation $\\mathbb{E}[\\|h_a\\|^2 \\nabla_{h_b}^T\\ell\\, h_b]$ need not factor into the positive quantity used in the proof; the averaged alignment result (Eq. 12) does not require this assumption.","fun_headline_variants_meta":{"raw":{"variants":["Weight decay alone mimics Hebbian","L2 weight decay creates Hebbian illusion","Noise flips Hebbian to anti-Hebbian","Regularized learning mimics Hebbian dynamics","Weight decay mimics Hebbian, noise flips it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000868,"raw_usage":{"total_tokens":3797,"prompt_tokens":1020,"completion_tokens":2777,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":2704}},"tokens_in":636,"tokens_out":2777,"duration_ms":21190,"temperature":1.0,"reasoning_tokens":2704,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:35:56.454703+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a ReLU network on heterogeneous inputs to a nonzero stationary point with at least two weight-decay strengths, and compute per-example alignment $\\mathbb{E}_x[\\langle -\\nabla_W\\ell(x;W), h_b(x)h_a(x)^T\\rangle]$ and its average. If the per-example quantity is not positive or does not increase with $\\gamma$, the stronger version of the claim fails. Separately, in a noisy regression setting, vary $\\sigma^2$ and $\\gamma$ independently and check whether the zero-alignment contour in the $(\\sigma^2,\\gamma)$ plane follows a line $\\gamma\\propto\\sigma^2$; a different boundary shape would falsify the anti-Hebbian tradeoff prediction.","supporting_citations":[{"cited_title":"Prevalence of neural collapse during the terminal phase of deep learning training","cited_arxiv_id":null,"evidence_quote":"Gives neural collapse as a setting where representation norms become nearly constant, used to justify Assumption 2."}],"review_version":1}