{"id":"51536677-f8d5-42be-8f87-ef5bc2a7d0ac","arxiv_id":"2505.23369","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":10,"one_line_summary":"DSBP claims faster, flatter training by projecting gradients onto principal activation eigenvectors, with reported gains over SAM, LoRA, and MAML, but the math is internally inconsistent and the results are not reproducible from the paper.","lead":"DSBP is a proposed training method that projects gradients onto the top eigenvectors of layer covariance matrices and adds a sharpness penalty, claiming lower compute and flatter minima. The reported experiments show small accuracy gains over SAM, LoRA, and MAML on four image datasets, but the derivation has unresolved dimensional inconsistencies and no code or data is released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The DSBP update rule is not well-defined: §3.1 subtracts a d_l×d_l rank-one matrix from d_l×d_{l-1} weights, so the core algorithm cannot be executed as written.","rationale":"The strongest claim is that DSBP is a low-cost training method with superior accuracy and a theoretical guarantee. For that claim to hold, the update rule must be a well-defined operation on the weights. The dimensional mismatch in §3.1 is not a subtle statistical assumption; it makes the algorithm unimplementable for typical layers where input and output dimensions differ. This is an internal inconsistency, and it is more decisive than the eigenspace-drift concern: even if the top-k eigenvectors were perfectly stationary, the update is still undefined. The reader's rationale noted the dimensional problem, though their formal 'weakest_assumption' focused elsewhere, hence partial agreement. The reported experiments cannot compensate because no code, seeds, or complete configurations are released, and the theoretical analysis inherits the same undefined operations. There is also no machine-checked proof or reproducible artifact that independently verifies the method. A corrected paper would need a fully specified projection rule, a rederived SDE and generalization bound, and new experiments; until then REJECT is appropriate.","tokens_in":9235,"tokens_out":3373,"duration_ms":33228,"concrete_test":"Implement Algorithm 1 exactly as written for a single linear layer with input dimension 2 and output dimension 3 (so d_l=3, d_{l-1}=2), using random activations A∈R^{4×3}, a random gradient G∈R^{3×2}, and the given projection and update equations. The implementation will fail at the first update because βλ e e^T is 3×3 while W is 3×2. To rule out a typo, ask the authors to specify the intended inner product ⟨·,·⟩ and the intended update shape; then re-derive §3.1 and §4.1 under that definition and check that the 'streamlined' terms, Proposition 1, and Theorem 1 still follow. Also verify whether the claimed O(k d_l) cost includes computing the covariance A^T A and the projection itself.","verdict_should_be":"REJECT","load_bearing_attack":"The central algorithmic object is dimensionally inconsistent. In §3.1, the gradient is ∇fγ(W_l^(t)) ∈ R^{d_l×d_{l-1}}, while each eigenvector e_{l,i} is in R^{d_l}. Step 3 defines ⟨∇fγ(W_l^(t)), e_{l,i}⟩ as a scalar and multiplies it by e_{l,i}, producing a vector in R^{d_l}; summing gives a vector, not a matrix update for the d_l×d_{l-1} weight. Step 4 then subtracts βλ_{l,1} e_{l,1} e_{l,1}^T, a d_l×d_l matrix, from W_l^(t), a d_l×d_{l-1} matrix. For any layer with d_l ≠ d_{l-1}, both operations are undefined. One could reinterpret the projection as left multiplication, PV ∇f, but this is not what is written, and the 'streamlined' trace computations in §4.1 would still need to be redone. Because the update rule is not a well-defined operation on the weights, the discrete dynamics described by Algorithm 1 do not exist as stated; consequently the claimed O(k d_l) complexity, the SDE convergence analysis, and the PAC-Bayes bound are not attached to a concrete algorithm. This is an internal inconsistency, not a disagreement with a prior or a matter of tuning.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Dynamic Spectral Backpropagation (DSBP), a training method that projects gradients onto the top-k eigenvectors of layer-wise activation covariance matrices, adds a sharpness regularization term, and claims to reduce computational complexity from O(d_l d_{l-1}) to O(k d_l). It also introduces five extensions (dynamic spectral inference, spectral architecture optimization, spectral meta learning, spectral transfer regularization, and Lie algebra inspired dynamics), presents a third-order SDE analysis and a PAC-Bayes generalization bound, and reports experiments on CIFAR-10, Fashion MNIST, MedMNIST, and Tiny ImageNet showing accuracy gains over SAM, LoRA, and MAML.","tokens_in":9532,"tokens_out":7295,"duration_ms":72327,"significance":"If the core claims were correct, DSBP would be a broadly applicable low-cost training method with theoretical guarantees, which would be a significant contribution. However, the central algorithm in Section 3.1 is dimensionally inconsistent and cannot be executed as written, invalidating the complexity claim, the SDE analysis, the PAC-Bayes bound, and the empirical validation. The paper does not provide reproducible code, machine-checked proofs, or falsifiable predictions independent of its own tuning choices. The strengths are limited to the identification of an interesting research direction (spectral projection of gradients) and an ambitious attempt to connect it to SDE and PAC-Bayes theory, but these are not realized in the current form.","major_comments":[{"comment":"The update rule is dimensionally inconsistent. The gradient ∇fγ(W_l^(t)) is a d_l×d_{l-1} matrix, while each eigenvector e_{l,i} is a vector in R^{d_l}. The inner product ⟨∇f, e_{l,i}⟩ is undefined as a scalar; if interpreted as e_{l,i}^T ∇f, it is a row vector of length d_{l-1}, not a scalar, and the sum in Step 3 produces a vector in R^{d_l}, not a matrix. Step 4 then subtracts βλ_{l,1} e_{l,1} e_{l,1}^T, which is a d_l×d_l matrix, from the d_l×d_{l-1} weight matrix W_l^(t). For any layer with d_l ≠ d_{l-1}, both operations are undefined. Consequently, Algorithm 1 cannot be executed as written, and all subsequent claims (complexity, SDE, PAC-Bayes) are not attached to a concrete algorithm.","section":"Section 3.1, Steps 3–4; Algorithm 1, lines 13–14"},{"comment":"The SDE derivation is invalid. The Taylor expansion of f(W+ΔW) uses trace notation for matrix arguments, but the terms are dimensionally inconsistent; for instance, tr(∇f^T e e^T) equals e^T ∇f e, a scalar, whereas the manuscript writes it as (e)^T ∇f, a row vector. The second-order and third-order 'streamlined' expressions similarly mix scalars, vectors, and matrices in ways that do not follow from the preceding line. Moreover, the noise covariance Σ_DSBP is defined as E[(∇̃f−∇f)^T(∇̃f−∇f)], but ∇̃f and ∇f are matrices of different sizes (or one is a vector under the manuscript's own Step 3), so the covariance is undefined. Since the discrete update itself is undefined, the claim 'As η→0, the discrete updates converge to the SDE' is vacuous.","section":"Section 4.1"},{"comment":"The claimed complexity reduction from O(d_l d_{l-1}) to O(k d_l) is not supported. Computing the activation covariance A^T A requires O(n d_l^2) operations, and the power iteration for k eigenvectors costs O(k d_l^2) per layer. Even under the charitable reinterpretation of the projection as left multiplication by P_{V_k^l} (a d_l×d_l matrix), multiplying P by the d_l×d_{l-1} gradient costs O(d_l^2 d_{l-1}) (or O(k d_l d_{l-1}) if computed as a sum of k outer products), not O(k d_l). Thus the paper's central efficiency claim is not established.","section":"Section 3.1, first paragraph"},{"comment":"The PAC-Bayes bound does not follow from the stated derivation. The standard PAC-Bayes inequality (Alquier et al. [7]) bounds f_D(Q) by f_S(Q) plus a KL-dependent term; the manuscript replaces f_S(Q) by f_S(W) + (σ^2/2) Σ_l λ_{l,1}(∇^2 f_S) + C d^3 σ^3/6, but for a Gaussian posterior N(W, σ^2 I_d) the expected loss is E[f_S(W+ϵ)] = f_S(W) + (σ^2/2) tr(∇^2 f_S) plus higher-order terms, not a sum over per-layer top eigenvalues. Additionally, the bound as stated contains a factor d σ^2/2 multiplying the layer sum, which would double-count parameters across layers. The proof is therefore not a derivation of Theorem 1.","section":"Section 4.2, Theorem 1"},{"comment":"The experimental evaluation is not reproducible and cannot support the claimed superiority. No code is provided; the hyperparameters for baselines (SAM, LoRA, MAML) are not reported; and the text in Section 6.4 states that k and p were chosen after iterative experimentation, without clarifying whether the reported test accuracies are selected from a validation-based search. Moreover, Section 6.2 claims that DSBP's loss curve 'closely matched the discrete updates,' but since the discrete updates are not well-defined (see Major Comment 1), this validation is meaningless. The reported improvements over SAM are small (e.g., 96.3% vs 95.5% on CIFAR-10) and no confidence intervals or significance tests are provided beyond the standard deviation of three runs.","section":"Section 6"}],"minor_comments":[{"comment":"The paper contains numerous typos and inconsistent notation, including 'CIF AR 10' and 'F ashion MNIST' throughout; these should be corrected to 'CIFAR-10' and 'Fashion MNIST'.","section":"Throughout"},{"comment":"The symbol C_l^(t) is defined twice, once for weights as W^T W ∈ R^{d_{l-1}×d_{l-1}} and once for activations as A^T A ∈ R^{d_l×d_l}; this overloaded notation is confusing, especially since the eigenvectors e_{l,i} are used for both objects.","section":"Section 2"},{"comment":"Figures 1–4 are referenced in the text and described in captions, but no actual figures appear in the manuscript; this must be fixed before any further review.","section":"Figures 1–4"},{"comment":"The reference list contains apparent errors (e.g., [3] gives page numbers 1234–1243 for a NeurIPS paper) and several citations are to preprints without DOIs; please verify all references.","section":"References"},{"comment":"The Lie bracket approximation [e_{l,i}, W]_approx is not a standard Lie bracket and is not defined as an operator; the exponential update is also undefined because the exponent is not a linear operator on the weight space.","section":"Section 5.3, Lie Algebra Inspired Dynamics"}],"recommendation":"reject","confidential_remarks":"This manuscript appears to be an early draft: it contains missing figures, undefined mathematical objects, and a core algorithm that cannot be executed. I recommend rejection. The most serious issue is the dimensional inconsistency in Section 3.1, which invalidates the entire theoretical and empirical analysis. If the authors can fix the update rule and re-derive the theory, a future submission could be considered, but the current version is not publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The stress-test is right, and it lands hard. In §3.1, Step 3 takes an inner product between a d_l×d_{l-1} gradient and a d_l-dimensional eigenvector, then multiplies by the eigenvector to get another vector; Step 4 subtracts a d_l×d_l matrix from a d_l×d_{l-1} weight update. For any layer where d_l ≠ d_{l-1}, these operations are undefined. So Algorithm 1 does not describe a concrete training procedure, and the claimed O(k d_l) complexity, the SDE convergence, and the PAC-Bayes bound are all attached to an update rule that is not actually an operation on the weights. This is not a notation quibble; it is the central object of the paper.\n\nTo give credit where it is earned: the underlying intuition is not crazy. Projecting gradient updates onto top eigenvectors of the activation covariance and penalizing movement along high-curvature directions is a reasonable thing to try, and the paper is honest that the projection dimension and update interval require tuning. The five extensions—dynamic spectral inference, architecture optimization, meta-learning, transfer regularization, and Lie-algebra dynamics—form a genuinely broad set of proposals, some with concrete implementation details. The writing is also readable in places, and the authors openly describe their tuning struggles, which is a good sign of intellectual honesty.\n\nThe soft spots are proportional to the central flaw. Section 4.1's SDE derivation is a Taylor expansion with unjustified expectation swaps, and Proposition 1 is asserted rather than proved. Theorem 1 mixes f_D(W) and f_D(Q), leaves the posterior width σ unspecified, and replaces the trace of the Hessian with a sum of top eigenvalues without justification. The complexity claim ignores the cost of computing covariances and projecting the full gradient. No code, data, seeds, or complete baseline configurations are provided, so even the moderate accuracy gains cannot be checked. Worse, the dynamic spectral inference coefficient α was selected after observing its effect on MedMNIST test accuracy, which is fitting, not prediction. Self-citation is not an issue here; those citations are standard. But the paper's own text contains the admission that α was tuned on test performance, which undercuts the empirical claims.\n\nMy verdict: this version does not deserve peer review as a paper because the algorithm it analyzes does not exist. It is a research proposal with a plausible seed. If the authors fix the update rule—for example, by defining a proper matrix projection, e.g., PV_k∇f or ∇f P_{k'}, and redoing the theory and experiments with released code—it could become a real contribution. But as submitted, the load-bearing equation is undefined. For a desk editor, reject.","headline":"The core update rule is dimensionally inconsistent, so the paper's central algorithm cannot be executed as written; the underlying idea is plausible but the current version does not support its claims.","tokens_in":10143,"tokens_out":2163,"would_cite":false,"duration_ms":25522,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Dynamic Spectral Backpropagation claims that projecting gradients onto top activation eigenvectors cuts per-layer training cost from $O(d_l d_{l-1})$ to $O(k d_l)$ and steers optimization toward flat minima.","keywords":["dynamic spectral backpropagation","gradient projection","flat minima","sharpness-aware minimization","PAC-Bayes generalization","stochastic differential equations","few-shot meta learning"],"falsifier":"Train a two-layer network on a synthetic regression whose target weights are known, and choose input features so that the informative gradient lies in the bottom eigenspace of the activation covariance; if DSBP's projected updates stall or diverge while full-gradient SGD converges, the top-$k$ projection is discarding essential information. Alternatively, on any benchmark, record $\\|\\nabla f - \\tilde{\\nabla} f\\| / \\|\\nabla f\\|$ across training; if this discarded-gradient norm stays large while accuracy improves, the small-error premise of the SDE analysis is violated.","tokens_in":8870,"feed_emoji":"⚡","tokens_out":7306,"duration_ms":70953,"temperature":0.7,"pith_summary":"This paper proposes Dynamic Spectral Backpropagation (DSBP), a training rule that replaces the full gradient with its projection onto the top $k$ eigenvectors of each layer's activation covariance matrix, recomputed every $p$ iterations. The paper claims this reduces per-layer gradient computation from $O(d_l d_{l-1})$ to $O(k d_l)$, and that adding a sharpness penalty tied to the largest eigenvalue steers training toward flat minima. A third-order stochastic differential equation and a PAC-Bayes bound are offered as theoretical support. If the claims hold, DSBP would be a low-cost alternative to standard backpropagation that also generalizes better, with reported accuracy gains over SAM, LoRA, and MAML on CIFAR-10, Fashion MNIST, MedMNIST, and Tiny ImageNet.","feed_headline":"Gradient projection onto top eigenvectors beats SAM, LoRA, MAML","feed_subtitle":"DSBP projects each layer's gradient onto its top-k activation eigenvectors, cutting cost and sharpness on four benchmarks.","key_machinery":"The load-bearing object is the top-$k$ eigenvector subspace $V_l^k = \\mathrm{span}\\{e_{l,1}^{(t)}, \\ldots, e_{l,k}^{(t)}\\}$ of the layer activation covariance $C_l^{(t)} = A_l^{(t)T}A_l^{(t)}$, with projection $P_{V_l^k} = \\sum_{i=1}^k e_{l,i}^{(t)} e_{l,i}^{(t)T}$. DSBP's update is the gradient projected onto this subspace minus a sharpness penalty $\\beta \\lambda_{l,1}^{(t)} e_{l,1}^{(t)} e_{l,1}^{(t)T}$; the same eigenvectors also drive the pruning rule in spectral architecture optimization. The claimed complexity reduction and the flat-minima effect both follow from replacing the full $d_l \\times d_{l-1}$ gradient by a $k$-dimensional coordinate along these eigenvectors.","core_discovery":"DSBP's central claim is that the directions of maximum variance in layer activations are the directions that matter for weight updates. For each layer, it forms $C_l^{(t)} = A_l^{(t)T}A_l^{(t)}$, estimates its top $k$ eigenvectors by power iteration, projects the gradient onto their span, and updates with $W_{l,t+1} = W_l^{(t)} - \\eta \\tilde{\\nabla} f_\\gamma(W_l^{(t)}) - \\beta \\lambda_{l,1}^{(t)} e_{l,1}^{(t)} e_{l,1}^{(t)T}$. The paper argues this projection cuts per-layer complexity to $O(k d_l)$ while the regularization term penalizes movement along the highest-curvature direction, flattening the loss landscape. The theoretical sections model the discrete updates as a third-order SDE and prove a PAC-Bayes generalization bound in which the dominant Hessian eigenvalues enter additively. On the reported benchmarks, including a 5-shot MedMNIST task, DSBP is claimed to outperform SAM, LoRA, and MAML on both accuracy and training time.","pith_inferences":["A direct test of the paper's core assumption is to measure subspace drift: compute the overlap between the top-$k$ activation eigenspace at time $t$ and at $t+p$; if the overlap decays quickly, the fixed-eigenvector approximation inside the SDE would need to be revised.","The $O(k d_l)$ complexity claim counts the projection but not the cost of estimating eigenvectors from a batch of size $n$; the real speedup depends on whether power iteration over $A_l^T A_l$ is cheaper than the full gradient for the chosen $n$, $d_l$, and $k$.","The sharpness penalty uses only the top activation eigenvalue; a natural extension is to weight all $k$ eigenvalues or to tie the penalty to the Hessian along the projected subspace, which could tighten the PAC-Bayes bound.","The PAC-Bayes bound contains $d\\sigma^2\\lambda$ terms and a $\\|W\\|^2$ term; for large networks this may be loose, and a practical check would be whether the bound's predicted test risk tracks the observed generalization gap across different projection dimensions $k$."],"forward_implications":["Per-layer gradient cost drops from $O(d_l d_{l-1})$ to $O(k d_l)$, so training becomes feasible on hardware where full-matrix gradient computations are too expensive.","The largest activation eigenvalue is penalized during updates, so the top Hessian eigenvalue is driven down and the model converges to flatter regions of the loss landscape.","The same eigenvector machinery extends to few-shot learning: spectral meta learning with a memory buffer of past eigenvectors reportedly beats MAML on 5-shot MedMNIST by 5.5 percentage points.","Spectral architecture optimization prunes weights along eigen-directions and reportedly cuts training time by 35% on CIFAR-10 with ResNet18 while keeping accuracy at 92.8%.","The third-order SDE gives an order-1 weak approximation of the discrete updates, so the dynamics of DSBP can be analyzed and predicted with $O(\\eta)$ error."],"supporting_citations":[{"why":"Provides the flat-minima generalization hypothesis that DSBP's sharpness penalty is designed to exploit.","marker":"[1]"},{"why":"Supplies the idea of regularizing networks through spectral properties of weight and activation matrices.","marker":"[3]"},{"why":"Defines the SAM baseline that DSBP claims to outperform and whose sharpness-aware objective motivates the penalty term.","marker":"[4]"},{"why":"Defines the LoRA baseline used for comparison on fine-tuning and Tiny ImageNet.","marker":"[5]"},{"why":"Defines the MAML baseline that spectral meta learning is compared against on few-shot MedMNIST.","marker":"[6]"},{"why":"Provides the PAC-Bayes framework from which the generalization bound in Theorem 1 is derived.","marker":"[7]"},{"why":"Grounds the third-order SDE derivation by connecting discrete stochastic updates to modified equations.","marker":"[9]"},{"why":"Supports the idea that explicit eigenvalue regularization improves sharpness-aware minimization, which DSBP extends.","marker":"[10]"}],"fun_headline_variants":["Eigenvector gradient projection cuts training cost and sharpness","Project onto top activation eigenvectors to flatten loss landscape","DSBP: faster training by projecting gradients onto principal directions","Spectral gradient projection beats SAM, LoRA, MAML on four benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the top-$k$ eigenvectors of each layer's activation covariance matrix, recomputed only every $p$ updates, span the gradient directions that matter for optimization.","fun_headline_variants_meta":{"raw":{"variants":["Eigenvector gradient projection cuts training cost and sharpness","Project onto top activation eigenvectors to flatten loss landscape","DSBP: faster training by projecting gradients onto principal directions","Spectral gradient projection beats SAM, LoRA, MAML on four benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1302,"prompt_tokens":908,"completion_tokens":394,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":325}},"tokens_in":524,"tokens_out":394,"duration_ms":4731,"temperature":1.0,"reasoning_tokens":325,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:47:57.498392+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a two-layer network on a synthetic regression whose target weights are known, and choose input features so that the informative gradient lies in the bottom eigenspace of the activation covariance; if DSBP's projected updates stall or diverge while full-gradient SGD converges, the top-$k$ projection is discarding essential information. Alternatively, on any benchmark, record $\\|\\nabla f - \\tilde{\\nabla} f\\| / \\|\\nabla f\\|$ across training; if this discarded-gradient norm stays large while accuracy improves, the small-error premise of the SDE analysis is violated.","supporting_citations":[{"cited_title":"Yoshida, T","cited_arxiv_id":null,"evidence_quote":"Supplies the idea of regularizing networks through spectral properties of weight and activation matrices."},{"cited_title":"Foret, A","cited_arxiv_id":null,"evidence_quote":"Defines the SAM baseline that DSBP claims to outperform and whose sharpness-aware objective motivates the penalty term."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the MAML baseline that spectral meta learning is compared against on few-shot MedMNIST."},{"cited_title":"Alquier, J","cited_arxiv_id":null,"evidence_quote":"Provides the PAC-Bayes framework from which the generalization bound in Theorem 1 is derived."},{"cited_title":"Exploiting Data Reduction Principles in Cloud-Based Data Management for Cryo-Image Data","cited_arxiv_id":"1703.10105","evidence_quote":"Grounds the third-order SDE derivation by connecting discrete stochastic updates to modified equations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the idea that explicit eigenvalue regularization improves sharpness-aware minimization, which DSBP extends."}],"review_version":1}