Pith. sign in

REVIEW 2 major objections 34 references

Accuracy and cosine similarity miss silent failures in feedback alignment

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-26 14:39 UTC pith:JFA2WRVN

load-bearing objection The paper shows that accuracy plus aggregate cosine can miss real problems in feedback alignment evaluation, but its new checks are not shown to be exhaustive. the 2 major comments →

arxiv 2606.21126 v1 pith:JFA2WRVN submitted 2026-06-19 cs.LG stat.ML

What Accuracy and Gradient Cosine Miss: Evaluating Feedback Alignment via Scale Stability, Reference Validity, and Depth Utility

Yuren Hao , Xiang Wan , ChengXiang Zhai This is my paper
classification cs.LG stat.ML
keywords feedback alignmentgradient cosine similarityevaluation protocoldeep learningcredit assignmentscale stabilityresidual networks
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that relying on task accuracy and the cosine similarity between a feedback alignment method's credit signals and backpropagation gradients is not enough to confirm the method works. These metrics can remain high even when the method fails to train deep layers due to two independent problems: the backpropagation reference gradient collapsing to near zero in some architectures, and the overall cosine hiding that only one part of the network is getting credit. To fix this, the authors introduce checks for whether gradient scales stay stable, whether the reference is valid, and whether signals reach all depths, plus reporting similarity for each layer separately. Their tests show the old pair never flags problems while the new protocol catches every failure case examined.

Core claim

Task accuracy and aggregate gradient cosine similarity are insufficient to evaluate feedback alignment because they miss measurement degeneracy, where the backpropagation reference gradient collapses numerically in terminal-LayerNorm residual architectures, and aggregation collapse, where layerwise heterogeneity in credit assignment is masked. The proposed protocol using scale stability, reference validity, and depth utility checks with per-layer cosine reporting detects all such failures where the standard pair does not, and the modes are causally independent as shown by targeted interventions.

What carries the argument

Diagnostic protocol of scale stability, reference validity, and depth utility checks combined with per-layer cosine similarity reporting to evaluate feedback alignment methods.

Load-bearing premise

The three checks and per-layer reporting are necessary and sufficient to reveal all relevant failure modes of feedback alignment across different network architectures.

What would settle it

Finding a feedback alignment implementation or architecture where the proposed protocol reports success but the method fails to train deep layers according to other functional tests, or where standard metrics detect an issue missed by the protocol.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The paper claims that the standard evaluation pair for feedback alignment (FA) methods—task accuracy and aggregate cosine similarity to the backpropagation gradient—is insufficient, as it misses two independent silent failure modes: (1) measurement degeneracy, where the BP reference gradient collapses to numerical floor in terminal-LayerNorm residual architectures, and (2) aggregation collapse, where aggregate cosine masks layerwise heterogeneity that concentrates credit at one end of the network. It proposes a new diagnostic protocol using three checks (scale stability, reference validity, depth utility) plus per-layer cosine reporting; across tested architectures the standard pair signals no failure while the protocol detects all with wide margins. The modes are causally independent, as a per-block scale penalty fixes Mode 1 without fixing Mode 2.

Significance. If the protocol reliably surfaces silent failures that accuracy and aggregate cosine miss, the work would improve evaluation standards for biologically plausible and hardware-efficient training methods, reducing the risk of building on non-functional credit assignment. The explicit separation of two independent modes and the demonstration of a partial causal fix constitute a concrete, actionable contribution beyond critique of existing metrics.

major comments (2)
  1. [Abstract] Abstract: the claim that the protocol 'identifies all failures with wide calibration margins' is load-bearing for the central argument that the new checks are jointly sufficient. The manuscript demonstrates detection of the two described modes (reference collapse and aggregation masking) in the audited cases and shows that per-block scaling addresses only one, but provides no theoretical bound or enumeration of the space of other possible degeneracies that could remain invisible to both accuracy and aggregate cosine; the sufficiency assertion therefore rests on an unproven assumption that the chosen diagnostics are exhaustive.
  2. [Experimental results] The experimental section (implied by the abstract's 'across multiple architectures and methods'): the audited cases are presented as representative, yet the selection criteria and coverage of possible FA degeneracies are not specified. This makes it difficult to assess whether the two modes and the protocol's detection power generalize beyond the reported instances.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive report. The two major comments correctly identify that the original manuscript overstates the generality of the proposed protocol. We respond point by point below and will revise the text to remove unsupported claims of exhaustiveness while preserving the core empirical contribution on the two identified failure modes.

read point-by-point responses

  1. Referee: [Abstract] the claim that the protocol 'identifies all failures with wide calibration margins' is load-bearing for the central argument... provides no theoretical bound or enumeration of the space of other possible degeneracies that could remain invisible to both accuracy and aggregate cosine; the sufficiency assertion therefore rests on an unproven assumption that the chosen diagnostics are exhaustive.

    Authors: We agree that the manuscript provides no theoretical bound or enumeration of all conceivable degeneracies that might evade both accuracy and aggregate cosine. The original phrasing therefore overclaims sufficiency. We will revise the abstract to state that the protocol 'detects the two described failure modes (measurement degeneracy and aggregation collapse) with wide margins in the audited architectures,' removing any implication that the checks are jointly exhaustive or that all possible silent failures have been ruled out. This revision will be implemented in the next version. revision: yes

  2. Referee: [Experimental results] the audited cases are presented as representative, yet the selection criteria and coverage of possible FA degeneracies are not specified. This makes it difficult to assess whether the two modes and the protocol's detection power generalize beyond the reported instances.

    Authors: The experiments used standard benchmark architectures (ResNet-style networks with terminal LayerNorm residuals) and representative FA methods to illustrate the failure modes in common settings. However, the manuscript did not articulate explicit selection criteria or discuss the scope of covered degeneracies. We will add a dedicated paragraph in the experimental setup section stating the rationale for the chosen architectures and methods, together with an explicit limitation note that generalization beyond these instances remains to be verified. This addition will clarify the empirical scope. revision: yes

Circularity Check

0 steps flagged

No circularity: new diagnostics are independent empirical checks, not reductions to fitted inputs or self-citations

full rationale

The manuscript introduces three new diagnostic checks (scale stability, reference validity, depth utility) plus per-layer cosine reporting as independent evaluation tools. These are motivated by observed failure modes in experiments but are not derived from or fitted to the same quantities they critique (accuracy and aggregate cosine). No equations, parameter fits, or self-citations appear in the provided text that would make any claim reduce to its own inputs by construction. The sufficiency claim for catching all failures is an empirical assertion open to falsification rather than a definitional or self-referential loop.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No free parameters, mathematical axioms, or invented entities are described in the abstract; the contribution consists of a proposed diagnostic protocol rather than a new model or derivation.

pith-pipeline@v0.9.1-grok · 5809 in / 1072 out tokens · 24741 ms · 2026-06-26T14:39:22.121476+00:00 · methodology

0 comments
read the original abstract

Despite the success of deep learning, training deep networks in biologically plausible and hardware-efficient ways remains an open challenge. Feedback alignment (FA) methods address this by replacing backpropagation's symmetric backward weights with fixed random matrices, but their effectiveness depends critically on whether they can be accurately evaluated. The standard evaluation relies on two quantities: task accuracy and cosine similarity between the method's credit signal and the backpropagation gradient. We show that this reporting pair is insufficient by identifying two independent failure modes, both silent under current reporting: (1) measurement degeneracy, where the BP reference gradient collapses to the numerical floor in terminal-LayerNorm residual architectures, rendering cosine uninterpretable; and (2) aggregation collapse, where the aggregate cosine masks layerwise heterogeneity that concentrates credit at one end of the network. To address these limitations, we propose a diagnostic evaluation protocol based on three checks -- scale stability, reference validity, and depth utility -- together with per-layer rather than aggregate cosine reporting. Across multiple architectures and methods, the standard reporting pair gives no signal of failure in any audited case, while our protocol identifies all failures with wide calibration margins. The two failure modes are causally independent: a per-block scale penalty alleviates Mode 1 (residual scale explosion driving reference collapse) without affecting Mode 2 (cosine ranking that contradicts every functional metric we measured). Identifying these silent failures prevents researchers from building on non-functional credit assignment and provides actionable guidance for developing FA methods that genuinely train deep layers.

Figures

Figures reproduced from arXiv: 2606.21126 by ChengXiang Zhai, Xiang Wan, Yuren Hao.

Figure 1
Figure 1. Figure 1: Standard reporting pair on the representa [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Backward architectures Backpropagation (BP) computes the exact gradient via the chain rule: al = ∂L/∂hl . The two FA variants replace BP’s exact backward pass with progressively simpler credit paths ( [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Per-layer state on the standard primary-audit setting (ResMLP [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Temporal evolution of ∥hL∥, ∥gL∥, and test accuracy on ResMLP d = 256, L = 4, terminal LN. 4.1 Mode 1: Residual scale explosion drives reference collapse In Mode 1, the BP reference gradient at deep layers is compressed below the numerical floor used by the cosine implementation, so the cosine reported by the standard pair is computed against a normalized floating-point residual rather than against the BP … view at source ↗
Figure 5
Figure 5. Figure 5: Temporal evolution of ∥hL∥, ∥gL∥, and test accuracy across three architectures: ViT-Mini (terminal LN), ResMLP no-LN, and StudentNet (no LN). ∥gL∥ y-axis ranges differ across rows. 0 5 10 15 20 25 30 Epoch 10 0 10 1 10 2 10 3 10 4 10 5 10 6 10 7 10 8 10 9 hL 2 hL under penalty = 0 = 10 4 = 10 2 DFA = 0 DFA = 10 4 DFA = 10 2 Fresh-B null BP reference 0.0 0.2 0.4 0.6 0.8 1.0 Deep cosine Deep cosine to BP gra… view at source ↗
Figure 6
Figure 6. Figure 6: Penalty intervention on DFA, ResMLP d = 256, L = 4, three seeds, 30 epochs. Left: ∥hL∥ under λ ∈ {0, 10−4 , 10−2}. Middle: deep cosine across conditions. Right: test accuracy against frozen-blocks baseline. We sweep λ ∈ {0, 10−4 , 10−2} on DFA, ResMLP d = 256, L = 4, three seeds, 30 epochs ( [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Depth-utility ladder: CIFAR-10 test accuracy versus the number of trainable blocks [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 8 canonical work pages · 2 internal anchors

  1. [1]

    Nature Communications , volume =

    Random synaptic feedback weights support error backpropagation for deep learning , author =. Nature Communications , volume =. 2016 , doi =

    2016

  2. [2]

    The Forward-Forward Algorithm: Some Preliminary Investigations

    The Forward-Forward Algorithm: Some Preliminary Investigations , author =. arXiv preprint arXiv:2212.13345 , year =

    work page Pith review arXiv

  3. [3]

    International Conference on Machine Learning (ICML) , year =

    Error-driven Input Modulation: Solving the Credit Assignment Problem without a Backward Pass , author =. International Conference on Machine Learning (ICML) , year =

  4. [4]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Direct Feedback Alignment Provides Learning in Deep Neural Networks , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  5. [5]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  6. [6]

    2019 , eprint =

    Principled Training of Neural Networks with Direct Feedback Alignment , author =. 2019 , eprint =

    2019

  7. [7]

    International Conference on Machine Learning (ICML) , year =

    Align, then Memorise: The Dynamics of Learning with Feedback Alignment , author =. International Conference on Machine Learning (ICML) , year =

  8. [8]

    2009 , institution =

    Learning Multiple Layers of Features from Tiny Images , author =. 2009 , institution =

    2009

  9. [9]

    Decoupled Weight Decay Regularization

    Decoupled Weight Decay Regularization , author =. International Conference on Learning Representations (ICLR) , year =. 1711.05101 , archivePrefix =

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    IEEE Transactions on Pattern Analysis and Machine Intelligence , year =

    ResMLP: Feedforward Networks for Image Classification With Data-Efficient Training , author =. IEEE Transactions on Pattern Analysis and Machine Intelligence , year =. doi:10.1109/TPAMI.2022.3206148 , note =

    work page doi:10.1109/tpami.2022.3206148 2022

  11. [11]

    International Conference on Machine Learning (ICML) , year =

    On Layer Normalization in the Transformer Architecture , author =. International Conference on Machine Learning (ICML) , year =

  12. [12]

    2016 , eprint =

    Layer Normalization , author =. 2016 , eprint =

    2016

  13. [13]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  14. [14]

    2018 , eprint =

    Feedback alignment in deep convolutional networks , author =. 2018 , eprint =

    2018

  15. [15]

    Frontiers in Neuroscience , year =

    Direct Feedback Alignment With Sparse Connections for Local Learning , author =. Frontiers in Neuroscience , year =. doi:10.3389/fnins.2019.00525 , note =

    work page doi:10.3389/fnins.2019.00525 2019

  16. [16]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Deep Learning without Weight Transport , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  17. [17]

    International Conference on Learning Representations (ICLR) , year =

    Biologically-Plausible Learning Algorithms Can Scale to Large Datasets , author =. International Conference on Learning Representations (ICLR) , year =

  18. [18]

    Neural Computation , year =

    An Approximation of the Error Backpropagation Algorithm in a Predictive Coding Network with Local Hebbian Synaptic Plasticity , author =. Neural Computation , year =

  19. [19]

    International Conference on Machine Learning (ICML) , year =

    Greedy Layerwise Learning Can Scale to ImageNet , author =. International Conference on Machine Learning (ICML) , year =

  20. [20]

    International Conference on Learning Representations (ICLR) , year =

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , author =. International Conference on Learning Representations (ICLR) , year =

  21. [21]

    Frontiers in Neuroscience , year =

    Learning Without Feedback: Fixed Random Learning Signals Allow for Feedforward Training of Deep Neural Networks , author =. Frontiers in Neuroscience , year =

  22. [22]

    Frontiers in Computational Neuroscience , volume =

    Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation , author =. Frontiers in Computational Neuroscience , volume =. 2017 , doi =

    2017

  23. [23]

    Difference Target Propagation

    Difference Target Propagation , author =. European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML/PKDD) , year =. doi:10.1007/978-3-319-23528-8_31 , note =

    work page doi:10.1007/978-3-319-23528-8_31

  24. [24]

    AAAI Conference on Artificial Intelligence , year =

    How Important Is Weight Symmetry in Backpropagation? , author =. AAAI Conference on Artificial Intelligence , year =. doi:10.1609/aaai.v30i1.10279 , note =

    work page doi:10.1609/aaai.v30i1.10279

  25. [25]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    A Theoretical Framework for Target Propagation , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  26. [26]

    International Conference on Machine Learning (ICML) , year =

    Implicit Regularization in Feedback Alignment Learning Mechanisms for Neural Networks , author =. International Conference on Machine Learning (ICML) , year =

  27. [27]

    Proceedings of the National Academy of Sciences (PNAS) , year =

    Streamlined Optical Training of Large-Scale Modern Deep Learning Architectures with Direct Feedback Alignment , author =. Proceedings of the National Academy of Sciences (PNAS) , year =. doi:10.1073/pnas.2532022123 , note =

    work page doi:10.1073/pnas.2532022123

  28. [28]

    International Conference on Learning Representations (ICLR) , year =

    Scaling Direct Feedback Learning with Jacobian Alignment Guarantees , author =. International Conference on Learning Representations (ICLR) , year =

  29. [29]

    Toward Practical Equilibrium Propagation: Brain-inspired Recurrent Neural Network with Feedback Regulation and Residual Connections

    Toward Practical Equilibrium Propagation: Brain-inspired Recurrent Neural Network with Feedback Regulation and Residual Connections , author =. arXiv preprint arXiv:2508.11659 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  30. [30]

    2026 , eprint =

    Overcoming Rank Collapse in Feedback Alignment , author =. 2026 , eprint =

    2026

  31. [31]

    IEEE Journal of Solid-State Circuits , year =

    A 65-nm Neuromorphic Image Classification Processor With Energy-Efficient Training Through Direct Spike-Only Feedback , author =. IEEE Journal of Solid-State Circuits , year =

  32. [32]

    Scaling Learning Algorithms Towards

    Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards

  33. [33]

    and Osindero, Simon and Teh, Yee Whye , journal =

    Hinton, Geoffrey E. and Osindero, Simon and Teh, Yee Whye , journal =. A Fast Learning Algorithm for Deep Belief Nets , volume =

  34. [34]

    2016 , publisher=

    Deep learning , author=. 2016 , publisher=

    2016