Pith. sign in

REVIEW 3 major objections 5 minor 47 references

Conditioned Direct Feedback Alignment via Activity and Error Geometry

T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Direct feedback alignment can fail through the conditioning of its local outer-product update, not just feedback quality; damped inverse-second-moment preconditioning of either factor yields large, seed-replicated gains.

desk verdict A genuinely new factor-level decomposition of DFA conditioning with honest boundaries; the activity-side gain replicates, but the nuisance-dominance mechanism is unvalidated on real tasks and the theory doesn't cover the alignment-rescue experiments. read the letter →

arxiv 2607.18574 v1 pith:JNBO3RZK submitted 2026-07-20 cs.LG cs.NEq-bio.NC

Houman Safaai , Varun Reddy , Bernardo L. Sabatini This is my paper · ORCID
classification cs.LGcs.NEq-bio.NC MSC 68T07
keywords directfeedbackalignmentlocallearningrulessecond-momentpreconditioningouter-productanisotropyactivityconditioningerrornuisancedirectionsspectralidentity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Direct feedback alignment (DFA) trains hidden layers with fixed random projections of the output error, avoiding backpropagation's transposed-weight pass, but it is brittle in practice. This paper identifies a failure mode distinct from feedback quality: the local DFA update is an outer product of a presynaptic activity vector and a local error vector, so anisotropy in either factor can dominate the update even when the feedback direction itself is useful. The proposed correction is a symmetric family of normalized DFA rules that multiply the update on the right by the damped inverse of the activity second moment, on the left by the damped inverse of the local-error second moment, or both; in an aligned linear-Gaussian model, the right factor provably replaces each eigendirection's gain λ_i by λ_i/(λ_i + λ_A), flattening the input-side condition number. Empirically, activity conditioning yields roughly 40-percentage-point gains over raw DFA in controlled nuisance-dominant regimes, while error conditioning adds 1.77–7.53 points and the two-sided rule adds 0.40–0.90 points, with signs replicating across fresh seeds on clean MNIST and Fashion-MNIST. The paper is deliberately scoped: it offers a factor-level diagnosis of when outer-product local rules fail, not a general replacement for backpropagation, and it openly reports that its prospective predictor of nuisance energy failed on held-out vision and that the error factor is fragile.

What carries the argument

The load-bearing object is the damped inverse second moment of each factor of the DFA outer-product update. With presynaptic activity h and local DFA error δ, the raw step is G = δh^T; activity nDFA right-multiplies by P_A = (C_A + λ_A I)^{-1}, error nDFA left-multiplies by P_E = (C_E + λ_E I)^{-1}, and K-nDFA applies both, where C_A = E[hh^T] and C_E = E[δδ^T] are minibatch uncentered second moments with separately tuned damping. Its work is carried by Proposition 1: in the aligned linear-Gaussian population model, the right preconditioner replaces each input eigendirection's gain λ_i by λ_i/(λ_i + λ_A), so the input-side condition number drops from κ(Σ) to κ(Σ)(λ_min + λ_A)/(λ_max + λ_A).

What would settle it

Compute the paper's prospective nuisance-energy ratio b_r from an untrained forward pass and class-conditional mean activations on a fresh vision task, and correlate it with realized nDFA-minus-DFA test-accuracy gains; the regime-dependence claim predicts a positive Spearman correlation, while the paper's own held-out vision measurement found ρ = −0.61. Alternatively, demonstrate a large (>10-point) nDFA gain over raw DFA on a task where the high-variance activity directions provably carry the label — either observation would settle whether the transfer holds.

Watch

Extended reading notes

Core claim

The paper's central claim is that a major, separable failure mode of DFA is the conditioning of the update's two factors. In the aligned linear-Gaussian population model, Proposition 1 gives an exact spectral identity: damped inverse-second-moment conditioning on the input side replaces the per-eigendirection gain λ_i by λ_i/(λ_i + λ_A), reducing the input-side condition number from κ(Σ) to κ(Σ)(λ_min+λ_A)/(λ_max+λ_A). The realized benefit is regime-dependent — large when high-variance directions carry task-irrelevant nuisance, small under isotropy or when the task lives in high-variance directions — and the synthetic stress suites confirm that dependence with an approximately 40-point activ

Load-bearing premise

The empirical case transfers a linearized, post-alignment spectral identity to nonlinear, pre-alignment, finite-sample DFA training, and in particular assumes that high-variance activity directions in real tasks are predominantly task-irrelevant nuisance — an assumption the paper's own prospective pre-training estimator failed to confirm on held-out vision (Spearman ρ = −0.61).

Editorial extensions

If this is right

  • DFA brittleness splits into two addressable problems — feedback quality and update conditioning — and the latter can be fixed locally, without backpropagated errors, once minibatch-wide second moments are available.
  • Activity-side preconditioning recovers roughly 40 percentage points over raw DFA when high-variance activity directions are nuisance-dominated, while conferring little advantage over a tuned exact-gradient baseline when task directions already carry most of the variance.
  • Error-side conditioning is a smaller but independent benefit (1.77–7.53 points) that requires substantially heavier damping, and the two-sided rule adds 0.40–0.90 points without joint tuning, provided per-example error second moments are correctly normalized.
  • Conditioning collapses sensitivity to the random feedback draw — the median feedback-seed standard deviation of final accuracy drops roughly sixfold — which matters for neuromorphic or photonic settings where the feedback matrix is fixed at fabrication.
  • The gain is not a scalar step-size or norm-matching effect: after layerwise gradient-norm matching, nDFA still improves over norm-matched DFA by 4–15 points in hard cells, and applying the same input preconditioner to exact BP reproduces the nuisance-regime gain, locating the mechanism in activity geometry rather than in DFA's error pathway.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper's own pre-training estimator of nuisance energy reversed sign on held-out vision (Spearman ρ = −0.61), a skeptical reader should treat the practical payoff as unvalidated: the controlled-regime mechanism is the established claim, while a deployable rule for deciding when to condition remains open.
  • The spectral identity λ_i → λ_i/(λ_i + λ_A) does not depend on the error being produced by a random feedback projection, so the same regime-dependent benefit should appear in any outer-product local rule (three-factor Hebbian updates, perturbation-based credit) — a directly testable extension the paper does not run.
  • The finding that right-preconditioning lifts exact BP by +18.3 points in the nuisance-dominant cell suggests the activity-side mechanism is a property of preconditioned gradient descent generally; a natural next experiment is error-side conditioning of exact BP under class imbalance or gating, which would separate the conditioning story from DFA-specific error routing.
  • The negative ReLU Fashion-MNIST pilot and partial convnet results delimit the two-sided rule's generality; extending K-nDFA beyond tanh/MNIST-class settings is the immediate open problem, and the paper's protocol (independent damping selection, frozen test evaluation, seed-level sign consistency, correction of mis-scaled sweeps) is a reusable template for such claims.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper studies a failure mode of direct feedback alignment that is distinct from feedback quality: the local DFA update is an outer product, so anisotropy can enter through either its presynaptic-activity factor or its local-error factor. The paper proposes a conditioned-DFA family: activity nDFA right-preconditions by (C_A + λ_A I)^{-1}, error nDFA left-preconditions by (C_E + λ_E I)^{-1}, and K-nDFA applies both with separately tuned damping. The theoretical centerpiece is Proposition 1 (Appendix A): in an aligned linear-Gaussian model, nDFA replaces the per-eigendirection input factor λ_i by λ_i/(λ_i+λ_A), reducing the input-side condition number and yielding a falsifiable prediction that gains should track nuisance loading of high-variance directions. Empirically, on a synthetic 128-cell stress suite with fixed full-rank feedback, activity nDFA improves raw DFA by roughly 22–40 pp in nuisance-stressed regimes; clean three-hidden-layer confirmations on MNIST/Fashion-MNIST give error-nDFA gains of 1.77–4.76 pp and K-nDFA increments of 0.40–0.90 pp, with signs replicated on eight fresh seeds in a ReLU/softmax MNIST model (p=0.0078). The paper runs extensive controls (BP-norm matching, decorrelation, Adam, BatchNorm, BP-preconditioning) and explicitly scopes its claims: convnet gains are partial; all-layer convolutional credit assignment is unsolved; its own prospective nuisance/task-energy diagnostic fails on held-out vision (Table 6, ρ=-0.61); and the alignment-rescue e

Significance. Methodologically, the paper sets a high bar for the local-learning literature: Proposition 1 is a clean derivation with explicit assumptions; the confirmations are preregistered with independently validated damping choices; the excluded historical runs and the withdrawn source-swap comparator are disclosed; and the controls (especially BP-preconditioning and norm-matching) distinguish anisotropic reweighting from scalar step-size effects. If the results hold, the factor-level decomposition is a genuine contribution to understanding when local outer-product rules fail. The paper's scoped honesty is itself an asset: it does not claim a universal BP replacement. The main gap is that the mechanism underlying the headline activity gains is not the mechanism analyzed: the theory is post-alignment, while the large gains occur where raw DFA anti-aligns and conditioning rescues alignment acquisition (Appendix E). The paper's own prospective test of the regime-dependence hypothesis fails on real data (Table 6). These limitations are honestly disclosed, but they are load-bearing for the central attribution claims, so the paper is a strong controlled study with a scoped theory rather than an e

major comments (3)
  1. [§2.1, Proposition 1 vs. Appendix E] The paper's strongest activity gains occur in the nuisance-dominant synthetic cells and the clean DFA-stall confirmations, where Appendix E reports raw DFA anti-aligns with its feedback ('15/15 runs') and nDFA 'rescues the alignment phase itself'—behavior the text explicitly says 'the post-alignment theory does not explain' (§4.2, Appendix E). Proposition 1 is stated in the post-alignment regime and is an input-side spectral identity; as the paper notes, it does not model feedback-alignment acquisition. The central mechanism attribution for the headline results is therefore unsupported by the paper's own theory. Either reframe the contribution as the discovery of an alignment-rescue effect, with Proposition 1 restricted to the regime where it applies, or add a diagnostic that tests the spectral factor's contribution during the phase where the gain actually occurs.
  2. [Table 6] The paper's sole prospective test of its core regime-dependence hypothesis fails: the nuisance/task-energy estimator ranks realized held-out vision gains backwards (Spearman ρ=-0.61; −0.85 within CIFAR-10), is no better than the task-blind κ(C) baseline (0.54 vs 0.59 on the 128-cell grid), and is beaten by an 'always helps' classifier (LORO 0.80 vs 0.92 base rate). This is the manuscript's own evidence, and the honest framing ('We report this as a negative result') is a strength, but it follows that the abstract's claim about 'task-irrelevant nuisance' as the operative condition is validated only in designed synthetic regimes. The practical-relevance claim should be proportionately weakened in the abstract and introduction, not merely in the appendix.
  3. [§4.2 / Appendix D] The incremental error/K-nDFA claims rest on delicate statistics. The tanh MNIST and Fashion-MNIST confirmations average three feedback seeds within each of only n=5 model/data-order seeds, so the two-sided Wilcoxon floor is p=0.0625; the MNIST error damping λE=10 lies at the upper grid boundary with validation still increasing, and the Fashion-MNIST activity damping λA=0.03 lies at the lower boundary. The K-nDFA increments are 0.40–0.90 pp. Only the ReLU/softmax row (n=8, p=0.0078) has conventional significance, and it is confined to MNIST. Before the 'two-sided gain' is presented as established, the paper should provide a minimal-detectable-effect analysis at n=5, an additional replication, or a downgrade of these increments to descriptive findings.
minor comments (5)
  1. [Table 1 / Abstract] The 'approximately 40-percentage-point' figure should carry the same caveat as Table 1: fixed full-rank feedback, synthetic cells, with 32 cells of each regime sharing only five data seeds. The caption explains this, but the abstract does not.
  2. [§4.1] The normalization correction and exclusion of archived mis-scaled results are exemplary, but at first mention of 'archived' results in §4.1 the reader cannot tell which numbers are excluded. Consider a consistent symbol for excluded historical results.
  3. [Fig. 5A] The marker 'open: η = 0.01 (not LR-matched)' is cryptic; state in the caption which entries use a separately selected learning rate and what 'open' signifies.
  4. [Fig. 6B / Appendix A.2] The minibatch-SGD agreement is a self-consistency check by construction of the simulator. Label it as such so it is not read as an independent empirical confirmation.
  5. [Table 4] Raw DFA's test loss of 187.339±95.506 is orders of magnitude above the other methods; the footnote explains it, but reporting a median or invoking a robust loss statistic would be clearer.

Circularity Check

0 steps flagged · score 1.0 of 10

No load-bearing circularity; the central spectral identity is definitional but explicitly scoped, and the empirical claims rest on external confirmations.

full rationale

The paper's derivation chain is not circular in any load-bearing sense. Proposition 1 (Appendix A) states that in the aligned linear-Gaussian model nDFA replaces per-eigendirection λ_i by λ_i/(λ_i+λ_A); this is an algebraic restatement of the update rule g_nDFA = -B ε Σ(Σ + λ_A I)^-1. However, the paper explicitly presents it as motivation ('we use it to motivate, rather than prove, a signed empirical hypothesis') and does not use the identity as evidence for the empirical gains. The headline empirical comparisons are made against raw DFA and tuned BP on designed synthetic regimes and on clean MNIST/Fashion-MNIST/ReLU confirmations, with dampings selected on fixed validation splits, fresh model/data-order seeds, and no joint tuning of K-nDFA; those are external to the identity. The only prospective test of the mechanism, the nuisance-energy estimator of Table 6, failed on held-out vision (Spearman ρ = -0.61), which is a disclosed negative result and is the opposite of a fitted prediction being relabeled as confirmation. Self-citations are not load-bearing: the NMNC baseline by Kang et al. (2026) is a comparison method, not the argument for nDFA. No fitted parameter is renamed as a prediction. Minor presentational overhang from calling the algebraic identity a 'spectral prediction' is not circularity under the stated rules.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim is not parameter-free: λA and λE are selected on validation splits, and the synthetic regime scales are hand-chosen to exhibit the effect. The theoretical derivation uses a linear-Gaussian aligned model with population moments; the empirical gains rely on designed nuisance conditions and validation-selected damping. No new physical or mathematical entities are introduced.

free parameters (3)
  • activity damping λA = 0.3 (MNIST tanh), 0.03 (Fashion-MNIST), 3 (ReLU/softmax), 0.3 (synthetic suite)
    Selected on fixed validation splits; central to the preconditioner; some choices lie at grid boundaries (Appendix D).
  • error damping λE = 10 (MNIST tanh, upper boundary), 30 (Fashion-MNIST), 0.1 (ReLU/softmax)
    Selected independently on validation; the error factor is fragile and requires heavy damping; MNIST choice not fully localized.
  • synthetic regime scales (s_task, s_nuis) = nuisance-dominant (0.45, 2.0); low-sample (0.7, 1.5); mixed (0.75, 1.2); task-aligned (1.3, 0.25)
    Hand-chosen designed conditions; the headline ~40pp gain is measured in the nuisance-dominant regime constructed to match the hypothesis.
assumptions (5)
  • domain assumption DFA update is δ h^T with fixed random feedback B (Eq. 1).
    The paper's entire object of study is this update; no derivation from biology or optimization theory.
  • domain assumption Post-alignment linear-Gaussian regime: B = α (W^(2))^T, x ~ N(0, Σ), population second moments (Proposition 1).
    The spectral identity is exact only in this idealized aligned regime; transfer to nonlinear pre-aligned training is assumed.
  • ad hoc to paper Matrix-normal/KFAC factorization C_E ⊗ C_A approximates the outer-product covariance.
    Motivates K-nDFA; the paper explicitly states it is not an exact Fisher identity for DFA because C_E depends on random feedback, gating, and current residuals.
  • domain assumption Same-sample covariance at λ=0 cancels exactly in the empirical update (Appendix A.1).
    Justifies using an empirical minibatch covariance and treating damping as an empirical rule parameter rather than a theoretically derived optimum.
  • ad hoc to paper High-variance presynaptic directions are task-irrelevant nuisance in the regimes of interest.
    The core empirical hypothesis; the prospective estimator of this premise failed on held-out vision (Table 6), leaving its real-world validity open.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Conditioned Direct Feedback Alignment via Activity and Error Geometry." pith.science (2026). https://pith.science/paper/JNBO3RZK

@misc{pith2026260718574,
  author       = {Pith},
  title        = {Pith review of: Conditioned Direct Feedback Alignment via Activity and Error Geometry},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JNBO3RZK}},
  note         = {Machine review of arXiv:2607.18574}
}
read the original abstract

Direct feedback alignment (DFA) trains hidden layers with fixed random projections of the output error, avoiding the transposed-weight backward pass of backpropagation (BP). We study a failure mode of DFA training that is distinct from feedback quality: the local weight update is calculated by an outer product, so anisotropy can enter through either its presynaptic-activity factor or its local-error factor. Our analyses with controlled synthetic regimes isolate the first failure mode and show an approximately 40-percentage-point activity-conditioning gain when high-variance directions contain task-irrelevant nuisance. Three clean confirmations isolate a different regime: error conditioning improves raw DFA by 1.77--7.53 percentage points, and combining independently selected activity and error factors adds 0.40--0.90 points over activity conditioning. The signs hold for tanh/one-vs-rest MNIST and preregistered Fashion-MNIST, and replicate on eight fresh seeds in a ReLU/softmax MNIST model. This factorization yields a symmetric block-local family of normalized DFA (nDFA): activity nDFA right-preconditions by an inverse activity second moment, error nDFA left-preconditions by an inverse local-error second moment, and K-nDFA applies both factors with separately tuned damping. A linearized post-alignment calculation gives an exact input-side spectral identity and a Kronecker-factor motivation for the two-sided rule, whereas norm matching rules out a scalar step-size explanation. The error factor is fragile when under-damped, BatchNorm is a strong activity-side alternative, and convnet gains remain partial. We therefore frame conditioned DFA as a factor-level study of when local outer-product rules fail, not as a general replacement for BP or a solution to all-layer convolutional credit assignment.

Figures

Figures reproduced from arXiv: 2607.18574 by the authors.

Figure 1
Figure 1. Inverse-second-moment preconditioning helps most when anisotropy is nuisance. A, When the task lies along the low-variance axis of the activity second moment, the raw update (weighted by C) points mostly along the high-variance nuisance axis; conditioning rebalances it toward the task. B–C, In the aligned linear model, BP/DFA scale eigendirections by λi , nDFA flattens the spectrum by λi/(λi + λA), and the condition… view at source ↗
Figure 2
Figure 2. Conditioned DFA and the activity-side regime. A–C, Fixed-full-rank nDFA separates from raw DFA in the three stressed regimes; stars mark separately tuned per-regime BP endpoints. Curves average the fixed grid of cells and are descriptive, not independent cell-level confidence bands. D, The activity-side endpoint effect in focused 100-epoch controls. selects λA=3 and λE=0.1; the protocol was frozen before test evalua… view at source ↗
Figure 3
Figure 3. Error-side and two-sided replication. A, Error nDFA improves raw DFA in both tanh confirmations and the fresh-seed ReLU/softmax confirmation. B, Adding the error factor after activity conditioning improves all seed-level pairs in all three settings. Dots are model/data-order seeds after averaging three feedback seeds; large points are mean ± SEM. C, Validation dynamics in the frozen ReLU/softmax confirmation; bands … view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Main controls (hard synthetic cells; gains in pp over the stated baseline). A, The factor gains survive BP-norm matching, so the improvement is not a scalar learning-rate shift. B, The nDFA gain over raw DFA grows with the empirical nuisance/task energy ratio (diamonds…
Figure 5
Figure 5. Figure 5: Convolutional diagnostics. A, ImageNet-100 pretrained fine-tuning with raw block-DFA and distinct block-output whitening controls; none implements Eq. 3, and full ZCA uses a separately selected learning rate at the two deepest settings. B, The kernel-patch spatial cond…
Figure 6
Figure 6. Figure 6: Mode-timing illustration (Proposition 2). A, Per-mode fitting curves in the aligned linear model (d=32, κ=50): raw DFA/BP fit the high-variance nuisance mode κ=50× earlier than the low-variance task mode; conditioning compresses the gap to ρc=5.5 at λA=0.1 while preser…
Figure 7
Figure 7. Figure 7: separates the three cases in Eq. 3. Development uses model/data-order seeds 42–44 and one feedback seed. Activity and error damping are selected independently from {0.03, 0.1, 0.3, 1, 3, 10} on a fixed 5,000-example split of the MNIST training set. The selected values …
Figure 8
Figure 8. Figure 8: Preregistered clean Fashion-MNIST replication. A–B, Independent validation-only damping sweeps; whiskers are SEM over three development seeds and dashed lines mark the frozen choices. C, Test confirmation after averaging three feedback seeds within each of five fresh m…
Figure 9
Figure 9. Figure 9: ReLU/softmax activity, error, and K-nDFA confirmation. A–B, Independent validation-only damping sweeps; whiskers are SEM over three development seeds and dashed lines mark the frozen values. C, Test endpoints for eight fresh model/data-order seeds after averaging three…
Figure 10
Figure 10. Figure 10: Feedback-draw variance collapse. A, Per-cell feedback-seed standard deviation of final accuracy, raw DFA versus nDFA (log-log; diagonal = no change), colored by regime; vision and Mixer cells are filled diamonds/triangles, and cells where raw DFA sits at the chance fl…
Figure 11
Figure 11. Figure 11: Exploratory kernel-patch nuisance and damping sweeps. A, D = acc(spatial-Kron) − acc(channel-only nDFA) rises monotonically with the amplitude α of an added class-independent low-frequency spatial nuisance on both the CIFAR-10 and CIFAR-100 convnets (identical archite…
Figure 12
Figure 12. Figure 12: Projected BP-step diagnostic. A, Regime-level projected-step means for DFA and nDFA. B, Matched nDFA accuracy gains and projected steps. C–D, Rank-restricted feedback is a historical sensitivity analysis; Π is a viability diagnostic rather than an objective to maximiz…
Figure 13
Figure 13. Figure 13 [PITH_FULL_IMAGE:figures/full_fig_p035_13.png]
Figure 14
Figure 14. Figure 14: Learning-speed gains over DFA. Activity nDFA gains appear early in the strongest synthetic regimes and remain positive. Vision and CIFAR curves are exploratory for the protocol reasons in [PITH_FULL_IMAGE:figures/full_fig_p036_14.png]
Figure 15
Figure 15. Figure 15: Exploratory vision learning dynamics. A–B, On noisy-label Fashion-MNIST and CIFAR-10 MLPs, nDFA closes most of the raw-DFA gap during training. C, On the harder CIFAR-100 convnet, nDFA learns above raw DFA but remains below BP and local auxiliary losses. Bands summari…
Figure 16
Figure 16. Figure 16: Descriptive endpoint nDFA gains over raw DFA. Whiskers summarize the archived crossed designs and should not be read as independent-replicate confidence intervals. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_16.png]
Figure 17
Figure 17. Figure 17: ImageNet-100 spatial-routing dynamics. In the matched 90-epoch diagnostic, broadcast block feedback remains strongest among the tested routing choices; none closes the deeper-substitution gap. 38 [PITH_FULL_IMAGE:figures/full_fig_p038_17.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 6 linked inside Pith

  1. [1]

    S., Saxe, A

    Advani, M. S., Saxe, A. M., and Sompolinsky, H. (2020). High-dimensional dynamics of generalization error in neural networks. Neural Networks, 132, 428--446

    2020

  2. [2]

    Ahmad, N. (2024). Correlations are ruining your gradient descent. arXiv:2407.10780

    arXiv 2024

  3. [3]

    Akrout, M., Wilson, C., Humphreys, P., Lillicrap, T., and Tweed, D. (2019). Deep learning without weight transport. In Advances in Neural Information Processing Systems

    2019

  4. [4]

    Amari, S. (1998). Natural gradient works efficiently in learning. Neural Computation, 10(2), 251--276

    1998

  5. [5]

    Amari, S., Karakida, R., and Oizumi, M. (2019). Fisher information and natural gradient learning in random deep networks. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 694--702

    2019

  6. [6]

    Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D. (2019). Invariant risk minimization. arXiv:1907.02893

    arXiv 2019

  7. [7]

    A., Marris, L., Hinton, G

    Bartunov, S., Santoro, A., Richards, B. A., Marris, L., Hinton, G. E., and Lillicrap, T. P. (2018). Assessing the scalability of biologically-motivated deep learning algorithms and architectures. In Advances in Neural Information Processing Systems

    2018

  8. [8]

    Bernacchia, A., Lengyel, M., and Hennequin, G. (2018). Exact natural gradient in deep linear networks and its application to the nonlinear case. In Advances in Neural Information Processing Systems

    2018

Show all 47 references
  1. [9]

    Boeshertz, G., Pascanu, R., and Clopath, C. (2026). Overcoming rank collapse in feedback alignment. arXiv:2606.11123

    2026 arXiv

  2. [10]

    Braun, G., Bao, H., Huang, W., and Imaizumi, M. (2026). Spectral gradient descent mitigates anisotropy-driven misalignment: A case study in phase retrieval. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research

    2026

  3. [11]

    Caillon, P., Fagnou, E., Delattre, B., and Allauzen, A. (2026). Scaling direct feedback learning with Jacobian alignment guarantees. In Proceedings of the International Conference on Learning Representations

    2026

  4. [12]

    and Abbott, L

    Chung, S. and Abbott, L. F. (2021). Neural population geometry: An approach for understanding biological and artificial neural networks. Current Opinion in Neurobiology, 70, 137--144

    2021

  5. [13]

    Y., Lee, D

    Chung, S. Y., Lee, D. D., and Sompolinsky, H. (2018). Classification and geometry of general perceptual manifolds. Physical Review X, 8, 031003

    2018

  6. [14]

    Crafton, B., Parihar, A., Gebhardt, E., and Raychowdhury, A. (2019). Direct feedback alignment with sparse connections for local learning. Frontiers in Neuroscience, 13, 525

    2019

  7. [15]

    Dalm, S., van Gerven, M., and Ahmad, N. (2023). Node perturbation can effectively train multi-layer neural networks. arXiv:2310.00965

    2023 arXiv

  8. [16]

    Dalm, S., Offergeld, J., Ahmad, N., and van Gerven, M. (2024). Efficient deep learning with decorrelated backpropagation. arXiv:2405.02385

    2024

  9. [17]

    and Kreiman, G

    Dellaferrera, G. and Kreiman, G. (2022). Error-driven input modulation: Solving the credit assignment problem without a backward pass. In Proceedings of the 39th International Conference on Machine Learning

    2022

  10. [18]

    Desjardins, G., Simonyan, K., Pascanu, R., and Kavukcuoglu, K. (2015). Natural neural networks. In Advances in Neural Information Processing Systems

    2015

  11. [19]

    R., Fee, M

    Fiete, I. R., Fee, M. S., and Seung, H. S. (2007). Model of birdsong learning based on gradient estimation by dynamic perturbation of neural conductances. Journal of Neurophysiology, 98(4), 2038--2057

    2007

  12. [20]

    Frenkel, C., Lefebvre, M., and Bol, D. (2021). Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks. Frontiers in Neuroscience, 15, 629892

    2021

  13. [21]

    George, T., Laurent, C., Bouthillier, X., Ballas, N., and Vincent, P. (2018). Fast approximate natural gradient descent in a Kronecker-factored eigenbasis. In Advances in Neural Information Processing Systems

    2018

  14. [22]

    and Vempala, S

    Garg, S. and Vempala, S. (2022). How and when random feedback works: A case study of low-rank matrix factorization. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 4070--4108

    2022

  15. [23]

    and Martens, J

    Grosse, R. and Martens, J. (2016). A Kronecker-factored approximate Fisher matrix for convolution layers. In Proceedings of the 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 573--582

    2016

  16. [24]

    Hinton, G. E. (2022). The forward-forward algorithm: Some preliminary investigations. arXiv:2212.13345

    2022 arXiv

  17. [25]

    Huang, L., Yang, D., Lang, B., and Deng, J. (2018). Decorrelated batch normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    2018

  18. [26]

    Huang, L., Zhou, Y., Zhu, F., Liu, L., and Shao, L. (2019). Iterative normalization: Beyond standardization towards efficient whitening. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    2019

  19. [27]

    Kang, B., Richards, M., and Sabatini, B. (2026). Credit assignment via neural manifold noise correlation. arXiv:2601.02636

    2026

  20. [28]

    Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations

    2015

  21. [29]

    Kolen, J. F. and Pollack, J. B. (1994). Backpropagation without weight transport. In Proceedings of the IEEE International Conference on Neural Networks, volume 3, pages 1375--1380

    1994

  22. [30]

    and Wei, X.-X

    Kriegeskorte, N. and Wei, X.-X. (2021). Neural tuning and representational geometry. Nature Reviews Neuroscience, 22, 703--718

    2021

  23. [31]

    Launay, J., Poli, I., Boniface, F., and Krzakala, F. (2020). Direct feedback alignment scales to modern deep learning tasks and architectures. In Advances in Neural Information Processing Systems

    2020

  24. [32]

    Lee, D.-H., Zhang, S., Fischer, A., and Bengio, Y. (2015). Difference target propagation. In Machine Learning and Knowledge Discovery in Databases

    2015

  25. [33]

    and Lee, S

    Lee, Y. and Lee, S. (2026). Enabling fine-tuning of direct feedback alignment via feedback-weight matching. In Proceedings of the International Conference on Learning Representations

    2026

  26. [34]

    P., Cownden, D., Tweed, D

    Lillicrap, T. P., Cownden, D., Tweed, D. B., and Akerman, C. J. (2016). Random synaptic feedback weights support error backpropagation for deep learning. Nature Communications, 7, 13276

    2016

  27. [35]

    and Grosse, R

    Martens, J. and Grosse, R. (2015). Optimizing neural networks with Kronecker-factored approximate curvature. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2408--2417

    2015

  28. [36]

    H., Litwin-Kumar, A., and Abbott, L

    Moskovitz, T. H., Litwin-Kumar, A., and Abbott, L. F. (2018). Feedback alignment in deep convolutional networks. arXiv:1812.06488

    2018 arXiv

  29. [37]

    N kland, A. (2016). Direct feedback alignment provides learning in deep neural networks. In Advances in Neural Information Processing Systems

    2016

  30. [38]

    and Eidnes, L

    N kland, A. and Eidnes, L. H. (2019). Training neural networks with local error signals. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 4839--4850

    2019

  31. [39]

    and Latham, P

    Pogodin, R. and Latham, P. E. (2020). Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks. Advances in Neural Information Processing Systems

    2020

  32. [40]

    Refinetti, M., d'Ascoli, S., Ohana, R., and Goldt, S. (2021). Align, then memorise: The dynamics of learning with feedback alignment. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8925--8935

    2021

  33. [41]

    P., Das Biswas, S., and Roy, K

    Roy, A., Apolinario, M. P., Das Biswas, S., and Roy, K. (2026). Feedback alignment meets low-rank manifolds: A structured recipe for local learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2984--2992

    2026

  34. [42]

    and Bengio, Y

    Scellier, B. and Bengio, Y. (2017). Equilibrium propagation: Bridging the gap between energy-based models and backpropagation. Frontiers in Computational Neuroscience, 11, 24

    2017

  35. [43]

    Vershynin, R. (2018). High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge University Press

    2018

  36. [44]

    Wang, Z., M \"u ller, K., Filipovich, M., Launay, J., Ohana, R., Pariente, G., Mokaadi, S., Brossollet, C., Moreau, F., Cappelli, A., Poli, I., Carron, I., Daudet, L., Krzakala, F., and Gigan, S. (2026). Streamlined optical training of large-scale modern deep learning architec...

    2026

  37. [45]

    Whittington, J. C. R. and Bogacz, R. (2017). An approximation of the error backpropagation algorithm in a predictive coding network with local Hebbian synaptic plasticity. Neural Computation, 29(5), 1229--1262

    2017

  38. [46]

    Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8, 229--256

    1992

  39. [47]

    and Nitanda, A

    Yoshida, K. and Nitanda, A. (2025). How does preconditioning guide feature learning in deep neural networks? arXiv:2509.25637

    2025

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.