Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

One-Time Soft Alignment Enables Resilient Learning without Weight Transport

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single soft alignment of forward and feedback weights at initialization lets deep networks train with fixed random feedback to backpropagation-level accuracy, with no weight transport during learning.

desk verdict A useful, well-controlled study of a simple initialization trick for feedback alignment, but the abstract overstates deep-network parity; the core results hold for shallow MLPs. read the letter →

arxiv 2505.20892 v1 pith:IKST5AYP submitted 2025-05-27 cs.LG

Jeonghwan Cheon , Jaehyuk Bae , Se-Bum Paik This is my paper
classification cs.LG
keywords initialfeedbackalignmentweighttransportproblembiologicallyplausiblelearninglosslandscapeHessianspectralanalysisadversarialrobustnessneuralnetworkinitialization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Backpropagation's need to copy forward weights into the backward pass at every step is both an energy cost and a biological implausibility, known as the weight transport problem. This paper claims that a single, approximate alignment of forward and feedback weights at initialization removes that need: networks then train with fixed random feedback, never re-synchronizing, and still reach test accuracy statistically indistinguishable from backpropagation on CIFAR-10. The broader point is that what makes feedback alignment unstable is largely a bad starting configuration, and a one-time fix suffices. If the claim holds, biologically plausible learning and neuromorphic hardware get a minimal recipe: mirror the weights once at startup, then let learning run with local signals.

What carries the argument

The load-bearing object is the initialization rule $W_l = B_l^\top \cos(\theta_{\mathrm{init}}) + R \sin(\theta_{\mathrm{init}})$, where $B_l$ is the fixed random feedback matrix, $R$ is an independent random matrix of matching variance, and $\theta_{\mathrm{init}}$ is the soft-alignment angle. At $\theta_{\mathrm{init}} = 0^\circ$ the forward weights equal the feedback weights exactly; at $\theta_{\mathrm{init}} = 90^\circ$ the rule reproduces standard feedback alignment; intermediate angles interpolate between the two. This single scalar controls everything the paper reports: whether the angle between pathways relaxes or tightens during training, how smooth the loss landscape remains, how far the trainability envelope extends in depth, variance, and data scarcity, and the trade-off between clean accuracy and adversarial robustness.

What would settle it

Track the per-layer alignment angle between forward and feedback weights during IFA training on networks of depth 2 through 10: if the angle returns toward 90 degrees within the first few hundred update steps on the deeper nets while their final accuracy still trails backpropagation, the one-time alignment has stopped being load-bearing and the method collapses to baseline feedback alignment. A sharper version is to measure the early-training decay rate of alignment, $\mathrm{d}\theta/\mathrm{d}t$, and check whether it grows with depth in a way that predicts the performance gap already visible in Table S1.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the learning trajectory of a feedback-alignment network is governed by its initial alignment state rather than by ongoing coupling between pathways. When forward weights are initialized to coincide with the fixed random feedback weights, or at a controlled angle to them, training proceeds by gradually relaxing that alignment while preserving backpropagation-like error signals; standard feedback alignment, which starts with orthogonal weights, spends the entire run trying to align them and produces erratic, unstable trajectories. Across weight-variance settings, network depths up to ten layers, dataset sizes from 100 to 50,000 samples, and four image benchmarks, the aligned start raises final accuracy and steers convergence to flatter minima with a compressed Hessian spectrum. A corollary the paper emphasizes is that the natural 'loosening' of alignment during fixed-feedback training gives IFA networks stronger resistance to FGSM, BIM, and PGD adversarial attacks than networks trained by exact backpropagation.

Load-bearing premise

The load-bearing premise is that the one-time alignment stays informative for the whole training run: forward and feedback weights must not drift back out of alignment faster than learning can use the error signal, and the paper supports this only with empirical trajectories (Fig. 4b, Fig. S8), not an analytical bound. Its own Table S1 shows IFA still trails backpropagation by 6 to 8 points on deeper convolutional variants, indicating the premise weakens with depth.

Editorial extensions

If this is right

  • Fixed random feedback plus a one-time aligned start reaches accuracy statistically indistinguishable from backpropagation on moderate feedforward networks (CIFAR-10, $P = 0.082$), with no weight transport during training.
  • Initial alignment widens the trainable region: IFA trains successfully across a much broader range of forward and backward weight variances, at greater depth, and with far fewer training samples than baseline FA.
  • Aligned starts converge to flatter minima, with lower Hessian trace and smaller maximum eigenvalue, and this tracks improved robustness across all fifteen CIFAR-10-C corruption types at multiple severity levels.
  • Misalignment is not pure loss: IFA-trained networks retain more accuracy under FGSM, BIM, and PGD adversarial attacks than backpropagation-trained networks at equal perturbation strength.
  • Soft alignment interpolates smoothly between these regimes, so a moderate initial angle (around $30^\circ$ to $60^\circ$) captures most of the accuracy gain while preserving some of the adversarial robustness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The natural next quantity is a bound on alignment drift: a proof that $\|W_l - B_l^\top\|$ grows slowly relative to the learning rate would turn the paper's empirical stability claim into a predictive theorem, including its depth limits.
  • The initialization rule reads like a hardware recipe: mirror forward weights into the feedback pathway once at chip power-on, then update locally forever — a cheap, testable prescription for neuromorphic and analog platforms where per-iteration weight transport is precisely what is expensive.
  • The angle $\theta_{\mathrm{init}}$ is a tunable robustness knob that backpropagation lacks, since clean accuracy and adversarial robustness move in opposite directions with the alignment angle; a designer could set it to match the deployment threat level.
  • Because the paper's developmental story invokes noise-driven pretraining as the source of natural alignment, one could test that story directly: pretrain a network on random noise and measure whether the emergent alignment angle matches the empirically optimal $\theta_{\mathrm{init}}$.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript proposes initial feedback alignment (IFA), which initializes the forward weights as a soft projection of fixed random feedback weights (Algorithm 1) and then trains with standard feedback alignment. The main empirical claim is that this one-time initialization, without any further weight transport, yields learning comparable to backpropagation on shallow MLPs, smoother Hessian spectra, better corruption robustness, and improved adversarial robustness. Experiments on CIFAR-10, SVHN, CIFAR-100, STL-10, and two CNN variants are reported.

Significance. The idea is of interest: if a single initialization event is sufficient, it offers a simple and arguably more biologically plausible alternative to exact weight transport. The MLP experiments are carefully controlled (ten runs, rank-sum tests), and the Hessian and robustness analyses are systematic. However, the abstract's deep-network claim is not supported by Table S1, and the central mechanism (why the initial alignment remains informative) is analyzed only empirically. With revision, the paper could still support a useful, narrower claim.

major comments (4)
  1. [Abstract; Section 6; Table S1] The abstract states that "a one-time soft alignment ... enables deep networks to achieve performance comparable to backpropagation," but Table S1 shows a 7.55-point gap against BP on the 5-conv+3-FC network (65.55% vs 73.10%) and a 5.76-point gap on the 2-conv+3-FC network (56.37% vs 62.13%). Section 6 explicitly concedes that "a performance gap remains between our approach and full backpropagation in deeper or more complex models." The headline claim should be restricted to the shallow MLP regime or accompanied by results that actually support deep-network parity.
  2. [Section 3, Algorithm 1; Section 4.3, Fig. 4] At theta_init = 0 degrees, Algorithm 1 line 4 sets W_l = B_l^T, so the first backward pass is exactly backpropagation; the initial improvement over FA at this point is therefore partly definitional. The paper's only evidence that the fixed B_l remains useful as W_l moves away from B_l^T is the angle trajectory in Fig. 4b and Fig. S8 for a three-layer MLP. No analytical bound, layer-wise gradient alignment measure, or error-signal quality metric is provided, and Table S1 shows the approach degrades with depth. Please add a quantitative analysis of how the instructive power of B_l decays, or substantially narrow the claim.
  3. [Section 4.1, Fig. 1h] The claim that IFA achieves performance "not significantly different from BP" rests on a two-sided rank-sum test with n = 10 and P = 0.082. Absence of significance at this sample size is not evidence of comparability; an equivalence test, a confidence interval for the accuracy difference, or an effect-size report is needed if the abstract's "comparable to backpropagation" wording is retained.
  4. [Section 4.4, Fig. 7] The adversarial-robustness claim is confounded by clean accuracy. Fig. 7c reports that larger initial misalignment (larger theta_init) improves accuracy under FGSM, but Fig. 4d shows that the same angles reduce clean accuracy. Since FA (theta = 90 degrees) has the lowest clean accuracy, the robustness gain may simply reflect the known accuracy-robustness tradeoff rather than an advantage of IFA. Please report robust accuracy at matched clean accuracy, or otherwise separate the effect of initial alignment from the accuracy penalty.
minor comments (4)
  1. [Section 4.3] The word "emplyoed" should be "employed"; the manuscript also contains LaTeX artifacts such as "f anin" in Algorithm 1 and Section B.2 that should be cleaned up.
  2. [Supplementary A.3.1] The text refers to "Figure S1b" and "Figure S1c" when describing layer-wise alignment dynamics for the second and final layers, but the relevant panels appear to be in Figure S8; the cross-reference should be corrected.
  3. [Section 7] The code availability statement says the code "will be made available after the paper is published"; since the paper is under review, please provide the code or a supplement to make the reported ten-run experiments reproducible.
  4. [Figure 1d] The memory-access comparison in Figure 1d lacks axes and units, making the claimed computational advantage difficult to quantify.

Circularity Check

2 steps flagged · score 6.0 of 10

IFA's headline comparison to BP is partially self-definitional: at θ=0 the initialization W_l = B_l^T makes the first backward pass exactly backpropagation, so the claimed performance benefit over FA is encoded in the initialization rather than derived; a self-cited prior paper is the sole support for the developmental-plausibility premise.

  1. self definitional [Algorithm 1 line 4; Section B.2 (Soft Initial Alignment); Section 4.1]
    "When θinit = 0◦, the forward and backward weights are fully aligned (Wl = B⊤l ), whereas θinit = 90◦ corresponds to the standard FA case, where Wl and B⊤l are orthogonal and thus unaligned."

    At θinit=0°, Algorithm 1's Wl = B⊤l cos(θinit) + R sin(θinit) reduces to Wl = B⊤l. Then the feedback-alignment error signal in Eq. (4), δl = (Bl δl+1)⊙ϕ′(ol), becomes (W⊤l δl+1)⊙ϕ′(ol), which is exactly the backpropagation error signal in Eq. (2). IFA's first backward pass is therefore the BP symmetric-weight condition by construction, not an emergent or discovered effect. The paper's central comparison ('IFA ... ultimately achieves performance comparable to BP') starts from this definitionally BP-aligned state, so the head-to-head advantage over random-feedback FA is built into Algorithm 1. The subsequent relaxation of alignment and the final accuracies are empirical and could have diverged, so the circularity is partial rather than total.

  2. self citation load bearing [Section 3 (One-time initial feedback alignment); Section 5 Discussion; Reference [38]]
    "Recent studies have shown that mimicking the brain’s developmental process [34–37], for example, through pretraining with random noise can yield weight alignment, thereby providing a “soft” alignment when the network begins learning from real data [38]."

    This sentence is the only support offered for the premise that the required W=B⊤ initialization can arise through a developmentally plausible process, and [38] is the authors' own prior work (Cheon, Lee, and Paik). The present paper uses that self-citation as if it were an established external fact and builds its 'developmental foundations' discussion on it. This is load-bearing for the biological-plausibility framing, although not for the measured training-accuracy results, which would stand even if [38] were absent. It therefore contributes to the overall score but is secondary to the definitional starting-condition issue.

full rationale

The clearest circularity is in the construction of IFA itself. Algorithm 1 defines W_l = B_l^T cos(θ_init) + R sin(θ_init); at θ_init=0 this gives W_l = B_l^T, so Eq. (4) coincides with Eq. (2) on the first backward pass. The paper presents the resulting BP-like trajectory (Figures 1g and 2d) as evidence that one-time alignment enables BP-comparable learning, but the initial state is exactly BP's weight-transport condition, so part of the claimed benefit is definitional. The later dynamics, the Hessian/spectral analyses, the corruption robustness, and the adversarial robustness are empirical measurements that do not follow from the initialization formula and are not circular. The paper's own Table S1 limits the claim: on the 5-conv AlexNet variant IFA reaches 65.55% vs BP 73.10%, and Section 6 concedes a remaining gap in deeper or more complex models, so the abstract's 'deep networks comparable to backpropagation' phrasing overreaches even on its own data. The self-citation [38] is a secondary, non-computational support for developmental plausibility; it is not the main derivation. Overall, the central trainability comparison is partially circular by construction, giving a score of 6 rather than a higher score.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method's only real free knob is theta_init along with the initialization variances; there is no fitted parameter in the core result. The main assumptions are domain-level: that feedback alignment signals remain useful after the initial alignment decays, and that flatness of the loss landscape is the causal driver of robustness. No new physical or mathematical entities are introduced.

free parameters (4)
  • initial alignment angle theta_init = 0, 30, 60, 90 degrees
    The central control parameter of the method. Main IFA uses 0 degrees, but the paper sweeps it to generate the accuracy and robustness relationships; no theoretical selection rule is given.
  • forward and backward weight variance scales a and b = a = b = sqrt(2) (He) for main runs; varied from 1e-6 to 2^7 in the trainability grid
    Initialization variances are chosen by hand, and the paper uses the grid to map the trainable region; they are standard initialization hyperparameters rather than fitted constants.
  • learning rate and optimizer betas = lr = 1e-4, beta1 = 0.99, beta2 = 0.999
    Fixed Adam hyperparameters; the method's stability is empirically shown only under these settings.
  • network depth and width = two 512-unit hidden layers in main; depths 2 to 10; CNN variants
    Architecture choices constrain the generality of the 'deep networks' claim; Table S1 shows performance worsens with depth.
assumptions (4)
  • domain assumption Fixed random feedback weights B_l provide a sufficiently informative error signal when forward weights are initialized close to B_l^T.
    This is the core assumption of the method; it is supported only empirically by Figures 1g and 4, not derived.
  • domain assumption Flatter minima, measured by lower Hessian trace and largest eigenvalue, cause better generalization and robustness.
    Invoked in Section 4.3 from references [45-49]; the paper does not test the causal direction.
  • ad hoc to paper The initial alignment can be realized biologically, e.g., by pretraining with random noise or by axonal guidance.
    Discussion Section 5 relies on the authors' own prior work [38] and speculative developmental mechanisms.
  • domain assumption ReLU networks trained with Adam on standard image benchmarks are representative of the regimes relevant to the weight transport problem.
    The benchmark claims are limited to this setup, and the authors acknowledge a gap in deeper or more complex models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of One-Time Soft Alignment Enables Resilient Learning without Weight Transport." pith.science (2026). https://pith.science/paper/IKST5AYP

@misc{pith2026250520892,
  author       = {Pith},
  title        = {Pith review of: One-Time Soft Alignment Enables Resilient Learning without Weight Transport},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IKST5AYP}},
  note         = {Machine review of arXiv:2505.20892}
}
read the original abstract

Backpropagation is the cornerstone of deep learning, but its reliance on symmetric weight transport and global synchronization makes it computationally expensive and biologically implausible. Feedback alignment offers a promising alternative by approximating error gradients through fixed random feedback, thereby avoiding symmetric weight transport. However, this approach often struggles with poor learning performance and instability, especially in deep networks. Here, we show that a one-time soft alignment between forward and feedback weights at initialization enables deep networks to achieve performance comparable to backpropagation, without requiring weight transport during learning. This simple initialization condition guides stable error minimization in the loss landscape, improving network trainability. Spectral analyses further reveal that initial alignment promotes smoother gradient flow and convergence to flatter minima, resulting in better generalization and robustness. Notably, we also find that allowing moderate deviations from exact weight symmetry can improve adversarial robustness compared to standard backpropagation. These findings demonstrate that a simple initialization strategy can enable effective learning in deep networks in a biologically plausible and resource-efficient manner.

Figures

Figures reproduced from arXiv: 2505.20892 by the authors.

Figure 1
Figure 1. Effect of initial weight alignment on learning without weight transport. (a) Schematic of the [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Loss landscape analysis of learning trajectories under different training rules. (a) Illustration [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Trainability of neural networks under various conditions. (a) Trainable parameter space: [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Effect of the degree of initial alignment on network performance. (a) Schematic of [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Spectral analysis of convergence across initial alignment angles. (a) Loss landscape [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Effect of initial weight alignment on corruption robustness. (a) Fifteen image corruption [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Effect of weight misalignment on adversarial robustness. (a) Illustration of an adversarial [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Eigen Neural Network: Unlocking Generalizable Vision with Eigenbasis

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    Reparameterizing layer weights in a learned orthonormal eigenbasis is claimed to improve ImageNet classification, cross-modal retrieval, and enable a faster backpropagation-free variant that surpasses standard backpro...

Reference graph

Works this paper leans on

67 extracted references · 39 canonical work pages · cited by 1 Pith paper

  1. [1]

    Rumelhart, Geoffrey E

    David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back- propagating errors. Nature, 323:533–536, 10 1986

    work page 1986

  2. [2]

    Deep learning

    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436–444, 2015

    2015

  3. [3]

    Imagenet classification with deep convolutional neural networks

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25, 2012

    2012

  4. [4]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  5. [5]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  6. [6]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    arXiv 2023

  7. [7]

    Energy and policy considerations for modern deep learning research

    Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for modern deep learning research. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 13693–13696, 2020

    work page 2020

  8. [8]

    Efficient processing of deep neural networks: A tutorial and survey

    Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S Emer. Efficient processing of deep neural networks: A tutorial and survey. Proceedings of the IEEE, 105(12):2295–2329, 2017

    2017

Show all 67 references
  1. [9]

    Green ai

    Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. Communications of the ACM, 63(12):54–63, 2020

    2020

  2. [10]

    Carbon emissions and large neural network training

    David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350, 2021

    2021 arXiv

  3. [11]

    Estimating the carbon footprint of bloom, a 176b parameter language model

    Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. Estimating the carbon footprint of bloom, a 176b parameter language model. Journal of Machine Learning Research, 24(253):1–15, 2023

    2023

  4. [12]

    Computing’s energy problem (and what we can do about it)

    Mark Horowitz. Computing’s energy problem (and what we can do about it). In 2014 IEEE international solid-state circuits conference digest of technical papers (ISSCC), pages 10–14. IEEE, 2014

    2014

  5. [13]

    Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks

    Yu-Hsin Chen, Joel Emer, and Vivienne Sze. Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks. ACM SIGARCH computer architecture news, 44(3):367–379, 2016

    2016

  6. [14]

    On computable numbers, with an application to the entscheidungsproblem

    Alan Mathison Turing et al. On computable numbers, with an application to the entscheidungsproblem. J. of Math, 58(345-363):5, 1936

    1936

  7. [15]

    First draft of a report on the edvac

    John V on Neumann. First draft of a report on the edvac. Technical report, Moore School of Electrical Engineering, University of Pennsylvania, 1945

    1945

  8. [16]

    Reconstruction and simulation of neocortical microcircuitry

    Henry Markram, Eilif Muller, Srikanth Ramaswamy, Michael W Reimann, Marwan Abdellah, Car- los Aguado Sanchez, Anastasia Ailamaki, Lidia Alonso-Nanclares, Nicolas Antille, Selim Arsever, et al. Reconstruction and simulation of neocortical microcircuitry. Cell, 163(2):456–492, 2015

    2015

  9. [17]

    Memory and information processing in neuromorphic systems

    Giacomo Indiveri and Shih-Chii Liu. Memory and information processing in neuromorphic systems. Proceedings of the IEEE, 103(8):1379–1397, 2015

    2015

  10. [18]

    A million spiking-neuron integrated circuit with a scalable communication network and interface

    Paul A Merolla, John V Arthur, Rodrigo Alvarez-Icaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan L Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science, 345(6197...

    2014

  11. [19]

    Distributed hierarchical processing in the primate cerebral cortex

    Daniel J Felleman and David C Van Essen. Distributed hierarchical processing in the primate cerebral cortex. Cerebral cortex (New York, NY: 1991), 1(1):1–47, 1991

    1991

  12. [20]

    Visual areas exert feedforward and feedback influences through distinct frequency channels

    André Moraes Bastos, Julien Vezoli, Conrado Arturo Bosman, Jan-Mathijs Schoffelen, Robert Oostenveld, Jarrod Robert Dowdall, Peter De Weerd, Henry Kennedy, and Pascal Fries. Visual areas exert feedforward and feedback influences through distinct frequency channels. Neuron, 85(...

    2015

  13. [21]

    Competitive learning: From interactive activation to adaptive resonance

    Stephen Grossberg. Competitive learning: From interactive activation to adaptive resonance. Cognitive science, 11(1):23–63, 1987

    1987

  14. [22]

    The recent excitement about neural networks

    Francis Crick. The recent excitement about neural networks. Nature, 337(6203):129–132, 1989

    1989

  15. [23]

    Random synaptic feedback weights support error backpropagation for deep learning

    Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature communications, 7(1):13276, 2016

    2016

  16. [24]

    Backpropagation and the brain

    Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience, 21(6):335–346, 2020

    2020

  17. [25]

    Dendritic solutions to the credit assignment problem

    Blake A Richards and Timothy P Lillicrap. Dendritic solutions to the credit assignment problem. Current opinion in neurobiology, 54:28–36, 2019

    2019

  18. [26]

    Assessing the scalability of biologically-motivated deep learning algorithms and architectures

    Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey E Hinton, and Timothy Lillicrap. Assessing the scalability of biologically-motivated deep learning algorithms and architectures. Advances in Neural Information Processing Systems, 31, 2018

    2018

  19. [27]

    How important is weight symmetry in backpropagation? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016

    Qianli Liao, Joel Leibo, and Tomaso Poggio. How important is weight symmetry in backpropagation? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016

    2016

  20. [28]

    Biologically-plausible learning algorithms can scale to large datasets

    Will Xiao, Honglin Chen, Qianli Liao, and Tomaso Poggio. Biologically-plausible learning algorithms can scale to large datasets. In International Conference on Learning Representations, 2019

    2019

  21. [29]

    An approximation of the error backpropagation algorithm in a predictive coding network with local hebbian synaptic plasticity

    James CR Whittington and Rafal Bogacz. An approximation of the error backpropagation algorithm in a predictive coding network with local hebbian synaptic plasticity. Neural computation, 29(5):1229–1262, 2017

    2017

  22. [30]

    Inferring neural activity before plasticity as a foundation for learning beyond backpropagation

    Yuhang Song, Beren Millidge, Tommaso Salvatori, Thomas Lukasiewicz, Zhenghua Xu, and Rafal Bogacz. Inferring neural activity before plasticity as a foundation for learning beyond backpropagation. Nature Neuroscience, 27(2):348–358, 2024

    2024

  23. [31]

    Equilibrium propagation: Bridging the gap between energy-based models and backpropagation

    Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between energy-based models and backpropagation. Frontiers in computational neuroscience, 11:24, 2017

    2017

  24. [32]

    Backpropagation without weight transport

    John F Kolen and Jordan B Pollack. Backpropagation without weight transport. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN’94), volume 3, pages 1375–1380. IEEE, 1994

    1994

  25. [33]

    Deep learning without weight transport

    Mohamed Akrout, Collin Wilson, Peter Humphreys, Timothy Lillicrap, and Douglas B Tweed. Deep learning without weight transport. Advances in Neural Information Processing Systems, 32, 2019

    2019

  26. [34]

    Spontaneous impulse activity of rat retinal ganglion cells in prenatal life

    Lucia Galli and Lamberto Maffei. Spontaneous impulse activity of rat retinal ganglion cells in prenatal life. Science, 242(4875):90–91, 1988

    1988

  27. [35]

    Retinal waves coordinate patterned activity throughout the developing visual system

    James B Ackman, Timothy J Burbridge, and Michael C Crair. Retinal waves coordinate patterned activity throughout the developing visual system. Nature, 490(7419):219–225, 2012

    2012

  28. [36]

    Prenatal activity from thalamic neurons governs the emergence of functional cortical maps in mice

    Noelia Antón-Bolaños, Alejandro Sempere-Ferràndez, Teresa Guillamón-Vivancos, Francisco J Martini, Leticia Pérez-Saiz, Henrik Gezelius, Anton Filipchuk, Miguel Valdeolmillos, and Guillermina López- Bendito. Prenatal activity from thalamic neurons governs the emergence of funct...

    2019

  29. [37]

    Spontaneous activity in developing thalamic and cortical sensory networks

    Francisco J Martini, Teresa Guillamón-Vivancos, Verónica Moreno-Juan, Miguel Valdeolmillos, and Guillermina López-Bendito. Spontaneous activity in developing thalamic and cortical sensory networks. Neuron, 109(16):2519–2534, 2021

    2021

  30. [38]

    Pretraining with random noise for fast and robust learning without weight transport

    Jeonghwan Cheon, Sang Wan Lee, and Se-Bum Paik. Pretraining with random noise for fast and robust learning without weight transport. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems...

    2024

  31. [39]

    Reading digits in natural images with unsupervised feature learning

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, number 2. Granada, 2011

    2011

  32. [40]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    2009

  33. [41]

    An analysis of single-layer networks in unsupervised feature learning

    Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215–223. JMLR Workshop and Conference Proceedings, 2011

    2011

  34. [42]

    Gradient-based learning applied to document recognition

    Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998

    1998

  35. [43]

    Visualizing the loss landscape of neural nets

    Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curr...

    2018

  36. [44]

    Pyhessian: Neural networks through the lens of the hessian

    Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data), pages 581–590. IEEE, 2020

    2020

  37. [45]

    Flat minima

    Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1):1–42, 1997

    1997

  38. [46]

    On large-batch training for deep learning: Generalization gap and sharp minima

    Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016

    2016 arXiv

  39. [47]

    Entropy-sgd: Biasing gradient descent into wide valleys

    Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124...

    2019

  40. [48]

    Averag- ing weights leads to wider optima and better generalization

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averag- ing weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018

    2018 arXiv

  41. [49]

    Sharpness-aware minimization for efficiently improving generalization

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021

    2021

  42. [50]

    Benchmarking neural network robustness to common corruptions and perturbations

    Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019

    2019

  43. [51]

    Explaining and harnessing adversarial examples

    Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014

    2014 arXiv

  44. [52]

    Understanding the difficulty of training deep feedforward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010

    2010

  45. [53]

    On the importance of initialization and momentum in deep learning

    Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139–1147. PMLR, 2013

    2013

  46. [54]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015

    2015

  47. [55]

    Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

    Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013

    2013 arXiv

  48. [56]

    Pretraining with random noise for uncertainty calibration

    Jeonghwan Cheon and Se-Bum Paik. Pretraining with random noise for uncertainty calibration. arXiv preprint arXiv:2412.17411, 2024

    2024 arXiv

  49. [57]

    Deep physical neural networks trained with backpropagation

    Logan G Wright, Tatsuhiro Onodera, Martin M Stein, Tianyu Wang, Darren T Schachter, Zoey Hu, and Peter L McMahon. Deep physical neural networks trained with backpropagation. Nature, 601(7894):549– 555, 2022

    2022

  50. [58]

    Backpropagation-free training of deep physical neural networks

    Ali Momeni, Babak Rahmani, Matthieu Malléjac, Philipp Del Hougne, and Romain Fleury. Backpropagation-free training of deep physical neural networks. Science, 382(6676):1297–1303, 2023. 12

    2023

  51. [59]

    Adversarial examples in the physical world

    Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial intelligence safety and security, pages 99–112. Chapman and Hall/CRC, 2018

    2018

  52. [60]

    Towards deep learning models resistant to adversarial attacks

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017

    2017 arXiv

  53. [61]

    Hessian-based analysis of large batch training and robustness to adversaries

    Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W Mahoney. Hessian-based analysis of large batch training and robustness to adversaries. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Pr...

    2018

  54. [62]

    Randomized algorithms for estimating the trace of an implicit symmetric positive semi-definite matrix

    Haim Avron and Sivan Toledo. Randomized algorithms for estimating the trace of an implicit symmetric positive semi-definite matrix. Journal of the ACM (JACM), 58(2):1–34, 2011

    2011

  55. [63]

    Calculation of gauss quadrature rules

    Gene H Golub and John H Welsch. Calculation of gauss quadrature rules. Mathematics of computation, 23(106):221–230, 1969

    1969

  56. [64]

    An investigation into neural net optimization via hessian eigenvalue density

    Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedi...

    2019

  57. [65]

    Additionally, we investigate several representative conditions to visualize learning curves: large forward weight variance at (24, √

    corresponds to He initialization (d; 1⃝). Additionally, we investigate several representative conditions to visualize learning curves: large forward weight variance at (24, √

  58. [66]

    For each variance condition, learning curves of training accuracy, test accuracy, training loss, and test loss are presented from top to bottom

    (e; 2⃝), large backward weight variance at ( √ 2, 24) (f; 3⃝), and large variance in both weights at (24, 24) (g; 4⃝). For each variance condition, learning curves of training accuracy, test accuracy, training loss, and test loss are presented from top to bottom. In the main r...

  59. [67]

    We also explored a wide range of variances by varying a and b from 10−6 to 100 (smaller variances), and from 20 to 27 (larger variances), using exponential step sizes of 1

    corresponds to He initialization. We also explored a wide range of variances by varying a and b from 10−6 to 100 (smaller variances), and from 20 to 27 (larger variances), using exponential step sizes of 1. Additional values of 10−0.5 and 2−0.5 were also included. This systema...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.