Pith. sign in

REVIEW 3 major objections 4 minor 62 references

Reconstructing Backpropagation from Forward Fluctuations in Noise-modulated Neural Networks

T0 review · 3 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A neural network that learns only in noise can reconstruct backpropagation using forward-pass statistics alone.

desk verdict A clean, honest small-scale demonstration that forward noise in NNNs can substitute for transposed weights; the diagonal approximation limits the scaling claim but the paper admits it. read the letter →

arxiv 2607.26483 v1 pith:NSGEPPLE submitted 2026-07-29 cs.NE

Shuhei Ikemoto This is my paper
classification cs.NE
keywords noise-modulatedneuralnetworkweighttransportforward-onlylearningcovariancemirrorcreditassignmentstochasticresonancespike-likesignalsneuromorphichardware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the noise-modulated neural network (NNN), whose binary neurons fire only when injected noise crosses a threshold, can recover the transposed weight matrices needed for backpropagation from covariances measured during ordinary forward passes. The central rule, cov jac, estimates each weight as the covariance between a next-layer pre-activation and a previous-layer firing, divided by the firing variance, then propagates output error recursively through the graph together with a locally estimated derivative of the activation. In experiments on regression and classification tasks, the reconstructed gradients match autograd directions with cosine similarity 0.998–1.000 at initialization, and final squared error equals backpropagation within seed variability. A variant, cov jac full, even replaces the output error with a covariance estimate, eliminating all analytical loss derivatives. If correct, this would mean noise is not just tolerated but is the mechanism that makes biologically plausible, hardware-friendly, forward-only learning possible.

What carries the argument

The covariance weight mirror: Ŵ_ji = Σ_n Cov_T(d_j^(l+1), z_i^(l)) / Σ_n Var_T(z_i^(l)), a per-unit regression that estimates each forward weight from fluctuations of binary firing z and the downstream pre-activation d. Combined with the common-random-number slope φ'_T = [z(d+h) − z(d−h)]/(2h), it feeds a recursion isomorphic to backpropagation, replacing each factor of the chain rule (W^T and ∂z/∂d) with a quantity estimated from forward samples.

What would settle it

Build a network with two or more hidden layers whose activations within a layer are strongly correlated (for example, duplicate input features or shared noise structure). Compute the mirror Ŵ with Eq. (20), compare it to the true W by Pearson correlation, and measure the recursion's gradient cosine similarity versus autograd. If correlation drops far below 0.99 or gradient cosine degrades with depth, the diagonal approximation fails.

Watch

Extended reading notes

Core claim

Backpropagation in a noise-modulated network can be rebuilt without transposed weights or a backward path: the noise that makes binary units fire also supplies the statistics. Regressing a next layer's pre-activation on previous-layer firing over T noisy samples yields a weight mirror Ŵ whose transpose replaces W^T; a symmetric finite-difference slope φ'_T supplies the local derivative. Recursing δ^(l) = (Ŵ^(l+1)T δ^(l+1)) ⊙ φ'_T(d^(l)) reproduces the backpropagation error signal. Verified on regression and classification tasks, this forward-only rule matches backpropagation's final MSE within seed variability and gives near-unbiased gradient directions.

Load-bearing premise

The weight mirror estimates each forward weight by regressing next-layer pre-activation on one unit's firing alone, assuming other units in the same layer are conditionally uncorrelated; the paper acknowledges that in deeper networks the resulting estimation error can accumulate through the recursion, and this is not tested at depth.

Editorial extensions

If this is right

  • Because gradients are estimated from forward samples, no transposed-weight readout or backward data path is needed, resolving the weight-transport problem within the NNN.
  • With local per-weight Adam updates, cov jac and cov jac full match backpropagation's final MSE on sin(x) regression (0.00056±0.00006 and 0.00057±0.00009 versus 0.00057±0.00010) and reach perfect accuracy on two linearly inseparable classification tasks.
  • When the readout error is also estimated by covariance with a third-central-moment correction, no analytical loss derivative appears anywhere, so training is driven entirely by forward statistics.
  • Uniform noise reduces the activation to comparators and XOR and the local derivative to a linear expression, allowing the whole system, learning rule included, to be built from digital gates and LFSR-based random number generators.
  • Gradient-direction fidelity persists during training (cosine similarity 0.89–0.99), with norm-ratio deviations absorbed by Adam's normalization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The per-unit univariate regression is a diagonal approximation; in wide layers with correlated activations, weight leakage should grow, and in deep networks the estimation error could accumulate through the recursion. This scaling limit is untested in the paper and is the clearest next experiment.
  • Because the mirror is estimated from the same noise samples used for inference, the rule could in principle track non-stationary weight changes continuously, without a separate perturbation phase or phase synchronization.
  • The skewness-bias analysis for the readout covariance is likely generic: any quadratic loss with asymmetric output fluctuations, combined with Adam's scale normalization, will drift after convergence. This predicts similar behavior in other perturbation-based learning rules.
  • The noise field that controls which units receive credit could double as a power-gating mechanism, coupling learning and sparse activity; this is an explicit design direction the paper leaves for future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a family of forward-only learning rules for the Noise-modulated Neural Network (NNN), a stochastic binary-unit network whose crossing activation fires only when threshold noise is present. The central idea is to reconstruct backpropagation's transposed-weight path from forward-pass statistics: a 'weight mirror' (Eq. 20) estimates each forward weight matrix from the covariance between a previous layer's activations and the next layer's pre-activations, and a common-random-number finite-difference estimator (Eq. 8) supplies the local derivative of the crossing activation. These ingredients are combined into a recursive credit-assignment rule (Eqs. 21--22) that is isomorphic to backpropagation except that W^T is replaced by the estimated W-hat^T. A further variant, 'cov jac full', also replaces the readout error by a covariance regression with a third-central-moment correction for the skewness bias. Experiments on sin(x) regression, Friedman #1, two moons, and concentric circles with a 1--64--64--1 network show final MSE matching backpropagation when Adam is used, and the gradient direction at initialization is close to autograd (cosine 0.998--1.000). The paper also discusses digital-hardware implementation under uniform noise, where the activation and derivative reduce to comparators and linear expressions.

Significance. If the central claim holds, the paper offers a clean solution to the weight transport problem within a noise-based spiking model: the same stochastic samples used for inference provide both the weight mirror and the local derivatives, with no separate perturbation phase and no backward data path. The derivations of the weight-mirror regression, the local derivative estimator, and the readout skewness bias are transparent, and the skewness correction is empirically verified with correlation 0.998 against the predicted bias. The paper also releases code and reports careful ablations that isolate the contribution of credit structuring, per-input pooling, slope estimation, and optimizer choice. The main limitation is that the load-bearing 'diagonal approximation' in the weight mirror is untested at depth; the current evidence is confined to shallow two-hidden-layer networks, and the paper itself acknowledges that error accumulation through the recursion is an open question.

major comments (3)
  1. [Sec. 4.3.2, Eq. (20); Secs. 6.1 and 7] The covariance weight mirror W-hat_ji = Sum_n Cov(d_j,z_i) / Sum_n Var(z_i) is a per-unit univariate regression. For layers l >= 2, the T-sample fluctuations of z_i and z_k within a layer share the lower-layer random activity z^(l-1), so they are correlated even after conditioning on the input; the independence of per-unit threshold noise does not remove this correlation. Consequently, as T -> infinity the estimator converges to W_ji plus an omitted-variable term Sum_{k != i} W_jk Cov(z_k,z_i|x)/Var(z_i|x) (or the corresponding pooled version), not to W_ji. This is a systematic bias, not a finite-sample effect. Because Eq. (22) recurses this biased estimate through the layers, the error can accumulate. The experiments only reach depth 2 (1--64--64--1), where the measured correlation is high (r >= 0.988), and Section 7 explicitly leaves deep-network accumulation open. This assumption is l
  2. [Sec. 5.3, Fig. 3(c)] The claim that the reconstructed gradient is 'near-unbiased' is supported only at initialization and after 300 epochs of backpropagation pretraining. During training, the cosine similarity drops to 0.89 and the norm ratio expands to 2.2 relative to the autograd reference. Since Adam's 1/sqrt(v) normalization absorbs scale errors, final MSE parity on the small tasks does not by itself establish that the recursive gradient estimate tracks the true backpropagation gradient along the actual cov_jac / cov_jac_full training trajectory. Please report cosine similarity and norm ratio for the cov_jac update as training proceeds under the proposed learning rule, not only under backpropagation pretraining. This would directly support or qualify the 'empirically near-unbiased' statement.
  3. [Sec. 5.3] The exact reference for the cosine comparisons is underspecified. 'Backpropagation via autograd' for a network with stochastic binary units and ensemble averaging is ambiguous: autograd cannot differentiate the sampling path without a surrogate or a specific computational graph. State clearly whether the reference gradient is computed on the expected network using the analytical phi-bar' and ensemble-averaged activations, or on the realized sample graph using the finite-difference estimator phi'_T. Without this, the numerical fidelity values in Fig. 3 are difficult to interpret and reproduce.
minor comments (4)
  1. [Sec. 3.1 / Sec. 4.2] The statement that the learning rules require 'no additional stochastic forward passes' should be qualified. The local derivative phi'_T requires evaluating z(d+h) and z(d-h) under the same noise samples, which is additional shifted-threshold computation even though no new random draws are needed.
  2. [Sec. 5.1 / Sec. 5.6] Clarify whether the reported MSE and classification accuracies are on a held-out test set or on the same data used for parameter updates. No train/test split is described, and the datasets are synthetic; this affects the interpretation of 'final accuracy' claims.
  3. [Sec. 4.4] The notation for the third central moment m3 is introduced as y^3_n - 3 y_n y^2_n + 2 y_n^3, but overbar notation for T-sample averages is used elsewhere. Please align the notation or define the averages explicitly to avoid ambiguity.
  4. [Fig. 3 / Table 4] The labels w0, w1, wout in Fig. 3(c) are not defined in the main text; please name the layers explicitly. In Table 4, state whether the memory figure for cov_jac includes the additional y^3 accumulation used by cov_jac_full.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central reconstruction is an estimator construction validated against external autograd and backprop baselines; the diagonal-approximation bias is an acknowledged correctness limitation, not a circular dependency.

full rationale

The paper's derivation chain is not circular. The weight mirror in Eq. (20) estimates W from forward covariances, Cov(d_j,z_i)/Var(z_i), and is never fitted to the autograd gradients it later matches; Fig. 3 and Tables 2, 3, and 5 compare the resulting learning signals against independent autograd and backpropagation references. The recursion in Eqs. (21)-(22) is openly a substitution of W-hat^T for W^T in backpropagation, so the claim is architectural: it shows that forward fluctuations suffice to recover the quantities backpropagation needs. The approximation in Eq. (20) is explicitly called a 'diagonal approximation' (Sections 4.3.2 and 6.1), and Section 7 states that its estimation error may accumulate across layers in deep networks, which is a correctness and scalability limitation rather than a circular step. The cov_jac_full skewness correction is derived from a moment identity (Eq. 24) and verified with a correlation of 0.998, not tuned to the target. Self-citations to prior NNN work ([9]-[11]) provide background; the required crossing-activation equations are re-derived in Section 3, so no load-bearing self-citation chain is used. Hyperparameters such as the EMA constant, noise scale, bandwidth h, and T are tuning choices, not fitted parameters renamed as predictions. Overall, the core result is externally validated and the acknowledged approximation is a limitation, not a circular dependency.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. Crossing activation and noise fields come from prior NNN work by the same author, while weight mirror and Kolen-Pollack are cited prior methods. The load-bearing assumptions are the independent-noise field and the diagonal approximation to the covariance regression.

free parameters (4)
  • noise scale (Gaussian σ / uniform half-width r) = Gaussian σ=0.5; uniform r=1.0
    Chosen as experimental hyperparameters; not fitted to targets, but they set the width of the crossing activation and thus the support of all covariance statistics.
  • finite-difference bandwidth h for φ'_T = not reported in main text (set in code)
    Controls bias and variance of the distribution-free derivative estimator in Eq. (8); not swept or specified in the paper.
  • weight-mirror EMA smoothing constant = 0.9
    Set in Section 4.3.1; determines how quickly the estimated mirror Ŵ tracks forward weights via Kolen-Pollack updates.
  • ensemble size T = 64
    Number of stochastic forward samples per input; drives variance of all covariance statistics and is a primary compute cost (Section 5.1, Table 1).
assumptions (5)
  • domain assumption Crossing activation is z = 1[(d≥η1) XOR (d≥η2)] with i.i.d. noise; E[z] = 2F(1-F).
    Model definition inherited from NNN prior work [9-11]; all covariance and derivative estimators start from this form.
  • domain assumption Noise samples are independent across units and samples, with arbitrary distribution F.
    Used in Eq. 20 so that Cov(d_j,z_i) is attributable to the direct weight W_ji; also underlies common-random-number derivative estimation.
  • domain assumption Activations within a layer are uncorrelated enough that univariate regression Cov(d_j,z_i)/Var(z_i) recovers W_ji.
    Eq. 20 uses only Var(z_i) and drops Cov(z_i,z_k); the paper acknowledges this diagonal approximation and its unknown depth scaling in Sections 4.3.2, 6.1, and 7.
  • standard math The finite difference (z^+ - z^-)/(2h) with common random numbers converges to the local derivative as T→∞ and h→0.
    Eqs. (7)-(8); standard kernel-density/finite-difference result with antithetic variance reduction.
  • standard math For readout covariance regression, Cov(L,y)/Var(y) = 2(E[y]-t) + E[ε^3]/Var(ε) for quadratic loss L=(y-t)^2.
    Eq. (24); exact algebraic identity, empirically verified in Fig. 5 with correlation 0.998.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reconstructing Backpropagation from Forward Fluctuations in Noise-modulated Neural Networks." pith.science (2026). https://pith.science/paper/NSGEPPLE

@misc{pith2026260726483,
  author       = {Pith},
  title        = {Pith review of: Reconstructing Backpropagation from Forward Fluctuations in Noise-modulated Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NSGEPPLE}},
  note         = {Machine review of arXiv:2607.26483}
}
read the original abstract

A Noise-modulated Neural Network (NNN) learns and infers only in the presence of noise, treating noise as a computational resource rather than a disturbance. The noise lets it learn efficiently by backpropagation while transmitting spike-like signals, but backpropagation needs a reverse path through transposed weights, the weight transport problem, which undermines biological and neuromorphic plausibility. Forward-only alternatives typically substitute a different objective or fixed random feedback, sacrificing stability and accuracy. We show that backpropagation itself can be reconstructed in the NNN from forward-pass statistics alone: a weight mirror estimates each weight matrix from the covariance between a previous-layer unit's output and the next-layer unit's input, and combining it with local differential estimation inside the units propagates the output error recursively along the computational graph, with no transposed-weight readout and no backward data path. The resulting gradient is empirically near-unbiased, and with local per-weight Adam updates it matches the final accuracy of backpropagation on simple regression tasks. With uniformly distributed noise, the local operations reduce to polynomials and comparators, making the whole system, learning rule included, well suited to digital circuits. Thus, in the NNN, noise is a resource not only for inference but also for reconstructing backpropagation.

Figures

Figures reproduced from arXiv: 2607.26483 by the authors.

Figure 1
Figure 1. Overview of the problem and proposed approach. (A) Conventional backpropagation requires a backward path and access to transposed forward weights, resulting in the weight transport problem. (B) The crossing activation function generates stochastic binary activity and permits distribution-free estimation of its local derivative by kernel density estimation based on forward noise samples. (C) Noise-induced fluctuation… view at source ↗
Figure 2
Figure 2. Learning performance on the sin(x) regression task using backprop and proposed method variants. (a) Evaluation MSE over 1500 training epochs for backprop, cov only, cov deriv, cov jac, and cov jac full. (b) Target function, learned outputs, and residuals after training for backprop, cov jac, and cov jac full. backpropagation via autograd ( [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Fidelity of the reconstructed weights and gradient directions. (a, b) Recovered mirror weights Wˆ versus the true forward weights W for the hidden and readout layers, respectively, in the untrained network; r denotes the Pearson correlation coefficient. (c) Cosine similarity between the layer-wise update directions produced by cov jac or cov deriv and the exact autograd gradients at initialization and after 300 epoc… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Ablation study of the proposed learning rules. Bars show the final evaluation MSE on a logarithmic scale, with the mean and standard deviation evaluated over three different seeds. The comparisons examine the effects of optimizer selection, per-input versus pooled cova…
Figure 5
Figure 5. Figure 5: Readout-error estimation from forward covariances and verification of the skewness bias. (a) Evaluation MSE for cov jac and three cov jac full readout-error estimators: uncorrected covari￾ance regression (full cov), third central moment correction (full cov m3), and th…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

62 extracted references · 7 linked inside Pith

  1. [1]

    Stochastic resonance.Reviews of Modern Physics, 70:223–287, 1998

    Luca Gammaitoni, Peter H¨ anggi, Peter Jung, and Fabio Marchesoni. Stochastic resonance.Reviews of Modern Physics, 70:223–287, 1998

    1998

  2. [2]

    Stochastic resonance and sensory information processing: a tutorial and review of application.Clinical Neurophysiology, 115(2):267–281, 2004

    Frank Moss, Lawrence M Ward, and Walter G Sannita. Stochastic resonance and sensory information processing: a tutorial and review of application.Clinical Neurophysiology, 115(2):267–281, 2004

    2004

  3. [3]

    Fiete and H

    Ila R. Fiete and H. Sebastian Seung. Gradient learning in spiking neural networks by dynamic perturbation of conductances.Physical Review Letters, 97(4):048104, 2006

    2006

  4. [4]

    Fiete, Michale S

    Ila R. Fiete, Michale S. Fee, and H. Sebastian Seung. Model of birdsong learning based on gradient estimation by dynamic perturbation of neural conductances.Journal of Neurophysiology, 98(4):2038– 2057, 2007. 19

  5. [5]

    Probabilistic inference in general graphical models through sampling in stochastic networks of spiking neurons.PLoS Computational Biology, 7(12):e1002294, 2011

    Dejan Pecevski, Lars Buesing, and Wolfgang Maass. Probabilistic inference in general graphical models through sampling in stochastic networks of spiking neurons.PLoS Computational Biology, 7(12):e1002294, 2011

    2011

  6. [6]

    Neural variability and sampling-based probabilistic representations in the visual cortex.Neuron, 92(2):530–543, 2016

    Gerg˝ o Orb´ an, Pietro Berkes, J´ ozsef Fiser, and M´ at´ e Lengyel. Neural variability and sampling-based probabilistic representations in the visual cortex.Neuron, 92(2):530–543, 2016

    2016

  7. [7]

    Aldo Faisal, Luc P

    A. Aldo Faisal, Luc P. J. Selen, and Daniel M. Wolpert. Noise in the nervous system.Nature Reviews Neuroscience, 9(4):292–303, 2008

    2008

  8. [8]

    McDonnell and Lawrence M

    Mark D. McDonnell and Lawrence M. Ward. The benefits of noise in neural systems: bridging theory and experiment.Nature Reviews Neuroscience, 12(7):415–425, 2011

    2011

Show all 62 references
  1. [9]

    Noise-modulated neural networks as an ap- plication of stochastic resonance.Neurocomputing, 277:29 – 37, 2018

    Shuhei Ikemoto, Fabio DallaLibera, and Koh Hosoda. Noise-modulated neural networks as an ap- plication of stochastic resonance.Neurocomputing, 277:29 – 37, 2018

    2018

  2. [10]

    Noise-modulated neural networks for selectively functionalizing sub-networks by exploiting stochastic resonance.Neurocomputing, 448:1–9, 2021

    Shuhei Ikemoto. Noise-modulated neural networks for selectively functionalizing sub-networks by exploiting stochastic resonance.Neurocomputing, 448:1–9, 2021

    2021

  3. [11]

    Spatial partial functionalization of neural networks based on noise fields, 2026

    Shuhei Ikemoto and Fabio DallaLibera. Spatial partial functionalization of neural networks based on noise fields, 2026. arXiv:2606.24588 [cs.NE]

    2026 arXiv

  4. [12]

    Neftci, Hesham Mostafa, and Friedemann Zenke

    Emre O. Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51–63, 2019

    2019

  5. [13]

    Neftci, Charles Augustine, Somnath Paul, and Georgios Detorakis

    Emre O. Neftci, Charles Augustine, Somnath Paul, and Georgios Detorakis. Event-driven random back-propagation: Enabling neuromorphic deep learning machines.Frontiers in Neuroscience, 11, 2017

    2017

  6. [14]

    Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks.Frontiers in Neuroscience, 15, 2021

    Charlotte Frenkel, Martin Lefebvre, and David Bol. Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks.Frontiers in Neuroscience, 15, 2021

    2021

  7. [15]

    Lillicrap, Adam Santoro, Luke Marris, Colin J

    Timothy P. Lillicrap, Adam Santoro, Luke Marris, Colin J. Akerman, and Geoffrey Hinton. Back- propagation and the brain.Nature Reviews Neuroscience, 21:335–346, 2020

    2020

  8. [16]

    The recent excitement about neural networks.Nature, 337:129–132, 1989

    Francis Crick. The recent excitement about neural networks.Nature, 337:129–132, 1989

    1989

  9. [17]

    Lillicrap, Daniel Cownden, Douglas B

    Timothy P. Lillicrap, Daniel Cownden, Douglas B. Tweed, and Colin J. Akerman. Random synaptic feedback weights support error backpropagation for deep learning.Nature Communications, 7, 2016

    2016

  10. [18]

    Direct feedback alignment provides learning in deep neural networks

    Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. InAdvances in Neural Information Processing Systems, volume 29, 2016

    2016

  11. [19]

    Kolen and J.B

    J.F. Kolen and J.B. Pollack. Backpropagation without weight transport. InProceedings of 1994 IEEE International Conference on Neural Networks (ICNN’94), volume 3, pages 1375–1380, 1994

    1994

  12. [20]

    Humphreys, Timothy Lillicrap, and Douglas Tweed

    Mohamed Akrout, Collin Wilson, Peter C. Humphreys, Timothy Lillicrap, and Douglas Tweed. Deep learning without weight transport. InAdvances in Neural Information Processing Systems, volume 32, 2019

    2019

  13. [21]

    J.C. Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient ap- proximation.IEEE Transactions on Automatic Control, 37(3):332–341, 1992

    1992

  14. [22]

    The mechanism of stochastic resonance.Jour- nal of Physics A: Mathematical and general, 14:453–457, 1981

    Roberto Benzi, Alfonso Sutera, and Angelo Vulpiani. The mechanism of stochastic resonance.Jour- nal of Physics A: Mathematical and general, 14:453–457, 1981

    1981

  15. [23]

    Stochastic resonance in climatic change.Tellus, 34(1):10–15, 1982

    Roberto Benzi, Giorgio Parisi, Alfonso Sutera, and Angelo Vulpiani. Stochastic resonance in climatic change.Tellus, 34(1):10–15, 1982

    1982

  16. [24]

    Stochastic resonance and the benefits of noise: from ice ages to crayfish and SQUIDs.Nature, 373(6509):33–36, 1995

    Kurt Wiesenfeld and Frank Moss. Stochastic resonance and the benefits of noise: from ice ages to crayfish and SQUIDs.Nature, 373(6509):33–36, 1995

    1995

  17. [25]

    What is stochastic resonance? definitions, misconceptions, debates, and its relevance to biology.PLoS Computational Biology, 5(5):e1000348, 2009

    Mark D McDonnell and Derek Abbott. What is stochastic resonance? definitions, misconceptions, debates, and its relevance to biology.PLoS Computational Biology, 5(5):e1000348, 2009

    2009

  18. [26]

    Douglass, L

    J.K. Douglass, L. Wilkens, E. Pantazelou, and F. Moss. Noise enhancement of information transfer in crayfish mechanoreceptors by stochastic resonance.Nature, 365(6444):337–340, 1993. 20

    1993

  19. [27]

    Levin and J.P

    J.E. Levin and J.P. Miller. Broadband neural encoding in the cricket cercal sensory system enhanced by stochastic resonance.Nature, 380:165–168, 1996

    1996

  20. [28]

    Collins, Thomas T

    James J. Collins, Thomas T. Imhoff, and Peter Grigg. Noise-enhanced tactile sensation.Nature, 383(6603):770, 1996

    1996

  21. [29]

    Interpreting neural response variability as monte carlo sampling of the posterior

    Patrik Hoyer and Aapo Hyv¨ arinen. Interpreting neural response variability as monte carlo sampling of the posterior. InAdvances in Neural Information Processing Systems, volume 15, 2002

    2002

  22. [30]

    Haefner, Pietro Berkes, and J´ ozsef Fiser

    Ralf M. Haefner, Pietro Berkes, and J´ ozsef Fiser. Perceptual decision-making as probabilistic infer- ence by neural sampling.Neuron, 90(3):649–660, 2016

    2016

  23. [31]

    Spontaneous cortical activity reveals hallmarks of an optimal internal model of the environment.Science, 331(6013):83–87, 2011

    Pietro Berkes, Gerg˝ o Orb´ an, M´ at´ e Lengyel, and J´ ozsef Fiser. Spontaneous cortical activity reveals hallmarks of an optimal internal model of the environment.Science, 331(6013):83–87, 2011

    2011

  24. [32]

    Neural dynamics as sampling: A model for stochastic computation in recurrent networks of spiking neurons.PLOS Computational Biology, 7(11):e1002211, 2011

    Lars Buesing, Johannes Bill, Bernhard Nessler, and Wolfgang Maass. Neural dynamics as sampling: A model for stochastic computation in recurrent networks of spiking neurons.PLOS Computational Biology, 7(11):e1002211, 2011

    2011

  25. [33]

    Chris M. Bishop. Training with noise is equivalent to Tikhonov regularization.Neural Computation, 7(1):108–116, 1995

    1995

  26. [34]

    Dropout: A simple way to prevent neural networks from overfitting.Journal of Machine Learning Research, 15(56):1929–1958, 2014

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting.Journal of Machine Learning Research, 15(56):1929–1958, 2014

    1929

  27. [35]

    Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, and James Martens

    Arvind Neelakantan, Luke Vilnis, Quoc V. Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, and James Martens. Adding gradient noise improves learning for very deep networks, 2015. arXiv:1511.06807 [stat.ML]

    2015 arXiv

  28. [36]

    Networks of spiking neurons: The third generation of neural network models

    Wolfgang Maass. Networks of spiking neurons: The third generation of neural network models. Neural Networks, 10(9):1659–1671, 1997

    1997

  29. [37]

    Training deep spiking neural networks using backpropagation.Frontiers in Neuroscience, 10, 2016

    Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer. Training deep spiking neural networks using backpropagation.Frontiers in Neuroscience, 10, 2016

    2016

  30. [38]

    Friedemann Zenke and Tim P. Vogels. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks.Neural Computation, 33(4):899–925, 2021

    2021

  31. [39]

    Estimating or propagating gradients through stochastic neurons for conditional computation, 2013

    Yoshua Bengio, Nicholas L´ eonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation, 2013. arXiv:1308.3432 [cs.LG]

    2013 arXiv

  32. [40]

    Optimal spike-timing- dependent plasticity for precise action potential firing in supervised learning.Neural Computation, 18:1318–1348, 2006

    Jean-Pascal Pfister, Taro Toyoizumi, David Barber, and Wulfram Gerstner. Optimal spike-timing- dependent plasticity for precise action potential firing in supervised learning.Neural Computation, 18:1318–1348, 2006

    2006

  33. [41]

    Stochastic variational learning in recurrent spiking networks.Frontiers in Computational Neuroscience, 8, 2014

    Danilo Jimenez Rezende and Wulfram Gerstner. Stochastic variational learning in recurrent spiking networks.Frontiers in Computational Neuroscience, 8, 2014

    2014

  34. [42]

    Armin Alaghi and John P. Hayes. Survey of stochastic computing.ACM Transactions on Embedded Computing Systems, 12(2s), May 2013

    2013

  35. [43]

    Two routes to scalable credit assignment without weight symmetry

    Daniel Kunin, Aran Nayebi, Javier Sagastuy-Brena, Surya Ganguli, Jonathan Bloom, and Daniel Yamins. Two routes to scalable credit assignment without weight symmetry. InProceedings of the 37th International Conference on Machine Learning, volume 119 ofProceedings of Machine Lea...

    2020

  36. [44]

    How auto-encoders could provide credit assignment in deep networks via target propagation, 2014

    Yoshua Bengio. How auto-encoders could provide credit assignment in deep networks via target propagation, 2014. arXiv:1407.7906 [cs.LG]

    2014 arXiv

  37. [45]

    Difference target propagation

    Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. In Annalisa Appice, Pedro Pereira Rodrigues, V ´ ıtor Santos Costa, Carlos Soares, Jo˜ ao Gama, and Al ´ ıpio Jorge, editors,Machine Learning and Knowledge Discovery in Databases, pa...

    2015

  38. [46]

    Towards scaling difference target propagation by learning backprop targets

    Maxence Ernoult, Fabrice Normandin, Abhinav Moudgil, Sean Spinney, Eugene Belilovsky, Irina Rish, Blake Richards, and Yoshua Bengio. Towards scaling difference target propagation by learning backprop targets. InInternational Conference on Machine Learning, pages 5968–5987. PML...

    2022

  39. [47]

    Equilibrium propagation: Bridging the gap between energy- based models and backpropagation.Frontiers in Computational Neuroscience, 11, 2017

    Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between energy- based models and backpropagation.Frontiers in Computational Neuroscience, 11, 2017

    2017

  40. [48]

    Whittington and Rafal Bogacz

    James C.R. Whittington and Rafal Bogacz. An approximation of the error backpropagation algo- rithm in a predictive coding network with local hebbian synaptic plasticity.Neural Computation, 29:1229–1262, 2017

    2017

  41. [49]

    Leibo, and Tomaso Poggio

    Qianli Liao, Joel Z. Leibo, and Tomaso Poggio. How important is weight symmetry in backpropa- gation? InProceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI’16, page 1837–1844, 2016

    2016

  42. [50]

    Biologically-plausible learning algorithms can scale to large datasets

    Will Xiao, Honglin Chen, Qianli Liao, and Tomaso Poggio. Biologically-plausible learning algorithms can scale to large datasets. InInternational Conference on Learning Representations (ICLR), 2019

    2019

  43. [51]

    Sebastian Seung

    Justin Werfel, Xiaohui Xie, and H. Sebastian Seung. Learning curves for stochastic gradient descent in linear feedforward networks.Neural Computation, 17(12):2699–2718, 2005

    2005

  44. [52]

    Williams

    Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning.Machine Learning, 8(3-4):229–256, 1992

    1992

  45. [53]

    Pearlmutter, Don Syme, Frank Wood, and Philip Torr

    Atılım G¨ une¸ s Baydin, Barak A. Pearlmutter, Don Syme, Frank Wood, and Philip Torr. Gradients without backpropagation, 2022. arXiv:2202.08587 [cs.LG]

    2022 arXiv

  46. [54]

    Scaling forward gradient with local losses

    Mengye Ren, Simon Kornblith, Renjie Liao, and Geoffrey Hinton. Scaling forward gradient with local losses. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  47. [55]

    The forward-forward algorithm: Some preliminary investigations, 2022

    Geoffrey Hinton. The forward-forward algorithm: Some preliminary investigations, 2022. arXiv:2212.13345 [cs.LG]

    2022 arXiv

  48. [56]

    Error-driven input modulation: Solving the credit as- signment problem without a backward pass

    Giorgia Dellaferrera and Gabriel Kreiman. Error-driven input modulation: Solving the credit as- signment problem without a backward pass. InProceedings of the 39th International Conference on Machine Learning, volume 162, pages 4937–4955, 2022

    2022

  49. [57]

    Training neural networks with local error signals

    Arild Nøkland and Lars Hiller Eidnes. Training neural networks with local error signals. InPro- ceedings of the 36th International Conference on Machine Learning, volume 97, pages 4839–4850, 2019

    2019

  50. [58]

    Greedy layerwise learning can scale to ImageNet

    Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Greedy layerwise learning can scale to ImageNet. InProceedings of the 36th International Conference on Machine Learning, volume 97, pages 583–593, 2019

    2019

  51. [59]

    A solution to the learning dilemma for recurrent networks of spiking neurons

    Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. A solution to the learning dilemma for recurrent networks of spiking neurons. Nature Communications, 11:3625, 2020

    2020

  52. [60]

    On the stability and scalability of node perturbation learning

    Naoki Hiratani, Yash Mehta, Timothy Lillicrap, and Peter E Latham. On the stability and scalability of node perturbation learning. InAdvances in Neural Information Processing Systems, volume 35, pages 31929–31941, 2022

    2022

  53. [61]

    Assessing the scalability of biologically-motivated deep learning algorithms and architectures

    Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey E Hinton, and Timothy Lil- licrap. Assessing the scalability of biologically-motivated deep learning algorithms and architectures. InAdvances in Neural Information Processing Systems, volume 31, 2018

    2018

  54. [62]

    Moskovitz, Ashok Litwin-Kumar, and L

    Theodore H. Moskovitz, Ashok Litwin-Kumar, and L. F. Abbott. Feedback alignment in deep convolutional networks, 2019. arXiv:1812.06488 [cs.NE]. 22

    2019 arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.