Pith. sign in

REVIEW 2 minor 62 references

Synthetic gradients can achieve lower mean squared error than backpropagation for gradient estimation under identified conditions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-29 10:16 UTC pith:NLVJQULR

load-bearing objection Synthetic gradients can beat backprop on MSE with arbitrarily large gaps under the new framework, but the practical edge still needs checking.

arxiv 2605.27946 v1 pith:NLVJQULR submitted 2026-05-27 stat.ML cs.LG

Is Backpropagation Optimal? When Synthetic Gradients Improve Sample Efficiency

Yibo Jacky Zhang , Zeyu Tang , Sanmi Koyejo This is my paper
classification stat.ML cs.LG
keywords backpropagationsynthetic gradientssample efficiencygradient estimationcomputational graphsreinforcement learningcontextual banditsmean squared error
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a unified vectorized feedback framework that models gradient estimation for both loss-based and reward-based learning on computational graphs. Within this model, synthetic gradients appear as an alternative to backpropagation, and the authors derive conditions where the former yields strictly lower gradient-estimation mean squared error. They further construct explicit examples in which the sample-efficiency advantage of synthetic gradients can be made arbitrarily large. Experiments on contextual bandits and reinforcement learning tasks illustrate that the theoretical distinction can appear in practice.

Core claim

We introduce a unified vectorized feedback framework for loss-based and reward-based learning on computational graphs, in which synthetic gradients emerge as a natural alternative to backpropagation. We characterize the conditions under which synthetic gradients can achieve a lower gradient-estimation mean squared error than backpropagation. We construct examples illustrating that this sample efficiency advantage can be arbitrarily large.

What carries the argument

The unified vectorized feedback framework that represents gradient estimation on computational graphs and enables direct mean-squared-error comparison between synthetic gradients and backpropagation.

Load-bearing premise

The unified vectorized feedback framework accurately represents the gradient estimation process for both loss-based and reward-based learning on computational graphs without introducing unmodeled biases in the MSE comparison.

What would settle it

An explicit computational graph and loss/reward structure satisfying the paper's derived conditions in which the measured gradient-estimation MSE of backpropagation is nevertheless lower than that of synthetic gradients, or a family of examples in which the efficiency gap remains bounded.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • In identified graph structures, synthetic gradients require fewer samples than backpropagation to reach a target estimation accuracy.
  • The advantage applies equally to loss-driven supervised settings and reward-driven reinforcement learning settings.
  • There exist families of problems where the sample-efficiency gap between the two methods grows without bound.
  • Practical tasks such as contextual bandits and reinforcement learning can exhibit the predicted efficiency difference.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Designers of learning systems could select between backpropagation and synthetic gradients on a per-graph basis according to the derived conditions rather than defaulting to one method.
  • The framework suggests a route to hybrid algorithms that switch estimation methods mid-training when sample budgets are tight.
  • Extensions to non-differentiable or partially observable graphs could be tested by relaxing the vectorized feedback assumptions while preserving the MSE comparison.
  • The result raises the question of whether analogous efficiency gaps exist when comparing other credit-assignment rules beyond these two.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 2 minor

Summary. The paper introduces a unified vectorized feedback framework for loss-based and reward-based learning on computational graphs. It characterizes conditions under which synthetic gradients achieve lower gradient-estimation mean squared error than backpropagation, constructs examples where this sample-efficiency advantage can be arbitrarily large, and reports experiments on contextual bandits and reinforcement learning tasks.

Significance. If the central derivations hold, the work is significant for providing a theoretical comparison of gradient estimators that unifies supervised and reinforcement settings, with explicit constructions showing unbounded MSE advantages for synthetic gradients. This supplies falsifiable conditions and reproducible examples that could inform algorithm design beyond the default use of backpropagation.

minor comments (2)
  1. The abstract states that 'arbitrarily large' advantages exist but does not indicate the precise graph size or signal dimension at which the advantage begins to grow; adding a brief reference to the relevant theorem or construction in §4 would improve clarity.
  2. Notation for the vectorized feedback signals (e.g., the distinction between observed signals and estimated gradients) is introduced without an explicit table of symbols; a short notation summary would aid readers comparing the MSE expressions for backpropagation and synthetic gradients.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive evaluation, the recognition of the unified vectorized feedback framework, and the recommendation for minor revision. The report correctly summarizes our contributions on characterizing conditions for synthetic gradients to outperform backpropagation in MSE and on constructing examples with arbitrarily large sample-efficiency advantages. No major comments requiring point-by-point rebuttal were raised.

Circularity Check

0 steps flagged

No significant circularity detected in derivation chain

full rationale

The paper introduces an independent unified vectorized feedback framework for gradient estimation on computational graphs, derives explicit MSE expressions for backpropagation versus synthetic gradients as functions of observed signals, and constructs explicit examples where the MSE advantage grows without bound. These steps consist of mathematical definitions and comparisons between two distinct estimators on the same graph; the MSE comparison does not reduce to a fitted parameter, self-referential definition, or self-citation chain. The framework treats both methods symmetrically without embedding unmodeled asymmetries by construction, making the central claim self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; no explicit free parameters, axioms, or invented entities are stated or derivable from the provided text.

pith-pipeline@v0.9.1-grok · 5633 in / 1017 out tokens · 29126 ms · 2026-06-29T10:16:43.197326+00:00 · methodology

0 comments
read the original abstract

Backpropagation is the default learning rule for artificial neural networks and is often treated as the settled approach whenever differentiability is available. In this work, we revisit this convention through a theoretical lens of sample efficiency. We introduce a unified vectorized feedback framework for loss-based and reward-based learning on computational graphs, in which synthetic gradients emerge as a natural alternative to backpropagation. We characterize the conditions under which synthetic gradients can achieve a lower gradient-estimation mean squared error than backpropagation. We construct examples illustrating that this sample efficiency advantage can be arbitrarily large. Experiments on contextual bandits and reinforcement learning tasks demonstrate the potential of our theoretical findings.

Figures

Figures reproduced from arXiv: 2605.27946 by Sanmi Koyejo, Yibo Jacky Zhang, Zeyu Tang.

Figure 1
Figure 1. Figure 1: From a network path to a state evolution trajectory. Left: a single path through a layered network is highlighted in black. Right: after treating layer steps as time steps, the same path is viewed as the trajectory of a state machine. Shaded nodes denote local observations available at each step. Dashed arrows indicate information that skips across time. Curved arrow indicates a vectorized feedback gL. It … view at source ↗
Figure 2
Figure 2. Figure 2: Finite-partition and MNIST bandit results. (a) In the expert simulation, synthetic gradients reduces gradient estimation MSE relative to backpropagation, where the reduction ratio fits the theoretical prediction. (b) In the MNIST contextual bandit, synthetic gradients requires fewer steps (numer of samples seen) to converge. (c) Mixing weights sensitivity in the MNIST contextual bandit experiment. 6 Experi… view at source ↗
Figure 3
Figure 3. Figure 3: POPGym labyrinth escape and labyrinth explore. Training dynamics for all methods with 20 sampled trials per gradient update and 40 sampled trials per gradient update. Across these settings, synthetic gradients improve over backpropagation in few-sample regimes. Implementation details and ablations on learning rates and adaptive mixing weights are presented in Appendix H.4. 7 Related Work Synthetic Gradient… view at source ↗
Figure 4
Figure 4. Figure 4: Finite-partition expert simulation. MSE ratio ∆2/∆˜ 2 between the squared error of the backpropagation estimator and the squared error of the synthetic-gradient estimator for the full population gradient. In all three cases, the ratio grows with dimension reduction d − k, matching the predicted advantage from theory. where q is called the neuron dimension. Each neuron connects to nparents other neurons, wh… view at source ↗
Figure 5
Figure 5. Figure 5: MNIST contextual bandit. Mean evaluation rewards along training steps. Each curve aggregates six seeds, with the standard error of the mean (SEM) reported. For SparseNet, synthetic gradient improves sample efficiency relative to backpropagation, out performing the dense MLP baseline. The fixed-λ comparison shows that the benefit depends on balancing synthetic and backpropagated gradients. H.3 MNIST Context… view at source ↗
Figure 6
Figure 6. Figure 6: MNIST contextual bandit ablations. (a) Each method has different optimal learning rate due. (b) Synthetic gradient propagation shows advantages in high variance regime. (c) Synthetic gradient propagation favors sparse connectivity. (d) Dynamics of the adaptive mixing weight along training steps. maze in labyrinth_escape or successfully visits all maze squares in labyrinth_explore, a terminal reward r = 1 i… view at source ↗
Figure 7
Figure 7. Figure 7: POPGym labyrinth results. Each row shows one environment and parallel-environment setting; columns show training dynamics, learning-rate sensitivity measured by the area-under-curve (AUC) of the success rate curve, and the adaptive mixing weight λ along steps. The more parallel environments, the more sampled trials the agent sees before each parameter update. Results show that synthetic gradient propagatio… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 2 canonical work pages · 1 internal anchor

  1. [1]

    Uncertainty, neuromodulation, and attention.Neuron, 46(4):681–692, 2005

    J Yu Angela and Peter Dayan. Uncertainty, neuromodulation, and attention.Neuron, 46(4):681–692, 2005

    2005

  2. [2]

    Decoupled greedy learning of cnns

    Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Decoupled greedy learning of cnns. InInternational Conference on Machine Learning, pages 736–745. PMLR, 2020

    2020

  3. [3]

    Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets

    Guillaume Bellec, Franz Scherr, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets.arXiv preprint arXiv:1901.09049, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1901

  4. [4]

    A non-asymptotic analysis of non-parametric temporal-difference learning.Advances in Neural Information Processing Systems, 35:7599–7613, 2022

    Eloïse Berthier, Ziad Kobeissi, and Francis Bach. A non-asymptotic analysis of non-parametric temporal-difference learning.Advances in Neural Information Processing Systems, 35:7599–7613, 2022

    2022

  5. [5]

    A finite time analysis of temporal difference learning with linear function approximation

    Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation. In Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet, editors,Proceedings of the 31st Conference On Learning Theory, volume 75 ofProceedings of Machine Learning Research, pages 1691–1692. PMLR, 06–09 Jul 2018. 11

    2018

  6. [6]

    A finite time analysis of temporal difference learning with linear function approximation

    Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation. InConference on learning theory, pages 1691–1692. PMLR, 2018

    2018

  7. [7]

    Neural temporal-difference learning converges to global optima.Advances in Neural Information Processing Systems, 32, 2019

    Qi Cai, Zhuoran Yang, Jason D Lee, and Zhaoran Wang. Neural temporal-difference learning converges to global optima.Advances in Neural Information Processing Systems, 32, 2019

    2019

  8. [8]

    Finite-time analysis of natural actor-critic for pomdps.SIAM Journal on Mathematics of Data Science, 6(4):869–896, 2024

    Semih Cayci, Niao He, and R Srikant. Finite-time analysis of natural actor-critic for pomdps.SIAM Journal on Mathematics of Data Science, 6(4):869–896, 2024

    2024

  9. [9]

    On the statistical benefits of temporal difference learning

    David Cheikhi and Daniel Russo. On the statistical benefits of temporal difference learning. In International Conference on Machine Learning, pages 4269–4293. PMLR, 2023

    2023

  10. [10]

    Exploring the use of synthetic gradients for distributed deep learning across cloud and edge resources

    Yitao Chen, Kaiqi Zhao, Baoxin Li, and Ming Zhao. Exploring the use of synthetic gradients for distributed deep learning across cloud and edge resources. In2nd USENIX Workshop on Hot Topics in Edge Computing (HotEdge 19), 2019

    2019

  11. [11]

    The surprising efficiency of temporal difference learning for rare event prediction.Advances in Neural Information Processing Systems, 37:81257–81286, 2024

    Xiaoou Cheng and Jonathan Weare. The surprising efficiency of temporal difference learning for rare event prediction.Advances in Neural Information Processing Systems, 37:81257–81286, 2024

    2024

  12. [12]

    Learning phrase representations using rnn encoder–decoder for statistical machine translation

    Kyunghyun Cho, Bart Van Merriënboer, Ça ˘glar Gulçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder–decoder for statistical machine translation. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1724–1734, 2014

    2014

  13. [13]

    Sobolev training for neural networks.Advances in neural information processing systems, 30, 2017

    Wojciech M Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszcz, and Razvan Pascanu. Sobolev training for neural networks.Advances in neural information processing systems, 30, 2017

    2017

  14. [14]

    Understanding synthetic gradients and decoupled neural interfaces

    Wojciech Marian Czarnecki, Grzegorz ´Swirszcz, Max Jaderberg, Simon Osindero, Oriol Vinyals, and Koray Kavukcuoglu. Understanding synthetic gradients and decoupled neural interfaces. In International Conference on Machine Learning, pages 904–912. PMLR, 2017

    2017

  15. [15]

    MIT press, 2005

    Peter Dayan and Laurence F Abbott.Theoretical neuroscience: computational and mathematical modeling of neural systems. MIT press, 2005

    2005

  16. [16]

    Vectorized instructive signals in cortical dendrites.Nature, 652(8112):1254–1263, 2026

    Valerio Francioni, Vincent D Tang, Enrique HS Toloza, Zilan Ding, Norma J Brown, and Mark T Harnett. Vectorized instructive signals in cortical dendrites.Nature, 652(8112):1254–1263, 2026

    2026

  17. [17]

    Optimality of lstd and its relation to mc

    Steffen Grunewalder, Sepp Hochreiter, and Klaus Obermayer. Optimality of lstd and its relation to mc. In2007 International Joint Conference on Neural Networks, pages 338–343. IEEE, 2007

    2007

  18. [18]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  19. [19]

    The Forward-Forward Algorithm: Some Preliminary Investigations

    Geoffrey Hinton. The forward-forward algorithm: Some preliminary investigations.arXiv preprint arXiv:2212.13345, 2(3):5, 2022

    work page Pith review arXiv 2022

  20. [20]

    Empirical bayes transductive meta-learning with synthetic gradients

    Shell Xu Hu, Pablo Garcia Moreno, Yang Xiao, Xi Shen, Guillaume Obozinski, Neil Lawrence, and Andreas Damianou. Empirical bayes transductive meta-learning with synthetic gradients. In International Conference on Learning Representations, 2020

    2020

  21. [21]

    Decoupled parallel backpropagation with convergence guarantee

    Zhouyuan Huo, Bin Gu, Heng Huang, et al. Decoupled parallel backpropagation with convergence guarantee. InInternational Conference on Machine Learning, pages 2098–2106. PMLR, 2018

    2098

  22. [22]

    Decoupled neural interfaces using synthetic gradients

    Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Sil- ver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. InInternational conference on machine learning, pages 1627–1635. PMLR, 2017

    2017

  23. [23]

    Ada-gp: Accelerating dnn training by adaptive gradient prediction

    Vahid Janfaza, Shantanu Mandal, Farabi Mahmud, and Abdullah Muzahid. Ada-gp: Accelerating dnn training by adaptive gradient prediction. InProceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture, pages 1092–1105, 2023

    2023

  24. [24]

    Synaptic plasticity dynamics for deep continuous local learning (decolle).Frontiers in Neuroscience, 14:424, 2020

    Jacques Kaiser, Hesham Mostafa, and Emre Neftci. Synaptic plasticity dynamics for deep continuous local learning (decolle).Frontiers in Neuroscience, 14:424, 2020

    2020

  25. [25]

    The fusiform face area: a module in human extrastriate cortex specialized for face perception.Journal of neuroscience, 17(11):4302–4311, 1997

    Nancy Kanwisher, Josh McDermott, and Marvin M Chun. The fusiform face area: a module in human extrastriate cortex specialized for face perception.Journal of neuroscience, 17(11):4302–4311, 1997

    1997

  26. [26]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. InInternational conference on machine learning, pages 5156–5165. PMLR, 2020. 12

    2020

  27. [27]

    Bias-variance error bounds for temporal difference updates

    Michael J Kearns and Satinder Singh. Bias-variance error bounds for temporal difference updates. In COLT, pages 142–147, 2000

    2000

  28. [28]

    Is temporal difference learning optimal? an instance-dependent analysis.SIAM Journal on Mathematics of Data Science, 3(4):1013–1040, 2021

    Koulik Khamaru, Ashwin Pananjady, Feng Ruan, Martin J Wainwright, and Michael I Jordan. Is temporal difference learning optimal? an instance-dependent analysis.SIAM Journal on Mathematics of Data Science, 3(4):1013–1040, 2021

    2021

  29. [29]

    Presynaptic release probability influences the locus of long-term potentiation.Nature, 360(6399):70–73, 1992

    Alan Larkman, Timo Hannay, Ken Stratford, and Julian Jack. Presynaptic release probability influences the locus of long-term potentiation.Nature, 360(6399):70–73, 1992

    1992

  30. [30]

    Finite-sample analysis of lstd

    Alessandro Lazaric, Mohammad Ghavamzadeh, and Rémi Munos. Finite-sample analysis of lstd. In ICML-27th International Conference on Machine Learning, pages 615–622, 2010

    2010

  31. [31]

    Deep learning.nature, 521(7553):436–444, 2015

    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning.nature, 521(7553):436–444, 2015

    2015

  32. [32]

    Backpropagation applied to handwritten zip code recognition

    Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541–551, 1989

    1989

  33. [33]

    Gradient-based learning applied to document recognition.Proceedings of the IEEE, 86(11):2278–2324, 2002

    Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition.Proceedings of the IEEE, 86(11):2278–2324, 2002

    2002

  34. [34]

    Difference target propagation

    Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. InJoint european conference on machine learning and knowledge discovery in databases, pages 498–515. Springer, 2015

    2015

  35. [35]

    High-probability sample complexities for policy evaluation with linear function approximation.IEEE transactions on information theory, 70(8):5969–5999, 2024

    Gen Li, Weichen Wu, Yuejie Chi, Cong Ma, Alessandro Rinaldo, and Yuting Wei. High-probability sample complexities for policy evaluation with linear function approximation.IEEE transactions on information theory, 70(8):5969–5999, 2024

    2024

  36. [36]

    Accelerated and instance-optimal policy evalua- tion with linear function approximation.SIAM Journal on Mathematics of Data Science, 5(1):174– 200, 2023

    Tianjiao Li, Guanghui Lan, and Ashwin Pananjady. Accelerated and instance-optimal policy evalua- tion with linear function approximation.SIAM Journal on Mathematics of Data Science, 5(1):174– 200, 2023

    2023

  37. [37]

    Random synap- tic feedback weights support error backpropagation for deep learning.Nature communications, 7(1):13276, 2016

    Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synap- tic feedback weights support error backpropagation for deep learning.Nature communications, 7(1):13276, 2016

    2016

  38. [38]

    Taylor expansion of the accumulated rounding error.BIT, 16(2):146–160, June 1976

    Seppo Linnainmaa. Taylor expansion of the accumulated rounding error.BIT, 16(2):146–160, June 1976

    1976

  39. [39]

    Context-dependent computation by recurrent dynamics in prefrontal cortex.nature, 503(7474):78–84, 2013

    Valerio Mante, David Sussillo, Krishna V Shenoy, and William T Newsome. Context-dependent computation by recurrent dynamics in prefrontal cortex.nature, 503(7474):78–84, 2013

    2013

  40. [40]

    Synthetic gradient methods with virtual forward-backward networks

    Takeru Miyato, Daisuke Okanohara, Shin-ichi Maeda, and Masanori Koyama. Synthetic gradient methods with virtual forward-backward networks. 2017

    2017

  41. [41]

    Popgym: Benchmarking partially observable reinforcement learning

    Steven Morad, Ryan Kortvelesy, Matteo Bettini, Stephan Liwicki, and Amanda Prorok. Popgym: Benchmarking partially observable reinforcement learning. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  42. [42]

    Feed-forward on-edge fine- tuning using static synthetic gradient modules

    Robby Neven, Marian Verhelst, Tinne Tuytelaars, and Toon Goedemé. Feed-forward on-edge fine- tuning using static synthetic gradient modules. InEuropean Conference on Computer Vision, pages 131–146. Springer, 2020

    2020

  43. [43]

    A mesoscale connectome of the mouse brain

    Seung Wook Oh, Julie A Harris, Lydia Ng, Brent Winslow, Nicholas Cain, Stefan Mihalas, Quanxin Wang, Chris Lau, Leonard Kuan, Alex M Henry, et al. A mesoscale connectome of the mouse brain. Nature, 508(7495):207–214, 2014

    2014

  44. [44]

    Climbing fibers encode a temporal-difference prediction error during cerebellar learning in mice.Nature neuroscience, 18(12):1798–1803, 2015

    Shogo Ohmae and Javier F Medina. Climbing fibers encode a temporal-difference prediction error during cerebellar learning in mice.Nature neuroscience, 18(12):1798–1803, 2015

    2015

  45. [45]

    Neural variability and sampling-based probabilistic representations in the visual cortex.Neuron, 92(2):530–543, 2016

    Gerg˝o Orbán, Pietro Berkes, József Fiser, and Máté Lengyel. Neural variability and sampling-based probabilistic representations in the visual cortex.Neuron, 92(2):530–543, 2016

    2016

  46. [46]

    Cortico-cerebellar networks as decoupling neural interfaces.Advances in neural information processing systems, 34:7745–7759, 2021

    Joseph Pemberton, Ellen Boven, Richard Apps, and Rui Ponte Costa. Cortico-cerebellar networks as decoupling neural interfaces.Advances in neural information processing systems, 34:7745–7759, 2021

    2021

  47. [47]

    BP($\mathbf{\lambda}$): Online learning via synthetic gradients.Transactions on Machine Learning Research, 2024

    Joseph Oliver Pemberton and Rui Ponte Costa. BP($\mathbf{\lambda}$): Online learning via synthetic gradients.Transactions on Machine Learning Research, 2024. 13

    2024

  48. [48]

    Learning representations by back-propagating errors.nature, 323(6088):533–536, 1986

    David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors.nature, 323(6088):533–536, 1986

    1986

  49. [49]

    Equilibrium propagation: Bridging the gap between energy- based models and backpropagation.Frontiers in computational neuroscience, 11:24, 2017

    Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between energy- based models and backpropagation.Frontiers in computational neuroscience, 11:24, 2017

    2017

  50. [50]

    Long short-term memory.Neural Comput, 9(8):1735– 1780, 1997

    Jürgen Schmidhuber, Sepp Hochreiter, et al. Long short-term memory.Neural Comput, 9(8):1735– 1780, 1997

    1997

  51. [51]

    A neural substrate of prediction and reward

    Wolfram Schultz, Peter Dayan, and P Read Montague. A neural substrate of prediction and reward. Science, 275(5306):1593–1599, 1997

    1997

  52. [52]

    Learning by directional gradient descent

    David Silver, Anirudh Goyal, Ivo Danihelka, Matteo Hessel, and Hado van Hasselt. Learning by directional gradient descent. InInternational Conference on Learning Representations, 2021

    2021

  53. [53]

    The highly irregular firing of cortical cells is inconsistent with temporal integration of random epsps.Journal of neuroscience, 13(1):334–350, 1993

    William R Softky and Christof Koch. The highly irregular firing of cortical cells is inconsistent with temporal integration of random epsps.Journal of neuroscience, 13(1):334–350, 1993

    1993

  54. [54]

    Highly nonrandom features of synaptic connectivity in local cortical circuits.PLoS biology, 3(3):e68, 2005

    Sen Song, Per Jesper Sjöström, Markus Reigl, Sacha Nelson, and Dmitri B Chklovskii. Highly nonrandom features of synaptic connectivity in local cortical circuits.PLoS biology, 3(3):e68, 2005

    2005

  55. [55]

    Finite-time analysis of adaptive temporal difference learning with deep neural networks.Advances in Neural Information Processing Systems, 35:19592–19604, 2022

    Tao Sun, Dongsheng Li, and Bao Wang. Finite-time analysis of adaptive temporal difference learning with deep neural networks.Advances in Neural Information Processing Systems, 35:19592–19604, 2022

    2022

  56. [56]

    Learning to predict by the methods of temporal differences.Machine learning, 3(1):9–44, 1988

    Richard S Sutton. Learning to predict by the methods of temporal differences.Machine learning, 3(1):9–44, 1988

    1988

  57. [57]

    Policy gradient methods for reinforcement learning with function approximation.Advances in neural information processing systems, 12, 1999

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation.Advances in neural information processing systems, 12, 1999

    1999

  58. [58]

    Shared and distinct transcriptomic cell types across neocortical areas.Nature, 563(7729):72–78, 2018

    Bosiljka Tasic, Zizhen Yao, Lucas T Graybuck, Kimberly A Smith, Thuc Nghi Nguyen, Darren Bertagnolli, Jeff Goldy, Emma Garren, Michael N Economo, Sarada Viswanathan, et al. Shared and distinct transcriptomic cell types across neocortical areas.Nature, 563(7729):72–78, 2018

    2018

  59. [59]

    CRC press, 2025

    Xiao-Jing Wang.Theoretical neuroscience: Understanding cognition. CRC press, 2025

    2025

  60. [60]

    Applications of advances in nonlinear sensitivity analysis

    Paul J Werbos. Applications of advances in nonlinear sensitivity analysis. Springer, 1981

    1981

  61. [61]

    Simple statistical gradient-following algorithms for connectionist reinforcement learning.Machine learning, 8(3):229–256, 1992

    Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning.Machine learning, 8(3):229–256, 1992. 14 Appendix A Proof of Lemma 1 Lemma 1.Let ˆ˜gθ v = 1 n P i∈[n] ˜gθ v,i be the empirical mean ofni.i.d. samples. Then ˜∆2 v = 1 n ˜ρ2 v + ˜ν2 v + ˜b2 v. In particular, for an conditionally unbiased estimator, i...

    1992

  62. [62]

    Rollout length is the same as the episode length in both environments. As mentioned in Appendix H.2, to estimate the mixing weight adaptively, since the true gradient in equation 3 is not available, we simply use the downstream propagated gradient to replace the true gradient. This, however, could accumulate excessive bias in the gradient estimation, espe...