Pith. sign in

REVIEW 5 major objections 5 minor 60 references

Improving Performance of Spike-based Deep Q-Learning using Ternary Neurons

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Symmetric ternary spiking neurons fail in deep Q-learning because their expected gradient vanishes; an asymmetric variant with a trainable negative threshold restores nonzero gradients and lifts scores to 130 percent of the binary…

desk verdict Promising asymmetric ternary neuron for spiking RL, but the 30% improvement is an artifact of ratio averaging and the theoretical explanation is shaky. read the letter →

arxiv 2506.03392 v1 pith:HKNEHJXH submitted 2025-06-03 cs.LG cs.NEcs.SYeess.SY

Aref Ghoreishee , Abhishek Mishra , John Walsh , Anup Das , Nagarajan Kandasamy This is my paper
classification cs.LGcs.NEcs.SYeess.SY
keywords spikingneuralnetworksdeepQ-learningternaryneuronsgradientestimationbiassurrogateneuromorphiccomputingreinforcementlearningasymmetricfiringthresholds
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to explain why the recently introduced ternary spiking neuron, which can emit $-1$, $0$, or $1$ and therefore has more representational capacity than a binary spiking neuron, performs badly in deep Q-learning, and to fix it. Its explanation is gradient estimation bias: with symmetric positive and negative firing thresholds and an approximately Gaussian membrane potential, positive and negative spikes become equally likely, so the expected spike-generation gradient is zero. The proposed fix is an asymmetric ternary spiking neuron with a fixed positive threshold and a trainable negative threshold, which keeps the extra capacity while making the expected gradient nonzero. Built into a deep spiking Q-network (DATSQN), it reaches on average 130 percent of the binary DSQN score across seven Atari games, a 30 percent improvement, while keeping the network fully spiking and multiplication-free. The paper also argues, through a block dynamical isometry analysis, that the asymmetric neuron trains at least as stably as a ReLU network.

What carries the argument

The object that carries the argument is the asymmetric ternary LIF neuron of Equation (19), which spikes $s_l(t)=1$ when $m_l(t) \ge v_{th}^p$, $s_l(t)=-1$ when $m_l(t) \le -v_{th}^n$, and $0$ between the thresholds, with $v_{th}^p \neq v_{th}^n$. Around it, the paper builds three pieces of analysis: a Gaussian approximation of the membrane-potential distribution derived from the Fokker-Planck equation for the subthreshold LIF; an information-entropy comparison showing $H(\tilde{s}_{AT})>H(\tilde{s}_B)$ for the same firing rate; and an expected-gradient identity that makes the symmetric neuron's expected gradient vanish when $p_+=p_-$. The trainable negative threshold is the mechanism that lets the optimizer move the neuron away from the vanishing-gradient point while keeping the extra ternary bit of capacity. Training stability is then carried by a block dynamical isometry calculation, Lemma 4.6, which gives $\phi(s_m s_m^T)=1-r$ and $\varphi(s_m s_m^T)=r-r^2$ for the spiking block, matching or beating the ReLU values used as the stability benchmark.

What would settle it

Run DTSQN training on Breakout while recording the actual surrogate gradient values that backpropagation uses, for instance the arctan or straight-through estimator from Table 1, and compare their empirical average for symmetric and asymmetric ternary neurons. The paper's Equation 16 implies this average is positive for both, so if the symmetric neuron's measured surrogate gradient is not smaller by an amount that tracks the score gap, the vanishing-expected-gradient mechanism is falsified even if the performance difference remains.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the poor Q-learning performance of ternary spiking neurons is caused by the symmetry of their thresholds, not by ternary encoding itself. For a symmetric ternary LIF, the Gaussian-like distribution of the subthreshold membrane potential makes the probabilities of a $+1$ spike and a $-1$ spike equal, so $\mathbb{E}[\partial \tilde{s}_T/\partial m] = \partial(p_+ - p_-)/\partial m$ is zero; the paper identifies this vanishing expected gradient, through the chain rule, with a biased gradient estimator that corrupts training. The asymmetric ternary LIF of Equation (19), with $v_{th}^p \neq v_{th}^n$, preserves $p_+ \neq p_-$, keeps the expected gradient nonzero, and still has higher information entropy than a binary LIF (Theorem 4.3). Empirically, the symmetric ternary network (DTSQN) falls to about 50 percent of the binary DSQN score, whereas the asymmetric network (DATSQN) reaches about 130 percent, and the training-gradient norm shows no vanishing or exploding behavior.

Load-bearing premise

The explanation stands on identifying the training signal with the derivative of spike probabilities: if a symmetric ternary neuron produces positive and negative spikes equally often, its expected gradient is claimed to be zero, and that zero is what hurts learning.

Editorial extensions

If this is right

  • DATSQN can be deployed as a fully spiking Q-network with a rate-coded first layer and a shorter simulation window, without paying the performance penalty that forced earlier DSQNs to use a non-spiking, floating-point first layer.
  • Ternary representation capacity is usable in reinforcement learning, not just in classification or language tasks, once the threshold asymmetry removes the vanishing expected gradient.
  • The neuron adds only a trainable threshold parameter and no multiplications, so the energy-efficiency advantage of spiking networks is preserved.
  • Because block dynamical isometry holds at least as well as for ReLU, networks built from asymmetric ternary neurons should avoid gradient vanishing and exploding in other deep architectures, not only in Q-learning.
  • The learned thresholds drift toward an excitatory-dominated regime, with positive spikes outnumbering negative spikes by a ratio of about 65 percent or higher, which the paper connects to the biological excitatory-inhibitory balance known to support efficient learning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence the paper leaves implicit: if the vanishing-expected-gradient mechanism is right, symmetric ternary neurons should degrade most in settings where the training signal is bootstrapped from the network's own predictions, as in Q-learning, while supervised tasks should be comparatively robust; this predicts where ternary spiking will need asymmetric thresholds.
  • The analysis compares a strictly positive surrogate-gradient expectation with a zero expected derivative of the stochastic spike probability; the paper's explanation lives in that gap, so measuring the empirical average of the actual surrogate gradient during DTSQN training would directly test whether the bias mechanism, rather than some other property of symmetric thresholds, causes the drop.
  • A cheaper alternative to threshold asymmetry would be a trainable bias that shifts the membrane-potential distribution, achieving $p_+ \neq p_-$ with symmetric thresholds; comparing that variant with DATSQN would show whether asymmetry itself or simply the nonzero probability difference is the active ingredient.
  • For neuromorphic deployment, the need to train $v_{th}^n$ per neuron adds a gradient path through a threshold parameter that inference-only chips may not support, so the practical benefit may depend on whether on-chip or offline training can update this parameter.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper studies ternary spiking neurons in deep Q-learning. It reports that the existing symmetric ternary LIF neuron (DTSQN) performs worse than a binary-neuron DSQN on seven Atari games, hypothesizes that this is due to a zero expected gradient that arises when positive and negative spikes are equally probable, and proposes an asymmetric ternary neuron (DATSQN) with unequal positive and negative firing thresholds. The authors claim DATSQN improves over DSQN by 30% on average and provide a dynamical-isometry argument for training stability. The evaluation uses a fully spiking Bernoulli input encoder and a simulation time window of 20, which are more neuromorphic-friendly than the baseline setup.

Significance. If the central empirical claim were established, the paper would make a useful contribution to energy-efficient spiking reinforcement learning: it introduces a simple, multiplication-free neuron variant, evaluates it under a fully spiking input encoding, and includes an ablation separating the effect of a trainable threshold from that of asymmetric thresholds. The paper also explicitly attempts a theoretical account of why ternary neurons fail in Q-learning, which is a question of genuine interest. However, the current evidence does not support the headline 30% improvement: the number is an average of per-game ratios rather than an aggregate or statistically grounded effect, the experiments use a single training run per condition, and the theoretical mechanism confuses the expected surrogate gradient with the derivative of the stochastic spike expectation. The idea is promising and the empirical trend is suggestive, but the quantitative and statistical claims need substantial rework before the paper can be accepted.

major comments (5)
  1. [Section 5, Tables 4 and 5] The headline "30% improvement" is the mean of seven per-game DATSQN/DSQN ratios, not an aggregate improvement. Summing the raw scores in Table 4 gives roughly 61,537 for DATSQN versus 58,070 for DSQN, an aggregate gain of about 6%, not 30%. The ratio average is dominated by Beam Rider (193%), which has the largest absolute score scale. Moreover, Table 4 shows that in Boxing DATSQN (79.1 ± 7.1) is not above DSQN (79.3 ± 9.5), so the statement that DATSQN "consistently achieves higher scores than DSQN" is not accurate. Please report aggregate scores, per-game differences with confidence intervals, and avoid averaging scale-dependent ratios.
  2. [Section 5, 'Testing and Metrics' and Table 4] The experiments use a single training run per condition, and the reported standard deviation is over 10 evaluation episodes of the best checkpoint selected during training. This standard deviation reflects evaluation noise, not training variability, and best-checkpoint selection can inflate scores. With one seed, the differences are not statistically established; for example, Breakout is 207.7 ± 64 (DSQN) versus 252.14 ± 70 (DATSQN) and Gopher is 1842 ± 465 versus 2111.1 ± 648, both well within one standard deviation. Multiple training seeds (at least 3–5) and a fixed evaluation protocol are needed to support the central improvement claim.
  3. [Section 5, Hyperparameters and Table 3] The decay factor β and learning rate were tuned on Breakout, and Breakout is also one of the seven reported test games. This makes the reported improvement on Breakout optimistic and means the other games are not fully held out with respect to these hyperparameters. The authors should either tune on a separate game or environment, or report sensitivity to these hyperparameters across games.
  4. [Section 4.1, Equations (16)–(18) and Lemma 4.1] The theoretical explanation conflates two distinct quantities. Equation (16) defines the expected surrogate gradient E[GE(m,t)] = ∫ GE(m)p(m,t)dm, which is strictly positive for the listed surrogate functions and is the quantity used in backpropagation. Equations (17)–(18) instead compute dE[˜s]/dm = d(p+ − p−)/dm, which is the derivative of the expected stochastic spike, not the surrogate gradient used in training. Lemma 4.1 then follows almost by construction: maximum entropy is defined as p+ = p−, and Equation (18) is defined so that p+ = p− gives zero. This does not establish an independent trade-off between representation capacity and gradient bias, and it does not directly connect to the surrogate-gradient training described in Algorithm 1. The mechanism for DTSQN's degradation is therefore not supported by the presented mathematics; at minimum, the relationship between d(p+ − p−)/dm and the actual surrogate gradient update must be made explicit and justified.
  5. [Section 4.3, Lemma 4.6 and Theorem 4.7] The dynamical-isometry analysis assumes that the Jacobian of the threshold function has entries equal to 1 when m(t) ∈ (−vthn, vthp) and 0 otherwise. This is a straight-through estimator with a boxcar gradient, but the surrogate gradients listed in Table 1 and used elsewhere in the paper (Atan, STE, Sigmoid) are smooth positive functions, not boxcar functions. The isometry result is therefore not connected to the actual training procedure. Please clarify which gradient estimator is used in the isometry analysis and in the experiments, or revise the analysis to match the surrogate used in training.
minor comments (5)
  1. [Section 4.1, Remark 4.2] Remark 4.2 states that "the expected values of the gradient estimators are nonzero" and then attributes high bias to ternary LIF; this appears to contradict the preceding bullet that says the expected gradient is zero for symmetric ternary thresholds. Please rephrase to distinguish the expected surrogate gradient (positive) from the expected stochastic-spike gradient (zero for symmetric ternary).
  2. [Algorithm 1, line 10] There is a missing closing parenthesis in "s_k(t) = Threshold(mk(t)"; please fix the typo.
  3. [Section 3, Equation (2)] The reset potential v_reset is set to 0 in Table 3, but Equation (2) and the surrounding text treat v_reset as a general parameter. Consider stating explicitly that the analysis and experiments use v_reset = 0, since this is used in the Gaussian membrane-potential approximation.
  4. [Section 5, Figure 3] The caption of Figure 3(b) mentions "DATSQN with one versus two trainable threshold parameters," but the text says DATSQN uses two independent trainable thresholds. Please clarify whether the comparison is between one and two trainable thresholds within the asymmetric model, and report the corresponding numeric scores.
  5. [Throughout] The manuscript uses "SSN" in a few places (e.g., "SSNs currently lag") where "SNN" is intended; please correct the terminology.

Circularity Check

1 steps flagged · score 6.0 of 10

Lemma 4.1 is definitional: 'maximum entropy' (p+=p−) and the Eq.-18 'expected gradient' d(p+−p−)/dm are the same condition, so the claimed explanation of DTSQN degradation is built into the definitions; the empirical DATSQN-vs-DSQN comparison remains an independent experiment.

  1. self definitional [Section 4.1, Equations (15), (18), and Lemma 4.1]
    "the maximum information entropy is achieved when p− = p+, reflecting a balanced and symmetric distribution ... For ternary neurons with symmetric thresholds, where vthn = vthp, we have p+ = p− ... Thus ... the expected gradient becomes zero. Lemma 4.1. The network achieves maximum representation capacity when the expected value of the gradient through the network approaches zero."

    Equation (18) defines the 'expected value of the spike generation gradient' as ∂(p+ − p−)/∂m(t), while Equation (15) defines maximum ternary entropy (at fixed firing rate) as p+ = p−. Lemma 4.1 then 'proves' that maximum capacity forces the expected gradient to zero; this is just restating the equivalence p+ = p− ⇔ p+ − p− = 0. No independent constraint or mechanism is introduced, so the claimed capacity-bias trade-off is built into the definitions. The step is also disconnected from the actual training signal: Equation (16) defines the surrogate-gradient expectation E[GE(m)] used in backpropagation, which the paper itself says is strictly positive for all neuron types, so the Eq.-18 quantity is not the training gradient.

full rationale

The paper's main empirical claim—DATSQN outperforms DSQN—rests on Table 4/5 experiments and is not circular: the asymmetric-threshold network is trained and evaluated independently of the Lemma, and the comparison to DSQN is a direct measurement. The circularity is confined to the theoretical framing in Section 4.1. The paper defines the 'expected gradient' in Eq. 18 as d(p+−p−)/dm, and separately defines maximum entropy as p+=p−; Lemma 4.1 then announces that maximum capacity makes this gradient zero. That is a tautology, not a derived trade-off, and it is further disconnected from the surrogate gradient GE of Eq. 16 actually used in training. Because the theoretical explanation of DTSQN's degradation and the design motivation for the asymmetric thresholds rest on this definitional step, the paper has one partial circular step. There is no load-bearing self-citation: the cited Gaussian membrane-potential result [54], ternary model [31], and dynamic-isometry results [33] come from external groups. The empirical effect size and single-seed evaluation are separate correctness concerns, not circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central performance claim depends on the asymmetric threshold model and on hyperparameters tuned on Breakout. The theoretical explanation additionally rests on a specific stochastic spike model and on a definition of expected gradient that is not the one used in training.

free parameters (4)
  • v_thn (negative threshold) = initialized to 2, trainable
    The asymmetry is implemented by making the negative firing threshold a trainable parameter (Section 4.2, Algorithm 1). Its final values depend on training data, so the performance gain is partly a fitted outcome rather than a test of a fixed model.
  • decay factor beta = 0.9
    Selected by trial and error on the Breakout game (Section 5, Experimental Setup), which is also one of the seven evaluation games.
  • learning rate = 0.00005
    Selected by trial and error on the Breakout game (Section 5, Experimental Setup).
  • v_thp (positive threshold) = 1
    Fixed based on prior work [31,51]; not fitted here but is a hyperparameter that affects firing rates and entropy.
assumptions (5)
  • domain assumption Membrane potential distribution is Gaussian-like and symmetric about Vreset = 0
    Used in Section 4.1 to claim p+ = p- for symmetric thresholds, which drives the zero-expected-gradient result (Equations 12-14). Real membrane potential distributions are skewed and biased.
  • ad hoc to paper The gradient estimator used in backpropagation equals dE[s]/dm
    Equations 17-18 define the expected gradient as the derivative of marginal spike probability. This differs from the surrogate gradient GE(m) defined in Equation 16 and used in training; the equivalence is asserted without proof.
  • domain assumption Stochastic spike model of Guo et al. [31]
    Spikes are treated as random variables with probabilities p0, p+, p-; the entire analysis is built on this model.
  • domain assumption Firing rate r is the same for binary and ternary neurons in the comparison
    Used in the entropy and gradient comparisons (Equation 15, Theorem 4.3).
  • domain assumption Firing rate r is between 0.1 and 0.5 and the input distribution is zero-mean for the isometry comparison
    Theorem 4.7 relies on these ranges to claim asymmetric ternary matches or beats ReLU isometry.
invented entities (1)
  • Asymmetric ternary LIF neuron
    purpose: Output -1, 0, or +1 with unequal positive and negative thresholds to keep the expected gradient nonzero while increasing entropy
    Only evaluated on the paper's own seven-game Atari setup; no external benchmarks, hardware implementation, or independent replication. The biological excitation-inhibition argument is post hoc.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Performance of Spike-based Deep Q-Learning using Ternary Neurons." pith.science (2026). https://pith.science/paper/HKNEHJXH

@misc{pith2026250603392,
  author       = {Pith},
  title        = {Pith review of: Improving Performance of Spike-based Deep Q-Learning using Ternary Neurons},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HKNEHJXH}},
  note         = {Machine review of arXiv:2506.03392}
}
read the original abstract

We propose a new ternary spiking neuron model to improve the representation capacity of binary spiking neurons in deep Q-learning. Although a ternary neuron model has recently been introduced to overcome the limited representation capacity offered by the binary spiking neurons, we show that its performance is worse than that of binary models in deep Q-learning tasks. We hypothesize gradient estimation bias during the training process as the underlying potential cause through mathematical and empirical analysis. We propose a novel ternary spiking neuron model to mitigate this issue by reducing the estimation bias. We use the proposed ternary spiking neuron as the fundamental computing unit in a deep spiking Q-learning network (DSQN) and evaluate the network's performance in seven Atari games from the Gym environment. Results show that the proposed ternary spiking neuron mitigates the drastic performance degradation of ternary neurons in Q-learning tasks and improves the network performance compared to the existing binary neurons, making DSQN a more practical solution for on-board autonomous decision-making tasks.

Figures

Figures reproduced from arXiv: 2506.03392 by the authors.

Figure 1
Figure 1. Approximate Gaussian distribution of the input term [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. (a) Learning curves, (b) average gradient norm across the network during training for each [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Learning curves for (a) DSQN with and without trainable thresholds, and (b) DATSQN [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

60 extracted references · 47 canonical work pages

  1. [1]

    Target-driven visual navigation in indoor scenes using deep reinforcement learning

    Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In2017 IEEE international conference on robotics and automation (ICRA), pages 3357–3364. IEEE, 2017

    2017

  2. [2]

    Automated deep reinforcement learning environment for hardware of a modular legged robot

    Sehoon Ha, Joohyung Kim, and Katsu Yamane. Automated deep reinforcement learning environment for hardware of a modular legged robot. In2018 15th international conference on ubiquitous robots (UR), pages 348–354. IEEE, 2018

    work page 2018

  3. [3]

    A sim-to-real pipeline for deep reinforcement learning for autonomous robot navigation in cluttered rough terrain.IEEE Robotics and Automation Letters, 6(4):6569–6576, 2021

    Han Hu, Kaicheng Zhang, Aaron Hao Tan, Michael Ruan, Christopher Agia, and Goldie Nejat. A sim-to-real pipeline for deep reinforcement learning for autonomous robot navigation in cluttered rough terrain.IEEE Robotics and Automation Letters, 6(4):6569–6576, 2021

    work page 2021

  4. [4]

    Tactical decision-making for autonomous driving using dueling double deep q network with double attention.IEEE Access, 9:151983–151992, 2021

    Shuwei Zhang, Yutian Wu, Harutoshi Ogai, Hiroshi Inujima, and Shigeyuki Tateno. Tactical decision-making for autonomous driving using dueling double deep q network with double attention.IEEE Access, 9:151983–151992, 2021

    work page 2021

  5. [5]

    High-performance temporal reversible spiking neural networks with o(l) training memory and o(1) inference cost

    JiaKui Hu, Man Yao, Xuerui Qiu, Yuhong Chou, Yuxuan Cai, Ning Qiao, Yonghong Tian, XU Bo, and Guoqi Li. High-performance temporal reversible spiking neural networks with o(l) training memory and o(1) inference cost. InForty-first International Conference on Machine Learning, 2024

    work page 2024

  6. [6]

    Spiking neural networks.J

    Samanwoy Ghosh-Dastidar et al. Spiking neural networks.J. Neural Systems, pages 295–308, 2009

    work page 2009

  7. [7]

    E. M. Izhikevich. Simple model of spiking neurons.IEEE Trans. Neural Net., 14(6):1569–1572, Nov 2003

    work page 2003

  8. [8]

    Spiking deep convolutional neural networks for energy-efficient object recognition.Int’l J

    Yongqiang Cao, Yang Chen, and Deepak Khosla. Spiking deep convolutional neural networks for energy-efficient object recognition.Int’l J. Computer Vision, 113:54–66, 2015

    work page 2015

Show all 60 references
  1. [9]

    Spatio-temporal backpropagation for training high-performance spiking neural networks.Frontiers in neuroscience, 12:323875, 2018

    Yujie Wu, Lei Deng, Guoqi Li, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks.Frontiers in neuroscience, 12:323875, 2018

    2018

  2. [10]

    Enabling spike-based backpropagation for training deep neural network architec- tures.Frontiers in neuroscience, 14:497482, 2020

    Chankyu Lee, Syed Shakib Sarwar, Priyadarshini Panda, Gopalakrishnan Srinivasan, and Kaushik Roy. Enabling spike-based backpropagation for training deep neural network architec- tures.Frontiers in neuroscience, 14:497482, 2020

    2020

  3. [11]

    Spiking convolutional neural networks for text classification

    Changze Lv, Jianhan Xu, and Xiaoqing Zheng. Spiking convolutional neural networks for text classification. InInt’l Conf. Learning Representations (ICLR), 2022

    2022

  4. [12]

    Francisco Naveros, Niceto R Luque, Jesús A Garrido, Richard R Carrillo, Mancia Anguita, and Eduardo Ros. A spiking neural simulator integrating event-driven and time-driven computation schemes using parallel cpu-gpu co-processing: a case study.IEEE transactions on neural netwo...

    2014

  5. [13]

    Generalized leaky integrate-and-fire models classify multiple neuron types.Nature communications, 9(1):709, 2018

    Corinne Teeter, Ramakrishnan Iyer, Vilas Menon, Nathan Gouwens, David Feng, Jim Berg, Aaron Szafer, Nicholas Cain, Hongkui Zeng, Michael Hawrylycz, et al. Generalized leaky integrate-and-fire models classify multiple neuron types.Nature communications, 9(1):709, 2018

    2018

  6. [14]

    Opportunities for neuromorphic computing algorithms and applications.Nature Computational Science, 2(1):10–19, 2022

    Catherine D Schuman, Shruti R Kulkarni, Maryam Parsa, J Parker Mitchell, Prasanna Date, and Bill Kay. Opportunities for neuromorphic computing algorithms and applications.Nature Computational Science, 2(1):10–19, 2022

    2022

  7. [15]

    Moradi, N

    S. Moradi, N. Qiao, F. Stefanini, and G. Indiveri. A scalable multicore architecture with hetero- geneous memory structures for dynamic neuromorphic asynchronous processors (DYNAPs). Biomedical Circuits and Systems, IEEE Transactions on, 12(1):106–122, Feb. 2018

    2018

  8. [16]

    Loihi: A neuromorphic manycore processor with on-chip learning.Ieee Micro, 38(1):82–99, 2018

    Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning.Ieee Micro, 38(1):82–99, 2018. 14

    2018

  9. [17]

    Brien M. Posey. Akida event domain neural processor. Technical report, Brainchip, 2023

    2023

  10. [18]

    Debole, Brian Taba, Arnon Amir, et al

    Michael V . Debole, Brian Taba, Arnon Amir, et al. TrueNorth: Accelerating from zero to 64 million neurons in 10 years.Computer, 2019

    2019

  11. [19]

    An efficient FPGA-based overlay inference archi- tecture for fully connected DNNs

    Ahmed M Abdelsalam, Felix Boulet, et al. An efficient FPGA-based overlay inference archi- tecture for fully connected DNNs. InProc. IEEE Conf. Reconfigurable Computing & FPGAs (ReConFig), pages 1–6, 2018

    2018

  12. [20]

    Gyro: A digital spiking neural network architecture for multi-sensory data analytics

    Federico Corradi, Guido Adriaans, and Sander Stuijk. Gyro: A digital spiking neural network architecture for multi-sensory data analytics. InProc. ACM Drone Syst. Eng. & Rapid Simulation & Perf. Eval., page 9–15, 2021

    2021

  13. [21]

    A fast and energy-efficient SNN processor with adaptive clock/event-driven computation scheme and online learning.IEEE Trans

    Sixu Li, Zhaomin Zhang, Ruixin Mao, Jianbiao Xiao, Liang Chang, and Jun Zhou. A fast and energy-efficient SNN processor with adaptive clock/event-driven computation scheme and online learning.IEEE Trans. Circuits & Systems I, 68(4):1543–1552, 2021

    2021

  14. [22]

    A low power and low latency FPGA- based spiking neural network accelerator

    Hanwen Liu, Yi Chen, Zihang Zeng, Malu Zhang, et al. A low power and low latency FPGA- based spiking neural network accelerator. InIEEE Int’l Joint Conf. Neural Networks (IJCNN), pages 1–8, 2023

    2023

  15. [23]

    Spiker: An FPGA-optimized hardware accelerator for spiking neural networks

    Alessio Carpegna, Alessandro Savino, and Stefano Di Carlo. Spiker: An FPGA-optimized hardware accelerator for spiking neural networks. InIEEE Computer Society Annual Symp. VLSI (ISVLSI), pages 14–19, 2022

    2022

  16. [24]

    A fully-configurable open-source software-defined digital quantized spiking neural core architecture

    Shadi Matinizadeh et al. A fully-configurable open-source software-defined digital quantized spiking neural core architecture. InIEEE Int’l Midwest Symp. Circuits & Systems (MWSCAS), 2024

    2024

  17. [25]

    Brainqn: Enhancing the robustness of deep reinforcement learning with spiking neural networks.Advanced Intelligent Systems, 6(9):2400075, 2024

    Shuo Feng, Jian Cao, Zehong Ou, Guang Chen, Yi Zhong, Zilin Wang, Juntong Yan, Jue Chen, Bingsen Wang, Chenglong Zou, et al. Brainqn: Enhancing the robustness of deep reinforcement learning with spiking neural networks.Advanced Intelligent Systems, 6(9):2400075, 2024

    2024

  18. [26]

    Deep rein- forcement learning with population-coded spiking neural network for continuous control

    Guangzhi Tang, Neelesh Kumar, Raymond Yoo, and Konstantinos Michmizos. Deep rein- forcement learning with population-coded spiking neural network for continuous control. In Conference on Robot Learning, pages 2016–2029. PMLR, 2021

    2016

  19. [27]

    Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation

    Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. arXiv preprint arXiv:2005.01807, 2020

    2005 arXiv

  20. [28]

    Improved robustness of reinforcement learning policies upon conversion to spiking neuronal network platforms applied to atari breakout game.Neural Networks, 120:108–115, 2019

    Devdhar Patel, Hananel Hazan, Daniel J Saunders, Hava T Siegelmann, and Robert Kozma. Improved robustness of reinforcement learning policies upon conversion to spiking neuronal network platforms applied to atari breakout game.Neural Networks, 120:108–115, 2019

    2019

  21. [29]

    Human-level control through directly trained deep spiking q-networks.IEEE transactions on cybernetics, 53(11): 7187–7198, 2022

    Guisong Liu, Wenjie Deng, Xiurui Xie, Li Huang, and Huajin Tang. Human-level control through directly trained deep spiking q-networks.IEEE transactions on cybernetics, 53(11): 7187–7198, 2022

    2022

  22. [30]

    Reinforcement co-learning of deep and spiking neural networks for energy-efficient mapless navigation with neuromorphic hardware

    Guangzhi Tang, Neelesh Kumar, and Konstantinos P Michmizos. Reinforcement co-learning of deep and spiking neural networks for energy-efficient mapless navigation with neuromorphic hardware. In2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), page...

    2020

  23. [31]

    Ternary spike: Learning ternary spikes for spiking neural networks

    Yufei Guo, Yuanpei Chen, Xiaode Liu, Weihang Peng, Yuhan Zhang, Xuhui Huang, and Zhe Ma. Ternary spike: Learning ternary spikes for spiking neural networks. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 12244–12252, 2024

    2024

  24. [32]

    Spikelm: Towards general spike-driven language modeling via elastic bi-spiking mechanisms.arXiv preprint arXiv:2406.03287, 2024

    Xingrun Xing, Zheng Zhang, Ziyi Ni, Shitao Xiao, Yiming Ju, Siqi Fan, Yequan Wang, Jiajun Zhang, and Guoqi Li. Spikelm: Towards general spike-driven language modeling via elastic bi-spiking mechanisms.arXiv preprint arXiv:2406.03287, 2024

    2024 arXiv

  25. [33]

    Zhaodong Chen, Lei Deng, Bangyan Wang, Guoqi Li, and Yuan Xie. A comprehensive and modularized statistical framework for gradient norm equality in deep neural networks.IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1):13–31, 2020. 15

    2020

  26. [34]

    Springer, 2023

    Shengbo Eben Li.Reinforcement learning for sequential decision and optimal control. Springer, 2023

    2023

  27. [35]

    Playing atari with deep reinforcement learning.arXiv preprint arXiv:1312.5602, 2013

    V olodymyr Mnih. Playing atari with deep reinforcement learning.arXiv preprint arXiv:1312.5602, 2013

    2013 arXiv

  28. [36]

    Human-level control through deep reinforcement learning.nature, 518(7540):529–533, 2015

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning.nature, 518(7540):529–533, 2015

    2015

  29. [37]

    Deep reinforcement learning with double q-learning

    Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. InProceedings of the AAAI conference on artificial intelligence, volume 30, 2016

    2016

  30. [38]

    Dueling network architectures for deep reinforcement learning

    Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. InInternational conference on machine learning, pages 1995–2003. PMLR, 2016

    1995

  31. [39]

    Mohit Sewak and Mohit Sewak. Deep q network (dqn), double dqn, and dueling dqn: A step towards general artificial intelligence.Deep reinforcement learning: frontiers of artificial intelligence, pages 95–108, 2019

    2019

  32. [40]

    Strategy and benchmark for converting deep q-networks to event-driven spiking neural networks

    Weihao Tan, Devdhar Patel, and Robert Kozma. Strategy and benchmark for converting deep q-networks to event-driven spiking neural networks. InProceedings of the AAAI conference on artificial intelligence, volume 35, pages 9816–9824, 2021

    2021

  33. [41]

    Solving the spike feature information vanishing problem in spiking deep q network with potential based normalization.Frontiers in Neuroscience, 16: 953368, 2022

    Yinqian Sun, Yi Zeng, and Yang Li. Solving the spike feature information vanishing problem in spiking deep q network with potential based normalization.Frontiers in Neuroscience, 16: 953368, 2022

    2022

  34. [42]

    Toward robust and scalable deep spiking reinforcement learning.Frontiers in Neurorobotics, 16:1075647, 2023

    Mahmoud Akl, Deniz Ergene, Florian Walter, and Alois Knoll. Toward robust and scalable deep spiking reinforcement learning.Frontiers in Neurorobotics, 16:1075647, 2023

    2023

  35. [43]

    Sf-dqn: Provable knowledge transfer using successor feature for deep reinforcement learning.arXiv preprint arXiv:2405.15920, 2024

    Shuai Zhang, Heshan Devaka Fernando, Miao Liu, Keerthiram Murugesan, Songtao Lu, Pin-Yu Chen, Tianyi Chen, and Meng Wang. Sf-dqn: Provable knowledge transfer using successor feature for deep reinforcement learning.arXiv preprint arXiv:2405.15920, 2024

    2024 arXiv

  36. [44]

    Path planning via an improved dqn-based learning policy.IEEE Access, 7:67319–67330, 2019

    Liangheng Lv, Sunjie Zhang, Derui Ding, and Yongxiong Wang. Path planning via an improved dqn-based learning policy.IEEE Access, 7:67319–67330, 2019

    2019

  37. [45]

    Deep q-learning from demonstrations

    Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  38. [46]

    Event- triggered deep reinforcement learning using parallel control: A case study in autonomous driving.IEEE Transactions on Intelligent Vehicles, 8(4):2821–2831, 2023

    Jingwei Lu, Liyuan Han, Qinglai Wei, Xiao Wang, Xingyuan Dai, and Fei-Yue Wang. Event- triggered deep reinforcement learning using parallel control: A case study in autonomous driving.IEEE Transactions on Intelligent Vehicles, 8(4):2821–2831, 2023

    2023

  39. [47]

    Distributed prioritized experience replay.arXiv preprint arXiv:1803.00933, 2018

    Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay.arXiv preprint arXiv:1803.00933, 2018

    2018 arXiv

  40. [48]

    The frequency of nerve action potentials generated by applied currents.Proceedings of the Royal Society of London

    RB Stein and Alan Lloyd Hodgkin. The frequency of nerve action potentials generated by applied currents.Proceedings of the Royal Society of London. Series B. Biological Sciences, 167(1006):64–86, 1967

    1967

  41. [49]

    Towards efficient spiking transformer: a token sparsification framework for training and inference acceleration

    Zhengyang Zhuge, Peisong Wang, Xingting Yao, and Jian Cheng. Towards efficient spiking transformer: a token sparsification framework for training and inference acceleration. In Forty-first International Conference on Machine Learning, 2024

    2024

  42. [50]

    Ndot: Neuronal dynamics-based online training for spiking neural networks

    Haiyan Jiang, Giulia De Masi, Huan Xiong, and Bin Gu. Ndot: Neuronal dynamics-based online training for spiking neural networks. InForty-first International Conference on Machine Learning, 2024. 16

    2024

  43. [51]

    Clif: Complementary leaky integrate-and-fire neuron for spiking neural networks.arXiv preprint arXiv:2402.04663, 2024

    Yulong Huang, Xiaopeng Lin, Hongwei Ren, Haotian Fu, Yue Zhou, Zunchang Liu, Biao Pan, and Bojun Cheng. Clif: Complementary leaky integrate-and-fire neuron for spiking neural networks.arXiv preprint arXiv:2402.04663, 2024

    2024 arXiv

  44. [52]

    Adaptive smoothing gradient learning for spiking neural networks

    Ziming Wang, Runhao Jiang, Shuang Lian, Rui Yan, and Huajin Tang. Adaptive smoothing gradient learning for spiking neural networks. InInternational Conference on Machine Learning, pages 35798–35816. PMLR, 2023

    2023

  45. [53]

    Enhancing the robustness of spiking neural networks with stochastic gating mechanisms

    Jianhao Ding, Zhaofei Yu, Tiejun Huang, and Jian K Liu. Enhancing the robustness of spiking neural networks with stochastic gating mechanisms. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 492–502, 2024

    2024

  46. [54]

    Cambridge university press, 2002

    Wulfram Gerstner and Werner M Kistler.Spiking neuron models: Single neurons, populations, plasticity. Cambridge university press, 2002

    2002

  47. [55]

    Direct training for spiking neural networks: Faster, larger, better

    Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. InProceedings of the AAAI conference on artificial intelligence, volume 33, pages 1311–1318, 2019

    2019

  48. [56]

    Going deeper with directly-trained larger spiking neural networks

    Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. InProceedings of the AAAI conference on artificial intelligence, volume 35, pages 11062–11070, 2021

    2021

  49. [57]

    Training spiking neural networks using lessons from deep learning.Proceedings of the IEEE, 111(9):1016–1054, 2023

    Jason K Eshraghian, Max Ward, Emre O Neftci, Xinxin Wang, Gregor Lenz, Girish Dwivedi, Mohammed Bennamoun, Doo Seok Jeong, and Wei D Lu. Training spiking neural networks using lessons from deep learning.Proceedings of the IEEE, 111(9):1016–1054, 2023

    2023

  50. [58]

    Why do we have so many excitatory neurons?bioRxiv, pages 2024–09, 2024

    Qingyang Wang, Albert Cardona, Marta Zlatic, Joshua T V ogelstein, and Carey E Priebe. Why do we have so many excitatory neurons?bioRxiv, pages 2024–09, 2024

    2024

  51. [59]

    Generation of diverse cortical inhibitory interneurons

    Khadeejah T Sultan and Song-Hai Shi. Generation of diverse cortical inhibitory interneurons. Wiley Interdisciplinary Reviews: Developmental Biology, 7(2):e306, 2018

    2018

  52. [60]

    Openai gym, 2016

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016. 17

    2016

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.