Pith. sign in

REVIEW 5 major objections 5 minor 51 references

Revisiting Reset Mechanisms in Spiking Neural Networks for Sequential Modeling: Specialized Discretization for Binary Activated RNN

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that reset and refractory mechanisms in spiking neural networks can be replaced by a fixed refractory period, because spikes only transmit information between layers and do not store history.

desk verdict An honest SSM+PSN baseline with a fresh but unsupported theory; Theorem 4.1 fails on standard hard-reset LIF. read the letter →

arxiv 2504.17751 v4 pith:VMKJK62O submitted 2025-04-24 cs.NE cs.AI

Enqi Zhang This is my paper · ORCID
classification cs.NEcs.AI
keywords SpikingNeuralNetworksSequentialModelingStateSpaceModelsResetMechanismRefractoryPeriodBinary-ActivatedRNNParallelTrainingSparse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that when spiking neural networks are used for sequence modeling, the memory and the spiking can be separated: a state-space module stores history, while spikes only transmit the stored information to the next layer. Reset and refractory mechanisms are therefore not essential to spiking; they are a special discretization that turns a continuous recurrent output into sparse sampling points. Because the next layer receives every spike as the same binary event, the paper claims the shape of the refractory function is unobservable, so a fixed refractory period with no reset is enough to keep spiking sparse. The paper builds two simple models on this idea and reports that the stronger one reaches 85.5% accuracy on Sequential CIFAR-10 while training in parallel. If the claim holds, complex nonlinear spiking dynamics are not needed for engineered sequence tasks.

What carries the argument

The carrying object is the spike encoding function $S[f(t)]$ of Theorem 4.1, which expresses any reset-and-refractory spiking scheme as firing exactly when the memory output $f(t)$ crosses the threshold $\theta$ and has stayed below it for the preceding $m(t)$ timesteps. The paper's move is to set $m(t)$ to a constant, removing reset and accumulation while keeping a fixed refractory period; because the next layer sums identical binary spikes linearly, the refractory function's shape cannot be perceived, so the fixed period is information-preserving. Theorem 4.2 supplies the parallel-training mechanism: if the memory output is constant over an interval of length $r$, the interval's effect equals one scaled input at the interval start, so a strided temporal convolution can merge the interval and satisfy the spike pattern with a single computation. The whole architecture is an SSM memory module followed by this fixed-refractory spiking module.

What would settle it

Take a fixed input sequence, generate two spike trains with identical spike times but different refractory functions, one fixed and one LIF-style, and feed both into a second layer whose synaptic integration is nonlinear in spike timing, such as a saturating or multiplicative synaptic model; if the two downstream outputs differ, the claim that the refractory function's shape is unobservable is false. A positive control would show identical outputs when the second layer is linear.

Watch

Extended reading notes

Core claim

The paper's central claim is that a sequence-modeling spiking neural network is a binary-activated recurrent network with two independent components: a linear memory module, equivalent to a state-space model, and a spiking module that only transmits the memory module's output between layers. It formalizes reset and refractory mechanisms as a time-varying spike-encoding function $S[f(t)]$ that fires when the memory output crosses the threshold and has not fired for the preceding $m(t)$ steps; this dependence on past spikes is what blocks training across timesteps. The paper then asserts that since every spike is perceived identically by the next layer, $m(t)$ can be fixed to a constant, giving a fixed refractory period with no reset and no accumulation; sparse spiking is preserved and training becomes a fixed convolution. A second theorem shows that when the memory output is constant over an interval, a single computation represents the whole interval. The resulting spikingPssm, an SSM memory followed by a specialized parallel spiking module, reaches 85.5% accuracy on Sequential CIFAR-10, which the paper presents as evidence that LIF/IF encoding with reset is not required.

Load-bearing premise

The argument stands on the premise that every spike is received by the next layer as the same binary value added linearly, so the shape of the refractory period cannot matter; if precise spike timing or the refractory waveform itself carries information, a fixed refractory period loses that information.

Editorial extensions

If this is right

  • Reset and refractory mechanisms can be replaced by a fixed refractory period without changing what a downstream layer perceives, provided spikes are binary and post-synaptic integration is linear.
  • Because the refractory period is fixed, spike suppression no longer depends on previous outputs, so training can run all timesteps in parallel through a fixed convolution.
  • SNN sequence models split cleanly into a memory module and a spiking module; any memory system usable in an ANN, such as state-space or linear attention, can be paired with the same spiking module.
  • Sparse spiking can be achieved by temporal output sharing: if the memory output is constant over an interval, compute once and reuse the pattern, which yields the same energy savings as suppressing spikes.
  • A simple SSM plus spiking module reaches 85.5% on Sequential CIFAR-10, so many complex nonlinear neuronal dynamics in current SNNs are not necessary for competitive sequence classification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the refractory-shape claim is right, neuromorphic hardware could drop per-neuron reset state and refractory-timing logic, replacing it with a single global fixed-pause rule; a testable extension is benchmarking such hardware-level simplification against reset-based designs.
  • The distribution-sampling framing suggests a new evaluation: compare the spike train's ability to reconstruct the memory module's continuous output, for instance by a distance between the original and spike-sampled sequences; this would measure information loss more directly than classification accuracy.
  • The argument implies the decoupling should transfer to any memory module, including attention-based or linear-attention memories; a natural test is swapping the SSM memory for linear attention and checking whether the fixed-refractory spiking module still matches reset-LIF baselines.
  • Because the central equivalence depends on linear integration, nonlinear post-synaptic effects such as dendritic saturation or timing-dependent plasticity are where the claim would likely break; measuring those effects in a spiking network is a direct stress test.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes viewing spiking neural networks as binary-activated recurrent neural networks for sequence modeling, decoupling them into a memory module (modeled by state space models) and a spiking transmission module. Its central theoretical claim is that reset and refractory mechanisms in SNNs act only as a special discretization or sparse sampling of the memory output, so that a fixed refractory period suffices and the spiking mechanism contributes no historical memory. On this basis the paper proposes two parallel-trainable architectures, spikingFRssm and spikingPssm, and reports accuracy on Sequential CIFAR-10.

Significance. If the central claim were correct, the paper would make a useful conceptual contribution: it would justify replacing reset-based LIF dynamics with a fixed refractory period, enabling parallel training while retaining sparse spike trains, and it would clarify the role of spikes in SSM-based sequence models. The paper is commendably candid about its limitations, provides implementation details in Algorithm 1, and Theorem 4.2 is a correct elementary calculation. However, the significance is substantially limited by the correctness problem in Theorem 4.1, by the informal 'distribution sampling' framework, and by the narrow empirical support.

major comments (5)
  1. [§4.2, Theorem 4.1 / Eq. (17)] Theorem 4.1 is not universal for the LIF model the paper claims to analyze. For the hard-reset LIF of Eq. (5) with V_reset = 0, θ = 1, and constant suprathreshold input I = 2, the neuron fires at every timestep because U(t+1) = (1 - 1/τ)V(t) + 2 = 2 at both t and t+1. Equation (17) requires f(t - Δt) ≤ θ, so it predicts no second spike. The theorem describes a threshold-and-blank encoder applied to the current memory output f(t), not an integrate-and-fire neuron whose firing depends on accumulated membrane potential. Since the fixed-refractory architecture in §4.2 is derived from this 'universal' encoding, the central claim that reset and refractory mechanisms do not contribute to historical memory is unsupported.
  2. [§4.2, refractory-function premise] The assertion that 'the choice of refractory function does not affect the system's perception of the spikes' assumes that the receiving layer sums binary spikes linearly and that spike timing within a timestep carries no information. This assumption is not derived from Theorem 4.1; it is an additional modeling premise. In the proposed spikingPssm itself, the spiking module applies a learnable temporal convolution in Eq. (32), so the refractory/window function shape changes the output. The premise therefore cannot support the conclusion that a fixed refractory period is information-lossless in general.
  3. [§5.3, Table 2] The empirical evaluation rests on a single dataset (Sequential CIFAR-10) with no error bars, no multiple-seed statistics, and no ablation isolating the fixed-refractory mechanism from the other components (SSM memory, GLU, learnable kernel). The comparative discussion in §5.3 makes claims such as 'SpikingPssm outperforms P-SpikeSSM... and PMSN' on the basis of a single number per method. This is insufficient support for the paper's architectural conclusions.
  4. [§4.3.1, Eq. (23)] The definition of the fixed convolutional kernel in Eq. (23) is garbled by inserted non-ASCII text and does not specify the kernel values beyond the final 1.0. As this kernel is the central implementation of the parallel training mechanism, the manuscript as written is not reproducible at this load-bearing point. The text around Eqs. (23)-(27) should be cleaned and the kernel fully specified.
  5. [§4.3.2, Theorem 4.2] Theorem 4.2 assumes the memory-module output x(t) is constant on the interval [0, r], and the theorem is then used to claim that 'it is entirely feasible to construct a sparse spiking neural network state-space model using regular discrete time steps.' For general inputs, which are not piecewise constant, the equivalence does not hold. The actual spikingPssm architecture uses a learnable convolutional kernel in Eq. (32) that approximates the desired aggregation rather than implementing this theorem exactly. The justification should be stated as a heuristic approximation, not as a feasibility proof.
minor comments (5)
  1. [Abstract] 'an another alternative perspective' should be 'another alternative perspective', and the comma before 'viewing' is misplaced.
  2. [Figure 6 caption] 'Persperctive' should be 'Perspective'.
  3. [§4.2] 'LIF models with either hard or soft resists cannot be directly parallelized' should read 'hard or soft reset'.
  4. [§4.1] The statement that a rigorous definition of the distribution interpretation is unnecessary sits awkwardly with the load the paper places on that interpretation; a formal definition or a precise approximation statement would strengthen the paper.
  5. [Table 2] Several baseline parameter counts are reported as 'N/A'; if the values are unavailable, this should be stated in the text rather than implied.

Circularity Check

2 steps flagged · score 6.0 of 10

Theorem 4.1 defines a threshold-and-refractory encoder and then 'derives' that fixed refractory suffices; the architecture instantiates that definition, so the central theoretical claim is partially circular.

  1. self definitional [Section 4.2, Theorem 4.1 / Eq. (17); applied in Section 4.3.1, Eqs. (18)-(27)]
    "This theorem universally describes the spike encoding scheme for all spiking neural networks with reset mechanisms and refractory periods, where different m(t) values correspond to different reset and refractory configurations. ... Thus, we can directly set m(t) as a time-invariant constant, remove the reset mechanism and the accumulation mechanism in Leaky Integrate-and-Fire , and only retain a fixed refractory period to ensure sparse spike firing."

    Eq. (17) is not derived from the LIF dynamics in Eq. (5); it defines S[f(t)] as a spike only when f(t) crosses theta and the previous m(t) values were below theta. That condition is precisely a refractory/blanking mechanism, and setting m(t) constant is the fixed refractory period later implemented in Eqs. (22)-(27). The paper's own hard-reset LIF in Eq. (5) can fire on consecutive timesteps under constant suprathreshold input (e.g., Vreset=0, input=2, theta=1 gives U=2 at both t and t+1), so the claimed universality is an assumption about what spiking is, not a consequence of the LIF equations.

  2. renaming known result [Section 4.2, final summary paragraph (Fig. 8 discussion)]
    "The sequence mapping of spiking neural networks can essentially be viewed as updating real-time memory for sequences and emitting spikes at irregular discrete time steps, approximating the output distribution of traditional continuous-form RNNs in the form of sampled points. The use of reset mechanisms and refractory periods constitutes this special discretization approach, distinguishing it from traditional RNNs with regular discrete time steps."

    The paper's Section 1 already frames the project by saying: 'We interpret these mechanisms as performing additional sparse sampling on discrete sampling points to achieve sparsity. This perspective leads to our fixed-refractory-period SNN model.' The Section 4.2 summary restates that same interpretive framing as a conclusion: SNNs are 'irregularly discretized' RNNs and reset/refractory are 'this special discretization approach.' Because the conclusion is identical to the initial framing, it does not provide independent support for the fixed-refractory architecture; the new terminology ('special discretization,' 'sampling points') does the explanatory work by renaming the known phenomenon of irregular spike emission.

full rationale

The principal circularity is in the theoretical derivation. Theorem 4.1 is introduced as a universal description of reset/refractory SNNs, but Eq. (17) is a definition of a threshold-and-blank encoder, and the fixed-refractory spiking module in Eqs. (18)-(27) is built to match that definition. The paper's claim that reset and refractory mechanisms are 'a specialized form of discretization' and that a fixed refractory period suffices is therefore substantially an unpacking of its initial interpretive framework rather than a result derived from the LIF dynamics in Eq. (5). The experimental comparisons on Sequential CIFAR-10 are real, and the paper honestly states that the model is simple and not state-of-the-art, so the empirical section is not fabricated; however, the experiments do not independently test the decoupling claim, since the architecture was designed from that claim. The self-citation to the author's prior work [49] in the appendix is not load-bearing for the central argument. Overall, one central definitional step plus a renaming-as-explanation yields partial circularity, score 6.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No new physical entities are introduced. The ledger captures three interpretive assumptions that carry the argument: the sampling view of spikes, the invariance of post-synaptic perception to refractory shape, and the equivalence of reset-free SNNs to SSMs.

free parameters (3)
  • spiking threshold theta = 0.5 (spikingFRssm), 0 (spikingPssm)
    Chosen per model in experiments, not derived from theory.
  • refractory period length r = 5 (also tested 3)
    Hyperparameter selected by hand; affects sparsity and receptive field.
  • surrogate gradient constants = h=0.15, l=gamma=0.5, sigma=l, sigma'=6l
    Chosen from prior surrogate gradient practice, not derived.
assumptions (3)
  • domain assumption The entire input sequence can be treated as a distribution, and spikes as discrete sampling points of that distribution.
    Stated in Section 4.1 as an analytical tool; the paper says a rigorous definition is unnecessary.
  • domain assumption The next layer cannot perceive differences between spikes produced under different refractory functions, because each spike produces the same post-synaptic potential.
    Stated in Section 4.2; used to justify replacing variable refractory with a fixed refractory period.
  • domain assumption When resets are disregarded, an SNN is a linear dynamical system whose memory is equivalent to an SSM.
    Section 4.1; relies on prior SSM theory (HiPPO) for memory characterization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Revisiting Reset Mechanisms in Spiking Neural Networks for Sequential Modeling: Specialized Discretization for Binary Activated RNN." pith.science (2026). https://pith.science/paper/VMKJK62O

@misc{pith2026250417751,
  author       = {Pith},
  title        = {Pith review of: Revisiting Reset Mechanisms in Spiking Neural Networks for Sequential Modeling: Specialized Discretization for Binary Activated RNN},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VMKJK62O}},
  note         = {Machine review of arXiv:2504.17751}
}
read the original abstract

In the field of image recognition, spiking neural networks (SNNs) have achieved performance comparable to conventional artificial neural networks (ANNs). In such applications, SNNs essentially function as traditional neural networks with quantized activation values. This article focuses on an another alternative perspective,viewing SNNs as binary-activated recurrent neural networks (RNNs) for sequential modeling tasks. From this viewpoint, current SNN architectures face several fundamental challenges in sequence modeling: (1) Traditional models lack effective memory mechanisms for long-range sequence modeling; (2) The biological-inspired components in SNNs (such as reset mechanisms and refractory period applications) remain theoretically under-explored for sequence tasks; (3) The RNN-like computational paradigm in SNNs prevents parallel training across different timesteps. To address these challenges, this study conducts a systematic analysis of the fundamental mechanisms underlying reset operations and refractory periods in binary-activated RNN-based SNN sequence models. We re-examine whether such biological mechanisms are strictly necessary for generating sparse spiking patterns, provide new theoretical explanations and insights, and ultimately propose the fixed-refractory-period SNN architecture for sequence modeling.

Figures

Figures reproduced from arXiv: 2504.17751 by the authors.

Figure 1
Figure 1. The Perspective of activation-quantized ANN [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. The Perspective of binary-activated RNN 6 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Modeling Recurrent Architecture Inputs as Probability Distributions [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Decoupling Spiking and Memory Modules in Spiking Neural Networks [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Special Discretization Perspective of Reset Mechanisms [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Special Discretization Persperctive of Refractory Period Mechanisms [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Approximation Perspective of SNNs with Refractory Periods to Conventional Binary RNNs [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Fixed Refractory Period Spiking Neural Network Memory Output Mechanism [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Surrogate gradient function used in our study [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Parallel training mechanism for SpikingFRssm’s spiking module [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: SpikingPssm spiking module architecture 5 Experiments We directly evaluate the model’s sequential modeling capability on the Sequential CIFAR-10 dataset. Given the model’s relatively simple architecture and our primary focus on analyzing the fundamental nature of rese…
Figure 12
Figure 12. Figure 12: Spiking frequency (Refractory=5) [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 14
Figure 14. Figure 14: Impact of refractory period settings 6 Conclusion In this study, we address the following questions: 1. Understanding the reset and refractory mechanisms from the perspective of binary-activated recur￾rent neural networks. The reset and refractory mechanisms can be in…
Figure 15
Figure 15. Figure 15: Generalized view of TTFS-SNN firing phases: (left) solving [PITH_FULL_IMAGE:figures/full_fig_p029_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 31 canonical work pages

  1. [1]

    Spatio-temporal backpropagation for training high-performance spiking neural networks

    Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018

    2018

  2. [2]

    Spike-driven transformer

    Man Yao, Jiakui Hu, Zhaokun Zhou, Li Yuan, Yonghong Tian, Bo Xu, and Guoqi Li. Spike-driven transformer. Advances in neural information processing systems , 36:64043–64058, 2023

    work page 2023

  3. [3]

    Ternary spike-based neuromorphic signal processing system

    Shuai Wang, Dehao Zhang, Ammar Belatreche, Yichen Xiao, Hongyu Qing, Wenjie Wei, Malu Zhang, and Yang Yang. Ternary spike-based neuromorphic signal processing system. Neural Networks , 187:107333, 2025

    work page 2025

  4. [4]

    Spike-based neuromorphic model for sound source localization

    Dehao Zhang, Shuai Wang, Ammar Belatreche, Wenjie Wei, Yichen Xiao, Haorui Zheng, Zijian Zhou, Malu Zhang, and Yang Yang. Spike-based neuromorphic model for sound source localization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024

    work page 2024

  5. [5]

    Hippo: Recurrent memory with optimal polynomial projections

    Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher R´ e. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33:1474–1487, 2020

    2020

  6. [6]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher R´ e. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396 , 2021

    arXiv 2021

  7. [7]

    Resurrecting recurrent neural networks for long sequences

    Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning, pages 26670–26698. PMLR, 2023

    2023

  8. [8]

    P-spikessm: Harnessing probabilistic spiking state space models for long-range dependency tasks

    Malyaban Bal and Abhronil Sengupta. P-spikessm: Harnessing probabilistic spiking state space models for long-range dependency tasks. arXiv preprint arXiv:2406.02923 , 2024

    arXiv 2024

Show all 51 references
  1. [9]

    Parallel spiking neurons with high efficiency and ability to learn long-term dependencies

    Wei Fang, Zhaofei Yu, Zhaokun Zhou, Ding Chen, Yanqi Chen, Zhengyu Ma, Timoth´ ee Masquelier, and Yonghong Tian. Parallel spiking neurons with high efficiency and ability to learn long-term dependencies. Advances in Neural Information Processing Systems , 36:53674–53687, 2023. 23

    2023

  2. [10]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  3. [11]

    Attention-based deep spiking neural networks for temporal credit assignment problems

    Lang Qin, Ziming Wang, Rui Yan, and Huajin Tang. Attention-based deep spiking neural networks for temporal credit assignment problems. IEEE Transactions on Neural Networks and Learning Systems , 2023

    2023

  4. [12]

    Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks

    Bojian Yin, Federico Corradi, and Sander M Boht´ e. Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks. Nature Machine Intelligence , 3(10):905–913, 2021

    2021

  5. [13]

    Long short-term memory and learning-to-learn in networks of spiking neurons.Advances in neural information processing systems, 31, 2018

    Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short-term memory and learning-to-learn in networks of spiking neurons.Advances in neural information processing systems, 31, 2018

    2018

  6. [14]

    An adaptive threshold neuron for recur- rent spiking neural networks with nanodevice hardware implementation

    Ahmed Shaban, Sai Sukruth Bezugam, and Manan Suri. An adaptive threshold neuron for recur- rent spiking neural networks with nanodevice hardware implementation. Nature Communications , 12(1):4234, 2021

    2021

  7. [15]

    Spik- ing neural networks with adaptive membrane time constant for event-based tracking.IEEE Transactions on Image Processing, 2025

    Jiqing Zhang, Malu Zhang, Yuanchen Wang, Qianhui Liu, Baocai Yin, Haizhou Li, and Xin Yang. Spik- ing neural networks with adaptive membrane time constant for event-based tracking.IEEE Transactions on Image Processing, 2025

    2025

  8. [16]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  9. [17]

    Com- bining recurrent, convolutional, and continuous-time models with linear state space layers

    Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher R´ e. Com- bining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 34:572–585, 2021

    2021

  10. [18]

    Simplified state space layers for sequence modeling

    Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208.04933 , 2022

    2022 arXiv

  11. [19]

    Hierarchically gated recurrent neural network for sequence modeling

    Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchically gated recurrent neural network for sequence modeling. Advances in Neural Information Processing Systems , 36:33202–33221, 2023

    2023

  12. [20]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran¸ cois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156–5165. PMLR, 2020

    2020

  13. [21]

    Efficient attention: At- tention with linear complexities

    Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: At- tention with linear complexities. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 3531–3539, 2021

    2021

  14. [22]

    Mega: Moving average equipped gated attention

    Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: Moving average equipped gated attention. arXiv preprint arXiv:2209.10655 , 2022

    2022 arXiv

  15. [23]

    Parallelizing linear transformers with the delta rule over sequence length

    Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. arXiv preprint arXiv:2406.06484 , 2024. 24

    2024 arXiv

  16. [24]

    Tc-lif: A two- compartment spiking neuron model for long-term sequential modelling

    Shimin Zhang, Qu Yang, Chenxiang Ma, Jibin Wu, Haizhou Li, and Kay Chen Tan. Tc-lif: A two- compartment spiking neuron model for long-term sequential modelling. In Proceedings of the AAAI conference on artificial intelligence , volume 38, pages 16838–16847, 2024

    2024

  17. [25]

    Pmsn: A parallel multi-compartment spiking neuron for multi-scale temporal processing

    Xinyi Chen, Jibin Wu, Chenxiang Ma, Yinsong Yan, Yujie Wu, and Kay Chen Tan. Pmsn: A parallel multi-compartment spiking neuron for multi-scale temporal processing. arXiv preprint arXiv:2408.14917, 2024

    2024 arXiv

  18. [26]

    Temporal dendritic heterogeneity incorporated with spiking neural networks for learning multi-timescale dynamics

    Hanle Zheng, Zhong Zheng, Rui Hu, Bo Xiao, Yujie Wu, Fangwen Yu, Xue Liu, Guoqi Li, and Lei Deng. Temporal dendritic heterogeneity incorporated with spiking neural networks for learning multi-timescale dynamics. Nature Communications, 15(1):277, 2024

    2024

  19. [27]

    Learning long sequences in spiking neural networks

    Matei-Ioan Stan and Oliver Rhodes. Learning long sequences in spiking neural networks. Scientific Reports, 14(1):21957, 2024

    2024

  20. [28]

    Spike-ssm: A sparse, precise, and efficient spiking state space model for long sequences learning

    Yan Zhong, Ruoyu Zhao, Chao Wang, Qinghai Guo, Jianguo Zhang, Zhichao Lu, and Luziwei Leng. Spike-ssm: A sparse, precise, and efficient spiking state space model for long sequences learning. arXiv preprint arXiv:2410.17268, 2024

    arXiv

  21. [29]

    Spikingssms: Learning long sequences with sparse and parallel spiking state space models

    Shuaijie Shen, Chao Wang, Renzhuo Huang, Yan Zhong, Qinghai Guo, Zhichao Lu, Jianguo Zhang, and Luziwei Leng. Spikingssms: Learning long sequences with sparse and parallel spiking state space models. arXiv preprint arXiv:2408.14909 , 2024

    2024 arXiv

  22. [30]

    Incorpo- rating learnable membrane time constant to enhance learning of spiking neural networks

    Wei Fang, Zhaofei Yu, Yanqi Chen, Timoth´ ee Masquelier, Tiejun Huang, and Yonghong Tian. Incorpo- rating learnable membrane time constant to enhance learning of spiking neural networks. In Proceedings of the IEEE/CVF international conference on computer vision , pages 2661–2671, 2021

    2021

  23. [31]

    Training high-performance low-latency spiking neural networks by differentiation on spike representation

    Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Zhi-Quan Luo. Training high-performance low-latency spiking neural networks by differentiation on spike representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , p...

    2022

  24. [32]

    A tandem learning rule for effective training and rapid inference of deep spiking neural networks

    Jibin Wu, Yansong Chua, Malu Zhang, Guoqi Li, Haizhou Li, and Kay Chen Tan. A tandem learning rule for effective training and rapid inference of deep spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems , 34(1):446–460, 2021

    2021

  25. [33]

    High-performance deep spiking neural networks with 0.3 spikes per neuron

    Ana Stanojevic, Stanis law Wo´ zniak, Guillaume Bellec, Giovanni Cherubini, Angeliki Pantazi, and Wul- fram Gerstner. High-performance deep spiking neural networks with 0.3 spikes per neuron. Nature Communications, 15(1):6793, 2024

    2024

  26. [34]

    Temporal-coded spiking neural networks with dynamic firing threshold: Learning with event-driven backpropagation

    Wenjie Wei, Malu Zhang, Hong Qu, Ammar Belatreche, Jian Zhang, and Hong Chen. Temporal-coded spiking neural networks with dynamic firing threshold: Learning with event-driven backpropagation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10...

    2023

  27. [35]

    Lc-ttfs: Toward lossless network con- version for spiking neural networks with ttfs coding.IEEE Transactions on Cognitive and Developmental Systems, 16(5):1626–1639, 2023

    Qu Yang, Malu Zhang, Jibin Wu, Kay Chen Tan, and Haizhou Li. Lc-ttfs: Toward lossless network con- version for spiking neural networks with ttfs coding.IEEE Transactions on Cognitive and Developmental Systems, 16(5):1626–1639, 2023. 25

    2023

  28. [36]

    Rectified linear postsynaptic potential function for backpropagation in deep spiking neural networks

    Malu Zhang, Jiadong Wang, Jibin Wu, Ammar Belatreche, Burin Amornpaisannon, Zhixuan Zhang, Venkata Pavan Kumar Miriyala, Hong Qu, Yansong Chua, Trevor E Carlson, et al. Rectified linear postsynaptic potential function for backpropagation in deep spiking neural networks. IEEE t...

    1947

  29. [37]

    Deep declarative networks

    Stephen Gould, Richard Hartley, and Dylan Campbell. Deep declarative networks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(8):3988–4004, 2021

    2021

  30. [38]

    Neural ordinary differ- ential equations

    Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differ- ential equations. Advances in neural information processing systems , 31, 2018

    2018

  31. [39]

    Augmented neural odes

    Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. Advances in neural information processing systems, 32, 2019

    2019

  32. [40]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  33. [41]

    nmode: neural memory ordinary differential equation

    Zhang Yi. nmode: neural memory ordinary differential equation. Artificial Intelligence Review , 56(12):14403–14438, 2023

    2023

  34. [42]

    Momentum residual neural networks

    Michael E Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyr´ e. Momentum residual neural networks. In International Conference on Machine Learning , pages 9276–9287. PMLR, 2021

    2021

  35. [43]

    Training feedback spiking neural networks by implicit differentiation on the equilibrium state

    Mingqing Xiao, Qingyan Meng, Zongpeng Zhang, Yisen Wang, and Zhouchen Lin. Training feedback spiking neural networks by implicit differentiation on the equilibrium state. Advances in neural infor- mation processing systems, 34:14516–14528, 2021

    2021

  36. [44]

    Diagonal state spaces are as effective as structured state spaces

    Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces. Advances in Neural Information Processing Systems , 35:22982–22994, 2022

    2022

  37. [45]

    Universality of linear recurrences followed by non-linear projections: finite-width guarantees and benefits of complex eigenvalues

    Antonio Orvieto, Soham De, Caglar Gulcehre, Razvan Pascanu, and Samuel L Smith. Universality of linear recurrences followed by non-linear projections: finite-width guarantees and benefits of complex eigenvalues. arXiv preprint arXiv:2307.11888 , 2023

    2023 arXiv

  38. [46]

    Language modeling with gated convolutional networks

    Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International conference on machine learning , pages 933–941. PMLR, 2017

    2017

  39. [47]

    State-space models with layer-wise nonlinearity are universal approxima- tors with exponential decaying memory

    Shida Wang and Beichen Xue. State-space models with layer-wise nonlinearity are universal approxima- tors with exponential decaying memory. Advances in Neural Information Processing Systems, 36:74021– 74038, 2023

    2023

  40. [48]

    On the parameterization and initialization of diagonal state space models

    Albert Gu, Karan Goel, Ankit Gupta, and Christopher R´ e. On the parameterization and initialization of diagonal state space models. Advances in Neural Information Processing Systems , 35:35971–35983, 2022

    2022

  41. [49]

    Research on spiking neural network with additional time dimension for sequential modeling

    Enqi Zhang. Research on spiking neural network with additional time dimension for sequential modeling. In 2024 21st International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP), pages 01–05. IEEE, 2024. 26

    2024

  42. [50]

    Deep equilibrium models

    Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in neural infor- mation processing systems, 32, 2019

    2019

  43. [51]

    Integration of neuromorphic ai in event-driven distributed digitized systems: Concepts and research directions

    Mattias Nilsson, Olov Schel´ en, Anders Lindgren, Ulf Bodin, Cristina Paniagua, Jerker Delsing, and Fredrik Sandin. Integration of neuromorphic ai in event-driven distributed digitized systems: Concepts and research directions. Frontiers in Neuroscience, 17:1074439, 2023. 27 A...

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.