REVIEW 6 cited by
Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Recently, brain-inspired spiking neuron networks (SNNs) have attracted widespread research interest because of their event-driven and energy-efficient characteristics. Still, it is difficult to efficiently train deep SNNs due to the non-differentiability of its activation function, which disables the typically used gradient descent approaches for traditional artificial neural networks (ANNs). Although the adoption of surrogate gradient (SG) formally allows for the back-propagation of losses, the discrete spiking mechanism actually differentiates the loss landscape of SNNs from that of ANNs, failing the surrogate gradient methods to achieve comparable accuracy as for ANNs. In this paper, we first analyze why the current direct training approach with surrogate gradient results in SNNs with poor generalizability. Then we introduce the temporal efficient training (TET) approach to compensate for the loss of momentum in the gradient descent with SG so that the training process can converge into flatter minima with better generalizability. Meanwhile, we demonstrate that TET improves the temporal scalability of SNN and induces a temporal inheritable training for acceleration. Our method consistently outperforms the SOTA on all reported mainstream datasets, including CIFAR-10/100 and ImageNet. Remarkably on DVS-CIFAR10, we obtained 83$\%$ top-1 accuracy, over 10$\%$ improvement compared to existing state of the art. Codes are available at \url{https://github.com/Gus-Lab/temporal_efficient_training}.
Forward citations
Cited by 6 Pith papers
-
AS-FedBridge: Pseudo-Spike Bridge Distillation for Heterogeneous ANN-SNN Federated Learning
AS-FedBridge trains a shared pseudo-spike bridge so ANN and SNN clients in federated learning can align representations and improve collaborative accuracy.
-
Adaptive Time-step Training for Enhancing Spike-Based Neural Radiance Fields
PATA learns a scene-wise inference time step for spike-based NeRF, cutting estimated energy by up to 68.90% with minimal PSNR loss.
-
SPEAR: Structured Pruning for Spiking Neural Networks via Synaptic Operation Estimation and Reinforcement Learning
SPEAR combines a linear-regression SynOps estimator with a target-aware reward to search structured pruning policies for spiking neural networks under a SynOps budget.
-
Latency Coding for Efficient and Low-Latency Deep Spiking Neural Networks
Deep TTFS/latency-coded SNNs can be trained directly with backpropagation, reaching ~93.6% on CIFAR-10 with an average inference latency near one timestep.
-
STI-SNN: A 0.14 GOPS/W/PE Single-Timestep Inference FPGA-based SNN Accelerator with Algorithm and Hardware Co-Design
A single-timestep SNN inference accelerator on FPGA, co-designed with TET-based temporal pruning, reports competitive accuracy with lower latency and energy than two-timestep implementations.
-
ReverB-SNN: Reversing Bit of the Weight and Activation for Spiking Neural Networks
ReverB-SNN replaces binary spikes with real-valued spikes and real weights with binary weights, keeping SNN inference addition-only while improving accuracy.
Discussion (0). Sign in to comment.