REVIEW 5 cited by
SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanisms
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Towards energy-efficient artificial intelligence similar to the human brain, the bio-inspired spiking neural networks (SNNs) have advantages of biological plausibility, event-driven sparsity, and binary activation. Recently, large-scale language models exhibit promising generalization capability, making it a valuable issue to explore more general spike-driven models. However, the binary spikes in existing SNNs fail to encode adequate semantic information, placing technological challenges for generalization. This work proposes the first fully spiking mechanism for general language tasks, including both discriminative and generative ones. Different from previous spikes with {0,1} levels, we propose a more general spike formulation with bi-directional, elastic amplitude, and elastic frequency encoding, while still maintaining the addition nature of SNNs. In a single time step, the spike is enhanced by direction and amplitude information; in spike frequency, a strategy to control spike firing rate is well designed. We plug this elastic bi-spiking mechanism in language modeling, named SpikeLM. It is the first time to handle general language tasks with fully spike-driven models, which achieve much higher accuracy than previously possible. SpikeLM also greatly bridges the performance gap between SNNs and ANNs in language modeling. Our code is available at https://github.com/Xingrun-Xing/SpikeLM.
Forward citations
Cited by 5 Pith papers
-
Improving Performance of Spike-based Deep Q-Learning using Ternary Neurons
An asymmetric ternary spiking neuron with a trainable negative threshold improves deep spiking Q-network scores on six of seven Atari games, but the theoretical explanation and the headline performance metric are not ...
-
Phi: Leveraging Pattern-based Hierarchical Sparsity for High-Efficiency Spiking Neural Networks
Phi decomposes SNN activations into pre-computed pattern rows plus sparse +/-1 corrections, yielding a 3.45x speedup and 4.93x energy savings over the Stellar accelerator.
-
LAS: Loss-less ANN-SNN Conversion for Fully Spike-Driven Large Language Models
LAS converts pre-trained ANN LLMs into fully spike-driven SNNs at 16 time steps using Outlier-Aware Threshold and Hierarchically Gated neurons, reporting near-lossless accuracy on OPT, BERT, GPT-2, and two vision-lang...
-
FAS: Fast ANN-SNN Conversion for Spiking Large Language Models
FAS converts pretrained LLMs to spiking LLMs by fine-tuning with QCFS and then calibrating thresholds and initial membrane potentials, reaching near-LLM accuracy at 8-16 timesteps with large claimed energy savings.
-
Darkit: A User-Friendly Software Toolkit for Spiking Large Language Model
DarwinKit is a new software toolkit that packages datasets, tokenizers, model graph inspection, and GUI tools to simplify building spiking large language models, but without experimental validation.
Discussion (0). Continue with ORCID to comment.