REVIEW 11 cited by
SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
As the size of large language models continue to scale, so does the computational resources required to run it. Spiking Neural Networks (SNNs) have emerged as an energy-efficient approach to deep learning that leverage sparse and event-driven activations to reduce the computational overhead associated with model inference. While they have become competitive with non-spiking models on many computer vision tasks, SNNs have also proven to be more challenging to train. As a result, their performance lags behind modern deep learning, and we are yet to see the effectiveness of SNNs in language generation. In this paper, inspired by the Receptance Weighted Key Value (RWKV) language model, we successfully implement `SpikeGPT', a generative language model with binary, event-driven spiking activation units. We train the proposed model on two model variants: 45M and 216M parameters. To the best of our knowledge, SpikeGPT is the largest backpropagation-trained SNN model to date, rendering it suitable for both the generation and comprehension of natural language. We achieve this by modifying the transformer block to replace multi-head self attention to reduce quadratic computational complexity O(N^2) to linear complexity O(N) with increasing sequence length. Input tokens are instead streamed in sequentially to our attention mechanism (as with typical SNNs). Our preliminary experiments show that SpikeGPT remains competitive with non-spiking models on tested benchmarks, while maintaining 20x fewer operations when processed on neuromorphic hardware that can leverage sparse, event-driven activations. Our code implementation is available at https://github.com/ridgerchu/SpikeGPT.
Forward citations
Cited by 11 Pith papers
-
SpikeWorld: Fast-State Adaptation for Frozen Spiking World Models
A frozen 1.45M-parameter spiking world model with a small external fast-state module raises frozen-policy reward by 7.90 (CI [2.48, 14.06]) and improves held-out prediction under shear and attenuation while inherited ...
-
The Sparsity Ceiling: Where Spiking Networks Can and Cannot Trade Activity for Energy
The sparsity dividend of spiking networks is task-dependent: perception and attention can run at 2–5% firing without quality loss, while recurrent sequence models are pinned near 50% firing.
-
TDFormer: A Top-Down Attention-Controlled Spiking Transformer
A top-down feedback module for spiking transformers improves temporal information flow, reduces temporal vanishing gradients, and reaches 86.83% top-1 accuracy on ImageNet.
-
LAS: Loss-less ANN-SNN Conversion for Fully Spike-Driven Large Language Models
LAS converts pre-trained ANN LLMs into fully spike-driven SNNs at 16 time steps using Outlier-Aware Threshold and Hierarchically Gated neurons, reporting near-lossless accuracy on OPT, BERT, GPT-2, and two vision-lang...
-
Neuromorphic Principles for Efficient Large Language Models on Intel Loihi 2
A 370M MatMul-free LLM is mapped onto Intel Loihi 2 and reported to achieve up to 3x higher generation throughput with about 2x less energy than transformer LLMs on an edge GPU, based on preliminary measurements that ...
-
FAS: Fast ANN-SNN Conversion for Spiking Large Language Models
FAS converts pretrained LLMs to spiking LLMs by fine-tuning with QCFS and then calibrating thresholds and initial membrane potentials, reaching near-LLM accuracy at 8-16 timesteps with large claimed energy savings.
-
Darkit: A User-Friendly Software Toolkit for Spiking Large Language Model
DarwinKit is a new software toolkit that packages datasets, tokenizers, model graph inspection, and GUI tools to simplify building spiking large language models, but without experimental validation.
-
Combining Aggregated Attention and Transformer Architecture for Accurate and Efficient Performance of Spiking Neural Networks
A spiking Transformer variant, SAFormer, drops the value matrix and uses downsampled spike query/key pairs plus depthwise convolution to reach 95.8% on CIFAR-10 and 81.3% on CIFAR10-DVS with lower estimated energy cost.
-
HiBerNAC: Hierarchical Brain-emulated Robotic Neural Agent Collective for Disentangling Complex Manipulation
HiBerNAC, a multi-agent 'brain-inspired' planner layered on a reactive VLA, is claimed to cut long-horizon task time by 23% and reach 12-31% success where VLA baselines fail, but the supporting data are inconsistent.
-
A Survey of RWKV
A review of the RWKV architecture, its versions, applications, benchmarks, and open-source ecosystem; it presents no new experimental results.
-
Word2Spike: Poisson Rate Coding for Associative Memories and Neuromorphic Algorithms
Word2Spike proposes a ternary quantization plus Poisson rate coding scheme for word embeddings, reporting 100% reconstruction on 10k words.
Discussion (0). Continue with ORCID to comment.