RECORD
SpikeGraphormer: A High-Performance Graph Transformer with Spiking Graph Attention
Neural and Evolutionary Computing · Submitted 21 Mar 2024
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim SpikeGraphormer consistently outperforms existing state-of-the-art approaches across various datasets and makes substantial improvements in training time, inference time, and GPU memory cost (10 ~ 20x lower than vanilla self-attention).
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Recently, Graph Transformers have emerged as a promising solution to alleviate the inherent limitations of Graph Neural Networks (GNNs) and enhance graph representation performance. Unfortunately, Graph Transformers are computationally expensive due to the quadratic complexity inherent in self-attention when applied over large-scale graphs, especially for node tasks. In contrast, spiking neural networks (SNNs), with event-driven and binary spikes properties, can perform energy-efficient computation. In this work, we propose a novel insight into integrating SNNs with Graph Transformers and design a Spiking Graph Attention (SGA) module. The matrix multiplication is replaced by sparse addition and mask operations. The linear complexity enables all-pair node interactions on large-scale graphs with limited GPU memory. To our knowledge, our work is the first attempt to introduce SNNs into Graph Transformers. Furthermore, we design SpikeGraphormer, a Dual-branch architecture, combining a sparse GNN branch with our SGA-driven Graph Transformer branch, which can simultaneously perform all-pair node interactions and capture local neighborhoods. SpikeGraphormer consistently outperforms existing state-of-the-art approaches across various datasets and makes substantial improvements in training time, inference time, and GPU memory cost (10 ~ 20x lower than vanilla self-attention). It also performs well in cross-domain applications (image and text classification). We release our code at https://github.com/PHD-lanyu/SpikeGraphormer.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Recently, Graph Transformers have emerged as a promising solution to alleviate the inherent limitations of Graph Neural Networks (GNNs) and enhance graph representation performance. | background 1.00 | 0.04 | 0.03 | asserted 0.62 |
| s1 | Unfortunately, Graph Transformers are computationally expensive due to the quadratic complexity inherent in self-attention when applied over large-scale graphs, especially for node tasks. | background 0.89 | 0.07 | 0.02 | asserted 1.00 |
| s2 | In contrast, spiking neural networks (SNNs), with event-driven and binary spikes properties, can perform energy-efficient computation. | background 1.00 | 0.05 | 0.02 | asserted 0.98 |
| s3 | In this work, we propose a novel insight into integrating SNNs with Graph Transformers and design a Spiking Graph Attention (SGA) module. | contribution 0.98 | 0.93 | 0.06 | asserted 0.99 |
| s4 | The matrix multiplication is replaced by sparse addition and mask operations. | method 0.99 | 0.78 | 0.04 | asserted 1.00 |
| s5 | The linear complexity enables all-pair node interactions on large-scale graphs with limited GPU memory. | result 0.31 | 0.65 | 0.10 | asserted 1.00 |
| s6 | To our knowledge, our work is the first attempt to introduce SNNs into Graph Transformers. | contribution 0.62 | 0.64 | 0.05 | asserted 0.96 |
| s7 | Furthermore, we design SpikeGraphormer, a Dual-branch architecture, combining a sparse GNN branch with our SGA-driven Graph Transformer branch, which can simultaneously perform all-pair node interactions and capture local neighborhoods. | method 0.75 | 0.94 | 0.09 | asserted 1.00 |
| s8 | SpikeGraphormer consistently outperforms existing state-of-the-art approaches across various datasets and makes substantial improvements in training time, inference time, and GPU memory cost (10 ~ 20x lower than vanilla self-attention). | result 1.00 | 0.94 | 0.77 | asserted 1.00 |
| s9 | It also performs well in cross-domain applications (image and text classification). | result 1.00 | 0.88 | 0.08 | asserted 0.98 |
| s10 | We release our code at https://github.com/PHD-lanyu/SpikeGraphormer. | contribution 0.86 | 0.86 | 0.02 | asserted 0.94 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Recently, Graph Transformers have emerged as a promising solution to alleviate the inherent limitations of Graph Neural Networks (GNNs) and enhance graph representation performance. Unfortunately, Graph Transformers are computationally expensive due to the quadratic complexity inherent in self-attention when applied over large-scale graphs, especially for node tasks. In contrast, spiking neural networks (SNNs), with event-driven and binary spikes properties, can perform energy-efficient computation. In this work, we propose a novel insight into integrating SNNs with Graph Transformers and design a Spiking Graph Attention (SGA) module. The matrix multiplication is replaced by sparse addition and mask operations. The linear complexity enables all-pair node interactions on large-scale graphs with limited GPU memory. To our knowledge, our work is the first attempt to introduce SNNs into Graph Transformers. Furthermore, we design SpikeGraphormer, a Dual-branch architecture, combining a sparse GNN branch with our SGA-driven Graph Transformer branch, which can simultaneously perform all-pair node interactions and capture local neighborhoods. SpikeGraphormer consistently outperforms existing state-of-the-art approaches across various datasets and makes substantial improvements in training time, inference time, and GPU memory cost (10 ~ 20x lower than vanilla self-attention). It also performs well in cross-domain applications (image and text classification). We release our code at https://github.com/PHD-lanyu/SpikeGraphormer.
Human review
How the literature uses it
1 arXiv paper cite this in 1 sentence
Papers citing it
Compares with it 1 paper
-
We compare the proposed with $12$ strong baselines of three categories: (1) ANN-based Euclidean GNNs: the popular GCN [ref] , GAT [ref] , GraphSAGE [ref] and SGC [ref] , (2) ANN-based Riemannian GNNs: HGCN [ref] and HyboNet [ref] of hyperbolic spaces, $κ-$GCN [ref] of the constant curvature space, and the recent $Q-$GCN [ref] of the quotient space, (3) The previous Euclidean Spiking GNNs: SpikeNet [ref] , SpikeGCN [ref] , SipkeGraphormer [ref] (termed as SpikeGT for short) and the recent SpikeGCL [ref] .
Spiking Graph Neural Network on Riemannian Manifolds · Experimental Setups · 0.98
What it cites · 3 sentences without a confident label
Compares with 1 paper
-
We compare SpikeGraphormer with several advanced methods, including GNNs (GCN [ref] , GAT [ref] , SGC [ref] ) and advanced GNNs ($H_2GCN$ [ref] and SIGN [ref] ); Transformers-based methods (Graphormer [ref] , GraphTrans [ref] ANS-GT [ref] and Nodeformer [ref] ).
SIGN: Scalable Inception Graph Neural Networks · Experiments · 0.99
Cites as background 1 paper
-
Some more extreme methods limit the attention scope to the local neighborhood of the central node (such as ego-graph) [ref] .
Gophormer: Ego-Graph Transformer for Node Classification · Salable Graph Transformers · 0.92
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.