REVIEW 5 major objections 6 minor 38 references
A Comparative Study on Dynamic Graph Embedding based on Mamba and Transformers
T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Mamba-based models match or beat transformers on dynamic graph link prediction.
desk verdict Useful empirical comparison, but the headline Mamba-vs-Transformer claim hinges on an ambiguous classifier split and a missing GraphSSM baseline. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the Mamba selective state-space layer, a linear-time sequence model whose state update uses input-dependent matrices B, C, and time step delta to selectively compress history. In DG-Mamba and GDG-Mamba, this layer processes the per-node sequence of graph-snapshot features, optionally preprocessed by GINE convolutions that fold node and edge attributes into node features, followed by mean pooling, a linear-tanh projection, and two heads emitting Gaussian embedding means and variances. The Mamba layer replaces the transformer's pairwise attention so that temporal mixing cost is linear in lookback length, and the hidden-attention identity is used to interpret the learned state matrix as cumulative attention-like weights between timestamps.
What would settle it
If the classifier were trained on node pairs sampled from test-snapshot edges, the reported numbers would not measure temporal generalization. A direct check would be to re-run the link prediction evaluation with a strict temporal split where the classifier sees no edges from the test interval, then compare MAP and MRR; if the performance drops substantially, the paper's central claim would be weakened.
Extended reading notes
Core claim
The central claim is that Mamba-based models achieve comparable or superior performance to transformer-based approaches in link prediction tasks while offering significant computational efficiency gains on longer sequences. On the paper's own terms, DG-Mamba and GDG-Mamba beat TransformerG2G and ST-TransformerG2G in MAP and MRR on UCI, Bitcoin-OTC, and Reality Mining, match them on the stable SBM benchmark, and fall short only on Slashdot, which has just 12 timestamps. The paper also claims that analysis of the learned state transition matrix A reveals an attention-like mechanism, so Mamba is not merely an efficient approximation but captures focused long-range temporal dependencies, and that measured FLOPs and MACs scale linearly with lookback while the transformer baselines grow quadratically.
Load-bearing premise
The load-bearing premise is that the logistic regression link prediction classifier is trained only on edges from training timestamps, so that the reported MAP and MRR measure prediction of future graph snapshots rather than reconstruction of already-seen edges.
Editorial extensions
If this is right
- Dynamic graph embedding can use longer lookback windows without the quadratic blowup of attention, so models can attend to entire histories rather than a few recent snapshots.
- Adding edge features through GINE convolutions helps most on transient datasets, suggesting that spatial edge information is key where links churn quickly.
- On datasets with very few timestamps, the extra complexity of SSMs and GINE can hurt, so transformer with GCN remains the better choice for short histories.
- The attention-like interpretation of the Mamba state matrix gives a way to inspect what temporal dependencies a trained model has learned, similar to attention weights.
- Linear complexity makes these models candidates for streaming or online link prediction on long-running dynamic graphs.
Reading between the lines
- If the hidden-attention view of Mamba is faithful, one could distill trained Mamba graph models into sparse or structured attention transformers that run fast on existing hardware, or vice versa.
- The same architecture could be extended to continuous-time temporal graphs by replacing fixed snapshots with event-driven state updates, where the linear scan maps naturally to event sequences.
- The reported efficiency suggests a testable extension: stress-test lookback length on large graphs to see where transformer memory fails and Mamba plateaus in accuracy.
- The probabilistic Gaussian output already gives per-node uncertainty; a natural next step is to calibrate those uncertainties on future-edge prediction and use them for anomaly detection, not just ranking.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents a comparative empirical study of dynamic graph embedding methods based on transformer and Mamba (state-space) architectures. The authors propose three models: ST-TransformerG2G, which augments a prior TransformerG2G model with graph convolutional layers; DG-Mamba, which replaces attention with a selective state-space model; and GDG-Mamba, which adds GINE edge convolutions before the Mamba layer. The models embed each node as a multivariate Gaussian and are trained with a triplet-based contrastive loss. The experiments evaluate temporal link prediction on five datasets (Reality Mining, UCI, SBM, Bitcoin-OTC, Slashdot) using MAP and MRR, with results reported across lookback values and against several baselines. The central claim is that Mamba-based models achieve comparable or superior link prediction performance to transformers while being computationally more efficient, especially on datasets with high temporal variability.
Significance. If the empirical claims hold, the paper would provide a useful data point for the dynamic-graph-embedding community: it shows that selective state-space models can replace attention in a probabilistic temporal graph encoder without losing accuracy, and it analyzes the learned state matrices to motivate this finding. The paper's strengths include the breadth of datasets, per-lookback reporting, multiple random initializations with standard deviations, and a direct FLOPs/MACs comparison. The main contribution is comparative rather than theoretical; its value depends on the correctness of the evaluation protocol, which is currently not fully specified. The absence of code, the ambiguous classifier split, and the omission of a cited SSM baseline (GraphSSM) are the main factors limiting the paper's current conclusiveness.
major comments (5)
- [Section 4.3] The temporal split for the logistic regression link-prediction classifier is unspecified. Section 4.2 states that after training, the model is used to 'predict and save the embeddings of the nodes for all timestamps,' and Section 4.3 describes the classifier as trained on positive and negative pairs 'sampled ... from each graph snapshot.' This wording does not restrict classifier training to the 70% training timestamps or to the validation timestamps. If test-snapshot edges enter classifier fitting, the MAP/MRR values in Tables 5-10 reflect memorization rather than temporal link prediction on held-out snapshots, which would invalidate the paper's central performance claim. The authors should state explicitly that classifier training pairs come only from training/validation timestamps, or, if they do not, revise the protocol and all reported numbers accordingly.
- [Tables 2-4 and Table 10] The hyperparameter and lookback selection procedure is not clearly separated from the test set. Tables 2-4 report per-dataset optimal lookbacks and Optuna-tuned hyperparameters, and Table 10 selects the best-performing lookback for each model. If these values were chosen by maximizing MAP on the test split, the reported results are optimistically biased and the comparison is not a fair evaluation of model selection. The paper should state whether the Optuna objective and lookback choices used the validation split only, and should provide validation-selected results or a nested selection procedure.
- [Section 2.2 and Section 4.3] GraphSSM [27] is cited as an existing state-space model for temporal graphs, yet it is not included as a baseline in Table 10 or anywhere in the experiments. For a paper whose title and abstract promise a comparative study of Mamba-based and transformer-based dynamic graph embedding, omitting the most relevant SSM baseline weakens the comparison. The authors should either add GraphSSM to the experiments or justify its exclusion, for example by explaining that it targets continuous-time event data or a different link prediction protocol.
- [Table 10 and Section 4.3] The baseline comparison is incomplete and inconsistently reported. The prose in Section 4.3 lists ROLAND as one of the six baselines, but Table 10 contains results for only DynGEM, dyngraph2vecAE, dyngraph2vecAERNN, EvolveGCN, DynG2G, and the proposed/prior transformer models; ROLAND results are absent. Additionally, several baseline rows report no standard deviations, making it unclear whether those numbers were obtained under the same five-seed protocol and the same evaluation split as the proposed models. The authors should provide complete baseline results under a shared protocol or explicitly state which numbers were taken from prior publications.
- [Table 9] The DG-Mamba results on Slashdot contain a striking anomaly: for lookback l=5 the MAP is 0.0006 +/- 0.0003, while at l=3 it is 0.0676 +/- 0.0009. This sudden collapse is not explained in the text and may indicate a training failure, a degenerate embedding, or a numerical issue in that configuration. Because Table 10 uses the best lookback for DG-Mamba on Slashdot, this anomaly does not directly affect the headline comparison, but it should be investigated and either explained or removed, and the reported best value should be verified.
minor comments (6)
- [Section 4.1] The Slashdot dataset description is inconsistent: Table 1 lists 50,825 nodes while the prose says 50,824 nodes; the two should be reconciled.
- [Abstract and Table 1] The abstract refers to 'Bitcoin' while the experiments use the Bitcoin-OTC dataset; use the full name consistently throughout the paper.
- [Section 4.3] The classifier is said to use 'Mean Average Precision (MAP) loss,' which is not a standard loss for logistic regression; the authors should define the exact objective or clarify that they mean a ranking-style loss.
- [Tables 3 and 4] Several hyperparameter values, such as dmodel = 1899 for UCI in Table 3 and dmodel = 1000 for SBM in Table 4, appear to equal the number of nodes in the dataset; this is surprising for a model dimension and should be explained in the text.
- [Section 4.5] The computational efficiency comparison reports FLOPs and MACs as a function of lookback, but it does not report wall-clock training or inference times; a brief runtime comparison would strengthen the efficiency claim.
- [Section 4.4] The visual analysis of state and attention matrices is qualitative and uses only two datasets; the authors should state whether the figures are representative or provide a quantitative summary of the claimed difference in temporal focus.
Circularity Check
No significant circularity: the empirical comparison is self-contained, includes independent baselines, and does not reduce its performance claims to fitted inputs or self-citations.
full rationale
This paper is an empirical comparative study rather than a formal derivation, so most circularity patterns do not apply. The proposed models (ST-TransformerG2G, DG-Mamba, and GDG-Mamba) are new architectures built on the authors' earlier TransformerG2G and DynG2G work, and the headline claim is supported by running these models and external baselines (EvolveGCN, DynGEM, dyngraph2vec) on five public benchmarks. The self-citations to [16] and [12] provide architectural provenance and baseline comparisons, but they are not load-bearing in the sense of substituting for evidence: the paper reports MAP/MRR values from executed experiments, and independent baselines are included in Table 10. The Mamba 'hidden attention' discussion in Section 4.4 is attributed to external reference [38], so it is not presented as an internally derived uniqueness result. No equation in the paper is defined in terms of the quantity it is said to predict, and no fitted parameter is renamed as a prediction. A separate methodological ambiguity exists in Section 4.3, where the logistic-regression classifier is described as trained on node pairs sampled 'from each graph snapshot' without explicitly excluding validation/test snapshots; if test-snapshot edges entered classifier fitting, the reported MAP/MRR would not measure held-out temporal link prediction. This is an evaluation-protocol concern and a correctness risk, not a circularity by construction, and is therefore not counted in the circularity score.
Assumptions & free parameters
free parameters (3)
- Per-dataset optimal lookback =
Varies by dataset and model; see Tables 2-4
- Optuna-tuned hyperparameters (dmodel, dstate, dconv, lr, weight decay, intermediate space, dropout) =
Per dataset in Tables 3-4
- Embedding dimension dout =
64 or 256 per dataset
assumptions (4)
- domain assumption Triplet-based contrastive loss with KL divergence produces embeddings that are useful for temporal link prediction.
- domain assumption Zero-padding adjacency matrices to the maximum node count n preserves the information needed for embedding.
- domain assumption Optuna validation-based hyperparameter search and early stopping on validation loss generalize to test timestamps.
- domain assumption MAP and MRR on the 1:10 positive-negative sampling are meaningful link prediction metrics.
Cite this review
Pith. "Pith review of A Comparative Study on Dynamic Graph Embedding based on Mamba and Transformers." pith.science (2026). https://pith.science/paper/6IKXNK2D
@misc{pith2026241211293,
author = {Pith},
title = {Pith review of: A Comparative Study on Dynamic Graph Embedding based on Mamba and Transformers},
year = {2026},
howpublished = {\url{https://pith.science/paper/6IKXNK2D}},
note = {Machine review of arXiv:2412.11293}
}
read the original abstract
Dynamic graph embedding has emerged as an important technique for modeling complex time-evolving networks across diverse domains. While transformer-based models have shown promise in capturing long-range dependencies in temporal graph data, they face scalability challenges due to quadratic computational complexity. This study presents a comparative analysis of dynamic graph embedding approaches using transformers and the recently proposed Mamba architecture, a state-space model with linear complexity. We introduce three novel models: TransformerG2G augment with graph convolutional networks, \mathcal{DG}-Mamba, and \mathcal{GDG}-Mamba with graph isomorphism network edge convolutions. Our experiments on multiple benchmark datasets demonstrate that Mamba-based models achieve comparable or superior performance to transformer-based approaches in link prediction tasks while offering significant computational efficiency gains on longer sequences. Notably, \mathcal{DG}-Mamba variants consistently outperform transformer-based models on datasets with high temporal variability, such as UCI, Bitcoin, and Reality Mining, while maintaining competitive performance on more stable graphs like SBM. We provide insights into the learned temporal dependencies through analysis of attention weights and state matrices, revealing the models' ability to capture complex temporal patterns. By effectively combining state-space models with graph neural networks, our work addresses key limitations of previous approaches and contributes to the growing body of research on efficient temporal graph representation learning. These findings offer promising directions for scaling dynamic graph embedding to larger, more complex real-world networks, potentially enabling new applications in areas such as social network analysis, financial modeling, and biological system dynamics.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[27]
J. Li, R. Wu, X. Jin, B. Ma, L. Chen, Z. Zheng, State space models on temporal graphs: A first-principles study, arXiv preprint arXiv:2406.00943 (2024)
arXiv 2024
-
[1]
T. N. Kipf, M. Welling, Semi-supervised classification with graph con- volutional networks, arXiv preprint arXiv:1609.02907 (2016)
arXiv 2016
-
[2]
Gilmer, S
J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, G. E. Dahl, Neural message passing for quantum chemistry, in: International conference on machine learning, PMLR, 2017, pp. 1263–1272
2017
-
[3]
P. Veliˇ ckovi´ c, G. Cucurull, A. Casanova, A. Romero, P. Lio, Y. Bengio, Graph attention networks, arXiv preprint arXiv:1710.10903 (2017)
arXiv 2017
-
[4]
S. Yun, M. Jeong, R. Kim, J. Kang, H. J. Kim, Graph transformer networks, Advances in neural information processing systems 32 (2019)
2019
-
[5]
J. Gou, X. Yuan, Y. Xue, L. Du, J. Yu, S. Xia, Y. Zhang, Discriminative and geometry-preserving adaptive graph embedding for dimensionality reduction, Neural Networks 157 (2023) 364–376
work page 2023
-
[6]
R. Trivedi, M. Farajtabar, P. Biswal, H. Zha, DyRep: Learning repre- sentations over dynamic graphs, in: International conference on learning representations, 2019
work page 2019
- [7]
Show all 38 references
-
[8]
Rossi, B
E. Rossi, B. Chamberlain, F. Frasca, D. Eynard, F. Monti, M. Bronstein, Temporal graph networks for deep learning on dynamic graphs, arXiv preprint arXiv:2006.10637 (2020)
2020 arXiv
-
[9]
Goyal, N
P. Goyal, N. Kamra, X. He, Y. Liu, DynGEM: Deep embedding method for dynamic graphs, arXiv preprint arXiv:1805.11273 (2018)
2018 arXiv
-
[10]
Goyal, S
P. Goyal, S. R. Chhetri, N. Mehrabi, E. Ferrara, A. Canedo, Dynam- icGEM: A library for dynamic graph embedding methods, arXiv preprint arXiv:1811.10734 (2018). 30
2018 arXiv
-
[11]
Pareja, G
A. Pareja, G. Domeniconi, J. Chen, T. Ma, T. Suzumura, H. Kanezashi, T. Kaler, T. Schardl, C. Leiserson, EvolveGCN: Evolving graph con- volutional networks for dynamic graphs, in: Proceedings of the AAAI conference on artificial intelligence, Vol. 34, 2020, pp. 5363–5370
2020
-
[12]
M. Xu, A. V. Singh, G. E. Karniadakis, DynG2G: An efficient stochastic graph embedding method for temporal graphs, IEEE Transactions on Neural Networks and Learning Systems 35 (1) (2022) 985–998
2022
-
[13]
J. You, T. Du, J. Leskovec, ROLAND: graph learning framework for dynamic graphs, in: Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, 2022, pp. 2358–2366
2022
-
[14]
Sharma, R
K. Sharma, R. Trivedi, R. Sridhar, S. Kumar, Temporal dynamics-aware adversarial attacks on discrete-time dynamic graph models, in: Proceed- ings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023, pp. 2023–2035
2023
-
[15]
Vaswani, N
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, Advances in neural information processing systems 30 (2017)
2017
-
[16]
A. J. Varghese, A. Bora, M. Xu, G. E. Karniadakis, TransformerG2G: Adaptive time-stepping for learning temporal graph embeddings using transformers, Neural Networks 172 (2024) 106086
2024
-
[17]
Y. Liu, S. Pan, Y. G. Wang, F. Xiong, L. Wang, Q. Chen, V. C. Lee, Anomaly detection in dynamic graphs via transformer, IEEE Transac- tions on Knowledge and Data Engineering 35 (12) (2021) 12081–12094
2021
-
[18]
G. Huo, Y. Zhang, B. Wang, J. Gao, Y. Hu, B. Yin, Hierarchical spatio– temporal graph convolutional networks and transformer network for traffic flow forecasting, IEEE Transactions on Intelligent Transporta- tion Systems 24 (4) (2023) 3855–3867
2023
-
[19]
Jiang, C
J. Jiang, C. Han, W. X. Zhao, J. Wang, PDFormer: Propagation delay- aware dynamic long-range transformer for traffic flow prediction, in: Proceedings of the AAAI conference on artificial intelligence, Vol. 37, 2023, pp. 4365–4373. 31
2023
-
[20]
Lezmi, J
E. Lezmi, J. Xu, Time series forecasting with transformer models and application to asset management, Available at SSRN 4375798 (2023)
2023
-
[21]
J. Wang, Z. Sun, C. Yuan, W. Li, A.-A. Liu, Z. Wei, B. Yin, Dynamic graphs attention for ocean variable forecasting, Engineering Applica- tions of Artificial Intelligence 133 (2024) 108187
2024
-
[22]
Hofmeister, K
M. Hofmeister, K. F. Lee, Y.-K. Tsai, M. M¨ uller, K. Nagarajan, S. Mos- bach, J. Akroyd, M. Kraft, Dynamic control of district heating networks with integrated emission modelling: A dynamic knowledge graph ap- proach, Energy and AI 17 (2024) 100376
2024
-
[23]
S. Ren, X. Pan, W. Zhao, B. Nie, B. Han, Dynamic graph transformer for 3D object detection, Knowledge-Based Systems 259 (2023) 110085
2023
-
[24]
A. Gu, K. Goel, C. R´ e, Efficiently modeling long sequences with struc- tured state spaces, arXiv preprint arXiv:2111.00396 (2021)
2021 arXiv
-
[25]
A. Gu, T. Dao, Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint arXiv:2312.00752 (2023)
2023 arXiv
-
[26]
Longa, V
A. Longa, V. Lachi, G. Santin, M. Bianchini, B. Lepri, P. Lio, F. Scarselli, A. Passerini, Graph neural networks for temporal graphs: State of the art, open challenges, and opportunities, arXiv preprint arXiv:2302.01018 (2023)
2023 arXiv
-
[28]
Holme, J
P. Holme, J. Saram¨ aki, Temporal networks, Physics reports 519 (3) (2012) 97–125
2012
-
[29]
S. M. Kazemi, R. Goel, K. Jain, I. Kobyzev, A. Sethi, P. Forsyth, P. Poupart, Representation learning for dynamic graphs: A survey, Jour- nal of Machine Learning Research 21 (70) (2020) 1–73
2020
-
[30]
D. Xu, C. Ruan, E. Korpeoglu, S. Kumar, K. Achan, Inductive repre- sentation learning on temporal graphs, in: International Conference on Learning Representations, 2020. 32
2020
-
[31]
Singer, I
U. Singer, I. Guy, K. Radinsky, Node embedding over temporal graphs, arXiv preprint arXiv:1903.08889 (2019)
2019 arXiv
-
[32]
Y. Seo, M. Defferrard, P. Vandergheynst, X. Bresson, Structured se- quence modeling with graph convolutional recurrent networks, in: Inter- national Conference on Neural Information Processing, Springer, 2018, pp. 362–373
2018
-
[33]
B. N. Patro, V. S. Agneeswaran, Mamba-360: Survey of state space models as transformer alternative for long sequence modelling: Methods, applications, and challenges, arXiv preprint arXiv:2404.16112 (2024)
2024 arXiv
-
[34]
Behrouz, F
A. Behrouz, F. Hashemi, Graph mamba: Towards learning on graphs with state space models, arXiv preprint arXiv:2402.08678 (2024)
2024 arXiv
-
[35]
Bojchevski, S
A. Bojchevski, S. G¨ unnemann, Deep gaussian embedding of graphs: Un- supervised inductive learning via ranking, in: International Conference on Learning Representations, 2018, pp. 1–13
2018
-
[36]
Akiba, S
T. Akiba, S. Sano, T. Yanase, T. Ohta, M. Koyama, Optuna: A next- generation hyperparameter optimization framework, in: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Dis- covery and Data Mining, 2019
2019
-
[37]
Goyal, S
P. Goyal, S. R. Chhetri, A. Canedo, dyngraph2vec: Capturing network dynamics using dynamic graph representation learning, Knowledge- Based Systems 187 (2020) 104816
2020
-
[38]
A. Ali, I. Zimerman, L. Wolf, The hidden attention of mamba models, arXiv preprint arXiv:2403.01590v2 (2024). 33
2024 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.