REVIEW 3 major objections 5 minor 1 cited by
Temporal graph models systematically fail to learn edge direction, density, and recency, while all succeed at preferential attachment.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-04 10:26 UTC pith:YKJ3GPTQ
load-bearing objection Useful probe framework for temporal GNNs, but the headline failures on recency and density don't survive close reading of the probes. the 3 major comments →
What Do Temporal Graph Learning Models Learn?
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On the paper's own terms, the central discovery is that none of the seven models studied assign measurably different probabilities to an edge (u,v) than to its reverse (v,u); none reproduce the true density of a graph when negative sampling ratios are varied; and none place more predictive weight on recently observed edges than on old ones, even on data with no other signal. By contrast, all seven models extrapolate degree-based preferential attachment: the higher a node's training degree, the higher the model's probability for unseen edges incident to it. The mixed results on temporal granularity, persistence, periodicity, and homophily show that capabilities are model-specific, with indivi
What carries the argument
The paper's experimental engine is a suite of property-isolating probes. For each of eight graph characteristics, it constructs a synthetic or curated dataset in which that characteristic is the only signal—scale-free graphs for preferential attachment, stochastic block models for homophily, fixed or alternating snapshots for persistence and periodicity, disjoint random edge sets for recency, and negative-sampling ratio sweeps for density—then reads model output probabilities for signatures of the characteristic. The probes convert the unobservable question 'does the model understand this property' into a measurable behavioral score, and the resulting eight-by-seven scoreboard is the paper's
Load-bearing premise
That a model's probability scores on these simplified synthetic probe graphs are a valid diagnostic of whether it has learned the corresponding characteristic of real temporal graphs.
What would settle it
Train a temporal graph model on a synthetic graph where edges recur with probability that decays with time since last observed; if the model assigns clearly higher probability to recently seen edges than to older ones, the recency '✗' is falsified. Similarly, a density probe that recalibrates output probabilities before comparing predicted and true density could falsify the density claim.
If this is right
- If the scoreboard is correct, high benchmark scores on temporal link prediction can be achieved without learning edge direction, graph density, or edge recency, so simple popularity-based heuristics may explain much of that performance.
- Practitioners who use these models with a confidence threshold should recalibrate probabilities: predicted density deviates from true density by orders of magnitude depending on negative sampling.
- Because all models learn preferential attachment, deployed models will tend to favor high-degree nodes, a bias that matters in recommender systems and link-prediction applications.
- The mixed per-model results mean model choice should be informed by which characteristic matters for the application, not by aggregate accuracy alone.
- Evaluation of temporal graph models should include interpretability probes alongside accuracy benchmarks; without them, benchmark rankings can hide systematic blind spots.
Where Pith is reading between the lines
- Editorial inference: the recency probe uses edge sets that are random and disjoint across timesteps, so flat probability scores may reflect appropriate uncertainty rather than an inability to learn recency; a probe with an actual recency signal would settle this.
- Editorial inference: the density result conflates learned density with probability calibration; recalibrating output scores to the empirical density could turn a 'fail' into a 'pass', so the failure may be partly an artifact of the readout.
- Editorial inference: if popularity and recency-blind shortcuts dominate, benchmark negative-sampling protocols should control for node degree and recent activity, or model rankings may reward shortcut learning rather than structural understanding.
- Editorial inference: the same probe suite could be applied to discrete-time models and heterogeneous graphs, where direction and recency may be learned more or less readily, offering a low-cost extension.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an evaluation framework for assessing whether temporal graph learning models capture eight fundamental properties of temporal graphs: temporal granularity, direction, density, persistence, periodicity, recency, homophily, and preferential attachment. Seven state-of-the-art models (DyGFormer, DyRep, JODIE, GraphMixer, TCL, TGAT, TGN) are tested on a mix of synthetic and real-world datasets. The central empirical result, summarized in Table 1, is that all models fail to learn direction, density, and recency, while all learn preferential attachment; performance on the other properties varies. The authors conclude that state-of-the-art models rely on popularity-like shortcuts and argue for interpretability-driven evaluations.
Significance. If the negative results are valid, the paper would provide a valuable cautionary complement to performance-oriented benchmarks, with practical implications for model selection and architecture design. The framework is systematic: it covers seven models, three empirical and several synthetic datasets, three training seeds, and a controlled probe for each property. The finding that all models capture preferential attachment is a concrete, falsifiable claim. However, the validity of the three headline failures is undermined by the design of the recency and density probes, as detailed below. Because the central message rests heavily on these failures, the paper in its current form overstates its conclusions. With better-controlled probes, the framework could be a useful contribution to interpretability-focused evaluations.
major comments (3)
- [Section 4.2.3 (Figure 5)] The recency probe does not test whether models learn recency. The training data are 10 timesteps of completely disjoint edge sets sampled uniformly at random from the same pool, with no temporal dependence. Under this generative process, the probability that an edge appears at t=11 is independent of its last-seen time, so a model that has correctly captured the data-generating process should assign equal probability to all training edges at test time. The flat average probabilities in Figure 5 are therefore the correct outcome, not evidence of a failure to learn recency. The conclusion that 'models do not place any predictive weight into the recency of edges' is not supported by this experiment. To test recency learning, the synthetic data must contain a recency signal (e.g., edges that recur sooner after last being observed) and the test should check whether model probabilities follow t
- [Section 4.1.3 (Figure 2)] The density experiment conflates calibration with learning. Models are trained with negative samples drawn at ratios that differ from the true density, so the sigmoid output is not an estimate of the marginal edge-existence probability under the true distribution. The observation that predicted density is much lower than the training ratio (or far from true density) may simply reflect miscalibration, as the paper itself acknowledges ('one may need to recalibrate output probability scores'). The conclusion that 'models do not appear to learn density' therefore does not follow from the experiment as designed. A valid test would either train with a negative-sampling ratio that matches the true density, apply a calibration step before measuring predicted density, or use rank-based metrics invariant to monotone transforms. Without this, the 'Density ✗' entries in Table 1 are not robust.
- [Section 4.1 (Table 1, findings paragraphs)] The assignment of ✓/∼/✗ in Table 1 is based on qualitative inspection of figures and aggregated scores without explicit, pre-registered criteria. For example, in Persistence (Section 4.2.1), DyGFormer is labelled ✓ because it 'appears to perfectly learn', while in Periodicity (Section 4.2.2), models are labelled based on visual separation in Figure 4. No thresholds for effect sizes, confidence intervals, or statistical tests across the three seeds are given. This makes the central summary table non-reproducible and leaves the headline claims insufficiently supported. I recommend specifying quantitative decision rules (e.g., a minimum AUC, difference in means with error bars, or a permutation test) for each of the eight characteristics.
minor comments (5)
- [Abstract] The abstract states 'systematically evaluating eight models' but the paper evaluates seven models (see Section 1 and Table 1). Please correct the number.
- [Table 3 and Figures 1–6] The typo 'TGA T' appears in several places (e.g., Table 3, Figure 1); it should be 'TGAT'.
- [Section 4.2.2] The findings paragraph contains contradictory statements: it first says 'GraphMixer and TGAT appear to distinguish odd and even timestamps very well (✓)', then says 'By contrast, TGAT and TCL appear to consistently separate edges between different timesteps; however, they still tend to be too confident ... (∼)'. According to Table 1, GraphMixer and TCL are ✓ and TGAT is ∼; the text should be corrected.
- [Figure 5] The y-axis labels in panels (a) and (b) are truncated ('0.5050.505'); please fix the formatting.
- [Section 5 (Limitations)] The limitations section does not mention that the recency experiment uses i.i.d. disjoint edges, which is a key design choice that prevents any recency signal from being present. A discussion of this limitation would be helpful.
Circularity Check
No circularity found: the paper is an empirical probe study whose conclusions are read off external synthetic and real datasets, not derived from fitted inputs or self-citations.
full rationale
This is an empirical evaluation, not a derivation. Each graph property is defined operationally, models are trained on external or synthetic datasets independent of the conclusion, and success/failure is read off measured probabilities or accuracies. There are no fitted parameters that are later renamed as predictions, no load-bearing self-citations (the cited works are benchmarks, implementations, and prior evaluations, not uniqueness theorems or justifications of the paper's own claims), and no ansatz is smuggled in via citation. The recency experiment is the closest thing to a concern: its training data has no predictive signal linking recency to the next timestep, so flat probabilities may reflect appropriate uncertainty rather than failure to learn recency. However, this is a measurement-validity / interpretation issue about what the probe can support; it is not circular reasoning, because the paper does not assume the flatness it then reports. The conclusion is a (possibly flawed) empirical inference from observed model outputs, not an equivalence to the inputs by construction. Accordingly, no circular step meeting the required standard is present.
Axiom & Free-Parameter Ledger
free parameters (4)
- Qualitative success thresholds (✓/∼/✗)
- Negative sampling ratio in density experiments =
varied from 1:1 upward by doubling
- Top-k thresholds in homophily evaluation =
1000, 10000, 100000
- Model hyperparameters per dataset =
Appendix A (lr=1e-4, batch=200, etc.)
axioms (5)
- domain assumption DyGLib implementations and the hyperparameters from Yu et al. faithfully represent each model.
- domain assumption Probe datasets isolate the target characteristic without confounding.
- domain assumption Probability-score behavior under these probes is the right operationalization of 'learning' a characteristic.
- domain assumption Flat scores in the recency probe indicate failure to learn recency, not absence of learnable signal.
- domain assumption Empirical datasets (Enron, UCI, Wikipedia) and three synthetic generators are representative enough for general statements about temporal graph models.
read the original abstract
Learning on temporal graphs has become a central topic in graph representation learning, with numerous benchmarks indicating the strong performance of state-of-the-art models. However, recent work has raised concerns about the reliability of benchmark results, noting issues with commonly used evaluation protocols and the surprising competitiveness of simple heuristics. This contrast raises the question of which characteristics of the underlying graphs temporal graph learning models actually use to form their predictions. We address this by systematically evaluating eight models on their ability to capture eight fundamental characteristics related to the link structure of temporal graphs. These include structural characteristics such as density, temporal patterns such as recency, and edge formation mechanisms such as homophily. Using both synthetic and real-world datasets, we analyze how well models learn these characteristics. Our findings reveal a mixed picture: models capture some characteristics well but fail to reproduce others. With this, we expose important limitations. Overall, we believe that our results provide practical insights for the application of temporal graph learning models and motivate more interpretability-driven evaluations in graph learning research.
Figures
Forward citations
Cited by 1 Pith paper
-
What Softmax Throws Away: Mass-Aware Attention for Evidence Accumulation
Mass-Aware Attention generalizes the softmax denominator from an L1 to an Lp norm, making the scale of attention outputs depend on how much repeated evidence has accumulated.
Reference graph
Works this paper leans on
-
[1]
Emergence of Scaling in Random Networks
Albert-László Barabási and Réka Albert. “Emergence of Scaling in Random Networks”. In: Science 286.5439 (Oct. 1999). Publisher: American Association for the Advancement of Science, pp. 509–512
1999
-
[2]
Position: Graph Learning Will Lose Relevance Due To Poor Benchmarks
Maya Bechler-Speicher et al. Position: Graph Learning Will Lose Relevance Due To Poor Benchmarks. arXiv preprint arXiv:2502.14546. Feb. 2025
Pith/arXiv arXiv 2025
-
[3]
Do We Really Need Complicated Model Architectures For Temporal Networks?
Weilin Cong et al. “Do We Really Need Complicated Model Architectures For Temporal Networks?” en. In: International Conference on Learning Representations. Kigali, Rwanda, 2023. 11 Hayes, Schumacher and Strohmaier
2023
-
[4]
Filip Cornell et al. Are We Really Measuring Progress? Transferring Insights from Evaluating Recommender Systems to Temporal Link Prediction. arXiv preprint arXiv:2506.12588. June 2025
Pith/arXiv arXiv 2025
-
[5]
On the Power of Heuristics in Temporal Graphs
Filip Cornell et al. “On the Power of Heuristics in Temporal Graphs”. en. In: I Can’t Believe It’s Not Better: Challenges in Applied Deep Learning. Singapore, Mar. 2025
2025
-
[6]
TGB 2.0: A Benchmark for Learning on Temporal Knowledge Graphs and Heterogeneous Graphs
Julia Gastinger et al. “TGB 2.0: A Benchmark for Learning on Temporal Knowledge Graphs and Heterogeneous Graphs”. en. In: Advances in Neural Information Processing Systems 37 (Dec. 2024), pp. 140199–140229
2024
-
[7]
Deep Learning for Dynamic Graphs: Models and Benchmarks
Alessio Gravina and Davide Bacciu. “Deep Learning for Dynamic Graphs: Models and Benchmarks”. en. In: IEEE Transactions on Neural Networks and Learning Systems 35.9 (Sept. 2024), pp. 11788–11801
2024
-
[8]
Stochastic blockmodels: First steps
Paul W. Holland, Kathryn Blackmond Laskey, and Samuel Leinhardt. “Stochastic blockmodels: First steps”. In: Social Networks 5.2 (June 1983), pp. 109–137
1983
-
[9]
Open Graph Benchmark: Datasets for Machine Learning on Graphs
Weihua Hu et al. “Open Graph Benchmark: Datasets for Machine Learning on Graphs”. In: Advances in Neural Information Processing Systems. V ol. 33. Curran Associates, Inc., 2020, pp. 22118–22133
2020
-
[10]
Benchtemp: A General Benchmark for Evaluating Temporal Graph Neural Networks
Qiang Huang et al. “Benchtemp: A General Benchmark for Evaluating Temporal Graph Neural Networks”. In: 2024 IEEE 40th International Conference on Data Engineering (ICDE). Utrecht, Netherlands, May 2024, pp. 4044–4057
2024
-
[11]
Temporal Graph Benchmark for Machine Learning on Temporal Graphs
Shenyang Huang et al. “Temporal Graph Benchmark for Machine Learning on Temporal Graphs”. In:Advances in Neural Information Processing Systems . Ed. by A. Oh et al. V ol. 36. New Orleans (LA), USA: Curran Associates, Inc., 2023, pp. 2056–2073
2023
-
[12]
UTG: Towards a Unified View of Snapshot and Event Based Models for Temporal Graphs
Shenyang Huang et al. “UTG: Towards a Unified View of Snapshot and Event Based Models for Temporal Graphs”. en. In: Proceedings of the Third Learning on Graphs Conference. PMLR, July 2025, 28:1–28:16
2025
-
[13]
Homophily influences ranking of minorities in social networks
Fariba Karimi et al. “Homophily influences ranking of minorities in social networks”. en. In: Scientific Reports 8.1 (July 2018). Publisher: Nature Publishing Group, p. 11077
2018
-
[14]
Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks
Srijan Kumar, Xikun Zhang, and Jure Leskovec. “Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks”. en. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. Anchorage, AK, USA, July 2019, pp. 1269–1278
2019
-
[15]
Evaluating Graph Neural Networks for Link Prediction: Current Pitfalls and New Benchmark- ing
Juanhui Li et al. “Evaluating Graph Neural Networks for Link Prediction: Current Pitfalls and New Benchmark- ing”. en. In: Advances in Neural Information Processing Systems 36 (Dec. 2023), pp. 3853–3866
2023
-
[16]
Patterns and dynamics of users’ behavior and interaction: Network analysis of an online community
Pietro Panzarasa, Tore Opsahl, and Kathleen M. Carley. “Patterns and dynamics of users’ behavior and interaction: Network analysis of an online community”. In: J. Am. Soc. Inf. Sci. Technol.60.5 (May 2009), pp. 911–932
2009
-
[17]
Towards Better Evaluation for Dynamic Link Prediction
Farimah Poursafaei et al. “Towards Better Evaluation for Dynamic Link Prediction”. en. In:Advances in Neural Information Processing Systems. Ed. by S Koyejo et al. V ol. 35. New Orleans (LA), USA: Curran Associates, Inc., 2022, pp. 32928–32941
2022
-
[18]
Rethinking Evaluation for Temporal Link Prediction through Counterfactual Analysis
Aniq Ur Rahman, Alexander Modell, and Justin Coon. “Rethinking Evaluation for Temporal Link Prediction through Counterfactual Analysis”. en. In: I Can’t Believe It’s Not Better: Challenges in Applied Deep Learning. Singapore, Mar. 2025
2025
-
[19]
Temporal Graph Networks for Deep Learning on Dynamic Graphs
Emanuele Rossi et al. “Temporal Graph Networks for Deep Learning on Dynamic Graphs”. en. In: ICML 2020 Workshop on Graph Representation Learning. Virtual Conference, Oct. 2020
2020
-
[20]
The Enron Email Dataset Database Schema and Brief Statistical Report
Jitesh Shetty and Jafar Adibi. The Enron Email Dataset Database Schema and Brief Statistical Report . en. Tech. rep. University of Southern California, 2004
2004
-
[21]
DyRep: Learning Representations over Dynamic Graphs
Rakshit Trivedi et al. “DyRep: Learning Representations over Dynamic Graphs”. en. In:International Conference on Learning Representations. New Orleans (LA), USA, Sept. 2018
2018
-
[22]
TCL: Transformer-based Dynamic Graph Modelling via Contrastive Learning
Lu Wang et al. TCL: Transformer-based Dynamic Graph Modelling via Contrastive Learning. arXiv:2105.07944 [cs]. 2021
Pith/arXiv arXiv 2021
-
[23]
Inductive Representation Learning on Temporal Graphs
Da Xu et al. “Inductive Representation Learning on Temporal Graphs”. en. In: International Conference on Learning Representations. Virtual Conference, Feb. 2020
2020
-
[24]
Evaluating link prediction methods
Yang Yang, Ryan N. Lichtenwalter, and Nitesh V . Chawla. “Evaluating link prediction methods”. en. In: Knowledge and Information Systems 45.3 (2015), pp. 751–782
2015
-
[25]
TGB-Seq Benchmark: Challenging Temporal GNNs with Complex Sequential Dynamics
Lu Yi et al. “TGB-Seq Benchmark: Challenging Temporal GNNs with Complex Sequential Dynamics”. en. In: The Thirteenth International Conference on Learning Representations. Singapore, 2025
2025
-
[26]
Towards better dynamic graph learning: new architecture and unified library
Le Yu et al. “Towards better dynamic graph learning: new architecture and unified library”. en. In:Proceedings of the 37th International Conference on Neural Information Processing Systems. NIPS ’23. Red Hook, NY , USA: Curran Associates Inc., May 2024, pp. 67686–67700
2024
-
[27]
A survey of dynamic graph neural networks
Yanping Zheng, Lu Yi, and Zhewei Wei. “A survey of dynamic graph neural networks”. en. In: Frontiers of Computer Science 19.6 (2025). Number: 6 Publisher: Higher Education Press, p. 196323. 12 What Do Temporal Graph Learning Models Learn? A Choice of Hyperparameters • Learning rate: 1e-4 • Batch size: 200 • Loss: BCELoss • Optimizer: Adam • Dropout: 0.0 •...
2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.