{"id":"4407b467-9a01-4f38-8e83-da8666e29f67","arxiv_id":"2502.02975","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TGB-Seq provides eight low-repetition temporal graph datasets and shows that current temporal GNNs struggle to capture sequential dynamics, generalizing poorly to unseen edges.","lead":"This paper introduces TGB-Seq, a set of eight temporal graph benchmarks with few repeated edges, designed to test whether temporal GNNs can learn sequential dynamics. It shows that current models such as GraphMixer and DyGFormer fail on simple sequential prediction tasks and drop sharply on the new benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The §3.2 toy experiment withholds node identity from temporal GNNs, making 'inherently incapable' an information-theoretic artifact rather than an architectural finding.","rationale":"The reader's weakest assumption points at the toy example's representativeness; my stress-test identifies a more specific and more damaging confound: the toy example makes the task information-theoretically impossible without node identity, so the observed 50% AP does not support the 'inherently incapable' conclusion. This is load-bearing because the abstract and §3.2 use the toy to motivate the entire benchmark. The benchmark itself remains useful: TGB-Seq is a large, reproducible collection of low-repeat datasets with a clear evaluation protocol, and the empirical observation that current temporal GNNs underperform there is valuable even if the interpretation is softened. The paper should either provide node-ID information to the temporal GNNs in the toy and benchmark, or explicitly frame the result as 'incapable without node identity/features' rather than 'inherently incapable.' The repeated-edge inconsistency for Yelp (25.18%) and Taobao (16.58%) noted by the reader is a real dataset-design issue but is secondary: it affects the 'minimize repeated edges' marketing claim, not the core architectural conclusion. My concrete test directly settles whether the toy result is an artifact of withheld identity; if it lands, the central claim must be revised, but the benchmark contribution can stand, hence CONDITIONAL.","tokens_in":21086,"tokens_out":5136,"duration_ms":49699,"concrete_test":"Re-run the §3.2 toy experiment with the same temporal GNNs but provide each item with a learnable per-node identity embedding (or one-hot ID input), matching the item-identity information available to SGNN-HN. If any temporal GNN (e.g., JODIE, TGN, GraphMixer, or DyGFormer) then exceeds chance-level AP on the i4-vs-i9 prediction, the 'inherently incapable' claim is falsified; the correct statement would be that these architectures fail only when node identities are withheld, not that they are inherently unable to learn sequential dynamics.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that GraphMixer and DyGFormer are 'inherently incapable of learning simple sequential dynamics' (Abstract, §3.2) rests on the toy experiment in §3.2, but that experiment is constructed so that the task is impossible for any model that cannot use item identity. The two item sets {i0..i4} and {i5..i9} are perfectly symmetric: corresponding items interact at exactly the same timestamps, and 'both nodes and edges lack features' (§3.2). With no features and identical timestamps, i4 and i9 are indistinguishable to a model that does not receive item IDs or per-node embeddings. The reported AP≈50% for all temporal GNNs is therefore exactly what an information-theoretically optimal model would achieve under this input restriction, not evidence of an architectural limitation. SGNN-HN reaches 100% because it is a sequential recommender that operates on item identities, giving it information the temporal GNNs are not given. The paper never states whether the temporal GNNs receive node IDs, and the benchmark datasets in §4 also 'exclude node and edge features,' so the same confound carries over to the main TGB-Seq experiments. Consequently, the toy result does not demonstrate that temporal GNN architectures cannot learn sequential dynamics; it demonstrates that models deprived of node identity cannot solve a task whose only solution requires node identity. The 'inherently incapable' wording overstates what the evidence supports.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that existing temporal-GNN benchmarks over-represent repeated edges and therefore miss a core challenge of future link prediction: complex sequential dynamics. It presents a toy experiment in which temporal GNNs score near chance while a session-based recommender (SGNN-HN) reaches perfect AP, and it uses this result to claim that models such as GraphMixer and DyGFormer are 'inherently incapable' of learning simple sequential dynamics. The authors then introduce TGB-Seq, eight large real-world temporal graphs (four bipartite recommendation networks and four non-bipartite networks) curated to have low repeat ratios, and benchmark eight temporal GNNs plus SGNN-HN under a chronological 70/15/15 split with MRR@100. They report substantial performance drops relative to Wikipedia/Reddit, dataset-dependent rankings, and high training costs, and release a pip package, datasets, and leaderboards.","tokens_in":21363,"tokens_out":15274,"duration_ms":133179,"significance":"The benchmark is a useful community resource: it targets an acknowledged weakness of existing temporal graph benchmarks (repeat-heavy evaluation), provides large real-world datasets with documented preprocessing, reports standard deviations over three seeds, and ships code and a leaderboard. The toy experiment is a clean attempt to isolate sequence order from other signals. However, the paper's central interpretive claim ('inherently incapable') is not supported as stated, because the toy construction removes the only information that can solve the task, and the real benchmark lacks a quantitative demonstration that its low repeat ratio translates into sequential dynamics rather than other dataset properties.","major_comments":[{"comment":"The toy experiment cannot support the 'inherently incapable' claim because it strips away the only cue needed to solve the task. In the construction, i4 and i9 (and the u and v groups) interact at identical timestamps, and the text states that nodes are represented solely by null features and interaction timestamps; the paper never states whether the temporal GNNs receive node IDs or per-node embeddings. Under that input restriction, an optimal identity-free model must assign equal score to i4 and i9, so AP≈50% is the information-theoretic ceiling rather than evidence of an architectural failure. SGNN-HN reaches 100% because it is given item identities. I ask the authors to (i) state explicitly whether node IDs/per-node embeddings were provided to the temporal GNNs; (ii) add a control condition in which the temporal GNNs receive one-hot or learnable per-node embeddings; and (iii) replace 'inherently incapable' (Abstract, Section 3.2, Section 6) with a statement scoped to the identity-free input setting.","section":"Section 3.2, Table 1"},{"comment":"The paper motivates TGB-Seq as a benchmark of 'complex sequential dynamics,' but the only reported property that operationalizes this notion is the repeat ratio, and a low repeat ratio is not evidence of sequential structure: a randomly shuffled sparse graph can also have few repeated edges. To make the central interpretation credible—that the observed MRR drops reflect the need to model ordered interaction patterns—the authors should quantify sequential dynamics in the datasets (e.g., next-item transition lift or entropy, repeat-vs-exploration statistics, temporal motif counts) or run an ablation that controls for dataset size, sparsity, temporal granularity, and feature availability. Without such evidence, the performance gaps reported in Tables 3 and 4 could be driven by these other factors.","section":"Section 4, Table 2"}],"minor_comments":[{"comment":"The x-axis labels in Figure 1 (Wikipedia, Reddit, GoogleLocal, ML-20M) do not match the caption and text, which name Yelp and Taobao as the recommendation datasets; please reconcile the figure with the reported experiments.","section":"Figure 1"},{"comment":"The sentence in Section 3.2 contains a typo: 'calcu the number of common neighbors' should read 'calculate the number of common neighbors.'","section":"Section 3.2"},{"comment":"The density entry for ML-20M is typeset as 1.49×100; it should be 1.49×10^0 (i.e., 1.49%), consistent with the 'Density (%)' column.","section":"Table 2"},{"comment":"The statement in Section 4 that 'All TGB-Seq datasets ... exclude node and edge features' conflicts with Appendix B.1, which states that ML-20M, Yelp, and GoogleLocal contain text features; please clarify that these features are excluded by the benchmark preprocessing.","section":"Section 4 vs. Appendix B.1"},{"comment":"The remark that existing proposed datasets 'still contain numerous repeated edges' should acknowledge the exception of tgbl-review (0.19% repeat ratio in Table 5), which is comparable to TGB-Seq's Yelp and Taobao.","section":"Section 4, Remark"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The manuscript appears to be a camera-ready ICLR paper, but the toy experiment's interpretation and the lack of direct evidence of sequential dynamics in TGB-Seq are central to the motivation; I recommend a major revision. The benchmark resources are valuable and should not be discarded."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the short version first: TGB-Seq is a solid, useful benchmark that deserves to be adopted, but the headline claim that temporal GNNs are 'inherently incapable' of learning simple sequential dynamics is overstated—the toy experiment in Section 3.2 is set up so that the task is impossible for any model that cannot use node identity, and the paper never tells us whether the models received node IDs.\n\nWhat's new and good: the benchmark itself. Eight large datasets with carefully documented preprocessing, low repeat ratios, chronological splits, and a clean evaluation protocol with 100 negatives and MRR. The authors did the work of curating datasets from recommendation and social domains that existing benchmarks miss, and they report standard deviations over three runs. The code, leaderboard, and pip package lower the barrier to entry. That's a real contribution. The observation that repeated edges inflate performance is not new—Poursafaei et al. (2022) and TGB already made that point—but TGB-Seq operationalizes it across a wider domain spread.\n\nThe soft spot is the toy experiment. The two item sets are perfectly symmetric: {i0..i4} and {i5..i9} interact at identical timestamps, and no features are provided. Under those conditions, i4 and i9 are indistinguishable to any model that represents nodes solely by features and timestamps—which is exactly what the paper says the aggregation and memory modules do. The 50% AP is then what an information-theoretically optimal model achieves, not evidence of an architectural flaw. SGNN-HN gets 100% because it operates on item IDs. The paper should either give the temporal GNNs access to node identity (e.g., one-hot IDs as features) or soften 'inherently incapable' to something like 'incapable given the current input representation.' This doesn't sink the benchmark, which may still reveal real weaknesses, but it does invalidate the strongest claim.\n\nMinor issues: the repeat ratios for Yelp (25%) and Taobao (16.6%) are hardly 'minimized' when tgbl-review has 0.19%; the paper should acknowledge these are moderate. The sequential recommender comparison uses only SGNN-HN, which is thin. And the exclusion of all node features is a design choice worth more discussion, since the appendix shows features help some models.\n\nBottom line: this is a valuable resource for the temporal GNN community and deserves careful peer review. The authors need to fix the toy-experiment interpretation and maybe add a baseline or two. I would cite it and put it on the reading group list.","headline":"TGB-Seq is a genuinely useful benchmark, but the 'inherently incapable' claim rests on a toy experiment that withholds node identity.","tokens_in":21914,"tokens_out":4487,"would_cite":true,"duration_ms":184711,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new benchmark shows that temporal GNNs fail to learn even simple sequential dynamics, because their memory and aggregation modules cannot distinguish order-based patterns, and that this failure is masked by datasets full of repeated…","keywords":["temporal graph neural networks","future link prediction","sequential dynamics","benchmark","unseen edges","repeat ratio","recommender systems","TGB-Seq"],"falsifier":"Re-run the toy example of Section 3.2 with a one-hot feature that identifies each item ($i_0,\\dots,i_9$) while keeping timestamps and neighborhoods identical. If any of the tested temporal GNNs then ranks $i_4$ above $i_9$ with AP well above 50%, the claim that the memory and aggregation modules are inherently incapable of learning this sequential pattern fails, because the limitation would be traceable to missing input features rather than to the modules' inability to represent order.","tokens_in":20879,"feed_emoji":"🔗","tokens_out":8860,"duration_ms":78187,"temperature":0.7,"pith_summary":"The paper sets out to show that current temporal graph neural networks do not actually learn sequential dynamics: they can memorize repeated edges, but they cannot learn a pattern as simple as \"a user who followed OpenAI and Anthropic is likely to follow AI at Meta next.\" Using a deliberately minimal toy graph in which the only signal is the order of item interactions and all events happen at identical timestamps, the authors find that all tested methods except a sequential-recommendation baseline score near chance, around 50-53% average precision. The paper then introduces TGB-Seq, eight large real-world datasets curated so that edges almost never repeat, forcing models to predict links they have never seen. On TGB-Seq, existing methods' performance drops substantially, several methods time out during training, and no single method wins across all domains; this is presented as evidence that temporal GNNs need to model sequence order rather than repetition.","feed_headline":"Temporal GNNs fail at learning sequential dynamics","feed_subtitle":"Eight real-world datasets with almost no repeated edges expose memorization instead of generalization.","key_machinery":"The load-bearing object is a toy temporal graph (Figure 3) built from two user groups, $u$ and $v$, and two item sequences, $\\{i_0,\\dots,i_4\\}$ and $\\{i_5,\\dots,i_9\\}$; every $u$-user interacts with items $\\{i_0,\\dots,i_3\\}$ in order and every $v$-user with $\\{i_5,\\dots,i_8\\}$, with both groups acting at identical timestamps, and the test asks whether a model can choose $i_4$ (for $u$) over $i_9$ after seeing only the four prior interactions. Because nodes and edges have no features, the only usable signal is the order of the sequence, which isolates sequential dynamics from any content cue. The second mechanism is the repeat ratio $r = |E_{\\text{seen}}|/|E|$, the fraction of edges that duplicate an earlier edge; TGB-Seq's curation targets $r = 0$ (or near-zero for Yelp and Taobao) to prevent the memorization shortcut that dominates existing benchmarks.","core_discovery":"The paper's central claim is that the architectural components shared by existing temporal GNNs—the memory module and the neighborhood-aggregation module—are structurally blind to sequential order. In the toy construction, items $i_4$ and $i_9$ are each connected to a mirrored group of users who interact at identical timestamps, so memory states and one-hop embeddings for the two candidates are indistinguishable; correlation-based modules such as common-neighbor counting and anonymous walks also fail because the neighborhoods mirror each other. Extending CAWN's walk length from one to two hops lifts AP only from 50.00% to 52.80% while consuming much more computation, and a three-hop walk exhausts an 80 GB GPU. The paper concludes that because existing datasets such as Wikipedia and Reddit contain 88-99% repeated edges, strong benchmark scores mostly reflect repetition-memorization, and it offers TGB-Seq—with repeat ratios of 0% on six of eight datasets—as a test that isolates whether a model can generalize to unseen edges.","pith_inferences":["Beyond the paper, the same toy construction can serve as a unit test for any new temporal GNN: a model that cannot push AP above 50% on it should not be expected to generalize on TGB-Seq, and one that solves it could be evaluated for whether the solution transfers to the real datasets.","The repeat-ratio statistic likely understates difficulty: a dataset with $r=0$ but highly repetitive neighborhood substructures could still be solved by aggregation shortcuts. A useful extension would be to report a 'novel-structure ratio' or to stratify MRR by whether a test edge's neighborhoods resemble training neighborhoods.","The paper's findings imply that temporal-GNN evaluation should adopt random negative sampling from all nodes, as TGB-Seq does, rather than historical negatives; otherwise the negative-sampling protocol itself rewards memorization and can invert rankings.","A testable prediction follows from the paper's mechanism analysis: adding an order-sensitive encoding (e.g., positional or sequential embeddings) to a memory or aggregation module should close much of the gap on TGB-Seq, especially on the bipartite recommendation datasets, without changing the underlying graph model."],"forward_implications":["On TGB-Seq, rankings reverse relative to Wikipedia and Reddit: memory-based JODIE and TGN lead the recommendation datasets, while CAWN and DyGFormer—the strongest methods on repeated-edge benchmarks—drop sharply, showing that benchmark choice changes conclusions about which methods work.","The MRR gap between repeated and unseen edges on existing datasets reaches up to eightfold, so high scores on Wikipedia, Reddit, MOOC, LastFM, Enron, and Social Evo. substantially overstate generalization ability.","Several methods cannot finish one training epoch within 24 hours on the larger TGB-Seq datasets (e.g., TGAT and CAWN on several datasets, DyGFormer on ML-20M), so efficiency and accuracy are not simultaneously achieved by any current method.","TGB-Seq ships datasets, a Python package for downloading, negative-sample generation, and evaluation, and public leaderboards, making it a ready testbed for sequence-aware temporal GNNs."],"supporting_citations":[{"why":"Provides EdgeBank and the repeated-historical-edge analysis that motivates the paper's repeat-ratio critique and the distinction between repeated and unseen edges.","marker":"Poursafaei et al. (2022)"},{"why":"Establishes the TGB benchmark and the MRR evaluation protocol with 20 negative samples that TGB-Seq extends to 100 negatives and to low-repeat datasets.","marker":"Huang et al. (2024b)"},{"why":"Introduces DyGFormer and the unified library used to run all benchmark experiments; DyGFormer is one of the methods the paper claims cannot learn simple sequential dynamics.","marker":"Yu et al. (2023)"},{"why":"Introduces GraphMixer, an aggregation-based baseline that scores 50.00% AP on the toy example and is a central target of the incapability claim.","marker":"Cong et al. (2023)"},{"why":"Introduces JODIE and the Wikipedia/Reddit/MOOC/LastFM datasets whose high repeat ratios are contrasted with TGB-Seq.","marker":"Kumar et al. (2019)"},{"why":"Introduces TGN, whose memory-module design is analyzed in Section 3.2 and benchmarked across TGB-Seq.","marker":"Rossi et al. (2020)"},{"why":"Introduces CAWN; its walk-length ablation (one-hop versus two-hop) is used to show that high-order aggregation gives only marginal gains on sequential dynamics.","marker":"Wang et al. (2021d)"},{"why":"Introduces DyRep, a memory-based method used in the toy example and the full benchmark.","marker":"Trivedi et al. (2019)"},{"why":"Introduces TGAT, an aggregation-based baseline used in the toy example and the full benchmark.","marker":"Xu et al. (2020)"}],"fun_headline_variants":["Temporal GNNs can't learn simple sequences","New benchmark: Temporal GNNs only memorize repeats","TGB-Seq: Exposing temporal GNNs' blind spot","Why temporal GNNs fail on unseen links","Sequential dynamics break temporal GNNs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the toy example, in which the only signal distinguishing the two candidate items is the order of a user's past interactions and no node or edge features exist, faithfully represents the sequential dynamics found in real-world systems; if that premise fails, the conclusion that chance-level AP scores reveal an architectural limitation rather than a task or optimization difficulty does not follow.","fun_headline_variants_meta":{"raw":{"variants":["Temporal GNNs can't learn simple sequences","New benchmark: Temporal GNNs only memorize repeats","TGB-Seq: Exposing temporal GNNs' blind spot","Why temporal GNNs fail on unseen links","Sequential dynamics break temporal GNNs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000179,"raw_usage":{"total_tokens":1331,"prompt_tokens":1008,"completion_tokens":323,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":246}},"tokens_in":624,"tokens_out":323,"duration_ms":3305,"temperature":1.0,"reasoning_tokens":246,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T10:21:48.856995+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the toy example of Section 3.2 with a one-hot feature that identifies each item ($i_0,\\dots,i_9$) while keeping timestamps and neighborhoods identical. If any of the tested temporal GNNs then ranks $i_4$ above $i_9$ with AP well above 50%, the claim that the memory and aggregation modules are inherently incapable of learning this sequential pattern fails, because the limitation would be traceable to missing input features rather than to the modules' inability to represent order.","supporting_citations":[{"cited_title":"Predicting dynamic embedding trajectory in temporal interaction networks","cited_arxiv_id":null,"evidence_quote":"Introduces JODIE and the Wikipedia/Reddit/MOOC/LastFM datasets whose high repeat ratios are contrasted with TGB-Seq."},{"cited_title":"Dyrep: Learning representations over dynamic graphs","cited_arxiv_id":null,"evidence_quote":"Introduces DyRep, a memory-based method used in the toy example and the full benchmark."}],"review_version":1}