{"id":"45395218-c6b0-4338-b774-2627fae1c8ff","arxiv_id":"2507.18977","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A global relation-similarity enhancement layer plus frequency-weighted sampling improves long-tail entity link prediction in incrementally trained temporal knowledge graphs on the ICEWS14 and ICEWS18 benchmarks.","lead":"The paper combines a relation-based global similarity enhancement layer with frequency-weighted sampling to improve incremental temporal knowledge graph completion, reporting MRR gains of 10% and 15% on ICEWS14 and ICEWS18. A generalist reader may care because the method targets rare and newly appearing entities, a central obstacle in dynamic knowledge graphs used for event forecasting and question answering.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 5.2's two-checkpoint protocol trains each next snapshot from a model already fine-tuned on earlier validation and test labels; the 10%/15% MRR gains may measure memorization, not incremental generalization.","rationale":"The reader's weakest assumption is also the load-bearing issue for me. The paper's own Section 5.2 explicitly describes maintaining a checkpoint that is post-trained on both the validation and test sets before moving to the next training snapshot. That is a direct use of future labels as learning signal, and it contaminates the initialization of every later model, including the models whose Average and Current metrics are reported in Table 2 and Table 3. Since all methods in the study are evaluated through this same contaminated pipeline, the comparison cannot cleanly support the central claim of improved incremental generalization. The concern is not merely about the absence of error bars or a single base model; it is a structural flaw in the evaluation protocol that the authors themselves document. A rerun that removes post-training on validation/test data is the minimal experiment that would decide whether the claimed gains survive. Because this is exactly the concern the reader raised, I agree with the REJECT verdict and see no reason to change it. I therefore recommend UNCHANGED: rejection remains appropriate unless the leakage-free rerun reproduces the reported margins.","tokens_in":16315,"tokens_out":6467,"duration_ms":66100,"concrete_test":"Re-run the ICEWS14 and ICEWS18 incremental experiments under a leakage-free protocol: keep only the checkpoint trained on D_train_t, use D_val_t only for hyperparameter selection with no parameter updates, never train on any D_test_t, and initialize M_{t+1} from the evaluation checkpoint. Then recompute Table 2 Current and Average MRR/H@10 and Table 3 inductive results. If Ours retains its margin over FT/ER/EWC, the concern does not land; if the margins shrink, reverse, or the 10%/15% numbers disappear, the reported gains are artifacts of post-training on test labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The protocol in Section 5.2 is explicitly leaky. It states that at each training step t, the authors keep one checkpoint trained on D_train_t for evaluation and another checkpoint obtained by 'post-training on both the validation and test sets for a few epochs before proceeding' to D_train_{t+1}. As a result, every model after t=1 is initialized from weights that have been supervised on D_test_1 through D_test_{t-1}, and the reported 'Average' metrics are computed over exactly those earlier test sets. This is not a subtle evaluation choice: future labels are used as training signal, so the comparison against FT, ER, and EWC does not isolate incremental generalization. The effect can be uneven across methods: the enhancement layer injects global relation-based similarity aggregations, so it may absorb leaked test-entity patterns more effectively than the baselines, and weighted sampling may further reinforce labels seen during that post-training. Without a leakage-free rerun, the claimed 10%/15% MRR improvements are unsubstantiated. Secondary issues such as single-base-model evaluation and absent error bars would not alone force rejection, but this leak is decisive for the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes an incremental training framework for temporal knowledge graph completion, consisting of (i) a model-agnostic enhancement layer that augments local GNN-based entity representations with a global relation-based similarity aggregation, and (ii) a weighted frequency-based sampling strategy that up-weights quadruples involving rare entities. The framework is instantiated with TiTer as the base model and evaluated on ICEWS14 and ICEWS18 under fine-tuning, EWC, and experience-replay baselines. The authors claim improved total link prediction, inductive link prediction, and long-tail entity performance, including 10% and 15% MRR gains, and they provide ablations, a forgetting analysis, and a complexity/runtime analysis.","tokens_in":16584,"tokens_out":10420,"duration_ms":104103,"significance":"If the empirical results were valid, the paper would contribute a practical, modular recipe for continual and inductive TKG completion, and the global-similarity enhancement is an interesting departure from purely local GNN aggregation. The two released benchmark datasets and the complexity analysis are also useful. However, the main evaluation protocol is contaminated by supervised use of validation and test labels during incremental training, so the central empirical claims cannot be assessed from the reported experiments. The model-agnostic claim is also supported only by a single base model, and no uncertainty quantification is provided.","major_comments":[{"comment":"The incremental training procedure is leaky: the text states that at each step the authors keep one checkpoint trained on D_train_t for evaluation and another obtained by 'post-training on both the validation and test sets for a few epochs before proceeding' to D_train_{t+1}. Consequently, every checkpoint after t=1 is initialized from weights that have been supervised on D_test_1 through D_test_{t-1} and on the corresponding validation sets, while the Average columns in Tables 2 and 3 are computed over exactly those earlier test sets. This is future-label leakage: the comparisons against FT, ER, and EWC in Tables 2-4 and Figure 4 measure partly memorization or leaked adaptation rather than incremental generalization. The claims in Section 6.1 of 10% (ICEWS14) and 15% (ICEWS18) MRR improvements, and the forgetting analysis in Figure 4a, are unsubstantiated without a rerun that uses only D_train_t for weight updates and initialization.","section":"Section 5.2"},{"comment":"The inductive link prediction evaluation is invalidated by the same leak. Entities are classified as unseen if they do not appear in previous training quadruples, but because earlier D_test sets were post-trained on before advancing to the next task, many of those 'unseen' entities have in fact appeared in the model's training history. The Average results over the union of inductive test sets are therefore not a measure of inductive generalization. In addition, Table 3 appears malformed: the rows for +ER and +EWC contain fewer entries than the columns require, and the claimed First-snapshot equality of FT, ER, and EWC is not visible from the table. The inductive claim cannot be evaluated from the current tables.","section":"Section 6.2 and Table 3"},{"comment":"The model-agnostic claim is not supported by the experiments. The abstract and introduction state that the enhancement layer can be added to 'any existing TKG completion method' or 'most GNN-based TKG completion models,' but all experiments use a single base model, TiTer. Even setting aside the leak, a one-base-model study cannot establish model-agnosticism; at minimum a second base model with a different inductive mechanism is needed.","section":"Abstract and Section 1"},{"comment":"The experiments report point estimates without seeds, error bars, or significance tests. Several headline differences are small (e.g., Table 2, ICEWS18 Average MRR: Ours 0.323 vs EWC 0.319; ICEWS14 Average MRR: Ours 0.421 vs FT 0.413), so it is not possible to tell whether the proposed components, rather than noise or the leaked data, drive the reported improvements. The hyperparameter search is performed on a validation set, which is itself later used for post-training, compounding the selection bias.","section":"Tables 2-4 and Section 5.2"}],"minor_comments":[{"comment":"In the Related Work section, 'corrsponding' should be 'corresponding'.","section":"Section 2"},{"comment":"The similarity set S_t(r) is defined without excluding the query entity s itself; if s has past events with relation r, the enhancement g(s,r,t) may include s's own embedding. The authors should clarify whether this is intentional and, if so, how self-reinforcement is handled.","section":"Equations (2)-(3)"},{"comment":"The notation '28k/3.7k/4k' and the header 'G1 Avg|Q| Gi>1' are not defined; the reader should be told whether these are train/validation/test counts and what the average is taken over.","section":"Section 5.1 and Table 1"},{"comment":"The text says '10% relative improvement' and '15% improvement' in MRR, but from Table 2 these percentages are relative to Titer trained only on the first snapshot, not relative to the best incremental baseline. The reference point should be stated explicitly in both the abstract and Section 6.1.","section":"Section 6.1"},{"comment":"The table formatting appears corrupted: the +ER and +EWC rows have fewer entries than the columns require, and the mapping between values and the 'First'/'Average' column groups for ICEWS18 and ICEWS14 is hard to infer. The table should be reconstructed carefully.","section":"Table 3"}],"recommendation":"reject","confidential_remarks":"The manuscript is transparent about the leaky protocol in Section 5.2, but the stated protocol is a fundamental evaluation flaw rather than a presentational issue. The reported numbers cannot support the central claims of improved continual and inductive TKG completion, and a corrected evaluation would require a substantially new experimental campaign. I therefore recommend rejection, despite the interesting architectural idea and the useful complexity analysis. I would be open to considering a substantially revised manuscript with a leakage-free evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing you should know before reading: the headline numbers are not trustworthy. Section 5.2 describes a two-checkpoint protocol where, before moving to the next temporal snapshot, the model is post-trained on both the validation and test sets for a few epochs. That means every later model is initialized from weights that have been supervised on earlier test labels, and the reported \"Average\" metrics are computed over exactly those test sets. The claimed 10% and 15% MRR improvements over FT, EWC, and ER could easily be memorization rather than incremental generalization. The enhancement layer and weighted sampling may absorb leaked patterns more effectively than the baselines, so the comparison is stacked in the authors' favor.\n\nWhat is actually new: the paper's own contribution is the incremental training regime with catastrophic-forgetting baselines. The enhancement layer and weighted sampling come from the authors' WWW 2024 companion paper. The problem setup—multiple temporal extrapolation tasks with new entities appearing over time—is useful and worth having as a benchmark. The ablation study is sensible, and the complexity analysis is fine. Credit where due: the authors are explicit about the protocol, so the leak is not hidden; that makes it easier to fix.\n\nSoft spots, in proportion: the leak is load-bearing. Aside from that, only Titer is used as the base model, so \"model-agnostic\" is an overstatement. There are no error bars or significance tests, and with two datasets and modest gains that matters. The relation-based similarity is a simple grouping rule rather than a learned measure, which is fine but less novel than the presentation implies. A minor issue: hyperparameters are selected on validation performance, and no sensitivity analysis is reported.\n\nWho this is for: researchers working on continual learning for temporal KGs will find the problem formulation and baseline comparisons a useful starting point, but they should not rely on the reported numbers. If the authors rerun without the leak, the core idea—global similarity plus weighted sampling for long-tail entities—might hold up, but that is currently an open question.\n\nRecommendation: reject the current version. If I were handling it, I would send it back with a clear request to remove the post-training on test sets, report results from the clean checkpoints, and include error bars over multiple runs. That revision, if it comes back clean, would deserve another look. For peer review: this is not a desk reject—the flaw is identifiable and fixable, and the problem is relevant enough to merit serious referee time.","headline":"The incremental TKG setup is worth discussing, but the reported 10%/15% MRR gains are uninterpretable because the Section 5.2 protocol trains on test labels before evaluating on them.","tokens_in":17084,"tokens_out":3156,"would_cite":false,"duration_ms":32081,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Incremental TKG completion gains 10–15% MRR from global similarity and tail sampling.","keywords":["temporal knowledge graphs","knowledge graph completion","incremental learning","continual learning","long-tail entities","inductive link prediction","graph neural networks","link prediction"],"falsifier":"Re-run the incremental protocol with each checkpoint trained only on the current snapshot's training split, with no post-training on that snapshot's validation or test sets, and recompute the Average MRR over all prior test sets; if the 10–15% advantage over FT, EWC, and ER shrinks to noise, the reported gains depend on the post-training step.","tokens_in":16144,"feed_emoji":"📈","tokens_out":8612,"duration_ms":86594,"temperature":0.7,"pith_summary":"The paper claims that temporal knowledge graph completion models can be made to handle an evolving graph—where new facts arrive and new, sparsely connected entities appear—without retraining from scratch. It proposes a model-agnostic enhancement layer that enriches an entity's representation using other entities that have participated in the same relation at earlier times, plus a sampling scheme that up-weights quadruples involving rare entities. On two ICEWS benchmarks, the combined additions improve overall and inductive link prediction over fine-tuning, experience replay, and elastic weight consolidation, with 10% and 15% relative MRR gains on ICEWS14 and ICEWS18. The authors' central point is that long-tail and unseen entities are both a representation problem and a data-balancing problem, and both can be addressed on top of an existing GNN-based TKG model.","feed_headline":"TKG link prediction gains 10-15% MRR via global similarity","feed_subtitle":"An enhancement layer and inverse-frequency sampling beat fine-tuning, replay, and EWC on evolving event graphs.","key_machinery":"The load-bearing pieces are (1) a relation-based global similarity set $S_t(r)=\\{s_i : (s_i,r,o_i,t_i)\\in G,\\ t_i