{"id":"9d815d05-8a19-48a8-9548-9f221308b0ee","arxiv_id":"2505.15245","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new benchmark (ETR) and a structure-aware LLM framework (GETER) inject temporal graph embeddings as a soft prompt to improve explainable temporal reasoning.","lead":"This paper introduces a benchmark for explainable temporal reasoning and a method that feeds temporal graph structure into LLMs as a soft token to improve their explanations. If it holds up, it offers a reusable testbed and a simple recipe for combining temporal knowledge graphs with LLM text generation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Explanation evaluation is circular: models are trained on GPT-4o-generated explanations, then scored by lexical similarity to those same GPT-4o texts, so the explainable-reasoning claim is unsupported without human or faithfulness checks.","rationale":"The central claim has two legs: prediction accuracy and explanation quality. The prediction results are credible: GETER consistently improves over LoRA-only by 7–11% F1 across three datasets, the ablation in Table 4 shows the adapter and reasoning-chain text contribute, and the gains are large enough that missing error bars are unlikely to invert the conclusion. The explanation results, however, are the paper's differentiator and its title's focus. They fail because the gold references are generated by GPT-4o under a prompt that leaks the label, the same references are used as training targets, and the chosen metrics are pure surface-similarity measures. This does not measure whether an explanation is faithful to the reasoning chains, logically valid, or even consistent with the predicted label. The human study in Appendix A.5 validates the benchmark's gold explanations, not the model outputs, so it cannot rescue the explanation claim. The reader identified this as the weakest assumption, and I agree. A secondary concern is the unspecified temporal split of the frozen RE-GCN encoder: if it was pre-trained on the full TKG including test-period events, the structural token could leak future information. That ambiguity should also be checked, but it does not displace the explanation-evaluation problem as the primary load-bearing issue. Since the prediction contribution is solid and the explanation claims are conditionally fixable via human evaluation or reframing, the reader's CONDITIONAL verdict stands unchanged.","tokens_in":29225,"tokens_out":10403,"duration_ms":94953,"concrete_test":"Run a blinded human evaluation on a random sample of 100 test instances per system (GETER, LoRA-only, GPT-4o zero-shot). For each instance, three annotators independently rate the generated explanation on: (1) factual/logical consistency with the provided reasoning-chain text, and (2) whether the explanation text supports the prediction label it states. Compare mean ratings with a paired significance test (e.g., Wilcoxon or bootstrap). If GETER does not significantly outperform LoRA (p < 0.05) on both criteria, the explanation-quality claims in Table 3 cannot be attributed to better explainable reasoning. As a quick complementary diagnostic, also compute BLEU-4 separately for GETER outputs whose predicted label matches vs. mismatches the gold label; high BLEU on mismatched outputs would confirm the metric is insensitive to reasoning correctness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2.2 constructs gold explanations by prompting GPT-4o with a template that already asserts the correct label ('We predict that [es] [r] [eo] will happen on [tq]...') and asking it to 'ensure that the prediction aligns with the reasoning steps.' These GPT-4o outputs are then used as the supervision target in instruction tuning (Section 4.3, Eq. 5, target tokens YA) and as the references for BLEU-4, ROUGE-L, METEOR, and BERTScore in Table 3. Consequently, the explanation metrics report how closely a fine-tuned 7B model reproduces GPT-4o's phrasing, not whether its explanation is logically consistent with the evidence or even supports the stated label. All four metrics are surface-level token/semantic similarities and do not condition on prediction correctness; a model can output the wrong label yet receive a high BLEU if the wording resembles the gold. The human evaluation (Appendix A.5) scores the benchmark's gold explanations, not any model output, so it does not establish that GETER's explanations are more convincing than LoRA's. The 'explainable' component of the central claim therefore rests on an invalid measurement. The prediction F1 gains (7–11% over LoRA) are not affected by this flaw and appear credible; but the paper's stated contribution—explainable temporal reasoning—is not supported by Table 3.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ETR, a benchmark for explainable temporal reasoning built from five temporal knowledge graph datasets, and GETER, a framework that prepends a soft graph token (derived from a frozen temporal encoder and a trained projection adapter) to LLM instruction-tuning inputs. The benchmark contains positive, negative, and neutral samples with GPT-4o-generated explanation text as gold output. GETER is evaluated against zero-shot LLMs, LoRA-tuned LLMs, and graph-based temporal reasoning baselines. The paper reports consistent F1 gains over LoRA-only tuning across five datasets, and improved BLEU/ROUGE/METEOR/BERTScore over GPT-4o and LoRA baselines, along with ablations supporting the contribution of the structural adapter and reasoning-chain text.","tokens_in":29509,"tokens_out":3462,"duration_ms":32962,"significance":"If the central claim were fully supported, the paper would make two useful contributions: a multi-granularity benchmark for temporal reasoning with explanation supervision, and a parameter-efficient method for injecting TKG structure into LLM generation. The prediction-side results are substantive and fairly consistent: GETER with Mistral improves overall F1 by roughly 7–11% over LoRA-only tuning on the three headline datasets, and the ablations in Table 4 show that both the structure-text adapter and the reasoning-chain text contribute. The code and dataset are claimed to be publicly available, and the benchmark construction addresses a genuine gap. However, the 'explainable' component is not established by the evaluation as presented. Explanation quality is scored by lexical/semantic similarity to GPT-4o-generated gold texts that also serve as training targets, and the human evaluation in Appendix A.5 rates only the gold explanations, not any model output.","major_comments":[{"comment":"The explanation-quality evaluation is circular and does not measure reasoning quality. The gold explanations are constructed by prompting GPT-4o to revise a template that already asserts the correct label ('We predict that ... will happen ... please revise ... ensure that the prediction aligns with the reasoning steps'). These same GPT-4o texts are used as the supervision target in the instruction-tuning objective (Eq. 5, tokens Y_A) and as the references for BLEU-4, ROUGE-L, METEOR, and BERTScore in Table 3. Consequently, the explanation metrics report how closely a fine-tuned model reproduces GPT-4o's phrasing, not whether the explanation is faithful to the evidence, logically consistent, or even supportive of the emitted label. The human evaluation in Appendix A.5 scores only the benchmark's gold explanations (query, reasoning chains, and explanation text consistency), not the outputs of GETER, LoRA-tuned, or zero-shot models; therefore it cannot validate the relative explanation quality claimed in Section 5.2. The prediction F1 results in Table 2 are not affected by this flaw, but the paper's stated 'explainable temporal reasoning' contribution is not supported by the current evidence.","section":"Sec. 3.2.2, Eq. (5), Sec. 5.2, Table 3"},{"comment":"The ablation study reports only prediction F1 for the variants (GETER without the structure-text adapter, without reasoning-chain text, and without both). Since the paper's central claim concerns explainable reasoning, the ablation should also report explanation metrics (or at least a human faithfulness check) for these variants. Without such results, it is impossible to tell whether the graph token improves explanation quality independently of prediction accuracy, or whether its only measurable effect is on the classification head. This is load-bearing because the Introduction and Abstract state that GETER advances explainable temporal reasoning by integrating graph structure, whereas Table 4 demonstrates only a prediction-side benefit.","section":"Sec. 5.3, Table 4"}],"minor_comments":[{"comment":"The abstract and introduction contain minor language issues, e.g., 'To address challenge' should be 'To address this challenge', and 'incoherence, unnatural flow, and insufficient logical consistency' is a run-on list. These do not affect the technical content but should be cleaned up.","section":"Abstract, Sec. 1"},{"comment":"The notation in Equation (1), displayed as 'C(es, eo) ← l^ i=1 (Ei, Ri, Ei+1, Ti)', is garbled or nonstandard. The authors should define the composition or concatenation operation explicitly, and clarify what 'l ∈ {1, 2} denotes the path length' means for the case l=1 versus l=2.","section":"Sec. 3.2.1, Eq. (1)"},{"comment":"Equation (4) calls the graph representation computation 'parameter-free message passing', but the projection matrix Wp is a learned parameter. The word 'parameter-free' is misleading; the authors likely mean that the aggregation over the reasoning chain is non-parametric. Please rephrase.","section":"Sec. 4.2, Eq. (4)"},{"comment":"No standard deviations or confidence intervals are reported for the F1 and explanation metrics, despite the fact that the test sets are of modest size (e.g., 949–2150 instances). Reporting results from multiple seeds or at least statistical significance tests would strengthen the reproducibility claims.","section":"Sec. 5.2, Tables 2–3"},{"comment":"The human evaluation scores average around 2.8–2.9 on a 1–3 scale, which is high, but no inter-annotator agreement measure (e.g., Cohen's kappa or Krippendorff's alpha) is reported. Adding such a measure would make the benchmark quality assessment more convincing.","section":"Appendix A.5, Table 7"}],"recommendation":"major_revision","confidential_remarks":"The prediction-side results appear solid and the paper is well scoped as an engineering contribution to structure-aware LLM fine-tuning. The blocker is that the 'explainable' claim is unsupported by the evaluation design. If the authors add a human study on model-generated explanations or a faithfulness/consistency metric that conditions on both the reasoning chains and the predicted label, and re-frame the explanation results accordingly, the paper could become acceptable. I would not recommend rejection, because the flaw is fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The prediction part of this paper is solid; the explanation part doesn't support the title claim. The new ETR benchmark is genuinely useful: five TKG-derived datasets across time granularities, with positive/negative/neutral queries and reasoning-chain text. And GETER's F1 improvements over LoRA tuning are consistent — roughly 7–16% across datasets with Mistral, and the ablation shows the soft graph token is responsible. I'd trust those numbers, modulo the lack of error bars.\n\nThe problem is the \"explainable\" claim. The gold explanations are generated by GPT-4o from a template that already asserts the label, and then the same GPT-4o texts are used both as training targets and as references for BLEU/ROUGE/METEOR/BERTScore. So Table 3 measures how well a 7B model imitates GPT-4o's phrasing, not whether the reasoning is faithful or even consistent with the predicted label. A model could output the wrong label and still score high on BLEU if the wording is similar. The human evaluation in Appendix A.5 only scores the gold explanations, not model outputs. The stress-test note is right: the central contribution is not supported by the current evaluation.\n\nOther soft spots are addressable. No error bars anywhere, so we don't know if the 1–5% BLEU differences are meaningful. It's unclear whether the frozen RE-GCN encoder was trained with a temporal split that prevents test-time leakage. And since the adapter is directly borrowed from G-Retriever / Filter-then-Generate, the paper should compare against those adapter baselines, not just LoRA and zero-shot.\n\nWho should read this: anyone building structure-text adapters for KGs or benchmarking LLM temporal reasoning. The benchmark and the prediction results are worth a serious referee. The explanation evaluation needs to be reframed — style matching, not reasoning fidelity — or supplemented with human judgment on model outputs and a faithfulness check that conditions on label correctness.\n\nVerdict: major revision, not reject. If the authors fix the evaluation and the split details, this becomes a solid contribution.","headline":"Solid prediction gains and a new benchmark, but the explanation evaluation is circular, so the explainable-reasoning claim needs a different yardstick.","tokens_in":30055,"tokens_out":2660,"would_cite":true,"duration_ms":23263,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that giving an instruction-tuned LLM a compact token derived from temporal knowledge graph structure boosts both its prediction accuracy and the quality of its explanations in explainable temporal reasoning.","keywords":["explainable temporal reasoning","temporal knowledge graph","large language models","structure-text adapter","soft graph token","instruction tuning","ETR benchmark","LoRA"],"falsifier":"Have human annotators write explanations for a random sample of ETR queries, then score GETER and text-only LoRA against those human texts; if GETER's explanation advantage shrinks or reverses, the reported gains come from imitating GPT-4o style rather than from deeper temporal understanding.","tokens_in":29023,"feed_emoji":"🕒","tokens_out":6408,"duration_ms":47054,"temperature":0.7,"pith_summary":"The paper tries to establish that large language models do explainable temporal reasoning better when they receive a structural token from a temporal knowledge graph in addition to plain text. The authors build the ETR benchmark, five datasets spanning minutes, days, and years, each with a query, a textualized reasoning chain, and a GPT-4o-written explanation. They then show that GETER, an instruction-tuned LLM with a soft graph token injected through a lightweight adapter, outperforms text-only LoRA tuning by 7 to 11 percent in F1 and produces explanations that score closer to the gold texts. If this is right, graph structure and text are complementary inputs for faithful temporal reasoning, and the gap can be bridged without retraining the LLM.","feed_headline":"Soft graph token lifts LLM temporal reasoning F1 up to 11%","feed_subtitle":"Structure-text adapter also beats GPT-4o zero-shot on explanation text by up to 251% BLEU-4.","key_machinery":"The load-bearing object is the soft graph token produced by GETER. A frozen temporal encoder (RE-GCN) embeds entities and relations; the query and its reasoning chains are pooled by parameter-free message passing over concatenated subject-relation-object embeddings; a trainable projection matrix $W_p \\in \\mathbb{R}^{3d_s \\times d_x}$ maps the pooled vector into the LLM embedding space; and the resulting token is prepended to the instruction-prompt tokens. LoRA adapts the LLM, which generates the target explanation text by maximizing the likelihood of the gold explanation given the graph token and text.","core_discovery":"The central claim is that structural information encoded in a temporal knowledge graph, when mapped into an LLM embedding space as a soft graph token, improves both the correctness and the explainability of temporal predictions. On the ETR benchmark, GETER with Mistral raises overall F1 to 79.08 on ICEWS14, 72.02 on GDELT, and 81.80 on ICEWS05-15, beating LoRA-only tuning by 11.10 percent, 10.71 percent, and 7.54 percent respectively, and beats GPT-4o zero-shot BLEU-4 by 75 to 251 percent. The explanation text is generated by the LLM from the soft graph token plus instruction prompt, and the paper argues that this lets the model capture evolving relation patterns that text-only models miss.","pith_inferences":["If explanation quality is measured by agreement with GPT-4o texts, the 75 to 251 percent BLEU gains may partly reflect stylistic mimicry rather than reasoning fidelity; a human-judged faithfulness test would separate the two.","A direct testable extension is to swap the trained RE-GCN embeddings for random or untrained embeddings while keeping the adapter; if the F1 gain persists, the benefit comes from extra trainable parameters or token position rather than from graph structure.","The same soft-token recipe could transfer to other structured reasoning tasks, such as multi-hop question answering over knowledge graphs or event forecasting, whenever the input has a graph plus a textual rendering.","The neutral-sample construction depends on an NLI threshold of 0.7; varying that threshold could change the difficulty of the Unsure split, so reported neutral F1 is partly a property of the benchmark construction."],"forward_implications":["GETER with Mistral raises overall F1 over LoRA-only tuning by 11.10 percent on ICEWS14, 10.71 percent on GDELT, and 7.54 percent on ICEWS05-15.","Explanation outputs beat GPT-4o zero-shot on BLEU-4 by 75.28 percent, 251.31 percent, and 99.07 percent on the same three datasets.","Removing the structure-text adapter drops overall F1 by roughly 7 to 12 points across datasets, and removing reasoning chains text drops it further, so both structural and textual inputs are load-bearing.","GETER stays competitive when paired with CEN, CENET, or SiMFy instead of RE-GCN, so the framework does not depend on a single temporal encoder.","Randomly ordering the reasoning-chain text still yields competitive F1 (77.57 versus 79.08 for the best ordering on ICEWS14), suggesting the adapter couples structure and text robustly."],"supporting_citations":[{"why":"Supplies the frozen RE-GCN temporal encoder whose entity and relation embeddings are the input structure for the soft graph token.","marker":"(Li et al., 2021)"},{"why":"LoRA is the parameter-efficient tuning method used to adapt the LLM in GETER and in the Tuned-only baseline.","marker":"(Hu et al., 2022)"},{"why":"Cited as evidence that a straightforward linear projection effectively aligns graph-based and textual representations.","marker":"(He et al., 2024)"},{"why":"GPT-4o generates and revises the gold explanation texts for the ETR benchmark.","marker":"(OpenAI, 2023)"},{"why":"Provides the ICEWS14 and ICEWS05-15 temporal knowledge graphs used to build benchmark instances.","marker":"(García-Durán et al., 2018)"},{"why":"Provides the GDELT dataset at 15-minute granularity.","marker":"(Liao et al., 2024)"},{"why":"Provides the WIKI dataset at yearly granularity.","marker":"(Leblay and Chekol, 2018)"},{"why":"Defines the extrapolation train/test split that ETR follows.","marker":"(Jin et al., 2020)"},{"why":"BLEU-4 is one of the explanation-quality metrics used to compare generated explanations against gold texts.","marker":"(Papineni et al., 2002)"},{"why":"BERTScore is used to measure semantic similarity of generated explanations to the gold texts.","marker":"(Zhang et al., 2020)"}],"fun_headline_variants":["Graph token boosts LLM temporal reasoning F1 by 11%","Explainable temporal reasoning: graph-aware LLM wins","GETER: LLMs reason over time with graph structure","Soft graph token improves LLM temporal explanations","LLMs + temporal knowledge graphs: 11% F1 gain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything about explanation quality rests on treating GPT-4o's revised explanations, written from prompts that already state the correct label, as ground truth; if those texts are not faithful reasoning, the BLEU, ROUGE, METEOR, and BERTScore improvements do not mean better reasoning.","fun_headline_variants_meta":{"raw":{"variants":["Graph token boosts LLM temporal reasoning F1 by 11%","Explainable temporal reasoning: graph-aware LLM wins","GETER: LLMs reason over time with graph structure","Soft graph token improves LLM temporal explanations","LLMs + temporal knowledge graphs: 11% F1 gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000227,"raw_usage":{"total_tokens":1455,"prompt_tokens":912,"completion_tokens":543,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":462}},"tokens_in":528,"tokens_out":543,"duration_ms":4667,"temperature":1.0,"reasoning_tokens":462,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:20:59.447754+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators write explanations for a random sample of ETR queries, then score GETER and text-only LoRA against those human texts; if GETER's explanation advantage shrinks or reverses, the reported gains come from imitating GPT-4o style rather than from deeper temporal understanding.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the GDELT dataset at 15-minute granularity."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the extrapolation train/test split that ETR follows."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BLEU-4 is one of the explanation-quality metrics used to compare generated explanations against gold texts."},{"cited_title":"Weinberger, and Yoav Artzi","cited_arxiv_id":null,"evidence_quote":"BERTScore is used to measure semantic similarity of generated explanations to the gold texts."}],"review_version":1}