{"id":"8bccba76-d424-4a13-93ad-87b2962b2068","arxiv_id":"2411.11046","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Injecting hand-built knowledge graph embeddings into transformer forecasters yields mixed results on Weather and ETT benchmarks, undercutting the paper's claim of consistent improvement.","lead":"This paper adds learnable knowledge graph embeddings, built from hand-crafted relationships between variables, to four transformer forecasting models and tests them on Weather and ETT datasets. The reported gains are large on some settings but negative on others, and key details of the graph construction are left to a missing appendix.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own Table III contradicts the central claim of consistent improvement: KGE substantially degrades ETTm2 and ETTh1 results for Transformer and Autoformer, so the headline result is not supported.","rationale":"The reader's verdict is REJECT, and my stress-test concurs: the paper's own tables undermine the central claim. However, the reader's weakest_assumption focused on the correctness of the hand-built adjacency matrix, whereas I identify the more decisive issue as the internal contradiction between the claimed universal improvement and the reported degradations in Table III. The graph-construction concern is real and relevant—the missing appendix prevents validating the knowledge graph—but it is secondary to the fact that even if the graph were perfect, the reported results show the method hurts several ETT settings. Given that the empirical evidence directly contradicts the strongest claim, the REJECT verdict stands without adjustment. The proposed sign test is a minimal, decisive check: it converts the qualitative overclaim into a countable comparison across all reported configurations. This respects the paper's own data and avoids relying on external baselines or unavailable code.","tokens_in":12936,"tokens_out":2195,"duration_ms":23521,"concrete_test":"Perform a paired sign test over every model-dataset-horizon row in Tables III and IV: count how many rows have KGE MSE lower than original MSE. If the proportion of wins is not significantly above 0.5, the phrase 'consistently surpasses benchmark results' is falsified. As a second check, rerun the ETTm2 Transformer configuration (horizon 96) with at least 5 random seeds and report mean and standard deviation; if the KGE version remains near MSE 1.116 versus 0.515 for the baseline, the degradation is robust and the abstract's claim of accuracy improvement fails for a substantial subset of the evaluated benchmarks.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV.D states 'our methodology consistently surpasses benchmark results,' and the abstract claims improved accuracy across multiple domains. Table III directly contradicts this. For ETTm2, Transformer-with-KGE at horizon 96 gives MSE 1.116 vs. 0.515 for the original Transformer, a more than twofold degradation; large degradations also appear at horizons 192 and 336. Autoformer-with-KGE on ETTh1 degrades at horizons 192 (0.675 vs. 0.582) and 336 (0.691 vs. 0.550), while Informer-with-KGE on ETTm1 at horizon 720 degrades from 1.031 to 1.461. The paper itself concedes 'sub-optimal results' for ETTm2 and ETTh1, attributing them to redundant parameters, but this concession is incompatible with the claimed universal superiority. The condition required for the central claim to hold—that adding KGE reliably improves forecasting accuracy—is thus false within the paper's own reported evidence. Because the appendix describing the knowledge graph and experimental configurations is missing, and no code or error bars are provided, the favorable Weather results cannot be independently checked against the concurrent degradations. The central claim as stated is therefore internally inconsistent, not merely outside consensus.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method for multivariate long-sequence time-series forecasting (LSTF) that augments transformer-based architectures (Transformer, Autoformer, Informer, PatchTST) with learnable Knowledge Graph Embeddings (KGE). The embeddings are constructed from a hand-built binary adjacency matrix that encodes conceptual relationships among variables, multiplied by learnable weight matrices, and added to the standard input embeddings. The method is evaluated on the Weather dataset and four ETT datasets (ETTm1, ETTm2, ETTh1, ETTh2) at horizons 96, 192, 336, and 720, with the central claim that adding KGE consistently improves forecasting accuracy across datasets and horizons.","tokens_in":13200,"tokens_out":4438,"duration_ms":40849,"significance":"The idea of injecting explicit variable-level relational knowledge into transformer time-series models is timely and potentially useful, and the results on the Weather dataset are encouraging. However, the paper's own tables contradict the central claim of consistent improvement, the appendix containing the knowledge-graph definitions and experimental configurations is missing, and the core KGE construction is under-specified. As presented, the contribution cannot be validated or reproduced, and the evidence does not support the headline claims.","major_comments":[{"comment":"The central claim that 'our methodology consistently surpasses benchmark results' is directly contradicted by the paper's own Table III. For ETTm2, Transformer with KGE has MSE 1.116 versus 0.515 without at horizon 96, 1.346 versus 0.764 at horizon 192, and 1.573 versus 1.117 at horizon 336, a more than twofold degradation in the first two cases. Autoformer with KGE on ETTh1 degrades at horizons 192 (0.675 versus 0.582) and 336 (0.691 versus 0.550). Informer with KGE on ETTm1 degrades at horizon 720 (1.461 versus 1.031). The text in Section IV.D acknowledges 'sub-optimal results' on these datasets, but that admission is inconsistent with the unqualified claims of consistent improvement in the abstract, introduction, and Section IV.D. The authors should either restrict the claims to datasets where KGE helps or provide an analysis of the conditions under which KGE does and does not help.","section":"Section IV.D, Table III"},{"comment":"The knowledge-graph construction is the central modeling choice, yet Section IV.B states that the relational frameworks 'are elaborated in the Appendix' and Section IV.C says that baseline configuration details are 'provided in the Appendix.' The submitted manuscript contains no appendix. Without the adjacency matrices for each dataset and the experimental configuration, the method is not reproducible and the reader cannot assess whether the favorable Weather results arise from the proposed mechanism or from ad-hoc graph choices. This is a load-bearing omission because the method's success depends entirely on the quality and validity of the hand-constructed graph, as the paper itself notes when discussing the failure on ETT datasets.","section":"Section IV.B and IV.C, missing Appendix"},{"comment":"The construction of WKGE is under-specified. The text says that the adjacency matrix A is 'multiplied by a learnable weight matrix Wl, followed by an Einstein summation with a learnable projection weight matrix Wp,' but no equation defines how the final WKGE is computed from A, Wl, and Wp. Given Wl ∈ R^{V×D} and Wp ∈ R^{L×D}, it is unclear how the Einstein summation yields a tensor of the shape needed to be added to X in Equation (2), or how this tensor varies across the M variables and L time steps. Since KGE is the paper's sole contribution, this mathematical ambiguity is a major technical gap that prevents replication.","section":"Section III.B, Equations (1)-(2)"},{"comment":"The text claims that 'the integration of KGE with PatchTST for ETT datasets' gives negligible improvement, but Table IV only reports PatchTST results on the Weather dataset. No PatchTST results on any ETT dataset are provided, so this claim is unverifiable. Given that PatchTST is one of the four architectures highlighted in the abstract, the absence of its performance on the ETT benchmarks weakens the paper's experimental completeness.","section":"Section IV.D, Table IV"}],"minor_comments":[{"comment":"There is a grammatical error: 'We observe a negligible improvement in the integration of KGE with PatchTST for ETT datasets' should be 'We observe negligible improvement from integrating KGE with PatchTST on ETT datasets.'","section":"Section IV.D"},{"comment":"The manuscript inconsistently uses 'Timeseries' and 'Time-series' (e.g., the title versus the abstract) and occasionally uses 'LTSF' instead of 'LSTF'; please standardize.","section":"Throughout"},{"comment":"No standard deviations, confidence intervals, or multiple random seeds are reported, so statements such as 'consistently surpasses' cannot be evaluated for statistical reliability.","section":"Section IV.D"},{"comment":"In Figure 2, the point at which the knowledge graph embedding WKGE is injected is not clearly marked; a clearer diagram would help the reader follow the proposed integration.","section":"Figure 2"}],"recommendation":"reject","confidential_remarks":"The missing appendix is a serious reproducibility concern, and the internal contradiction between the abstract's universal claim and Table III's degradations would need a substantial rewrite. The idea is not without merit, but the current manuscript does not meet the bar for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Core idea: build a binary adjacency matrix from hand-coded relations between variates, multiply it by learnable matrices, add the resulting embedding to transformer inputs. This specific form of static conceptual KG plus learnable projection is not something I've seen directly, and the paper does a clean within-model ablation of with vs. without KGE across four horizons on Weather and ETT. The Weather gains, especially for Informer, are large, and the integration is computationally cheap. The literature review is adequate and the method is simple enough to be reproducible in principle.\n\nThe problem is the central claim. Section IV.D says the methodology 'consistently surpasses benchmark results,' and the abstract promises improved accuracy across multiple domains. Table III directly contradicts that. Transformer with KGE on ETTm2 at horizon 96 gives MSE 1.116 versus 0.515 without, more than a two-fold degradation; Autoformer with KGE degrades on ETTh1 at horizons 192 and 336; Informer with KGE degrades on ETTm1 at horizon 720. The paper concedes 'sub-optimal results' for ETTm2 and ETTh1, but that concession is incompatible with the abstract's sweeping claim. This is not a minor blemish; it is the difference between the method helping and hurting on half the ETT datasets.\n\nAlso missing: the appendix describing the knowledge graph construction and experimental configurations is referenced but not present in the preprint. PatchTST results for ETT datasets are absent. There are no error bars, no seeds, and no code. That means the favorable Weather numbers cannot be independently checked, and the large Informer improvement is hard to trust without knowing the baseline configuration details. The hand-built graph itself is a legitimate modeling choice, but there is no sensitivity analysis—no test of whether a random graph or a fully learned adjacency would give similar results, which matters given the observed degradations.\n\nProportionate bottom line: the idea might be useful in narrow settings where strong domain knowledge exists, but as submitted the empirical case is internally inconsistent and the missing material prevents verification. The paper deserves attention as a cautionary example of overclaiming from a partial ablation, but not as a reliable forecasting contribution in its current form.\n\nRecommendation: desk reject with an invitation to resubmit after correcting the claims, adding the appendix and ETT PatchTST results, and providing code or at least error bars. A serious referee would spend their time explaining the contradiction the authors already concede.","headline":"A simple hand-built graph embedding injection idea with clean ablations, but the paper's own Table III contradicts the headline claim and the missing appendix blocks verification.","tokens_in":13696,"tokens_out":2113,"would_cite":false,"duration_ms":22495,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that injecting learnable knowledge-graph embeddings, derived from a hand-built graph of variable relationships, into transformer baselines improves multivariate long-sequence forecasting accuracy, with the largest…","keywords":["multivariate time-series forecasting","long sequence time-series forecasting","knowledge graph embeddings","transformer","attention mechanism","spatio-temporal dependencies","ETT dataset","Weather dataset"],"falsifier":"Permute the rows and columns of the adjacency matrix $A$ before training so the graph no longer matches the variables, while keeping the degree distribution unchanged; if forecast MSE does not worsen materially on the Weather dataset, then the specific relational content of the graph is not what drives the reported gains, and the improvement would instead come from added parameters or from any fixed auxiliary input.","tokens_in":12754,"feed_emoji":"📈","tokens_out":4468,"duration_ms":38556,"temperature":0.7,"pith_summary":"This paper argues that transformer forecasters for multivariate long-horizon time series overlook the structural relationships between variables, and that a small, learnable knowledge-graph embedding can supply that missing structure. The authors build a binary graph whose edges encode domain knowledge—physical laws for weather variables and operational principles for electric transformer temperature—and turn it into a dynamic embedding that is added to the input of standard transformer architectures. On Weather and ETT benchmarks, the augmented models mostly beat their unaugmented baselines, with the largest reported gains on Weather (for Informer, an average 54.5% MSE reduction) and on ETTm1 (for Autoformer, 21.9%). The paper also reports that gains shrink or reverse on datasets where the graph is simple, such as ETTm2 and ETTh1.","feed_headline":"Variable-relationship graphs cut transformer forecasting error","feed_subtitle":"Adding learnable knowledge-graph embeddings to transformer baselines cuts long-horizon forecast error on weather data.","key_machinery":"The central object is the dynamic and learnable knowledge-graph embedding $W_{\\text{KGE}}$, built by multiplying a hand-built binary adjacency matrix $A \\in \\{0,1\\}^{|V|\\times|V|}$ by a learnable weight matrix $W_l$ and a projection matrix $W_p$, then adding the result to the input embeddings together with positional, value, and temporal embeddings before the multi-head attention blocks. The graph itself supplies the initial relational basis, while backpropagation updates $W_l$ and $W_p$, making the static graph dynamic and dataset-adaptive; this is what carries the claimed transfer of inter-variable structural knowledge into the transformer.","core_discovery":"The central claim is that adding a learnable knowledge-graph embedding to the input of a transformer gives the attention mechanism access to inter-variable structure it otherwise ignores, improving multivariate long-sequence forecasting accuracy without requiring graph neural networks or graph convolution layers. The knowledge-graph embedding is constructed from a hand-built binary adjacency matrix that encodes relationships among variables, then made dynamic by learnable weight matrices updated through backpropagation during training. The authors assert that their method consistently surpasses the original baselines across four transformer architectures—Transformer, Informer, Autoformer, and PatchTST—and that the improvements are most pronounced for longer forecast horizons and for datasets with well-established variable relationships.","pith_inferences":["A testable extension of the paper's logic is that the same learned-embedding injection should help any attention-based forecaster on datasets with strong inter-variable relations, while hurting on datasets where the graph is uninformative; the authors do not run this broader sweep.","The reported failures on ETTm2 and ETTh1 suggest that the binary graph may be too crude; a graded edge-weight version could preserve the benefits where the binary skeleton currently adds noise.","Because the appendix describing the exact graph construction is missing from the preprint, an immediate reproducibility check is to reconstruct the graphs from the cited physical laws and operational principles and compare results.","The paper positions itself against dynamic graph neural methods but does not directly benchmark against those models, so the practical claim is limited to beating the same transformer without KGE, not to outperforming graph neural baselines."],"forward_implications":["If the central claim holds, transformer forecasters can be improved with a small plug-in embedding instead of redesigned graph neural network layers.","The reported gains grow with forecast horizon, suggesting the relational embedding helps most where long-range prediction makes temporal patterns harder to isolate.","The method transfers across four different transformer variants, so it is presented as architecture-agnostic rather than tied to one attention mechanism.","The approach is only beneficial where variable relationships are well established; the authors note that on datasets with weak or simple structure, the extra parameters can be redundant and degrade performance.","Datasets without conceptual relationships among variables are deliberately excluded, which limits the method's scope to domains where such domain knowledge exists."],"supporting_citations":[{"why":"Supplies the transformer architecture, multi-head attention, and sinusoidal positional embeddings that KGE augments.","marker":"[17]"},{"why":"Provides the Informer baseline and the ETT dataset family used for evaluation.","marker":"[18]"},{"why":"Provides the Autoformer baseline with its decomposition and auto-correlation blocks that KGE is plugged into.","marker":"[19]"},{"why":"Provides the PatchTST baseline with channel-independent patching that KGE is compared against and enhanced.","marker":"[21]"},{"why":"Supplies the Weather dataset with its 21 meteorological variables whose inter-relations are encoded in the knowledge graph.","marker":"[22]"},{"why":"Supplies the temporal-embedding construction from timestamps that is combined with KGE in the input embedding.","marker":"[60]"}],"fun_headline_variants":["Knowledge graph boosts transformer forecasting","Adding variable links sharpens long-horizon forecasts","Learnable knowledge embeddings improve transformer time-series","Graph-aware transformers beat baselines on long forecasts","Modeling variable relationships trims forecast error"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hand-built binary graph of variable relationships—derived from physical laws for weather and operational principles for ETT—actually captures the inter-variable structure that matters for forecasting; if those edges are wrong or irrelevant, the learnable embeddings add noise instead of signal, and the paper itself observes degraded results on ETTm2 and ETTh1 where the graph may be too simple.","fun_headline_variants_meta":{"raw":{"variants":["Knowledge graph boosts transformer forecasting","Adding variable links sharpens long-horizon forecasts","Learnable knowledge embeddings improve transformer time-series","Graph-aware transformers beat baselines on long forecasts","Modeling variable relationships trims forecast error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"total_tokens":1237,"prompt_tokens":894,"completion_tokens":343,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":277}},"tokens_in":510,"tokens_out":343,"duration_ms":87698,"temperature":1.0,"reasoning_tokens":277,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:58:16.648767+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Permute the rows and columns of the adjacency matrix $A$ before training so the graph no longer matches the variables, while keeping the degree distribution unchanged; if forecast MSE does not worsen materially on the Weather dataset, then the specific relational content of the graph is not what drives the reported gains, and the improvement would instead come from added parameters or from any fixed auxiliary input.","supporting_citations":[{"cited_title":"Weather dataset: Max planck institute for biogeochemistry, germany,","cited_arxiv_id":null,"evidence_quote":"Supplies the Weather dataset with its 21 meteorological variables whose inter-relations are encoded in the knowledge graph."}],"review_version":1}