{"id":"f2051175-edc6-4f5a-8fac-8f50e6c7afb4","arxiv_id":"2506.23141","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SARMP improves knowledge graph link prediction by selecting the Top-K semantically relevant edges per node and aggregating them with multi-head attention, beating some baselines on FB15k-237 and Kinship but not all benchmarks.","lead":"A new knowledge graph completion model uses Top-K edge selection and attention to pass only the most relevant relational messages between entities. It reports gains on two standard benchmarks, but the claimed state-of-the-art status is contradicted by its own table on the other two.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Forward pass in Eqs. (5)-(7) is circular: h^(l+1) is the mean of s^(l+1), while s^(l+1) depends on h^(l+1); no acyclic computation order is specified.","rationale":"The reader's verdict centers on evaluation protocol; that is a reasonable concern but not the most load-bearing. The strongest claim is empirical SOTA, but for that claim to even be meaningful, there must be a well-defined model. The equations in §3.2 create a circular dependency or at least an ambiguity that prevents replication. This is more fundamental: fixing baselines would not help if the architecture is unspecified. I recommend UNVERDICTED, because the paper can become verifiable after a clarification of the forward pass, but as submitted it is not. Credit where due: the high-level idea (Top-K relation-aware selection) is plausible, and the ablation results are consistent with the narrative, but the formal description needs repair.","tokens_in":10642,"tokens_out":8424,"duration_ms":86197,"concrete_test":"Request or reconstruct the layer-update algorithm. Concretely: (1) ask the authors for the forward-pass pseudocode or code; (2) check whether the dependency graph between h^(l+1), s^(l+1), m^(l+1) has a cycle. If a cycle exists, test both plausible orderings—(a) Top-K edge enrichment → node mean pooling → edge message → final edge state; (b) edge message → final edge state → node pooling—and see which one yields the reported Table 1 numbers. If neither ordering reproduces the results, the central claim is unsupported as written.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim is that SARMP achieves state-of-the-art results, but the method as written cannot be executed. In Section 3.2, Eq. (5) sets h_v^(l+1) = Mean({s_e_v^(l+1)}) over incident edges. Eq. (7) sets s_e^(l+1) = σ(Linear([s_e^(l), m_e^(l+1)])) where m_e^(l+1) = MLP([h_u^(l+1), h_v^(l+1)]) (Eq. 6). Thus h^(l+1) depends on s^(l+1) and s^(l+1) depends on h^(l+1). The text first describes a Top-K enrichment that produces s^(l+1), then an alternating update that also produces s^(l+1); it is unclear whether these are the same variable. No pseudocode or code is given. If the enrichment output is temporary, the equations reuse s^(l+1) ambiguously; if it is final, the system is cyclic. This is not a cosmetic issue: the reported MRR numbers in Table 1 cannot be reproduced or checked without knowing the actual layer ordering. The reader's concern about baseline fairness is valid, but even with perfectly reproduced baselines, the paper does not define a computable model.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SARMP, a knowledge graph completion model that performs semantic-aware Top-K edge selection and fuses selected edge states with a multi-head attention aggregator, iterating node and edge updates in an alternating relational message passing scheme. The model is evaluated on FB15k-237, WN18RR, Kinship, and UMLS, reporting MRR and Hits@1 under the filtered setting. The authors claim state-of-the-art performance across all datasets and analyze hyperparameter sensitivity, ablations, and parameter count.","tokens_in":10878,"tokens_out":4456,"duration_ms":41207,"significance":"If the method is made precise and the empirical claims are corrected, the core idea—restricting message passing to semantically relevant Top-K edges—is a reasonable response to noise and over-smoothing in KG completion, and the reported gains over FDM on FB15k-237 and Kinship are potentially useful. However, the manuscript currently does not substantiate the headline claim: its own Table 1 places SARMP second on two of four datasets. The method definition in Eqs. (5)-(7) is ambiguous to the point of appearing circular, and the absence of code, error bars, and explicit validation protocol limits reproducibility. The ablations in Table 3 and the sensitivity analysis in Figure 2 are useful and support the role of the proposed components.","major_comments":[{"comment":"The claim 'state-of-the-art performance across all datasets' is contradicted by Table 1: on WN18RR, SARMP's MRR is 0.535 versus 0.551 for NBFNet, and on UMLS, SARMP's MRR is 0.944 versus 0.964 for RED-GNN. Section 4.2 itself acknowledges second-best results on these datasets while still claiming 'remarkable improvement across all metrics on all four datasets.' The abstract and contributions must be revised to state accurately where SARMP is best and where it trails, and the discussion should interpret the gaps (e.g., whether they are within run-to-run variance).","section":"Abstract and Section 1, contributions bullet 4"},{"comment":"The forward computation is not well-defined. Eq. (5) sets h_v^(l+1) = Mean({s_e_v^(l+1)}) over incident edges; Eq. (6) constructs m_e^(l+1) from h_u^(l+1) and h_v^(l+1); Eq. (7) then defines s_e^(l+1) = σ(Linear([s_e^(l), m_e^(l+1)])). If s_e^(l+1) is the same variable in Eqs. (5) and (7), then h^(l+1) depends on s^(l+1) and s^(l+1) depends on h^(l+1), with no acyclic order. If the Top-K enrichment output is meant to be a temporary edge state distinct from the message-passing update, the notation reuses s^(l+1) ambiguously. Please rename the two edge-state variables, state the layer ordering explicitly, and provide pseudocode or a reference implementation; without this, the Table 1 numbers cannot be independently checked.","section":"Section 3.2, Eqs. (5)-(7)"},{"comment":"The evaluation protocol is underspecified. The paper does not say whether Top-K, τ, γ, number of hops, and hidden dimensions were selected on a validation split or tuned on test, and no error bars or repeated runs are reported. Several baseline cells are missing (PathCon on Kinship and UMLS; RED-GNN on Kinship), so the comparison is incomplete. Please specify the data split, hyperparameter selection procedure, number of seeds, and report standard deviations (or confidence intervals); complete or justify the missing baseline entries.","section":"Section 4.1 and Table 1"},{"comment":"The parametric analysis claims SARMP has complexity O(n·K^hops) and 0.34M parameters on FB15k-237, but the meaning of n and K is not defined, and the derivation of the parameter count is not shown. Since RQ3 is one of the stated research questions, include the formula for the parameter count and define all symbols (and state whether K refers to Top-K or the sampled neighborhood size).","section":"Section 4.4, Table 4"}],"minor_comments":[{"comment":"The text says ∥·∥_2 denotes the squared Euclidean distance, but the notation ∥·∥_2 usually denotes the norm itself; please use a distinct symbol (e.g., d(·,·)) or write ∥·∥_2^2 explicitly.","section":"Section 3.2, Eq. (3)"},{"comment":"The Bayes decomposition is not explicitly connected to the model components; please either state the correspondence (which term is modeled by the Top-K selection, which by the message passing, and how) or remove the decomposition as non-load-bearing.","section":"Section 3.1, Eqs. (1)-(2)"},{"comment":"The subplots lack clear axis labels and legends; add them so the sensitivity claims can be verified by the reader.","section":"Figure 2"},{"comment":"The row grouping 'Head Pred' / 'Tail Pred' is not formatted consistently; ensure the table header clarifies that these are MRR values for head and tail prediction, and align the column widths.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an arXiv submission without released code, which heightens reproducibility concerns given the ambiguity in Eqs. (5)-(7). The novelty is moderate and the empirical claim is overstated; I would not reject on novelty grounds, but the authors must correct the SOTA claim, clarify the algorithm, and provide a complete evaluation protocol before the paper can be considered further."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is an incremental KGC paper with a real but modest idea, and the writing has a load-bearing ambiguity in the update equations that needs to be fixed before the numbers mean anything. The stress-test note is correct, not a nitpick.\n\nWhat's actually new: the specific combination of a hard Top-K semantic edge selection based on a learnable RBF similarity, a multi-head attention aggregator, and alternating relational message passing in the style of PathCon. That combination is not in the cited RED-GNN, PathCon, or CompGCN. The ablation study is useful: removing Top-K or replacing the scoring function costs 1–3 MRR points on FB15k-237, and the hyperparameter sensitivity plots suggest the authors actually looked at what they changed. The parametric analysis comparing their 0.34M parameters against PathCon's 1.67M is also a point in their favor.\n\nWhere it falls down: first, the equations in Section 3.2 do not define a computable model as written. Eq. (5) sets h^(l+1) to the mean of enriched edge states s^(l+1), then Eqs. (6)–(7) construct messages from h^(l+1) and define s^(l+1) again from those messages. Two different s^(l+1) assignments in the same layer, no ordering, no pseudocode. Maybe the enrichment is intended as a temporary representation and Eq. (7) is the real edge update, but the paper doesn't say that. That is a serious reproducibility failure, not a cosmetic one.\n\nSecond, the headline claim of state-of-the-art across all datasets is contradicted by the paper's own Table 1: SARMP is second-best on WN18RR (0.535 vs 0.551) and UMLS (0.944 vs 0.964). The abstract and contribution list repeat the overclaim even though Section 4.2 quietly admits second-best. Baselines are missing for PathCon and RED-GNN on Kinship and UMLS, no error bars, no code, and no statement about whether hyperparameters were tuned on validation. These are standard expectations for a KGC submission, and their absence makes the empirical edge hard to trust. The Bayes decomposition in Section 3.1 is never used; it reads as decoration.\n\nNet: the core idea is plausible and the ablation supports it, but the paper needs a corrected claim, a clear computation order (or code), and a fairer baseline table. For a reader in KGC it's worth a look as a working-paper version; the ambiguity makes it a useful object lesson in why equations need to be executable. I'd send it to peer review because the idea is worth referee time, but I'd expect major revisions.","headline":"Plausible incremental KGC idea, but the update equations as written are ambiguous/circular and the empirical claim overreaches; worth referee time only if the authors fix the computation order and the baseline table.","tokens_in":11476,"tokens_out":3859,"would_cite":false,"duration_ms":39029,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Selecting only the Top-K most semantically relevant edges for message passing improves knowledge graph link prediction while using far fewer parameters than embedding-based models.","keywords":["knowledge graph completion","relational message passing","semantic-aware neighbor selection","Top-K selection","multi-head attention","link prediction","graph neural network"],"falsifier":"Rerun the Table 1 comparison on identical filtered splits with several random seeds and report mean and standard deviation; the superiority claim fails if the FB15k-237 and Kinship gains over the strongest embedding baseline vanish under seed variance, or if the two leading GNN baselines remain ahead on WN18RR and UMLS by margins beyond run-to-run noise.","tokens_in":10399,"feed_emoji":"🔗","tokens_out":9409,"duration_ms":95052,"temperature":0.7,"pith_summary":"This paper tries to establish that knowledge graph completion improves when a node's contextual message is built from only the Top-K most semantically relevant incident edges, rather than from all neighboring edges. The authors propose a semantic-aware relational message passing framework that scores edge relevance in a shared latent space, selects the Top-K edges, and fuses them with a multi-head attention aggregator. They report that the resulting model, SARMP, has the best MRR on FB15k-237 and Kinship and is close behind the best baseline on WN18RR and UMLS, all with about 0.34 million parameters. The practical interest is that a simple hard filtering step may mitigate the noise, over-smoothing, and information dilution that come with indiscriminate neighbor aggregation.","feed_headline":"Top-K semantic edges lift knowledge graph link prediction","feed_subtitle":"Restricting message passing to the K most relevant edges cuts noise and matches larger models with far fewer parameters.","key_machinery":"The load-bearing mechanism is the semantic-aware Top-K edge selection strategy inside an alternating relational message passing loop. The selection scores each candidate edge by negative squared Euclidean distance in a learned embedding space, keeps only the K most relevant edges, and therefore makes low-relevance edges unable to contribute at all, unlike soft-attention methods where every edge keeps a nonzero weight. The second component is a multi-head attention aggregator that fuses the selected edge states with the central edge's own state; the paper's ablation shows that replacing it with mean pooling degrades performance. This pair of components converts relational message passing from a noise-spreading process into a context-filtering one.","core_discovery":"The paper claims that semantic relevance, not topology alone, should decide which edges contribute to a node's message. For a central edge $e_v$, each neighboring edge $e_n$ receives a score $\\text{Score}(e_v,e_n)=\\exp(-\\|f(s^{(l)}_{e_v})-f(s^{(l)}_{e_n})\\|^2/\\tau)$ in a shared latent space, only the Top-K scores survive, the surviving states are mean-pooled, and a multi-head attention aggregator combines them with the central edge's own state. A node's updated representation is the mean of its enriched incident edge states, after which edge messages are rebuilt from endpoint node representations, alternating node and edge updates. On the paper's reported evidence, this gives best or near-best link prediction on four benchmarks while using no entity embeddings and a fraction of the parameter count of embedding-based rivals.","pith_inferences":["Because the node update drops the previous node state and defines the representation purely from incident edge states, the architecture is a natural candidate for inductive completion on entities unseen at training time, though the paper reports no such experiment.","A testable extension follows from the noise story: on denser knowledge graphs with larger average degree, the gap between Top-K selection and full-neighborhood aggregation should widen, because the number of irrelevant neighboring edges grows.","The fixed moderate K (10 in the paper) is tuned on one benchmark; if K must scale with degree on larger graphs, the mechanism's practical value would depend on a robust K-selection rule rather than a universal hyperparameter."],"forward_implications":["If the comparison is taken at face value, knowledge graph completion does not require entity embeddings or full-neighborhood aggregation: relation-centric Top-K message passing matches or beats embedding and GNN baselines with 0.34 million parameters.","The ablations indicate that Top-K selection is the main driver: replacing it with random sampling drops FB15k-237 MRR from 0.492 to 0.473 and WN18RR MRR from 0.535 to 0.509.","Performance peaks at Top-K=10 and two hops on FB15k-237, and degrades with larger K or deeper hops, which is consistent with the paper's noise-reduction explanation.","On WN18RR and UMLS the paper reports second-best MRR, not best, so the abstract's 'superior performance' holds against most baselines but not against the leading GNN models on those two benchmarks."],"supporting_citations":[{"why":"Supplies the relational message passing backbone that SARMP enhances with Top-K selection and a multi-head attention aggregator.","marker":"[18]"},{"why":"Provides the soft-attention baseline and the motivation: low-weight edges can still inject noise, which SARMP's hard Top-K filter is designed to remove.","marker":"[14]"},{"why":"Is the strongest GNN baseline on WN18RR, the benchmark where SARMP finishes second, and therefore a key comparison for the superiority claim.","marker":"[47]"},{"why":"Is the strongest embedding-based baseline on FB15k-237 that SARMP overtakes, making it the central non-GNN rival in the headline result.","marker":"[22]"},{"why":"Supplies the filtered evaluation protocol used for MRR and Hits@N and is one of the embedding baselines in the comparison.","marker":"[12]"}],"fun_headline_variants":["Semantic Top-K edges beat full-graph message passing","Choosing K relevant edges cuts noise, lifts link prediction","Edge selection by semantic relevance outperforms dense propagation","Top-K semantic neighbors: fewer params, better KG completion","Attention on relevant edges refines knowledge graph embeddings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes the benchmark comparison is fair: baseline numbers are correctly reproduced, hyperparameters such as Top-K, temperature, margin, and hop count were chosen on validation rather than the test set, and the reported gains exceed run-to-run noise.","fun_headline_variants_meta":{"raw":{"variants":["Semantic Top-K edges beat full-graph message passing","Choosing K relevant edges cuts noise, lifts link prediction","Edge selection by semantic relevance outperforms dense propagation","Top-K semantic neighbors: fewer params, better KG completion","Attention on relevant edges refines knowledge graph embeddings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000159,"raw_usage":{"total_tokens":1218,"prompt_tokens":921,"completion_tokens":297,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":221}},"tokens_in":537,"tokens_out":297,"duration_ms":3648,"temperature":1.0,"reasoning_tokens":221,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:47:32.087229+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the Table 1 comparison on identical filtered splits with several random seeds and report mean and standard deviation; the superiority claim fails if the FB15k-237 and Kinship gains over the strongest embedding baseline vanish under seed variance, or if the two leading GNN baselines remain ahead on WN18RR and UMLS by margins beyond run-to-run noise.","supporting_citations":[],"review_version":1}