{"id":"e520724e-cef0-41f5-bb95-de9d2d4fa0dd","arxiv_id":"2505.12272","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Small MRR gains on WN18RR and FB15K-237 follow from adding iterative message filtering and top-k probabilistic bilinear scoring to KGC models, but the merged-variant synergy claim is not supported by the reported numbers.","lead":"The paper adds two modules to knowledge graph completion models: an iterative message-filtering step for graph neural networks and a top-k probabilistic bilinear scoring module for embedding models. A generalist might read it as a check on whether cheap add-ons can squeeze small accuracy gains from existing KGC models, though the paper's own tables contradict its synergy claim.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation 2 never defines the Distill operator or alpha_i, so every DIST/MERGE result and over-smoothing claim in Table 1 is unverifiable; the claimed 'significant' gains also lack error bars or significance tests.","rationale":"The reader's weakest assumption is the right one: the Distill operator is genuinely load-bearing because the GNN-distillation contribution and all DIST/MERGE results are defined through it. Without a specification, an independent reader cannot tell whether the gains come from distillation, from the extra parameters alpha, or from the 4-layer setup; this is an internal completeness failure, not a disagreement with consensus. The additional absence of error bars and significance tests reinforces the concern, because several reported deltas are at the noise level and Observation 2's synergy claim is contradicted on WN18RR. I therefore keep the reader's CONDITIONAL verdict: the paper is potentially sound but must (i) define Distill and the alpha computation, (ii) report per-seed statistics with a paired test, and (iii) correct or remove the synergy claim. The proposed concrete check isolates the Distill operator by comparing it with identity, directly testing whether the distillation branch has any measurable effect.","tokens_in":12400,"tokens_out":6469,"duration_ms":64406,"concrete_test":"Inspect the released code to recover the exact implementations of Distill and alpha_i, then re-run Table 1 for all DIST and MERGE rows with Distill replaced by the identity operator. If the identity version matches the reported DIST/MERGE numbers, the distillation branch has no measurable effect and the central over-smoothing claim is vacuous. If it does not match, use the code to write a complete specification of Distill into Section 3.1, then have an independent re-implementation reproduce Table 1; report per-seed metrics and paired confidence intervals for the claimed gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that enhanced models 'surpass their baseline counterparts' (Section 3.4, Observation 1). For the GNN distillation half, this claim cannot currently be checked: Section 3.1 defines distilled messages only as Distill(m_i; alpha_i) (Eq. 2) and gives a decay schedule for alpha (Eq. 4), but never specifies what Distill does or how alpha_i is computed from the message. All DIST and MERGE rows in Table 1 depend on this operator, yet the paper provides no pseudocode, no derivation, and no ablation that isolates distillation from the shared 4-layer architecture. The empirical evidence is also insufficient: Table 1 reports 'averaged over 5 runs' without standard deviations or significance tests, so differences such as Simkgc-APIM's +0.001 MRR on FB15K-237 or KB-GAT-APIM's +0.005 MRR cannot support the word 'significant.' Table 1 also contradicts Observation 2's synergy claim on WN18RR: KB-GAT-MERG MRR 0.466 is below KB-GAT-APIM 0.478 and KB-GAT-DIST 0.469; CompGCN-MERG 0.444 is below CompGCN-APIM 0.472 and CompGCN-DIST 0.466; RGCN-MERG 0.436 is below RGCN-DIST 0.455. Thus the merged variant does not exceed the additive effect of its components as claimed. These problems are fixable, but as written the central claim rests on an underspecified operator and unquantified comparisons.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes two modules for knowledge graph completion: GNN distillation, which filters messages inside each GNN layer to mitigate over-smoothing, and Abstract Probabilistic Interaction Modeling (APIM), which projects entity embeddings to sparse probabilistic signatures and scores triples as an expectation under relation-specific transition matrices. The methods are attached to KB-GAT, RGCN, CompGCN, and SimKGC, with APIM, distillation, and merged variants, and are evaluated on WN18RR and FB15K-237. The paper claims consistent significant gains over baselines and a synergistic effect for the merged variant.","tokens_in":12821,"tokens_out":7689,"duration_ms":72994,"significance":"If the empirical claims were fully supported, this would be a useful and model-agnostic contribution: the two modules are simple, they can be grafted onto both GNN-based and embedding-based KGC models, and the evaluation covers standard benchmarks with five-run averages, ablations on top-k and decay schedules, and an appendix analysis. The main weakness is not the idea but verifiability: the distillation operator is unspecified, the merged-variant synergy claim is contradicted by Table 1, and the word \"significant\" is used without error bars or significance tests. The central claim may be true, but the manuscript as written does not allow a reader to check it.","major_comments":[{"comment":"Section 3.1, Eqs. (2)–(4): the operator Distill is never defined. Equation (2) writes \\tilde m_i^{(l)} = Distill(m_i^{(l)}; \\alpha_i^{(l)}), and Eq. (4) gives a scalar schedule for \\alpha(k), but the paper does not state what Distill does to a message, how \\alpha_i^{(l,k)} is computed from the message or node, or which features are retained. Because every DIST and MERGE row in Table 1 and the over-smoothing explanation depend on this operator, the GNN-distillation half of the central claim is currently unreproducible. Please provide the exact filtering operation, such as top-percentile masking, thresholding on message norms, or soft shrinkage, together with the formula for the per-message importance.","section":"3.1, Eqs. (2)–(4)"},{"comment":"Table 1, Observation 2: the claimed synergistic performance of MERGE is not visible in the reported numbers. On WN18RR, KB-GAT-MERG (MRR 0.466) is below both KB-GAT-APIM (0.478) and KB-GAT-DIST (0.469); CompGCN-MERG (0.444) is below CompGCN-APIM (0.472) and CompGCN-DIST (0.466); and RGCN-MERG (0.436) is below RGCN-DIST (0.455). The merged variant is often not even the best among the three variants, so the statement that it exceeds the additive effects of its components is contradicted by the paper's own data. Please either correct the claim or identify the specific sense of synergy that the table supports.","section":"Table 1, Observation 2"},{"comment":"Table 1 and Observation 1: the paper says results are averaged over 5 runs and repeatedly calls gains significant, but no standard deviations, confidence intervals, or significance tests are reported. Differences such as Simkgc-APIM's +0.001 MRR on FB15K-237 and KB-GAT-APIM's +0.005 MRR cannot be assessed without variance information, so the claims of \"significant\" improvements and \"consistent and significant improvements across all evaluated architectures\" are not supported. Please add error bars or paired significance tests, or soften the statistical language.","section":"Table 1 and Observation 1"},{"comment":"Section 5, Key Findings: the discussion names \"linear decay from 1.0 to 0.4\" as the moderate filtering ratio that yields the best performance, but Section 4.3 and Figure 5 identify the 1.0-to-0.2 setting as the best (MRR 0.3588 on FB15K-237) and describe the 1.0-to-0.4 setting as underperforming. This is an internal contradiction in the paper's practical recommendation and should be corrected.","section":"Section 5 vs. Section 4.3"},{"comment":"Appendix D: the cumulative-energy analysis does not provide independent evidence for the top-20 choice. The signatures a_e are taken from a model already trained with the TopK mask, so observing E(20) ≥ 0.85 partly reflects the imposed sparsity rather than a property of the underlying interaction distribution. In addition, the claim of \"statistical significance\" is not accompanied by any test. Please present this analysis as descriptive or provide an external validation, for example by comparing energy retention against random mode subsets or other values of K.","section":"Appendix D"}],"minor_comments":[{"comment":"There are numerous typos and inconsistencies, including \"Emebdding-based\" in Section 2.3, \"gradially\" and \"intial\" in Section 3.1, \"an transforming\" in Section 2.2, \"Methodologys\" in Section 3, \"ComPGCN\" versus \"CompGCN\" in Sections 3.3 and 4, and \"the the\" in Section 1.","section":"Throughout"},{"comment":"Observation 1 is incomplete: the bullet says the methods \"achieve comparable performance to their corresponding,\" which reads as a truncated sentence and should be finished.","section":"Section 4.2, Observation 1"},{"comment":"Equation (7) uses TopK(·,k) and the text says the mask preserves the indices of the k largest values in α_e, but the variable should be the signature a_e; the formula itself is otherwise clear.","section":"Section 3.2, Eq. (7)"},{"comment":"The phrase \"probabilistic interaction\" is used although Eq. (9) normalizes P_r via tanh, which allows negative entries; please clarify whether P_r is a probability matrix or a signed interaction-weight matrix.","section":"Section 3.2, Eq. (9)"},{"comment":"Equation (11) writes λ·|P_r|^2_F; use \\|P_r\\|_F^2 for the Frobenius norm to avoid ambiguity.","section":"Section 3.2, Eq. (11)"},{"comment":"Figure 5's axis ticks are dense and the bar labels partially overlap; add clear group labels and a caption defining what \"Linear decay 1.0 0.2\" means.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The authors should also position the contribution against existing message-passing sparsification and information-bottleneck methods, since the current related-work discussion is thin and the novelty may be questioned. Before any further round, the anonymous code link should be replaced with a persistent repository, and the baseline numbers should be checked against the original papers. The main obstacle to acceptance is the underspecified distillation operator and the unsupported significance and synergy claims; these are fixable within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper offers two cheap add-on ideas: iterative message filtering for GNNs and a top-k sparse bilinear interaction term for embedding models. The individual modules mostly nudge three GNN baselines and SimKGC upward on WN18RR and FB15K-237, and the gains, though small (0.001–0.032 MRR), are broadly consistent. That is genuinely useful for practitioners who want to graft something onto existing KGC models. The APIM top-k masking is a modest twist on bilinear scoring, but it does help SimKGC on WN18RR (MRR 0.626→0.658). The distillation idea—filtering messages iteratively to counter over-smoothing—is plausible, and Table 1 shows the DIST variants beat the base models on most rows. Code is released anonymously, and the ablations, limited as they are, probe the main hyperparameters. Credit where due: this is a reasonable empirical direction on standard datasets.\n\nThe soft spots are more serious than the reader's take suggests. Equation (2) defines the distilled message as Distill(m_i; alpha_i) and never says what Distill actually does. No pseudocode, no derivation, no ablation that isolates it. Every DIST and MERGE number rests on that operator, so the central claim of 'outperforming baselines' is not checkable from the text. The word 'significant' is used without standard deviations or significance tests; some differences are +0.001 MRR. Observation 2's synergy claim is contradicted by Table 1 on WN18RR: for KB-GAT, RGCN, and CompGCN the MERG variant lands below both single-module variants. Section 5 says the best filter ratio is linear decay to 0.4, while Figure 5 shows 0.2 wins. And the Appendix D energy-retention 'validation' is descriptive—it uses the trained signatures themselves, so it provides no independent support for the top-20 choice.\n\nNone of this is fatal to the underlying idea. But it is collective load-bearing, and fixing it requires real work: define Distill, report error bars, correct the synergy claim, harmonize Section 5 with Figure 5, and add a proper validation protocol for the hyperparameters. As written, I would not rely on the quantitative claims. The direction is worth a serious referee, though—this is the kind of paper where a careful review could turn a promising but underspecified submission into something reproducible and valuable.\n\nWho this is for: practitioners who want cheap add-ons to existing KGC models, and researchers working on over-smoothing in GNNs. Not for someone who needs reliable state-of-the-art numbers. I would bring it to a reading group as a case study in how a promising idea can be undercut by underspecification. Recommend: engage with it critically, but don't cite it until the method is actually specified.","headline":"A plausible pair of cheap add-on modules, but the undefined Distill operator and the unquantified 'significant' claims make the central results unverifiable as written.","tokens_in":13310,"tokens_out":2696,"would_cite":false,"duration_ms":27425,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that two add-on modules—iterative message distillation and abstract probabilistic interaction modeling—reliably improve knowledge graph completion across GNN-based and embedding-based baselines.","keywords":["knowledge graph completion","GNN distillation","over-smoothing","abstract probabilistic interaction modeling","probabilistic signatures","transition matrices","WN18RR","FB15K-237"],"falsifier":"Take the four-layer KB-GAT setup on FB15K-237 and replace the distillation operator with random feature masking that removes the same fraction of message dimensions under the same decay schedule; if MRR stays near the reported 0.3588 or higher, then the specific distillation mechanism is not the source of the gain. A second check: run the same models with APIM removed and distillation kept, and then with distillation removed and APIM kept, on both datasets; the gap between those two ablations reveals which module carries the improvement.","tokens_in":12208,"feed_emoji":"🕸️","tokens_out":8620,"duration_ms":78824,"temperature":0.7,"pith_summary":"Knowledge graph completion predicts missing links in incomplete graphs, but deep graph neural networks tend to collapse node representations through over-smoothing, and embedding models miss abstract relational structure. This paper tries to establish that two add-on modules fix those failures: an iterative GNN message-distillation step that filters out redundant features at each layer, and an abstract probabilistic interaction module that scores triples through learned relation-conditioned transition matrices over sparse entity interaction modes. If the reported results hold, any existing GNN or embedding KGC model can be strengthened by grafting on these modules without redesigning the model itself. The paper supports the claim with experiments on WN18RR and FB15K-237 across KB-GAT, RGCN, CompGCN, and SimKGc.","feed_headline":"Two add-on modules lift knowledge graph completion scores","feed_subtitle":"Filtering redundant GNN messages and modeling abstract relation interactions beat the base models on WN18RR and FB15K-237.","key_machinery":"The load-bearing object on the GNN side is the distillation operator $Distill(\\tilde{m}_i^{(l,k)}; \\alpha_i^{(l,k)})$, an iterative message filter that runs $K$ rounds per layer and progressively retains a fraction $\\alpha$ of the most informative features, with $\\alpha$ following a linear or exponential decay; the paper does not specify the operator's concrete form. On the embedding side the load-bearing object is the APIM score $f(h,r,t)=\\tilde{a}_h^\\top P_r \\tilde{a}_t$, built from a sigmoid entity signature $a_e=\\sigma(W_a h_e)$, a top-$k$ sparsification mask, and a relation-specific transition matrix $P_r=\\tanh(\\Theta_r)$. The distillation operator is meant to prevent over-smoothing while the bilinear expectation score is meant to capture abstract relational patterns. These two objects carry the argument: the rest of the paper is the claim that inserting them into existing models and training with a combined loss yields the reported benchmark gains.","core_discovery":"On its own terms, the paper's discovery is that two mechanisms—distillation and probabilistic interaction modeling—are complementary and portable. GNN distillation treats the aggregated message at each layer as a signal to be refined over several rounds: only features judged important survive, under a linear or exponential decay schedule, so nodes keep local distinctiveness while still absorbing global context. APIM encodes each entity as a sparse probabilistic signature over latent interaction modes, gives each relation a trainable transition matrix between modes, and scores a triple as the expected interaction $\\tilde{a}_h^\\top P_r \\tilde{a}_t$; the module is trained with binary cross-entropy and a Frobenius penalty. Applied separately or together to four-layer GNNs and to the embedding/LLM-based SimKGc, the enhanced variants outperform their unmodified baselines on MRR and Hits@N, with the largest reported single gain being the WN18RR SimKGc MRR improvement from 0.626 to 0.658. The paper further reports that retaining about 20 interaction modes and a linear distillation decay from 1.0 to 0.2 are the best operating points.","pith_inferences":["The author-defined distillation operator is a placeholder: a concrete implementation is needed, and a natural candidate is hard top-k masking of message dimensions with an energy threshold analogous to the APIM top-20 rule; this would make the over-smoothing mechanism directly testable.","Since the APIM score is a relation-conditioned bilinear form over entity signatures, it is mathematically close to the factored bilinear embedding family; models in that family could absorb APIM almost for free, and gains should be largest on datasets where relational patterns are hierarchical, matching the larger reported improvements on WN18RR.","The paper does not isolate the distillation effect through a module-level ablation; a natural test of the claimed synergy would compare KB-GAT-DIST, KB-GAT-APIM, and KB-GAT-MERG against a version that trains both branches but replaces distillation with the identity map.","If the distillation mechanism is what the paper says, it should transfer to node classification on non-relational graphs, where over-smoothing is also a known failure mode; running GCN or GAT on citation graphs would be a cheap transfer test."],"forward_implications":["GNN distillation should make deeper KGC architectures usable: the paper's four-layer models with distillation beat their unmodified counterparts, so the technique is a candidate remedy for over-smoothing in even deeper networks.","The merged APIM-plus-distillation variant is reported to exceed the additive effects of the individual modules, suggesting that the two mechanisms do not merely duplicate each other.","Moderate hyperparameter settings matter: retaining about 20 interaction modes and linearly decaying the distillation ratio from 1.0 to 0.2 gave the best results, while more aggressive filtering degraded performance.","Because the distillation module is inserted at every GNN layer and APIM is appended at the end, the combined loss $\\mathcal{L}=\\mathcal{L}_{\\text{GNN}}+\\lambda_{\\text{APIM}}\\mathcal{L}_{\\text{APIM}}$ is a reusable recipe for other GNN-based KGC models."],"supporting_citations":[{"why":"Identifies the over-smoothing failure mode in deep GNNs that the distillation module is designed to fix.","marker":"Li et al., 2018"},{"why":"Provides the theoretical result that GNNs exponentially lose expressive power with depth, motivating the distillation approach.","marker":"Oono and Suzuki, 2019"},{"why":"Supplies the RGCN baseline architecture into which the distillation module is inserted.","marker":"Schlichtkrull et al., 2018"},{"why":"Provides the KB-GAT baseline whose attention-based message passing is modified with distillation.","marker":"Nathani et al., 2019"},{"why":"Defines the TransE embedding baseline and the entity-ranking evaluation protocol used for MRR and Hits@N.","marker":"Bordes et al., 2013"},{"why":"Provides the ComplEx embedding baseline that APIM is meant to enrich with abstract interaction modeling.","marker":"Trouillon et al., 2016"},{"why":"Supplies the SimKGc embedding/LLM baseline to which APIM is applied in the main experiments.","marker":"Wang et al., 2022"},{"why":"Supplies the WN18RR dataset and its filtering methodology.","marker":"Toutanova and Chen, 2015"},{"why":"Supplies the FB15K-237 dataset and the filtered evaluation setting used for ranking metrics.","marker":"Dettmers et al., 2018"}],"fun_headline_variants":["GNN distillation and probabilistic modeling enhance KGC","Distilled GNNs and probabilistic interactions improve KGC","Two complementary modules improve knowledge graph completion","GNN distillation plus probabilistic interaction enhance KGC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument collapses if the distillation operator does not actually preserve informative features while removing redundant ones, because the paper never defines how the operator or its importance scores work.","fun_headline_variants_meta":{"raw":{"variants":["GNN distillation and probabilistic modeling enhance KGC","Distilled GNNs and probabilistic interactions improve KGC","Two complementary modules improve knowledge graph completion","GNN distillation plus probabilistic interaction enhance KGC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001445,"raw_usage":{"total_tokens":5867,"prompt_tokens":1036,"completion_tokens":4831,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":652,"completion_tokens_details":{"reasoning_tokens":4772}},"tokens_in":652,"tokens_out":4831,"duration_ms":33092,"temperature":1.0,"reasoning_tokens":4772,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:37:13.643012+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the four-layer KB-GAT setup on FB15K-237 and replace the distillation operator with random feature masking that removes the same fraction of message dimensions under the same decay schedule; if MRR stays near the reported 0.3588 or higher, then the specific distillation mechanism is not the source of the gain. A second check: run the same models with APIM removed and distillation kept, and then with distillation removed and APIM kept, on both datasets; the gap between those two ablations reveals which module carries the improvement.","supporting_citations":[],"review_version":1}