{"id":"0717f46a-41bc-4550-a4bc-cf835990cca2","arxiv_id":"2412.11016","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"CAB-KGC, a BERT classifier using head and relation context, reports Hit@1 of 0.322 on FB15k-237 and 0.637 on WN18RR, but the claimed SOTA gains do not match the paper's own comparison table.","lead":"A short workshop paper proposes CAB-KGC, a BERT-based model that scores candidate tail entities in knowledge graph completion using neighboring entities and relationships as context. It claims new state-of-the-art Hit@1 results on FB15k-237 and WN18RR, but the supporting tables and baselines contain inconsistencies.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claim fails: context may leak test labels through Eqs. 1–4, ranking protocol is unspecified, and Table 2 itself contradicts the reported MRR.","rationale":"The paper's only evidence for the headline claim is Table 2. For that evidence to be meaningful, the evaluation protocol must match the baselines: contexts must be computed from training triples only, and ranking must use the standard filtered setting. The text never defines the split for T in Eqs. 1–4; under the natural reading, the test triple itself contributes the true tail to both Hc and Rc, turning prediction into a copying task. This is the single most load-bearing issue because it would invalidate the reported improvements even if every other detail were correct. The omission of filtering in Eq. 9 is a second comparability risk. The table also contains an internal contradiction: on FB15k-237, the proposed MRR is below two embedding baselines, and the claimed improvement percentages do not match any baseline row. No code is provided, and there is no formal verification or independent support. A single rerun with training-only contexts and filtered ranking would settle whether the reported advances survive; until then, the central claim cannot be accepted. The reader's weakest assumption correctly identifies the context-split and ranking-protocol ambiguity, and the verdict of REJECT is appropriate.","tokens_in":5731,"tokens_out":6927,"duration_ms":67138,"concrete_test":"Obtain the exact train/validation/test split and rerun the identical model under two conditions: (a) contexts built from the full triple set as the paper ambiguously defines, and (b) contexts built from training triples only, with standard filtered ranking in both cases. If Hit@1 on WN18RR or FB15k-237 drops materially in condition (b), or if the two conditions differ, the reported benchmark is a leakage artifact and the SOTA claim is invalid.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that CAB-KGC sets a new SOTA on FB15k-237 and WN18RR. For this to hold, the reported Hit@1 numbers must measure the same quantity as the baselines. The paper never states whether T in Eqs. 1–4 is the training split or the full triple set. If T is the full graph, then for a test query (h, r, ?), the ground-truth tail t appears in E(h) via (h, r, t) and in Rc via (h, r, t), so the answer token is present in the input sequence of Eq. 5. That is direct label leakage and can inflate Hit@1. Separately, Eq. 9 defines rank without mentioning the filtered protocol used by the Table 2 baselines, so even absent leakage the numbers are not comparable. Moreover, Table 2 internally contradicts the SOTA claim: on FB15k-237 the proposed MRR is 0.350, below TuckER (0.358) and RESCAL (0.356), and the claimed 5.3% and 4.88% Hit@1 improvements correspond to no baseline values in the table (there is no row with Hit@1 0.306 or 0.607). The benchmark claim is therefore not supported by the evidence as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CAB-KGC, a BERT-based classifier for knowledge graph completion that predicts a missing tail entity from a head entity and relation together with graph-derived context: the head context Hc (neighbor entities and relations of the head) and the relation context Rc (entity pairs connected by the relation). The method avoids entity descriptions and negative triplet sampling. Experiments are reported on FB15k-237 and WN18RR, with the abstract claiming improvements in Hit@1 of 5.3% and 4.88% respectively and a new state of the art in KGC.","tokens_in":5961,"tokens_out":3637,"duration_ms":31451,"significance":"If the central claim were validated, the paper would offer a simple and computationally lighter alternative to description- and negative-sampling-based KGC methods, and the idea of feeding graph-derived context into a BERT classifier is reasonable. However, the paper's own Table 2 contradicts the state-of-the-art claim: on FB15k-237 the proposed method has MRR 0.350, below RESCAL (0.356) and TuckER (0.358), and the stated percentage improvements in Hit@1 do not match any baseline in the table. The experimental protocol is also underspecified: Eqs. (1)–(4) do not say whether the triple set T is the training split or the full graph, and Eq. (9) defines ranking without mentioning the filtered protocol used by the cited baselines. These issues are load-bearing because they directly affect whether the reported numbers measure the same quantity as the baselines. The paper does not provide code or data splits, so the experiments are not reproducible as described.","major_comments":[{"comment":"The central claim that CAB-KGC outperforms existing methods in MRR is contradicted by Table 2. On FB15k-237, the proposed MRR of 0.350 is below RESCAL (0.356) and TuckER (0.358); on WN18RR, the proposed Hit@3 of 0.687 is below SimKGC (0.731) and NNKGC (0.722). The conclusion's statement that the model \"outperforms existing methods in MRR and Hit@k\" is therefore not supported by the reported numbers.","section":"Section 2.2, Table 2, Section 3"},{"comment":"The claimed Hit@1 improvements of 5.3% and 4.88% cannot be derived from Table 2. No baseline row has Hit@1 = 0.306 on FB15k-237 or 0.607 on WN18RR, which would be the values implied by a percentage-point interpretation relative to the proposed 0.322 and 0.637. A relative-improvement interpretation also fails: 0.322/0.266 is about 21% over the best non-LLM baseline, and 0.637/0.585 is about 8.9% over SimKGC. The paper must specify the comparison basis for these percentages.","section":"Abstract, Section 2.2, Table 2"},{"comment":"The notation T in Eqs. (1)–(4) is never defined as the training split or the full triple set. If T contains all triples, then for a test query (h, r, ?), the ground-truth tail t appears in E(h) via (h, r, t) and in Rc via (h, r, t), so the answer token is present in the input sequence of Eq. (5). That would be direct label leakage and could inflate Hit@1. If T is the training split, this must be stated explicitly for the results to be interpretable.","section":"Eqs. (1)–(4), Eq. (5)"},{"comment":"Eq. (9) defines ranki as the position of the true entity in a sorted list of predicted scores but does not specify whether the evaluation uses the standard filtered protocol, in which other valid triples from the graph are removed from the candidate list. The baselines in Table 2 are reported under filtered ranking in their original papers, so without stating that the same protocol is used, the CAB-KGC numbers are not comparable. The paper also does not report the candidate set size or how ties are handled.","section":"Eq. (9), Section 2.1"}],"minor_comments":[{"comment":"The text contains a typo: \"will oftern struggle\" should be \"will often struggle.\"","section":"Section 2.2"},{"comment":"\"contexual\" should be \"contextual.\"","section":"Section 3"},{"comment":"KICGPT is cited inconsistently as Wei et al. [2023] and Wei et al. [2024] with two different reference entries; the paper should use one canonical citation.","section":"References"},{"comment":"The notation Hc is described as \"Head (h) or Entity context,\" which is ambiguous; it should be defined consistently as \"head context.\"","section":"Table 1"},{"comment":"The hyperparameter description is incomplete for reproducibility: no random seed, number of training epochs, early-stopping criterion, or maximum context length for Eq. (5) is given, and no code or data splits are released.","section":"Section 2.1"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be an extended version of a workshop paper, but the evaluation issues are not cosmetic. The state-of-the-art claim fails on the paper's own reported numbers, and the context-construction and ranking protocols are underspecified to the point that the reported Hit@1 values may not measure the same quantity as the baselines. These problems cannot be fixed by local edits; they require re-running the experiments under a clearly defined, leakage-free protocol with correctly chosen baselines. I therefore recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know: the model is a clean BERT-classifier variant for KGC that concatenates head-neighbor context and relation context, and it drops entity descriptions and negative sampling, which is practically nice. But the paper's central SOTA claim is not supported by its own Table 2, and the evaluation protocol may leak the answer into the input. Do not take the benchmark numbers at face value.\n\nWhat's genuinely new: the specific context construction in Eqs. 1–4 — combining R(h), E(h), and relation-specific entity pairs into one BERT sequence — is a real variant over KG-BERT, NN-KGC, and Sim-KGC, which use descriptions or neighborhoods differently. Removing the description dependence and negative sampling is a useful simplification, and the system is trainable on a single GPU. That part is worth keeping in mind.\n\nThe soft spots are serious. On FB15k-237, Table 2 gives CAB-KGC an MRR of 0.350, below TuckER (0.358) and RESCAL (0.356), yet the conclusion says it outperforms existing methods in MRR. The claimed 5.3% and 4.88% Hit@1 gains do not correspond to any baseline row in the table — there is no 0.306 or 0.607 Hit@1 baseline to beat. Second, Eqs. 1–4 never say whether T is the training split or the full graph. If it is the full graph, then for any test triple (h,r,t), t is included in E(h) or Rc, so the ground-truth token is physically present in the input sequence of Eq. 5. That is direct label leakage. Third, Eq. 9 defines rank without mentioning the filtered ranking protocol used by the baselines, so even without leakage the numbers are not comparable. No code is provided, so none of this can be checked.\n\nWho benefits: someone looking for a short example of how to build BERT-based KGC context features could skim the method section. But the empirical claims are the whole point of the paper, and they collapse on inspection.\n\nRecommendation: I would not send this to peer review in its current form. A serious editor should desk reject and invite a resubmission only after the authors specify the graph split, use filtered ranking, fix the internal contradiction, and run an honest comparison that includes the previously excluded prompt-based baselines or explains their exclusion more carefully.","headline":"A sensible context-based BERT classifier for KGC, but the SOTA claim is contradicted by its own table and the evaluation protocol may leak labels — reject as is.","tokens_in":6550,"tokens_out":4320,"would_cite":false,"duration_ms":37240,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A contextualized BERT model that reads only local graph neighborhoods predicts missing tail entities and sets new Hit@1 records on two standard knowledge graph benchmarks.","keywords":["knowledge graph completion","BERT","link prediction","contextual embeddings","graph context","tail entity prediction","FB15k-237","WN18RR"],"falsifier":"Run the released model through the standard filtered evaluation on FB15k-237 by removing all other true triples from the ranking before computing ranks, verify that the context extraction reads only training triples, and check whether Hit@1 remains 0.322.","tokens_in":5478,"feed_emoji":"🔗","tokens_out":6362,"duration_ms":47248,"temperature":0.7,"pith_summary":"CAB-KGC is a BERT-based model that completes knowledge graphs by predicting the missing tail entity in a triple (h, r, ?) using only local graph context: the relations and neighbor entities attached to the head, plus the entities involved in the target relation. This removes the need for entity descriptions and negative triplet sampling, which earlier text and LLM-based methods require. On FB15k-237 and WN18RR, the paper reports Hit@1 of 0.322 and 0.637, improving prior best results by 5.3% and 4.88% respectively, and an MRR of 0.685 on WN18RR. If correct, the recipe is cheaper and more universally applicable because every knowledge graph already contains the context it needs.","feed_headline":"Neighbor-context BERT tops two knowledge-graph completion benchmarks","feed_subtitle":"Reading only a head's neighbor relations and entities, it lifts Hit@1 to 0.322 and 0.637.","key_machinery":"The load-bearing mechanism is the construction of the two context sets — $H_c = \\mathcal{R}(h) \\cup \\mathcal{E}(h)$, the union of the head's incident relations and neighbor entities, and $R_c$, the entity pairs connected by the operational relation — which are concatenated into the input sequence $[CLS]\\, h, H_c\\,[SEP]\\, r, R_c$. This converts local graph structure into a text sequence so BERT's pretrained representations can classify the tail, entirely without entity descriptions or negative sampling.","core_discovery":"CAB-KGC predicts the tail t of an incomplete triple (h, r, ?) by constructing two context sets: head context Hc, the union of all relations incident to h and all entities directly connected to h, and relationship context Rc, all entity pairs connected by the relation r. These are assembled into the BERT input sequence [CLS] h, Hc [SEP] r, Rc, processed by BERT, and passed through a softmax classification layer over all entities. Trained with cross-entropy loss against the correct tail, the model reports Hit@1 of 0.322 and 0.637 on FB15k-237 and WN18RR, with MRR 0.350 and 0.685, outperforming embedding-based, text-based, and LLM-based baselines listed in the comparison.","pith_inferences":["The results suggest that BERT's pretrained lexical knowledge may encode enough of the semantics of entity names that graph structure alone suffices on these datasets, which would be a meaningful departure from embedding-based methods that learn entity vectors from scratch.","Because the context sets grow with node degree, a practical extension would cap or sparsify them for denser knowledge graphs.","The approach might transfer to inductive KGC settings where new entities appear only as named nodes with neighbor structure, since it does not learn entity embeddings.","A direct test of the context hypothesis would be to ablate Hc and Rc independently to see which component carries the reported gains."],"forward_implications":["Any knowledge graph can use the method, because neighbor entities and relations exist wherever a graph exists.","Training cost drops because negative triplet sampling is removed, and there is no dependence on external entity descriptions.","The reported numbers would displace the previous best Hit@1 on FB15k-237 and WN18RR by several relative percent.","The same context-construction recipe can be reused for head prediction and relation prediction by symmetric substitutions."],"supporting_citations":[{"why":"Supplies the FB15k-237 dataset used for the primary Hit@1 comparison.","marker":"Bollacker et al. [2008]"},{"why":"Supplies the WN18RR dataset used for the second benchmark.","marker":"Miller [1995]"},{"why":"SimKGC is the strongest text-and-description-based baseline that CAB-KGC claims to beat on both datasets.","marker":"Wang et al. [2022]"},{"why":"NNKGC is another neighborhood-based textual baseline in the comparison table.","marker":"Li and Yang [2023]"},{"why":"The paper takes most of the baseline numbers from this study's reported results.","marker":"Wei et al. [2024]"},{"why":"KG-BERT is the prior BERT-based KGC approach that this work extends.","marker":"Yao et al. [2019]"},{"why":"ComplEx is a standard embedding baseline whose Hit@1 and MRR anchor the comparison.","marker":"Trouillon et al. [2016]"},{"why":"RotatE is a strong embedding baseline whose performance CAB-KGC claims to surpass.","marker":"Sun et al. [2019]"}],"fun_headline_variants":["Contextual BERT uses neighbor links to beat LLM baselines in KGC","Neighbor-only BERT lifts Hit@1 by 5.3% on FB15k-237","Tail prediction via neighbor context: BERT sets KG completion benchmark","No negative sampling: neighbor-aware BERT tops KG completion benchmarks","Context-aware BERT improves KG completion without entity descriptions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that the reported Hit@1 and MRR numbers are computed under the same filtered ranking protocol as the baselines, with context sets derived only from training triples, so the numbers are directly comparable.","fun_headline_variants_meta":{"raw":{"variants":["Contextual BERT uses neighbor links to beat LLM baselines in KGC","Neighbor-only BERT lifts Hit@1 by 5.3% on FB15k-237","Tail prediction via neighbor context: BERT sets KG completion benchmark","No negative sampling: neighbor-aware BERT tops KG completion benchmarks","Context-aware BERT improves KG completion without entity descriptions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000336,"raw_usage":{"total_tokens":1854,"prompt_tokens":932,"completion_tokens":922,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":825}},"tokens_in":548,"tokens_out":922,"duration_ms":8131,"temperature":1.0,"reasoning_tokens":825,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:23:39.155780+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released model through the standard filtered evaluation on FB15k-237 by removing all other true triples from the ranking before computing ranks, verify that the context extraction reads only training triples, and check whether Hit@1 remains 0.322.","supporting_citations":[],"review_version":1}