{"id":"80c171d1-42ed-41a9-ae1c-44ce27c65824","arxiv_id":"2509.01166","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"SAT uses hierarchical contrastive alignment and a unified graph instruction to tune a lightweight adapter for knowledge graph completion, reporting large link prediction gains.","lead":"This paper proposes SAT, a framework that aligns knowledge graph embeddings with language via contrastive learning and then fine-tunes only a small adapter on a frozen large language model for knowledge graph completion. The authors report large gains on link prediction benchmarks, but the evaluation protocol may not be directly comparable to baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Top-3 beam-search evaluation makes SAT's link prediction metrics incomparable with full-ranking baselines.","rationale":"The reader's weakest assumption correctly identifies the evaluation protocol as the load-bearing weakness. The central claim is that SAT significantly outperforms state-of-the-art methods in link prediction. That claim depends entirely on the comparability of the reported Hits@1 and MRR values. Since SAT is evaluated by generating only three answers while traditional baselines rank the full entity set, the numbers are not directly comparable. The paper never shows SAT's performance under the standard protocol, leaving the headline improvement unsupported. This alone justifies the REJECT verdict. I also considered the potential computational infeasibility of the local alignment loss (Eq. 4) on large KGs like YAGO3-10, which requires an N×N similarity matrix over 123k nodes; the paper does not describe any sampling or batching, making the methodology as written impossible to run on the stated hardware. However, the evaluation flaw is more directly tied to the central claim, so it is the primary concern. The reader's verdict of REJECT stands, and no verdict change is needed.","tokens_in":17864,"tokens_out":5546,"duration_ms":62725,"concrete_test":"Re-evaluate SAT under the standard filtered ranking protocol: for each test query (h, r, ?), score every candidate entity in the KG (e.g., by computing a completion score with the frozen LLM and adapter) and rank all entities. Then compute Hits@1 and MRR based on this full ranking. Compare these numbers to Table 3 and Table 4. If Hits@1 drops by more than ~5 points (e.g., from 0.386 to below 0.34 on FB15k-237N), the reported improvements are an artifact of the restricted candidate set. The released code should allow this with moderate changes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 5.1, the paper states: 'For each query, the top-3 answers are generated using a beam-search strategy.' Hits@1 and MRR are then computed from this 3-element candidate set. This protocol is not commensurate with the standard link prediction evaluation used by the baselines (e.g., TransE, ConvE, RotatE, RED-GNN), which rank all entities in the KG. Under the paper's protocol, the model only needs to place the correct answer among three generated candidates; if it is absent, the reciprocal rank is effectively 0. This makes the task substantially easier: the model can exploit language priors and local subgraph structure to guess a few plausible names, rather than discriminating among all possible entities. The claimed relative improvements of 8.7%–29.8% (abstract and Tables 3–4) are therefore not comparable to the reported baseline numbers. No full-ranking results for SAT are provided, so the state-of-the-art claim is unsupported. Moreover, the appendix (Table 8) shows SAT has lower F1 than PKGC on FB15k-237 and YAGO3-10, yet the text claims 'F1 improvements' by substituting Accuracy values — an additional sign of selective reporting.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents SAT, an LLM-based framework for knowledge graph completion (KGC). It first aligns graph embeddings with natural-language representations through local (node-description) and global (subgraph-document) contrastive learning, then performs structural instruction tuning with a unified graph-instruction template and a lightweight knowledge adapter while freezing the LLM and graph encoder. The authors evaluate triple classification and link prediction on FB15k-237N, CoDeX-S, FB15k-237, and YAGO3-10, reporting large improvements over prior methods, especially in link prediction (8.7-29.8% relative Hits@1 gains). Source code is promised.","tokens_in":18191,"tokens_out":7315,"duration_ms":79478,"significance":"The proposed architecture is sensible and addresses a real problem: bridging structural and textual representations for LLM-based KGC. The hierarchical alignment is a reasonable way to inject graph structure, and the unified instruction/adapter design is practical and parameter-efficient. If the reported gains hold under a comparable evaluation protocol, the method would be a useful contribution. The strongest assets are the promise of reproducible code, clear ablation studies supporting the alignment and instruction components, and transferability experiments across LLMs and datasets. However, the paper's central comparative claim rests on an evaluation protocol mismatch and on an internal contradiction in the triple-classification results, so the significance cannot be assessed from the current evidence.","major_comments":[{"comment":"The link prediction evaluation is not commensurate with the baselines. §5.1 states 'For each query, the top-3 answers are generated using a beam-search strategy.' Hits@1 and MRR are then computed over this three-element candidate set, whereas the traditional baselines (ConvE, RotatE, RED-GNN, etc.) rank all entities under the standard filtered protocol. Under a 3-candidate protocol, a model need only place the correct entity in a short generated list; an absent correct entity yields reciprocal rank 0, so the scores are not comparable to full-ranking MRR/Hits@1. The claimed 8.7%–29.8% relative improvements are therefore unsupported. Please report SAT's Hits@1/MRR with full entity ranking (ideally filtered) and use the same protocol for all generative baselines, or clearly state that the comparison is against a different, easier evaluation.","section":"§5.1, Tables 3–4"},{"comment":"The text states 'SAT achieves relative F1 improvements of 1.3% and 2.4% over PKGC' on FB15k-237 and YAGO3-10, but Table 8 reports SAT F1=0.822 vs PKGC 0.846 on FB15k-237 and SAT F1=0.795 vs PKGC 0.831 on YAGO3-10. SAT's F1 is lower in both cases; only Accuracy is higher. This internal contradiction suggests selective reporting of the Accuracy metric under the label 'F1'. The claim must be corrected, and the triple-classification comparison on the large datasets should not be described as a universal F1 improvement.","section":"Appendix E.1, Table 8"},{"comment":"The query subgraph construction is underspecified. For each test query, the paper says it extracts 'the k-hop neighborhoods around the anchor entities in the query from the KG' (§4.2). If the KG used at inference includes the test triple or the target tail as a neighbor, the structural prompt may leak the answer. Please state explicitly that query subgraphs are extracted from the training KG only, and verify that no test edge appears in the subgraph. This is necessary to rule out an alternative explanation for the strong Hit@1 numbers.","section":"§4.2, §5.1"}],"minor_comments":[{"comment":"The efficiency analysis is internally inconsistent: Table 6 lists SAT training time as 70H on YAGO3-10, while Table 9 reports the full tuning stage as 28H (5H + 23H). Please reconcile these numbers or specify what is included in the 70H.","section":"Table 6 vs Appendix E.2"},{"comment":"The notation D is used both for the matrix of description embeddings and for the set of descriptions; D′ is similarly overloaded for the document set and the matrix in Eqs. (5)–(6). Please disambiguate.","section":"§4.1"},{"comment":"YAGO3-10 is typeset inconsistently as 'Y AGO3-10' in several tables and sentences. Please standardize.","section":"Tables and text"},{"comment":"The baseline 'TuckER' appears in Table 3 but has no corresponding entry in the reference list. Please add the citation.","section":"Table 3"},{"comment":"The ablation bars in Figure 4 are reported without error bars or significance tests. Given that some differences (e.g., in panel c) are small, please report variance or at least state how many independent runs were averaged.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is well-founded: the link prediction evaluation is not comparable to the baselines. I do not see a circularity problem in the alignment construction, but the protocol mismatch and the F1 contradiction must be resolved. I would not recommend rejection outright because the framework is reasonable and the evaluation can be repaired; however, the revision must include new full-ranking results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nTwo things to know up front. The method is a genuine combination—hierarchical contrastive alignment of node and subgraph embeddings with the text space, plus a unified graph instruction for both triple classification and link prediction. That is new compared with KoPA and GraphGPT, and the ablations suggest the alignment components matter. The paper also provides code and detailed implementation, which is helpful.\n\nThe problem is the headline result. The link prediction evaluation uses only the top-3 beam-search outputs, then computes Hits@1 and MRR over that tiny candidate set. The baselines rank every entity. That is not a fair comparison; it makes the task much easier and invalidates the 8.7–29.8% improvements claimed in the abstract. No full-ranking numbers are given for SAT, so the state-of-the-art claim is unsupported. This is a load-bearing flaw.\n\nThere is also a clear internal consistency problem. In Appendix E.1, Table 8 shows SAT with lower F1 than PKGC on both FB15k-237 and YAGO3-10, yet the text says SAT achieves relative F1 improvements of 1.3% and 2.4% over PKGC. That is contradictory and suggests selective reporting.\n\nWhat is solid: the triple-classification results on the two smaller datasets are plausible, the robustness tests with noisy or missing descriptions are useful, and the transferability and efficiency analyses give practical context. The GPT-4 subgraph extraction for global alignment is not circular—it uses external documents rather than the KG itself. The ablation study is well done.\n\nThis paper is for researchers working on LLM-based KGC who want to see how to align graph structure with language space. Read it for the method, not for the numbers. It deserves a serious referee because the idea has substance and the evaluation flaw is fixable. If the authors rerun with full ranking and correct the F1 claims, it could be a solid contribution.\n\nMy recommendation: send it to peer review with a clear demand for a common evaluation protocol and corrected abstract. Desk rejection would be defensible, but it would risk losing a worthwhile idea.","headline":"A sensible architecture with a real evaluation flaw: the headline link-prediction numbers come from a top-3 beam-search protocol that is not comparable to the baselines.","tokens_in":18623,"tokens_out":3686,"would_cite":false,"duration_ms":39394,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Aligning graph embeddings with language lets a frozen LLM complete knowledge graphs, with Hits@1 gains of up to 29.8% in link prediction.","keywords":["knowledge graph completion","large language models","graph-text alignment","contrastive learning","instruction tuning","link prediction","triple classification","structure-aware reasoning"],"falsifier":"Run SAT on FB15k-237N and CoDeX-S under the standard filtered link-prediction protocol, ranking the correct tail entity against all candidate entities, and compare Hits@1 and MRR with RED-GNN and KoPA evaluated under the identical protocol. If the large relative gains (29.1% and 29.8%) shrink or disappear, the paper's central claim is not established.","tokens_in":17801,"feed_emoji":"🔗","tokens_out":4034,"duration_ms":46552,"temperature":0.7,"pith_summary":"Knowledge graphs are incomplete, and large language models struggle to use their structure because graph embeddings and natural language live in different representation spaces. This paper claims that the gap can be closed by a two-stage recipe: first align graph embeddings with text embeddings through contrastive learning at both node and subgraph levels, then instruction-tune a frozen LLM using a lightweight adapter with a unified graph instruction. On triple classification and link prediction across four benchmarks, SAT reports large improvements, especially in link prediction Hits@1 (8.7% to 29.8% relative over the second-best baseline). If correct, the paper shows a cheap, task-unified way to inject graph structure into LLMs without per-task prompt engineering or full fine-tuning.","feed_headline":"Structure-aware tuning boosts LLM link prediction up to 29.8%","feed_subtitle":"Aligning graph embeddings with text lets a frozen LLM complete knowledge graphs without task-specific prompts.","key_machinery":"The load-bearing components are (1) hierarchical knowledge alignment, which uses multi-task contrastive learning to pull node embeddings toward their Wikipedia descriptions and subgraph embeddings toward corresponding documents, thereby placing graph structure in the language space; and (2) structural instruction tuning, which constructs a unified graph instruction that concatenates a human question with graph embeddings of a 2-hop query subgraph, and fine-tunes only a two-layer feed-forward knowledge adapter on top of a frozen LLM and frozen graph encoder. The graph encoder is a graph transformer and the text encoder is a vanilla transformer, both trained during the alignment stage and then","core_discovery":"The paper proposes SAT (Structure-Aware Alignment-Tuning), a framework that makes graph structure legible to LLMs for knowledge graph completion. The central claim is that hierarchical knowledge alignment — local contrastive alignment between entity nodes and their textual descriptions, plus global contrastive alignment between subgraphs and textual documents — resolves the representation-space mismatch between graph encoders and language models. Then, structural instruction tuning, which feeds a 2-hop query subgraph as graph embeddings through a unified instruction template and updates only a small knowledge adapter while freezing the LLM and graph encoder, enables structure-aware reasoning","pith_inferences":["The reported margins depend on the evaluation protocol: SAT computes Hits@1 and MRR over only the top-3 beam-search answers, whereas most baselines rank all entities; re-scoring under the standard filtered full-ranking protocol could narrow or alter the claimed improvements.","GPT-4 was used both to generate the subgraph-document pairs for global alignment and as a baseline, so part of the observed gain may reflect distillation of GPT-4's extraction behavior rather than the alignment mechanism alone.","The same alignment-tuning recipe could generalize beyond knowledge graphs to other structure-plus-language tasks, such as table-to-text generation, code understanding over abstract syntax trees, or schema-guided dialogue, wherever a structural encoder needs to speak the language model's language."],"forward_implications":["A single instruction template and adapter can serve multiple KGC tasks, reducing the duplication of designing separate prompts and lowering the cost of adding new completion tasks.","Freezing both the LLM and graph encoder while tuning only a small adapter keeps memory and computation low enough for practical deployment on standard hardware.","Structure-based instructions outperform text-flattened triples, implying that relational structure itself, rather than the surface text of neighboring triples, drives the improved reasoning.","The learned alignment transfers across different LLM backbones (Vicuna, Llama2, Llama3) without retraining the graph encoder, suggesting the alignment is model-agnostic.","The link-prediction gains are concentrated in Hits@1 rather than MRR, indicating that alignment sharpens the top prediction rather than merely re-ranking a few plausible candidates."],"supporting_citations":[{"why":"Supplies KoPA and Structural-aware IT, the embedding-based LLM-KGC baselines that SAT extends and compares against on triple classification and link prediction.","marker":"(Zhang et al., 2024b)"},{"why":"Provides PKGC, the strongest transformer-based baseline, and introduces the FB15k-237N benchmark dataset used for evaluation.","marker":"(Lv et al., 2022)"},{"why":"Provides RED-GNN, the GNN baseline with the best link-prediction results on FB15k-237 and YAGO3-10 that SAT reports beating.","marker":"(Zhang and Yao, 2022)"},{"why":"Supplies the Llama2-Chat-7B model used as the frozen LLM backbone in all main experiments.","marker":"(Touvron et al., 2023)"},{"why":"GPT-4 is used to build the subgraph-document pairs for global knowledge alignment and also serves as a text-based GPT baseline.","marker":"(Achiam et al., 2023)"},{"why":"Provides the graph transformer used as the graph encoder in both hierarchical knowledge alignment and structural instruction tuning.","marker":"(Yun et al., 2019)"},{"why":"Provides the vanilla transformer used as the text encoder to embed entity descriptions and documents for alignment.","marker":"(Vaswani, 2017)"},{"why":"Defines the TransE embedding baseline and the FB15k family of datasets that several benchmarks in this paper derive from.","marker":"(Bordes et al., 2013)"}],"fun_headline_variants":["Structure-aware alignment-tuning lifts KG link prediction up to 29.8%","Aligning graph embeddings with text boosts LLM link prediction up to 29.8%","SAT aligns graph and text spaces for better KG completion","Freeze LLM, align graph and text: KG completion improves up to 29.8%"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The reported gains assume that measuring Hits@1 and MRR on only the top-3 answers generated by beam search is measuring the same capability that baselines' full rankings measure; if those protocols are not comparable, the headline improvements are not supported.","fun_headline_variants_meta":{"raw":{"variants":["Structure-aware alignment-tuning lifts KG link prediction up to 29.8%","Aligning graph embeddings with text boosts LLM link prediction up to 29.8%","SAT aligns graph and text spaces for better KG completion","Freeze LLM, align graph and text: KG completion improves up to 29.8%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000903,"raw_usage":{"total_tokens":3700,"prompt_tokens":702,"completion_tokens":2998,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":446,"completion_tokens_details":{"reasoning_tokens":2912}},"tokens_in":446,"tokens_out":2998,"duration_ms":27778,"temperature":1.0,"reasoning_tokens":2912,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:48:20.945653+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SAT on FB15k-237N and CoDeX-S under the standard filtered link-prediction protocol, ranking the correct tail entity against all candidate entities, and compare Hits@1 and MRR with RED-GNN and KoPA evaluated under the identical protocol. If the large relative gains (29.1% and 29.8%) shrink or disappear, the paper's central claim is not established.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the TransE embedding baseline and the FB15k family of datasets that several benchmarks in this paper derive from."}],"review_version":1}