{"id":"f9b02f1a-a2ce-438d-a15e-25e9543ec7b7","arxiv_id":"2501.18119","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A self-supervised VQ-VAE with a GCN encoder and semantic distillation learns discrete entity codes that, when used as LLM tokens, improve link prediction and triple classification with only 16 tokens per entity.","lead":"The paper introduces SSQR, a method that compresses knowledge graph entities into short discrete code sequences, then fine-tunes large language models to solve graph tasks using these codes as tokens. The goal is to make knowledge graph information cheaper and more effective to feed into LLMs than long text prompts.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Link prediction comparisons are confounded by candidate-set evaluation; SSQR ranks only 20 AdaProp candidates while cited baselines rank the full entity vocabulary, and internal ablations show Hits@10 is already saturated.","rationale":"The paper's central comparative claim is that fine-tuned LLaMA2/LLaMA3.1 with SSQR codes achieves superior performance on KG link prediction using only 16 tokens per entity. That claim rests entirely on Table 3, whose evaluation protocol is not comparable to the cited baselines. The reader identified this exact weakness, and the stress-test confirms it is the most load-bearing concern. The internal ablation data make the issue concrete: removing the SSQR codes leaves Hits@10 essentially unchanged, while MRR and Hits@1 shift more. This indicates that the candidate set, not the learned codes, is what drives the top-10 performance, and that the reported Hits@10 numbers are close to the ceiling imposed by AdaProp's candidate recall. A full-corpus filtered evaluation or matched candidate-set baselines would settle the matter. I do not see a separate internal inconsistency in the SSQR learning objective or the instruction-tuning pipeline that would be more damaging; the main risk is the external validity of the link-prediction numbers. Lack of released code and error bars is secondary and does not change the conditional status.","tokens_in":21896,"tokens_out":5052,"duration_ms":50879,"concrete_test":"Run a matched-protocol experiment on both datasets: take the same test queries and the same 20 AdaProp candidates used in §4.3, and evaluate every baseline in Table 3 under this 20-candidate ranking protocol (using each model's scores restricted to the candidate set). Additionally, report recall@20 for the AdaProp candidates. If SSQR-LLaMA2/3.1 no longer beats the best baseline under matched candidate sets—or if recall@20 is below the reported Hits@10 values—the advertised superiority is an artifact of the evaluation mismatch. The cheaper diagnostic is the recall@20 number alone, since Hits@10 cannot exceed it.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 and Table 1 specify that link prediction is done by asking the fine-tuned LLM to rank 20 candidates generated by AdaProp; Appendix C confirms AdaProp is the candidate generator. The baselines in Table 3 (TransE, CompGCN, AdaProp, MA-GNN, TCRA, DiffusionE, and the LLM-based methods) are reported in their respective papers under the standard filtered setting, where scores are ranked against the full entity vocabulary. These two protocols are not comparable: MRR and Hits@10 over 20 candidates are bounded by the recall of AdaProp's candidate set, and a reranker can never recover a missing true tail. The paper never reports recall@20. Internal evidence makes the confound visible: in Table 5, removing SSQR codes entirely ('w/o SSQR') changes Hits@10 by only 0.74% (0.673 vs 0.668 on WN18RR) and 1.34% (0.597 vs 0.589 on FB15k-237), while MRR/Hits@1 change more. This is the signature of a Hits@10 metric saturated by the candidate set, not by the quantized codes. The authors' own note in §4.3 that 'our method generally achieve[s] a greater improvement in the Hits@1 metric, which is caused by the candidate selection and ranking strategies' supports this reading. Unless the same candidate sets are used for the baselines or full-corpus ranking is reported, the central claim of 'superior performance' is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SSQR, a two-stage framework that learns a discrete quantized representation for KG entities: a GCN-based encoder with a learnable codebook is trained through a VQ-style objective, a structural reconstruction loss, and a semantic distillation loss from text embeddings. The resulting 16-token entity codes are then injected into the LLM vocabulary and used as input features to fine-tune LLaMA2 and LLaMA3.1 on KG link prediction and triple classification. Experiments on WN18RR, FB15k-237, and FB15k-237N report improvements over quantized baselines and LLM-based methods, and the paper includes ablations, entropy/Jaccard distinguishability analyses, and case studies.","tokens_in":22230,"tokens_out":4936,"duration_ms":51220,"significance":"The central idea—turning KG structure into discrete, LLM-consumable tokens rather than continuous adapters or long textual prompts—is timely and potentially useful, and the paper grounds it with a substantial set of ablations and qualitative analyses. The self-supervised codebook learning with combined structural and semantic objectives is a reasonable contribution. However, the headline link-prediction comparison is currently not evaluable because SSQR ranks only 20 AdaProp candidates while the cited baselines rank over the full entity vocabulary; until this protocol mismatch is resolved and candidate recall is reported, the abstract's claim of 'superior performance' on link prediction is not established. If the comparison is fixed with same-protocol evaluation and recall@20 reporting, the contribution would be solid and worth publishing.","major_comments":[{"comment":"The link prediction evaluation protocol makes the headline results non-comparable to the baselines. The instruction format in Table 1 and the description in Section 4.3 state that only 20 candidates generated by AdaProp are provided to the LLM for ranking, whereas the general embedding and LLM-based baselines in Table 3 are evaluated in the standard filtered setting over the full entity set in their original papers. Consequently, SSQR's MRR and Hits@10 are upper-bounded by AdaProp's recall@20, and no recall@20 or coverage of the true tail in the candidate set is reported. The internal ablation in Table 5 is consistent with this concern: removing SSQR codes changes Hits@10 by only 0.74% on WN18RR (0.673 vs. 0.668) and 1.34% on FB15k-237 (0.597 vs. 0.589), suggesting the Hits@10 metric is largely saturated by the candidate set. Please rerun the baselines on the same candidate sets, or perform full-corpus ranking, and in either case report recall@20 and candidate coverage per test split.","section":"Section 4.3, Table 1, Table 3, Appendix C"},{"comment":"The semantic distillation loss in Eq. (8) is written with a leading minus sign: Lse = -1/|E| sum_i ||W_s q_{e_i} - t_{e_i}||_2^2. Since Eq. (9) minimizes L = Lq + Lst + Lse, minimizing this term would move the quantized codes away from the text embeddings instead of aligning them. Please correct the sign or clarify the objective actually optimized in the experiments; this is load-bearing for the semantic distilling component and for the ablation results labeled 'w/o sem'.","section":"Eq. (8) and Eq. (9)"},{"comment":"The comparison of SSQR against NodePiece, EARL, and RandomEQ in Table 2 is presented without stating whether all rows use the same scoring function and evaluation protocol. SSQR's structural reconstruction loss in Eq. (6) uses a ConvE-style scorer, while the baseline numbers are taken from Li et al. (2023) and may use different decoders or filtered settings. Please specify the shared protocol, decoder, negative sampling, and filtered/unfiltered setting used for each row so that the claimed improvements of 9.28%/7.84% and 16.45%/8.57% in Table 2 are verifiable.","section":"Section 4.1, Table 2"}],"minor_comments":[{"comment":"The summation limit N in the LLM fine-tuning loss is also used for the code sequence length N in Section 2; please use a distinct symbol for the number of output tokens to avoid ambiguity.","section":"Eq. (10)"},{"comment":"The text says 'In the WN8RR dataset' but the dataset is WN18RR; please fix the typo.","section":"Section 4.2, Figure 5"},{"comment":"Please report the exact LLM fine-tuning recipe for the link prediction experiments (number of epochs, trainable parameters, sequence length, and any LoRA/adapters), as only the learning rate and warmup ratio are given.","section":"Appendix C"},{"comment":"The claim that code tokens and word tokens occupy distinct representation zones is based on a qualitative t-SNE plot; please either add a quantitative separation measure or explicitly label the figure as illustrative.","section":"Section 4.4, Figure 8"}],"recommendation":"major_revision","confidential_remarks":"The main gate for publication is the link-prediction evaluation protocol: without recall@20 of the AdaProp candidate set or same-protocol baseline comparisons, the central 'superior performance' claim is not supported. The sign error in Eq. (8) also needs clarification. I would encourage a revision that addresses these two issues; the underlying approach and the self-supervised quantization contribution are worth preserving, and the triple-classification results and ablations provide useful evidence for the method's value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe thing to know about this paper is that the core idea is genuinely interesting, but the central link prediction comparison is not apples-to-apples. SSQR learns discrete quantized codes for KG entities via a VQ-VAE-style encoder with GCN structure and semantic distillation, then uses those codes as tokens to fine-tune LLaMA. That's a real contribution: prior quantization methods (NodePiece, EARL, RandomEQ) are unsupervised and don't integrate with LLMs in this token-like way. The paper is also honest about limitations and does solid ablation work.\n\nThe problem is in the evaluation. For link prediction, the LLM only reranks 20 candidates produced by AdaProp, while the cited baselines (TransE, CompGCN, AdaProp, etc.) rank against the full entity vocabulary. Those protocols produce incomparable MRR/Hits@10. The internal evidence supports this reading: dropping the SSQR codes entirely ('w/o SSQR') moves Hits@10 by only 0.7–1.3% on both datasets, while MRR/Hits@1 drop by 8–14%. That is exactly the signature of a metric saturated by the candidate set. The authors even concede in §4.3 that the Hits@1 gains are 'caused by the candidate selection and ranking strategies.' Until they either run baselines on the same 20-candidate sets or report filtered full-corpus ranking, the headline 'superior performance' claim is unsupported.\n\nThere is also a smaller issue: Table 2 quotes baselines from Li et al. (2023) rather than re-running them, which is common but reduces confidence in the quantization comparison. No code or data is released, and there are no error bars.\n\nWho is this for? Someone working on parameter-efficient KG representations or token-based LLM–KG integration would get useful ideas here. The token-efficiency argument (16 tokens per entity vs thousands) is compelling. But the empirical claims need a serious rework. I would send it to peer review—the idea deserves referee time—but I'd expect major revisions on the evaluation before acceptance.\n\nRecommendation: engage with it, but don't rely on the link prediction numbers as they stand.","headline":"Interesting token-based KG–LLM integration idea, but the link prediction evaluation is confounded by candidate-set ranking and needs rework before the performance claims can be trusted.","tokens_in":22782,"tokens_out":2584,"would_cite":false,"duration_ms":22625,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Compressing each knowledge-graph entity into 16 discrete tokens lets fine-tuned LLMs outperform earlier KG completion methods while using a fraction of the prompt tokens.","keywords":["knowledge graph completion","self-supervised learning","vector quantization","discrete code representations","large language models","link prediction","triple classification","token-efficient prompting"],"falsifier":"Check, for every test query on WN18RR and FB15k-237, whether the correct answer is among the 20 AdaProp candidates; if it is missing for a substantial fraction of queries, the reported MRR and Hits@10 values are measuring candidate recall rather than ranking ability and would not match full-corpus baseline numbers.","tokens_in":21691,"feed_emoji":"🧠","tokens_out":11803,"duration_ms":101373,"temperature":0.7,"pith_summary":"This paper seeks to close the format gap between knowledge graphs and large language models by compressing each entity into a short sequence of discrete tokens. The proposed SSQR method learns these tokens in a self-supervised way, training a graph encoder and a discrete codebook so that the codes can reconstruct the graph's triple structure and mirror entity text semantics. The authors then treat the codes as ordinary token IDs, extend the LLM's tokenizer vocabulary, and fine-tune LLaMA2 and LLaMA3.1 on instruction data for link prediction and triple classification. They report that this approach outperforms prior quantization-based and LLM-based baselines while using only 16 tokens per entity, in contrast to the thousands of tokens text-prompting approaches need.","feed_headline":"16 tokens per entity let LLMs beat KG completion baselines","feed_subtitle":"Self-supervised codes compress structure and meaning, replacing thousands of prompt tokens with a handful.","key_machinery":"The load-bearing object is the discrete codebook: a set of M learnable vectors, each identified by an integer index. A graph convolutional encoder produces an entity embedding, which is projected to N vectors; each vector is assigned to its nearest codebook entry by Euclidean distance, yielding a code sequence such as [q1, q2, ..., qN] that acts as the entity's token ID. The quantization loss is the VQ-VAE codebook-plus-commit objective with a straight-through estimator, structure knowledge is injected by scoring triples over quantized embeddings with a ConvE-style scorer, and semantic knowledge is distilled by regressing the codes toward large-model text embeddings. On the LLM side, the codes are appended to the tokenizer vocabulary and instruction-tuning data is built around them, so no adapter or architectural change is needed.","core_discovery":"The central claim is that a knowledge graph's structure and entity semantics can be distilled into a fixed-length sequence of discrete codebook indices, and that those indices are sufficient features for an LLM to carry out knowledge-graph tasks. Self-supervision comes from two complementary signals: structure reconstruction, scored by a convolutional model over quantized triple representations, and semantic distillation, which aligns the codes with embeddings of entity text descriptions. Ablations show that both signals matter, with structure dominating on the denser FB15k-237 graph and semantics mattering more on WN18RR. With instruction tuning, LLaMA2 and LLaMA3.1 rank candidate entities using only the code sequences, and the paper reports gains over both embedding-based and LLM-based completers on link prediction and triple classification.","pith_inferences":["If the codes preserve both structure and semantics, the same vocabulary could plausibly be reused for KG question answering and retrieval without retraining the quantizer, though the paper only demonstrates completion and classification.","The reported LLM gains depend on the quality of the 20-candidate generator; a stronger candidate model than AdaProp would likely push Hits@1 higher, and a weaker one would erode the gains.","Treating codes as opaque IDs raises the question of whether the LLM uses their learned structure or merely memorizes entity-specific tokens; a controlled ablation that replaces code sequences with random distinct IDs would separate these effects."],"forward_implications":["Every entity can be represented by 16 tokens, so tasks involving many entities fit within LLM context windows that text-prompting with sampled triples would overflow.","Extending the tokenizer vocabulary with codebook tokens is the only change needed to make a knowledge graph consumable by an LLM, replacing adapters and alignment layers.","The same learned codes transfer across at least two knowledge-graph tasks by swapping the instruction format, pointing toward a shared entity vocabulary for multiple KG applications.","Increasing the codebook size M and the sequence length N improves accuracy, with sequence length mattering more on sparser graphs."],"supporting_citations":[{"why":"Supplies the vector-quantization objective and straight-through gradient estimator that the discrete codes are built on.","marker":"Van Den Oord et al., 2017"},{"why":"NodePiece is the anchor-based quantized representation baseline whose fixed-size entity vocabulary SSQR improves on.","marker":"Galkin et al., 2022"},{"why":"EARL is the second unsupervised anchor-based quantized representation baseline compared in Table 2.","marker":"Chen et al., 2023"},{"why":"RandomEQ is the random-anchor quantization baseline, and its entropy and Jaccard-distance metrics are used to measure SSQR's distinguishability.","marker":"Li et al., 2023"},{"why":"CompGCN supplies the relation-composition message-passing operation used for structural modeling and is also a link-prediction baseline.","marker":"Vashishth et al., 2020"},{"why":"ConvE provides the triple-scoring function used for structure reconstruction and defines the WN18RR and FB15k-237 evaluation datasets.","marker":"Dettmers et al., 2018"},{"why":"AdaProp generates the 20 answer candidates that the fine-tuned LLMs rank in link prediction, making it load-bearing for the evaluation.","marker":"Zhang et al., 2023"},{"why":"KoPA is the adapter-based KG-LLM baseline for triple classification that SSQR is compared against.","marker":"Zhang et al., 2024b"},{"why":"KICGPT is an LLM-based link-prediction baseline and the source of the candidate-generation plus re-ranking evaluation strategy.","marker":"Wei et al., 2023"}],"fun_headline_variants":["16 tokens, not thousands: LLMs beat KG baselines","Self-supervised codes: KG knowledge in 16 tokens for LLMs","16 tokens distill KG structure and semantics for LLM integration","SSQR: self-supervised quantization compresses KG into 16 LLM tokens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported link-prediction numbers assume that the 20 candidate entities AdaProp generates for each query contain the correct answer for nearly every test query, making ranking over those 20 comparable to ranking over the entire entity vocabulary.","fun_headline_variants_meta":{"raw":{"variants":["16 tokens, not thousands: LLMs beat KG baselines","Self-supervised codes: KG knowledge in 16 tokens for LLMs","16 tokens distill KG structure and semantics for LLM integration","SSQR: self-supervised quantization compresses KG into 16 LLM tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001731,"raw_usage":{"total_tokens":6814,"prompt_tokens":886,"completion_tokens":5928,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":5852}},"tokens_in":502,"tokens_out":5928,"duration_ms":35585,"temperature":1.0,"reasoning_tokens":5852,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T00:35:40.866419+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check, for every test query on WN18RR and FB15k-237, whether the correct answer is among the 20 AdaProp candidates; if it is missing for a substantial fraction of queries, the reported MRR and Hits@10 values are measuring candidate recall rather than ranking ability and would not match full-corpus baseline numbers.","supporting_citations":[],"review_version":1}