{"id":"91a64019-6127-420b-a5f8-ff2ab6ffe0ec","arxiv_id":"2411.11531","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Injecting Wikidata entity embeddings through a linear adapter into frozen LLMs improves hallucination detection on HaluEval, True-False, and FEVER benchmarks.","lead":"This paper proposes a way to reduce language model hallucinations by adding knowledge graph embeddings as an extra input modality, with only a small adapter trained. On three fact-focused benchmarks the method improves accuracy, but the evaluation has gaps and the knowledge source overlaps with the test data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The gap between oracle KG embeddings used to train the adapter and noisy Text2Graph predictions at inference is unmeasured; a random-embedding control is needed to confirm the KG content is what drives the reported gains.","rationale":"The reader's weakest assumption identifies the same load-bearing risk: the adapter is trained with ground-truth Wikidata embeddings, but at inference it must consume Text2Graph predictions, and the paper never measures the quality of that transfer. My stress-test sharpens this into a concrete control: replacing predicted embeddings with random embeddings would reveal whether the adapter is actually using KG content or merely exploiting inserted tokens and distributional bias. This is the most important gap because every headline result in Tables 2–4 passes through Text2Graph at deployment, yet only the QA row even attempts an oracle-versus-predicted comparison, and that row is ambiguous. The concern is addressable rather than fatal, so the reader's CONDITIONAL verdict remains appropriate; no verdict change is needed.","tokens_in":12161,"tokens_out":4666,"duration_ms":46797,"concrete_test":"Run the +KG pipeline on HaluEval and on a 1k-sample subset of True-False/FEVER under three conditions: (A) ground-truth Wikidata embeddings, (B) Text2Graph predicted embeddings, and (C) shuffled ground-truth embeddings matched per position. Report per-condition accuracy, plus Text2Graph entity-linking F1 and mean cosine similarity between predicted and true embeddings. If (C) matches (A)/(B) within 1 point, the gains are not attributable to KG content. If (B) is substantially below (A), the deployed pipeline is not validated by the reported numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the full pipeline working at inference: Text2Graph (§3.2) must produce KG embeddings that are close enough to the ground-truth PyTorch-BigGraph embeddings on which the adapter was trained (§3.3) for the adapter to extract useful factual signal. The paper reports no entity-linking accuracy, no embedding prediction error, and no comparison of adapter behavior under predicted versus ground-truth inputs, except for the HaluEval QA row where §4.1 says 'we use the provided entity embeddings rather than generating them with Text2Graph' (labeled 'real KG embs' in Table 2), while the same table also lists 'predicted KG embs' for QA, leaving the actual setup ambiguous. Without this measurement, the positive results on True-False and FEVER could arise from correct linking only on easy statements, or from the adapter exploiting the inserted special tokens and shifting output bias rather than reading the KG embedding content. The method's promise—improving factual accuracy without retrieval—rests on this unvalidated transfer from oracle embeddings to noisy predictions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method to reduce hallucinations in large language models by injecting knowledge graph (KG) embeddings as an additional modality. The pipeline consists of a Text2Graph mapper (RoBERTa-large plus a linear layer) that predicts PyTorch-BigGraph TransE embeddings of Wikidata entities mentioned in the input text, and a linear adapter that projects these KG embeddings into the LLM's embedding space, inserted between trainable <GRAPH_START> and <GRAPH_END> tokens. The LLM itself remains frozen; only the mapper and adapter are trained. The authors introduce WikiEntities, a dataset of over 3.2 million Wikipedia texts annotated with Wikidata entities and their embeddings, and use it to train the adapter on ground-truth entity embeddings. They evaluate the method on HaluEval, True-False, and FEVER for hallucination detection, reporting improvements for Mistral 7B, LLaMA 2-7B, and LLaMA 3-8B, while also reporting results on MMLU, GSM8k, TruthfulQA, Winogrande, HellaSwag, and ARC to show no large degradation.","tokens_in":12361,"tokens_out":5330,"duration_ms":51122,"significance":"If the central claim holds, the paper offers a novel direction for incorporating factual knowledge into LLMs without external retrieval, and the WikiEntities dataset could be a valuable resource for entity linking and multimodal adaptation. The method is lightweight in that it only trains a mapper and a linear adapter. However, the current evidence is undermined by the unmeasured gap between the oracle embeddings used to train the adapter and the noisy predictions used at inference, an ambiguous QA setup, a lack of statistical significance testing, and an evaluation that is entirely within the Wikipedia/Wikidata distribution. These issues must be addressed before the claimed improvements can be considered reliable.","major_comments":[{"comment":"The adapter is trained on ground-truth Wikidata entity embeddings from WikiEntities (oracle embeddings), while at inference on True-False and FEVER it is fed embeddings predicted by Text2Graph. The paper never reports entity linking accuracy, embedding prediction error, or a direct comparison of adapter behavior under predicted versus oracle inputs. This transfer is load-bearing: without evidence that predicted embeddings are close enough to the oracle, the reported gains could be due to the special tokens, the linear projection, or Text2Graph only linking easy, unambiguous entities. Please report these metrics and include a control with random KG embeddings to confirm that the KG content, not the injection format, drives the improvement.","section":"§3.3 and §4.1"},{"comment":"The QA setup is internally contradictory. The text states, \"For this dataset, we use the provided entity embeddings rather than generating them with Text2Graph,\" yet Table 2 lists both \"predicted KG embs\" and \"real KG embs\" for the QA task, and the Discussion claims that predicted embeddings perform slightly better than real ones. This ambiguity matters because the QA row labeled \"real KG embs\" is an oracle condition and does not test the full pipeline. Please clarify which condition was used for each row, and if the predicted-KG-emb QA row is part of the official evaluation, describe exactly how Text2Graph was applied to the questions.","section":"§4.1, Table 2"},{"comment":"No error bars, confidence intervals, or significance tests are reported. Several improvements are small (e.g., Table 3, LLaMA 3 True-False average: 0.96 to 0.97; Table 5, Mistral TruthfulQA drops from 0.422 to 0.414), and it is impossible to judge whether the reported differences are meaningful without variance estimates. Please provide repeated-run results, bootstrap confidence intervals, or a paired significance test (e.g., McNemar's test for classification tasks).","section":"Tables 2–5"},{"comment":"All evaluation benchmarks (HaluEval, True-False, FEVER) are derived from Wikipedia, the entity embeddings come from Wikidata (the same knowledge source), and the adapter is trained on ground-truth Wikipedia entity annotations. This creates a circularity risk: the model may be exploiting benchmark-specific correlations between entity embeddings and answer patterns rather than learning to reason with KG embeddings in general. Please include an out-of-domain factual evaluation (e.g., questions from non-Wikipedia sources such as TriviaQA or a manually curated set of non-Wikipedia facts) and a control with shuffled or random entity embeddings to demonstrate that the KG content, rather than the injection mechanism, is responsible for the gains.","section":"§3.1, §4.1–4.3"},{"comment":"The input format for the KG modality is underspecified, which affects reproducibility. Concretely, the paper does not state how many KG embeddings are produced for a text containing multiple entities, how these embeddings are ordered relative to the <GRAPH_START>/<GRAPH_END> tokens, or what happens when no entities are detected. In addition, the Text2Graph mapper is described only at a high level (span size of 20 tokens, MSE loss, AdamW, 1 epoch), with no details on embedding dimension, number of training samples, or mapper accuracy. Please provide these specifications, and consider releasing the exact preprocessing code.","section":"§3.2–3.3"}],"minor_comments":[{"comment":"Typo: \"dialoque history\" should be \"dialogue history\".","section":"§4.1"},{"comment":"The column heading \"Cieacf\" is unclear and likely a typo (perhaps intended as \"Companies\" or another topic). Please correct.","section":"Table 3"},{"comment":"Typo: \"LLaMA3-8B suceeds\" should be \"succeeds\".","section":"§4.5"},{"comment":"The phrase \"a model harnessing approach\" is awkward; consider rephrasing to \"an approach that harnesses the model\" or similar.","section":"§4.1"},{"comment":"The qualitative examples are illustrative, but they should be accompanied by a small quantitative evaluation on a set of similar questions to demonstrate that the improvement is systematic rather than cherry-picked.","section":"Figure 3"},{"comment":"The availability statement refers to an anonymized repository; the final version should include a permanent DOI or link, and ideally the dataset itself should be released with a license and versioned.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"The core idea and the WikiEntities dataset are promising, but the current experiments do not yet establish the central claim because the oracle-to-predicted transfer is unmeasured and the QA setup is ambiguous. The requested additions—entity linking accuracy, embedding prediction error, random-embedding controls, significance testing, and an out-of-domain evaluation—are feasible and would materially strengthen the paper. I recommend major revision rather than rejection, as the identified weaknesses are fixable within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The strongest thing here is the WikiEntities dataset: 3.2M Wikipedia texts with Wikidata entity spans and PyTorch-BigGraph embeddings is a real resource, worth having on its own. The adapter-plus-modality recipe is borrowed from multimodal work, but applying it to hallucination reduction without retrieval is a sensible new combination, and the qualitative examples (Figure 3) are genuinely striking, especially the europium one.\n\nNow the soft spots. The central claim is that the full pipeline—Text2Graph mapper producing KG embeddings at inference, fed through the adapter—reduces hallucinations. The problem is that the adapter is trained on ground-truth embeddings, and the paper never measures how close Text2Graph's predictions are to those ground truths. No entity linking accuracy, no embedding prediction error, no control with random embeddings. The stress-test note is right: without that control, the gains could come from the inserted special tokens or output bias rather than the KG content. The QA setup is also muddled: Section 4.1 says they use provided entity embeddings for QA, yet Table 2 also reports \"predicted KG embs\" for QA on the same rows. Which one is the actual result? That ambiguity needs to be resolved.\n\nThe evaluation lacks error bars or significance tests, and the gains on HaluEval are small (1-4 points) with the big jump only on True-False for Mistral. No RAG baseline either, so it's unclear whether this beats something as simple as retrieval. The \"no degradation\" claim overstates Table 5, which shows small drops on MMLU and GSM8k for Mistral and LLaMA 3. These are not fatal, but they are real overstatements.\n\nThere's also a circularity/overlap issue: the benchmarks are Wikipedia-derived, the adapter is trained on Wikipedia text, and the KG embeddings are from Wikidata. So the method may be partially exploiting distributional overlap rather than adding general factual knowledge. That doesn't kill it, but it limits what the current numbers prove.\n\nThe fixes are all addressable: clarify QA, add a random-embedding ablation, measure Text2Graph accuracy, add significance tests and a RAG comparison. This is the right kind of paper to send to reviewers who can push on these points. The dataset alone justifies a citation, and the method is worth a careful look, so I'd send it to peer review.","headline":"A plausible KG-modality adapter for hallucination reduction, with a genuinely useful dataset, but the evaluation undercuts the central claim by not validating the oracle-to-predicted embedding transfer.","tokens_in":12927,"tokens_out":3010,"would_cite":true,"duration_ms":28176,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Injecting knowledge-graph embeddings as a second modality cuts LLM hallucinations.","keywords":["hallucination reduction","knowledge graph embeddings","adapter","entity linking","Wikidata","TransE","Wikipedia dataset","modality fusion"],"falsifier":"Measure the entity-linking precision and recall of the Text2Graph mapper on the evaluation inputs (HaluEval questions, True-False statements, FEVER claims) and correlate those errors with downstream accuracy; also compare the full pipeline against an oracle version that uses gold KG embeddings on every task. If downstream gains largely vanish when gold embeddings are used on tasks currently scored with predicted embeddings, the claim that predicted KGs drive the improvement is undercut.","tokens_in":11916,"feed_emoji":"🧠","tokens_out":5246,"duration_ms":45240,"temperature":0.7,"pith_summary":"The paper claims that adding knowledge-graph (KG) embeddings as an extra input modality—continuous vectors injected through a lightweight adapter, not retrieved text—reduces hallucination in large language models on fact-focused benchmarks. The authors build WikiEntities, a 3.2-million-text Wikipedia dataset annotated with Wikidata entities and TransE embeddings, train a Text2Graph mapper that predicts KG embeddings from text, and train a single linear adapter that projects those embeddings into the LLM's input space. Across Mistral 7B, LLaMA 2-7B, and LLaMA 3-8B, they report improved hallucination detection on HaluEval and True-False and improved fact verification on FEVER, with little change on general reasoning benchmarks. The practical upshot, if correct, is a parameter-efficient way to make LLMs more factual without retrieval pipelines or fine-tuning the base model.","feed_headline":"KG embeddings as second modality cut LLM hallucinations","feed_subtitle":"A lightweight adapter injects Wikidata/TransE embeddings into Mistral, LLaMA 2, and LLaMA 3, lifting HaluEval, True-False, and FEVER scores.","key_machinery":"The central mechanism is the two-stage injection of KG embeddings into the LLM's input sequence. First, a Text2Graph mapper—RoBERTa-large with an unfrozen encoder and a linear head, trained with MSE against PyTorch-BigGraph TransE embeddings of Wikidata entities—converts a text span into a predicted KG embedding. Second, a single linear adapter, trained with cross-entropy language modeling on WikiEntities texts, projects that KG embedding into the LLM embedding space, and the result is concatenated to the token embeddings between special tokens `<GRAPH_START>` and `<GRAPH_END>`. The mapper is independent of the LLM; the adapter is retrained per model. WikiEntities supplies the paired data: 3.2M Wikipedia texts with Wikidata entity spans, IDs, and corresponding embedding lookups.","core_discovery":"The central discovery is that a frozen LLM can consume knowledge-graph embeddings as a discrete additional modality—wrapped in special tokens `<GRAPH_START>` and `<GRAPH_END>`—and use them to judge and generate more factually. The authors show that the Text2Graph mapper, a RoBERTa-large encoder with a linear head trained under MSE loss on WikiEntities, yields KG embeddings that, when projected by a trained linear adapter, improve hallucination detection accuracy on HaluEval (e.g., LLaMA 2-7B from 0.468 to 0.546 average) and True-False (Mistral 7B from 0.81 to 0.97 average), and FEVER verification (Mistral from 0.665 to 0.743), while MMLU, GSM8k, and other general benchmarks stay approximately flat. The method explicitly avoids external retrieval and keeps the LLM frozen; only the adapter and the mapper are trained. For the HaluEval QA task, the authors use ground-truth entity embeddings from the dataset rather than Text2Graph predictions, while other tasks rely on predicted KG embeddings.","pith_inferences":["If the transfer from oracle to predicted embeddings holds up, the same adapter-modality trick could be applied to other structured knowledge sources (e.g., relation triples or numeric facts) and other base models, since the mapper and adapter are lightweight and model-agnostic.","A natural, testable extension is to report entity-linking accuracy and KG embedding prediction error on the evaluation sets; without those numbers, the observed gains could partly be driven by the dataset's entity distribution rather than by the KG semantics.","The method could be combined with retrieval-augmented generation: retrieved entities could be fed as KG embeddings rather than text, potentially reducing the token cost of RAG while keeping factual grounding.","Because the adapter is trained with ground-truth embeddings on Wikipedia text, its robustness on out-of-domain or time-sensitive facts—where Wikidata embeddings may be stale—is an open question the paper does not address."],"forward_implications":["Adding KG embeddings as a modality improves hallucination detection and fact verification by about 2 to 10 percentage points absolute on HaluEval, True-False, and FEVER across the three tested LLMs, with the largest gains on LLaMA 2-7B and Mistral 7B.","The approach can be adapted to any LLM by training only a linear adapter, since the Text2Graph mapper is model-agnostic and the underlying KG embedding space stays fixed.","Because the LLM is frozen and no retrieval is used, the method adds factual grounding without changing general-task behavior; the reported MMLU, GSM8k, Winogrande, HellaSwag, and ARC scores remain roughly flat.","The WikiEntities dataset of over 3 million Wikipedia texts annotated with Wikidata entities and spans can itself serve as training or evaluation data for entity linking models.","In the HaluEval QA task, using predicted KG embeddings slightly outperforms using the provided entity embeddings for Mistral (0.521 vs 0.516), suggesting the mapper adds signal beyond exact entity lookup."],"supporting_citations":[{"why":"PyTorch-BigGraph produces the TransE entity embeddings that define the KG embedding space used throughout the paper.","marker":"[21]"},{"why":"TransE is the KG embedding model whose training objective yields the entity vectors predicted by Text2Graph.","marker":"[3]"},{"why":"RoBERTa-large is the base encoder for the Text2Graph mapper that converts text spans into predicted KG embeddings.","marker":"[29]"},{"why":"Wikidata is the source of entities and entity IDs used to annotate the WikiEntities dataset.","marker":"[38]"},{"why":"HaluEval is the hallucination detection benchmark (QA, dialogue, summarization) used to evaluate the method.","marker":"[23]"},{"why":"The True-False dataset from Azaria and Mitchell is used to evaluate the models' ability to judge statement correctness.","marker":"[2]"},{"why":"FEVER is the fact verification dataset used to measure the method's effect on claim verification.","marker":"[36]"},{"why":"Mistral 7B is one of the three base LLMs for which a KG-adapter is trained and evaluated.","marker":"[16]"},{"why":"LLaMA 2-7B is one of the base LLMs for which a KG-adapter is trained and evaluated.","marker":"[37]"},{"why":"LLaMA 3-8B is one of the base LLMs for which a KG-adapter is trained and evaluated.","marker":"[1]"}],"fun_headline_variants":["KG embeddings as extra modality slash LLM hallucinations","Adapter injects knowledge graph embeddings to cut LLM falsehoods","Frozen LLMs with KG embeddings reduce hallucinations","No retrieval or fine-tuning: KG embeddings improve factual LLMs","Knowledge graph embeddings for LLMs curb hallucination errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training pipeline feeds the adapter ground-truth Wikidata entity embeddings, but at test time the adapter receives predicted KG embeddings from Text2Graph; the paper reports no entity-linking accuracy or embedding prediction error on the evaluation sets, and for the HaluEval QA task it falls back to provided embeddings, so the transfer from oracle to noisy inputs is an unmeasured load-bearing premise.","fun_headline_variants_meta":{"raw":{"variants":["KG embeddings as extra modality slash LLM hallucinations","Adapter injects knowledge graph embeddings to cut LLM falsehoods","Frozen LLMs with KG embeddings reduce hallucinations","No retrieval or fine-tuning: KG embeddings improve factual LLMs","Knowledge graph embeddings for LLMs curb hallucination errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000675,"raw_usage":{"total_tokens":3114,"prompt_tokens":1031,"completion_tokens":2083,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":647,"completion_tokens_details":{"reasoning_tokens":2004}},"tokens_in":647,"tokens_out":2083,"duration_ms":14933,"temperature":1.0,"reasoning_tokens":2004,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:24:02.586017+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the entity-linking precision and recall of the Text2Graph mapper on the evaluation inputs (HaluEval questions, True-False statements, FEVER claims) and correlate those errors with downstream accuracy; also compare the full pipeline against an oracle version that uses gold KG embeddings on every task. If downstream gains largely vanish when gold embeddings are used on tasks currently scored with predicted embeddings, the claim that predicted KGs drive the improvement is undercut.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PyTorch-BigGraph produces the TransE entity embeddings that define the KG embedding space used throughout the paper."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TransE is the KG embedding model whose training objective yields the entity vectors predicted by Text2Graph."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Wikidata is the source of entities and entity IDs used to annotate the WikiEntities dataset."}],"review_version":1}