{"id":"be8e01b0-ba17-4985-88d2-06e2d35cf172","arxiv_id":"2607.17935","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"DeLIVeR uses GRPO-trained question planning over knowledge graphs to report F1 83.7/84.6/79.7 on LIAR/FEVER/PolitiFact, claiming ~10-15% relative gains over HippoRAG2.","lead":"DeLIVeR is a fact-checking pipeline that trains a 'planner' AI to turn a claim into targeted questions, pull evidence from a knowledge graph, and let a frozen language model issue the verdict. The paper reports large F1 gains, but the experimental setup is underspecified and contains internal inconsistencies.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The F1 gains may be an artifact of KG leakage: §3.3 says claims/labels are excluded, but Table 4 lists LIAR KG as 'Political claims and entities'; LIAR has no evidence corpus, so the central comparison is unscorable as described.","rationale":"The reader's weakest assumption and my concern coincide: the empirical headline cannot be audited because the evidence corpus and train/test split are unspecified. I sharpen it with an internal inconsistency: §3.3's guarantee that claim text and labels are excluded is incompatible with Table 4's description of the LIAR KG as 'Political claims and entities', and LIAR's dataset format makes a ground-truth evidence corpus impossible without an external source that the paper never identifies. This is a correctness risk, not an accusation: the issue could be resolved by releasing the provenance trace and re-running the evaluation with leaked triples removed. Since the reader already issued a CONDITIONAL verdict with high correctness risk, my reading does not change that verdict; if the concrete test confirms leakage, the appropriate verdict would be REJECT.","tokens_in":16561,"tokens_out":7385,"duration_ms":72054,"concrete_test":"Request or reconstruct the provenance map for every KG triple (the paper says this metadata exists). For a random sample of 200 LIAR and 200 PolitiFact test claims, trace each retrieved evidence node's source_doc_uri. If any source document is the claim's own LIAR row or the fact-check article containing its verdict, label leakage is confirmed. Then re-run the Qwen2.5-7B evaluation after deleting all triples whose source URI contains the claim sentence or any verdict/rationale phrase, on the same split. If F1 drops by more than ~2 points on either dataset, the 10–15% headline cannot be attributed to the planner.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — peak F1 83.73/84.57/79.70 and a 10–15% improvement over HippoRAG2 (§4.2, Table 5) — depends on the KG evidence being non-leaky. §3.3 states that the KG is built from 'ground-truth evidence corpora associated with each dataset' and that claim text and veracity labels are explicitly excluded. That precondition is not met for at least two datasets. LIAR has no ground-truth evidence corpus: it is a collection of short statements with labels, so there is no corpus from which to build the described KG. Table 4's KG statistics describe LIAR as 'Political claims and entities', which suggests claim text is present in the graph despite the §3.3 exclusion. PolitiFact is described as a 'Fact-checking corpus', but fact-check articles typically restate the claim and contain the verdict rationale, so provenance metadata alone does not prevent label leakage. If KG nodes include claim statements or verdict-bearing text, the retrieval step can surface gold-label information directly; the reported F1 then measures retrieval of leaked evidence, not the GRPO question-planning policy. The paper also never specifies train/validation/test splits for the GRPO-trained planner, which is a second, independent route to inflated scores.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DeLIVeR, a fact-verification framework in which a GRPO-optimized 'Planner' LLM decomposes a claim into 4–8 questions that retrieve evidence from a knowledge graph; a frozen Verifier LLM then outputs True/False/NEI with an explanation. The central empirical claim is that, using Qwen2.5-7B, DeLIVeR achieves peak F1-scores of 83.73 on LIAR, 84.57 on FEVER, and 79.70 on PolitiFact, a 10–15% improvement over HippoRAG2. The paper also reports ablations on question-set size, a qualitative evaluation, and an error analysis of 200 PolitiFact failures.","tokens_in":16898,"tokens_out":4838,"duration_ms":44995,"significance":"If the reported numbers are genuine, the contribution is a useful and modular architecture: reinforced question planning over structured knowledge graphs for fact-checking, with auditable evidence chains and a stable GRPO training objective. The paper gives clear motivation, detailed hyperparameters, and an ablation of the central design choice (number of questions). However, the empirical claim is not currently supported: the evidence corpora used to build the knowledge graphs are not identifiable for two of the three datasets, the train/validation/test protocol is unspecified, and the reward used for GRPO training is not connected to gold labels. These are load-bearing gaps in an otherwise well-scoped system paper. No code or data is released, so the results cannot be independently checked.","major_comments":[{"comment":"The KG-construction protocol cannot be satisfied for LIAR and PolitiFact. LIAR has no ground-truth evidence corpus, and PolitiFact fact-check articles typically restate the claim and the verdict rationale. Table 4 describes the LIAR KG as 'Political claims and entities,' which directly contradicts §3.3's claim that claim text and veracity labels are excluded from graph construction. If the KG contains claim statements or verdict-bearing text, retrieval can surface gold-label information directly, making the reported F1 gains an artifact of leakage rather than of the GRPO planner. The authors must identify the exact source documents for each KG, demonstrate that no claim text or labels enter the graph, and report leakage checks (e.g., overlap between retrieved triples and gold evidence/verdict text).","section":"§3.3, Table 4"},{"comment":"No train/validation/test split is specified for the GRPO-trained planner or the final evaluation. Table 3 only reports total dataset counts, and Table 5 gives no confidence intervals, standard errors, or the macro/micro averaging scheme for precision/recall/F1 over the three verdict classes. Because Table 4 states that early stopping is done on validation F1, the absence of an explicit split means the reported test numbers cannot be audited; overlap between training and test claims is a second, independent route to inflated scores. Provide exact splits, per-class metrics, and a statement of how the three-class F1 is averaged.","section":"§4.1, Table 3; §4.2, Table 5"},{"comment":"The accuracy reward R_Accuracy is defined as a binary 'information is correct' signal from an LLM judge or fine-tuned classifier, but no mapping from the judge's SUPPORTS/REFUTES/NEI output to the 0/1 reward is given, and the judge's agreement with gold labels is not reported. If the reward is not calibrated to the gold veracity labels, GRPO may optimize a proxy that is unrelated to the F1 numbers in Table 5. Specify the reward computation, report judge accuracy, or change the reward to use the gold labels directly.","section":"§3.6, Eq. (8)"},{"comment":"The paper claims 'high-precision evidence' retrieval, yet no retrieval-quality metric (evidence precision, recall, hit rate against gold evidence, or path accuracy) is reported anywhere. The only reported quantities are final verdict F1 and accuracy. Since the contribution is explicitly about improving evidence retrieval, the absence of retrieval-side metrics means the mechanism responsible for any F1 gain cannot be identified—the gain could come from the verifier's prompt, the KG size, or label leakage rather than from the question-planning policy. At minimum, report retrieval precision/recall on FEVER, which has gold evidence annotations.","section":"§4.2, Table 5"},{"comment":"The error-analysis percentages sum to 112% (48 + 34 + 16 + 14), and the text says that Irrelevant Questions (16%) and Redundant Questions (14%) 'together account for only 30%,' which is internally inconsistent with both the sum and the claim that these two categories are a minor component. The four error categories are evidently not mutually exclusive; the overlap must be defined and the percentages corrected. As reported, the error analysis does not support the conclusion that GRPO effectively reduces irrelevant/redundant questions.","section":"§4.3, Table 6"}],"minor_comments":[{"comment":"The acronym in the abstract is rendered inconsistently ('DeLIVeR' vs. 'VeacityRecognition'), which interferes with readability.","section":"Abstract"},{"comment":"The table caption says 'MultiReQA Datasets,' but the experiments are on LIAR, FEVER, and PolitiFact. MultiReQA is not mentioned anywhere else and the caption should be corrected.","section":"§4.2, Table 5"},{"comment":"The qualitative evaluation text lists five scores for six categories: '83.2 (Knowledge-ability, Relevance), 75.12 (Comprehensiveness), 82.45 (Factuality), 68.4 (Logical Coherence), and 75.25 (Correctness)'—83.2 is assigned to two categories. Clarify whether these are separate or joint scores.","section":"§4.3"},{"comment":"The GRPO advantage in Eq. (5) uses a normalization function F_norm (e.g., standard deviation), but the worked example in Figure 3 computes advantages using only the group mean without normalization. Align the figure with the equation.","section":"§3.6, Figure 3"},{"comment":"The sentence 'The performance of the set of questions is refined iteratively using GRPO, detailed in Section 3.4' is a self-reference; GRPO is described in Section 3.6.","section":"§3.4"}],"recommendation":"reject","confidential_remarks":"The decisive issue is the contradiction between §3.3's exclusion of claim text from the KG and Table 4's description of the LIAR KG as 'Political claims and entities.' For LIAR, no ground-truth evidence corpus exists; for PolitiFact, fact-check articles normally encode the verdict. This is not a presentation issue: the experiments need to be redone with identifiable, non-leaky evidence corpora and explicit splits before the central claim can be evaluated. I therefore recommend rejection, although a carefully redone version of this system could be a reasonable contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: this paper's headline result — 10–15% F1 gains over HippoRAG2 from GRPO-optimized question planning over knowledge graphs — is plausible but currently unauditable. The method itself is a reasonable integration of existing pieces: claim decomposition, KG retrieval, GRPO with diversity and accuracy rewards. That integration is genuinely new, as far as I can tell from the cited prior work, and the auditable evidence-path framing is nice for real-world fact-checking.\n\nWhat it does well: the pipeline is clearly described, the GRPO setup is standard, and the reward design is sensible in principle. The ablation on question set size is a nice touch. The motivation — that static RAG queries are brittle for multi-hop claims — is valid.\n\nThe soft spots are not minor. First, the leakage problem. Section 3.3 says the KG is built only from ground-truth evidence corpora and that claim text and labels are explicitly excluded. But for LIAR there is no ground-truth evidence corpus; the original dataset contains only statements and labels. Table 4 describes the LIAR KG as 'Political claims and entities,' which is direct evidence that claim text is in the graph, contradicting §3.3. For PolitiFact, the 'fact-checking corpus' will restate claims and verdicts, so label leakage via retrieval is a real risk. If retrieval surfaces verdict-bearing text, the reported F1 measures leaked evidence, not the GRPO policy.\n\nSecond, the evaluation protocol is underspecified: no train/validation/test split for the GRPO-trained planner, no evidence corpora identified for LIAR/PolitiFact, no retrieval quality metrics, and no code/data. The baselines are black boxes. There are also internal slips: the error analysis percentages sum to 112%, Table 5 is labeled 'MultiReQA' though the datasets are LIAR/FEVER/PolitiFact, and the accuracy reward is defined by an LLM judge whose relation to gold labels is never stated. The 'ideal category distribution C*' in the structure reward is said to come from oracle analysis, which is a red flag unless it is defined on training data only.\n\nThe central claim is not definitionally circular — the test F1 numbers, if genuine, are on unseen claims. But because the KG may encode the answers, the comparison is unscorable as described. These are addressable issues: release data/code, specify splits, clarify how LIAR/PolitiFact KGs were constructed and audited for leakage, and add retrieval quality metrics.\n\nWho is this for: anyone working on RL-based retrieval or KG-grounded fact-checking will want to know this exists, but the results should not be cited as evidence until the protocol is fixed. A serious referee should see it, because the method is worth evaluating properly and the current version has specific, correctable flaws.\n\nMy recommendation: send it to peer review, but with a strong request for a revised evaluation. As it stands, I would not rely on the reported numbers.","headline":"Plausible new integration of GRPO question planning and KG retrieval, but the headline F1 gains are unauditable because the evidence protocol is underspecified and LIAR appears to leak claims into the KG.","tokens_in":17372,"tokens_out":4778,"would_cite":false,"duration_ms":43223,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DeLIVeR claims that fact-checking improves when a language model learns to plan a small, diverse set of questions for querying a knowledge graph, lifting F1 by 10–15 percent over static retrieval.","keywords":["fact-checking","knowledge graph","question generation","reinforcement learning","retrieval-augmented generation","veracity classification","GRPO","claim decomposition"],"falsifier":"An independent replication that builds the knowledge graph only from the official evidence documents, splits the planner's training, validation, and test sets before any training, and then reproduces the 10–15 percent F1 margin over the same static baselines would confirm the claim; a large drop or a reversal would falsify it.","tokens_in":16429,"feed_emoji":"🕵️","tokens_out":4867,"duration_ms":45149,"temperature":0.7,"pith_summary":"DeLIVeR aims to show that treating evidence retrieval as a learned, strategic search over a knowledge graph beats treating it as a single fixed query. A planner language model decomposes each claim into four to eight targeted questions; those questions retrieve structured evidence, and a frozen verifier renders a True/False/Not-Enough-Information verdict. The planner is trained with group-relative policy optimization, with the reward weighted heavily toward structural diversity of the question set. Reported results on three public fact-checking benchmarks put the approach about 10–15 percent higher in F1 than the strongest static retrieval baseline, with the best scores on a 7-billion-parameter model. If right, the lesson is that retrieval policy—not just model scale—is a major lever for veracity accuracy.","feed_headline":"Asking four well-chosen questions beats one fixed query in fact-checking","feed_subtitle":"A reinforcement-trained planner over knowledge graphs lifts F1 by 10–15 percent over static retrieval.","key_machinery":"The central object is the question-generation 'planner': a language model that maps a claim to a set of 4–8 questions, optimized by GRPO. The carrying mechanism is the structure reward, which classifies each question into semantic categories (who, what, when, where, how, contradiction) and rewards coverage of the ideal category set; it carries the most weight (0.60), meaning the policy is trained primarily to ask questions that span distinct subgraphs of the knowledge graph. Retrieval uses cosine similarity between question embeddings and graph element embeddings, and every graph element keeps provenance back to a source document so the verifier can audit paths.","core_discovery":"The paper claims that a reinforced question-planning policy over a knowledge graph substantially outperforms static retrieval for claim verification. Given a claim, a secondary LLM generates a small, diverse set of questions; each question is embedded and matched to nodes and edges of a knowledge graph built from ground-truth evidence; the combined evidence is passed to a frozen verifier. A GRPO loop with a reward weighted toward structural diversity (0.60 weight) refines the question set to cover different semantic facets of the claim. The empirical claim is peak F1 of about 83.7, 84.6, and 79.7 on LIAR, FEVER, and PolitiFact, respectively, which the authors attribute to better evidence cov","pith_inferences":["The same reinforced question-planning recipe could extend to other knowledge-intensive verification tasks, such as medical claims, product safety, or financial statements, wherever a clean evidence corpus can be turned into a knowledge graph; the paper does not test that transfer.","The paper's dependence on a ground-truth evidence corpus per dataset is the biggest open question: a testable extension is to build the knowledge graph from a generic encyclopedic corpus and measure how much the F1 margin shrinks.","The reward design could be strengthened by penalizing document mismatch directly, since the error analysis identifies it as the dominant failure mode; a version of the reward based on retrieved-document overlap with gold evidence would test that."],"forward_implications":["If correct, standard retrieval-augmented pipelines should be re-thought: replacing a single fixed query with a learned, diverse question set can yield large gains in evidence quality and verdict accuracy, independent of scaling the verifier.","The approach makes fact-checking decisions auditable end-to-end: each verdict is backed by the generated questions, the retrieved knowledge-graph paths, and their source documents–useful for high-stakes deployment.","The four-question optimum suggests a practical design rule: small, non-redundant question sets beat larger ones, because extra questions add retrieval noise.","The error analysis indicates that document mismatch and insufficient coverage together account for most failures, so the next incremental gain lies in aligning questions to evidence documents rather than in adding more questions."],"fun_headline_variants":["Fact-checking with a question-planning LLM beats static retrieval","Reinforced question sets lift fact-check F1 by 10–15%","LLM planner asks diverse questions to verify claims","DeLIVeR: RL-trained question planning improves fact-checking","Reinforcement learning for evidence retrieval in fact-checking"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that each dataset has a trustworthy, claim-free evidence corpus from which a knowledge graph can be built without label leakage, and that the reinforced planner is trained and evaluated on clean, non-overlapping splits; for two of the three benchmarks no such corpus is explicitly identified.","fun_headline_variants_meta":{"raw":{"variants":["Fact-checking with a question-planning LLM beats static retrieval","Reinforced question sets lift fact-check F1 by 10–15%","LLM planner asks diverse questions to verify claims","DeLIVeR: RL-trained question planning improves fact-checking","Reinforcement learning for evidence retrieval in fact-checking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000158,"raw_usage":{"total_tokens":1068,"prompt_tokens":758,"completion_tokens":310,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":221}},"tokens_in":502,"tokens_out":310,"duration_ms":3779,"temperature":1.0,"reasoning_tokens":221,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T16:34:29.615939+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An independent replication that builds the knowledge graph only from the official evidence documents, splits the planner's training, validation, and test sets before any training, and then reproduces the 10–15 percent F1 margin over the same static baselines would confirm the claim; a large drop or a reversal would falsify it.","supporting_citations":[],"review_version":1}