{"id":"303e4b3f-1707-4806-ad48-3e9d8cc59461","arxiv_id":"2411.11247","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"ZeFaV is a zero-shot prompting framework that adds relation triples and reorganized evidence to improve LLM fact-checking, with inconsistent gains over baselines.","lead":"ZeFaV combines relation extraction and evidence re-organization to prompt a large language model for zero-shot fact verification. It reports F1 gains on two multi-hop benchmarks, but the gains are mixed and the comparisons are not fully controlled.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ZeFaV's own ablation undercuts its core claim: adding the relation-extraction module to InfoRE lowers F1 on HoVer 3-hop/4-hop and FEVEROUS-S, so the reported boost is not established.","rationale":"The strongest version of the paper's claim is that ZeFaV, as a combination, beats prior methods; but the more precise scientific claim is that the proposed relation-extraction module is what boosts performance. The paper's only evidence for this is Table 2, and that table shows the module hurting on three of four settings when context is present and on all settings when context is absent. That is an internal inconsistency, not just an issue of statistical power: the conclusion asserted in the text contradicts the numbers reported. I therefore treat this as the load-bearing concern. The confounding baseline issue (QACheck on InstructGPT, InfoRE on Llama-2, only ProgramFC rerun on the same model) also matters for the headline comparison, but even if baselines were matched, the paper would still need to show that its own component, not just InfoRE, contributes; Table 2 fails to do so. A focused paired comparison with confidence intervals would settle whether the observed deficits are noise or real. The absence of relation-extraction quality metrics further weakens the causal story, but the primary failure is that the paper's own ablation does not support the 'boost' claim.","tokens_in":5333,"tokens_out":6277,"duration_ms":61926,"concrete_test":"Re-run the fact-verification stage on the same HoVer and FEVEROUS-S splits with relation extraction toggled on/off (InfoRE and all other settings fixed, same Meta-Llama-3-70B-Instruct, same decoding, 3+ seeds) and compute paired bootstrap 95% confidence intervals for per-claim F1 differences. If the interval for relation-on minus relation-off overlaps zero on either dataset, or is negative on 3-hop/FEVEROUS-S, the claimed relation-induced boost is not established. To identify the mechanism, also hand-evaluate extracted triples on 100 randomly sampled claims and count how often Algorithm 1 drops an evidence triple needed to verify the claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that ZeFaV's FewRel-fine-tuned relation extraction stage boosts zero-shot fact verification beyond InfoRE alone. The paper's ablation, Table 2, isolates this exactly: with evidence context, replacing ZeFaV full (Relation+InfoRE) with InfoRE-only raises F1 on HoVer 3-hop from 70.61 to 72.03, on HoVer 4-hop from 67.47 to 67.64, and on FEVEROUS-S from 86.74 to 86.89; without evidence context, the same replacement raises F1 on all four settings (e.g., FEVEROUS-S 75.97 to 76.11). Since these are paired comparisons of the proposed addition, and no error bars or significance tests are reported, the conclusion in Section 4.2 that 'both InfoRE and Relation help increase the performance' is not supported by its own table. The main comparative advantage in Table 1 could therefore be carried by InfoRE and by the stronger base model rather than by ZeFaV's novel relation-extraction/closure mechanism. The mechanism is also plausible to fail: Algorithm 1 discards evidence relations whose head entity is not in the claim-relation hypothesis set, and the paper gives no measurement of FewRel relation-extraction recall/precision on HoVer or FEVEROUS claims, so the observed drops may reflect silently removed relevant evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ZeFaV, a pipeline for zero-shot fact verification with large language models. The method first fine-tunes an LLM on FewRel for relation extraction, then reorganizes the evidence with the InfoRE technique, and finally combines the claim, the original evidence, the reorganized evidence, and extracted claim/evidence relations in a prompt that asks the LLM to return True or False. Experiments on HoVer and FEVEROUS-S with Meta-Llama-3-70B-Instruct report F1 scores that are claimed to outperform prior methods. However, the paper's own ablation in Table 2 shows that the relation-extraction component often lowers F1 relative to using InfoRE alone, and the main comparisons in Table 1 are confounded by the use of different base models across methods.","tokens_in":5577,"tokens_out":4869,"duration_ms":48188,"significance":"The idea of using extracted relations as structured hints for zero-shot fact verification is plausible, and the paper makes its code available and evaluates on public datasets, which are positive features. If the claimed gains were robust, ZeFaV would be a useful addition to the fact-checking toolkit. However, the central contribution—the FewRel-fine-tuned relation-extraction and closure-filtering component—is not supported by the paper's own ablation, and the headline comparison against InfoRE and QACheck does not isolate the proposed mechanism from the choice of the base LLM. The empirical support is therefore currently insufficient to establish the claimed improvement.","major_comments":[{"comment":"The main comparison against InfoRE and QACheck is not controlled: ZeFaV uses Meta-Llama-3-70B-Instruct, while InfoRE uses Llama-2-70B and QACheck uses InstructGPT (Section 4.1 and Table 1). Any observed improvement could be due to the stronger base model rather than to ZeFaV's relation-extraction and InfoRE components. The text also claims that 'ZeFaV obtained better results than other methodologies on the HoVer dataset,' but Table 1 shows ProgramFC scoring 70.84 on HoVer 3-hop, higher than ZeFaV's 70.61. In addition, the FEVEROUS-S F1 is reported as 86.54 in the text but 86.74 in Table 1. The comparison should be rerun with a common base model and the numerical inconsistencies corrected.","section":"4.1, Table 1"},{"comment":"The ablation directly contradicts the conclusion that 'both InfoRE and Relation help increase the performance.' Replacing the full ZeFaV (Relation+InfoRE) with InfoRE-only raises F1 on HoVer 3-hop from 70.61 to 72.03, on HoVer 4-hop from 67.47 to 67.64, and on FEVEROUS-S with evidence context from 86.74 to 86.89; the same replacement raises F1 in all four settings without evidence context (e.g., FEVEROUS-S from 75.97 to 76.11). No error bars, confidence intervals, or significance tests are reported, so even the direction of the paired comparison is not statistically established. The paper's central claim that the relation-extraction module boosts fact verification is therefore unsupported by its own ablation.","section":"4.2, Table 2"},{"comment":"The closure filter in Algorithm 1 is a plausible source of the observed degradation: it retains an evidence relation only if its head entity appears in the claim-hypothesis set, and the paper provides no measurement of relation-extraction precision or recall on HoVer or FEVEROUS claims. If the FewRel-fine-tuned model produces incomplete or noisy triples, the filter can silently discard evidence relations needed for verification, which would explain the F1 drops in Table 2. The authors should report relation-extraction quality on the target datasets and provide at least one qualitative example of the filter's behavior on the cases where adding the relation component lowers accuracy.","section":"Algorithm 1 and Section 3"},{"comment":"The method is described as 'zero-shot,' but the relation-extraction stage is fine-tuned on the FewRel dataset and the InfoRE stage uses few-shot prompting examples. If 'zero-shot' is intended only for inference on the target fact-verification datasets, this should be stated explicitly; otherwise the title's zero-shot claim is misleading and the comparison with few-shot methods such as ProgramFC needs to account for the fine-tuning budget used by ZeFaV's relation-extraction component.","section":"3 and 4.1"}],"minor_comments":[{"comment":"The phrase 'potential results results' contains a duplicated word and should be corrected, likely to 'promising results.'","section":"Abstract"},{"comment":"The caption contains the typo 'Emprical results' and the dataset name is rendered as 'HoV er'; both should be fixed.","section":"Table 1 caption"},{"comment":"The prompt template for relation extraction does not specify how the full FewRel relation list is inserted, whether all 80 relations are used, or how the LLM is fine-tuned (training split, number of examples, epochs, learning rate, LoRA settings). These details are needed for reproducibility.","section":"Section 3"},{"comment":"The prose acknowledges that performance 'slightly decreases when combined with the relation' on HoVer 3-hop and FEVEROUS-S, yet the following sentence claims both components help; this internal inconsistency should be resolved by reporting the paired differences with error bars and a clear interpretation.","section":"Section 4.2"},{"comment":"The challenge type 'Combining Tables and Tex' contains a typo and should be 'Combining Tables and Text.'","section":"Table 3"},{"comment":"The confusion matrix in Figure 2 is referenced but not described in sufficient detail; the claim that the model predicts refuted claims as supported claims should be accompanied by the actual counts or percentages from the confusion matrix.","section":"Figure 2"}],"recommendation":"reject","confidential_remarks":"The paper's central empirical claim is contradicted by its own ablation in Table 2, and the main comparison in Table 1 is confounded by different base models. The contribution is not established as submitted; a convincing revision would require a controlled comparison on a common base model, error bars or significance tests, and a demonstration that the relation-extraction component helps rather than hurts. The authors have made code available, which is a positive factor if they choose to resubmit after substantial additional experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ZeFaV is a prompt-engineering paper. The pipeline is: fine-tune a relation extractor on FewRel, extract claim and evidence relations, run a closure filter to drop evidence relations whose head entity is not in the claim's hypothesis set, reorganize the evidence with InfoRE, and feed all of that to Llama-3-70B for zero-shot fact verification. The specific combination is new, the authors release code, and they evaluate on HoVer and FEVEROUS-S with an ablation and a challenge-type breakdown. That is real work, and the full ablation table is honestly reported.\n\nThe problem is what the ablation says. Table 2 compares ZeFaV full (relation + InfoRE) with InfoRE-only. With evidence context, removing the relation module improves F1 on HoVer 3-hop (72.03 vs 70.61), HoVer 4-hop (67.64 vs 67.47), and FEVEROUS-S (86.89 vs 86.74). Without evidence context, InfoRE-only beats the full pipeline on all four settings. So the one genuinely new component — the FewRel relation extraction plus closure — does not help; it mostly hurts. Section 4.2 says \"both InfoRE and Relation help increase the performance,\" which is not a reading of its own table. The observed advantage over InfoRE in Table 1 is better explained by the base model (Llama-3 vs Llama-2) and by InfoRE itself.\n\nOther soft spots are smaller but real. The zero-shot label is imprecise because the relation extractor is fine-tuned on FewRel. The comparison against InfoRE and QACheck uses different base models, so the headline numbers are confounded; the only same-model baseline, ProgramFC, beats ZeFaV on HoVer 3-hop (70.84 vs 70.61), which undercuts the claim of being better on HoVer. There are no error bars or significance tests, and the code link has no commit hash or fine-tuning details. The paper also never measures relation extraction precision/recall on these datasets, so it cannot tell whether the closure filter is dropping useful evidence.\n\nNone of this makes the paper worthless. The method is plausible as a recipe, and the result \"InfoRE helps, relation filtering doesn't\" is a useful negative result if the authors are willing to report it that way. But the central claim — that ZeFaV boosts zero-shot fact verification — is not supported by the evidence in the paper. A serious referee should see it, mostly to force the authors to either drop the relation-module claims or rerun controlled same-base-model experiments with significance. As submitted, it should be rejected; with an honest reframing it could be a modest empirical note.","headline":"ZeFaV's own ablation shows its novel relation-extraction module hurts more than it helps, so the central claim that ZeFaV boosts zero-shot fact verification does not survive its data.","tokens_in":6113,"tokens_out":5387,"would_cite":false,"duration_ms":52133,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ZeFaV claims that adding claim-anchored relation triples and reorganized evidence to an LLM prompt improves zero-shot fact verification enough to beat earlier LLM methods on HoVer and FEVEROUS-S.","keywords":["fact verification","zero-shot learning","large language models","relation extraction","in-context learning","multi-hop reasoning","evidence reorganization","chain-of-thought prompting"],"falsifier":"One decisive check would be to annotate, for a random sample of claims, whether the closure-filtered evidence relations retain the evidence sentences that determine the verdict; if a large fraction of verdict-relevant evidence is filtered out, then the relation stage cannot be doing the grounding work claimed, and the reported gains would stem from InfoRE and context alone. A simpler probe is to reproduce Table 2 on 3-hop HoVer with a higher-quality relation extractor: the current numbers show adding relations to InfoRE lowers F1 from 72.03 to 70.61, so a corrected extractor should flip that drop if relations truly help.","tokens_in":5117,"feed_emoji":"✅","tokens_out":10816,"duration_ms":95789,"temperature":0.7,"pith_summary":"This paper tries to establish that zero-shot fact verification with large language models can be improved by changing what goes into the prompt rather than by adding training data or switching to a larger model. The proposed pipeline, ZeFaV, extracts entity-relation triples from the claim and evidence, keeps only evidence relations connected to the claim's entities through a closure procedure, rewrites the evidence as a hierarchical structure, and then asks the model to decide True or False with all of this context. On HoVer and FEVEROUS-S, the paper reports F1 scores that beat several prior LLM-based fact-checking methods, including a few-shot program-guided baseline, while using an open-weights 70B model in a zero-shot setting. If correct, this would mean a cheap, reusable recipe: few-shot relation extraction plus evidence reorganization can give a general-purpose LLM the structured context it needs for multi-hop verification.","feed_headline":"Relation-guided prompts beat ProgramFC on two fact-checking sets","feed_subtitle":"ZeFaV adds relation triples and hierarchical evidence to the prompt, lifting zero-shot F1 on two fact-checking benchmarks.","key_machinery":"The mechanism is a three-stage prompt construction pipeline. Stage one uses a relation-extraction prompt fine-tuned on FewRel to turn a sentence into (head, relation, tail) triples. Stage two applies a closure filter: seed a hypothesis set with the entities appearing in the claim's triples, then repeatedly add every evidence triple whose head entity is already in the set and add its tail to the set until no further addition is possible; this is meant to keep exactly the evidence relations connected to the claim and discard unrelated ones. Stage three uses InfoRE, a prompt that reorganizes evidence into a hierarchical structure with indentation marking subordination, and then the final verification prompt concatenates the original evidence, the organized evidence, the claim, and both relation lists with a chain-of-thought instruction.","core_discovery":"ZeFaV's claim is that a zero-shot LLM verifier can be made substantially better by restructuring the evidence and grounding it with relations before asking for a verdict. The pipeline extracts (head, relation, tail) triples from the claim and evidence, filters the evidence triples to those reachable from claim entities, rewrites the evidence into a hierarchical form, and then feeds the original evidence, rewritten evidence, and relation lists to the LLM with a chain-of-thought True/False instruction. On HoVer (2-/3-/4-hop) and FEVEROUS-S, the paper reports F1 scores of 77.85, 70.61, 67.47, and 86.74, respectively, comparing favorably with ProgramFC, QACheck, and InfoRE. The paper also reports an ablation in which removing the original evidence context or InfoRE hurts more than removing the relation component, and in which the relation component alone is the weakest single addition.","pith_inferences":["The FewRel-trained extractor is implicitly treated as a portable relation vocabulary for fact verification, so the same extractor could be dropped into other multi-hop QA tasks without per-task relation annotation.","A testable extension is to convert tables and numbers into triples before the closure step, since the weakest FEVEROUS-S category is numerical reasoning; if that conversion lifts the 72.83% F1, the bottleneck is representation rather than the verifier.","The closure filter is a graph-reachability selection rule, which suggests the active ingredient may be entity-anchored subgraph selection rather than relation semantics; replacing InfoRE with any evidence selector that keeps passages sharing claim entities would isolate whether the relational form itself matters."],"forward_implications":["Zero-shot verification can reach the level of a few-shot program-decomposition system when the prompt includes relation triples, hierarchically reorganized evidence, and the original evidence.","The original evidence context and the InfoRE rewrite carry most of the accuracy gain; the relation component is a smaller add-on that can reduce F1 on some subsets (3-hop HoVer and FEVEROUS-S) when InfoRE is already present.","The pipeline can be applied to new claims without in-domain training examples, so the same FewRel-tuned extractor and prompts transfer to any evidence text.","Structured and numerical evidence remains the main limitation, with the lowest F1 on FEVEROUS-S numerical reasoning."],"supporting_citations":[{"why":"This supplies the HoVer and FEVEROUS-S evaluation splits and the ProgramFC baseline that ZeFaV is compared against.","marker":"[14]"},{"why":"This provides the InfoRE evidence-reorganization method used in the third stage of the pipeline.","marker":"[5]"},{"why":"This provides the FewRel dataset and relation types used to fine-tune the relation extraction model.","marker":"[6]"},{"why":"This supplies the QACheck baseline and the recommendation to use locally run open-source LLMs.","marker":"[13]"},{"why":"This supplies the FEVEROUS-S dataset used in the zero-shot evaluation.","marker":"[2]"},{"why":"This supplies the HoVer multi-hop dataset used in the 2-, 3-, and 4-hop evaluations.","marker":"[9]"},{"why":"This gives the LoRA 4-bit quantization method that makes the fine-tuning step run on a single GPU.","marker":"[8]"},{"why":"This provides the chain-of-thought prompting mechanism used in the final verdict prompt.","marker":"[16]"}],"fun_headline_variants":["Relation triples and hierarchical evidence boost zero-shot fact checks","ZeFaV: zero-shot fact verification with relation-extracted prompts","Grounding LLMs with relations improves fact-checking F1","Zero-shot fact-checker: relation triples beat ProgramFC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the relation extractor fine-tuned on FewRel produces triples that are complete and accurate enough on HoVer and FEVEROUS-S that the closure filter keeps the evidence needed to decide the claim and the relation lists guide the verifier rather than mislead it, which the paper does not directly test and which the ablation suggests may fail on some subsets.","fun_headline_variants_meta":{"raw":{"variants":["Relation triples and hierarchical evidence boost zero-shot fact checks","ZeFaV: zero-shot fact verification with relation-extracted prompts","Grounding LLMs with relations improves fact-checking F1","Zero-shot fact-checker: relation triples beat ProgramFC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1277,"prompt_tokens":853,"completion_tokens":424,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":353}},"tokens_in":469,"tokens_out":424,"duration_ms":4476,"temperature":1.0,"reasoning_tokens":353,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:44:20.505764+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One decisive check would be to annotate, for a random sample of claims, whether the closure-filtered evidence relations retain the evidence sentences that determine the verdict; if a large fraction of verdict-relevant evidence is filtered out, then the relation stage cannot be doing the grounding work claimed, and the reported gains would stem from InfoRE and context alone. A simpler probe is to reproduce Table 2 on 3-hop HoVer with a higher-quality relation extractor: the current numbers show adding relations to InfoRE lowers F1 from 72.03 to 70.61, so a corrected extractor should flip that drop if relations truly help.","supporting_citations":[{"cited_title":"In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","cited_arxiv_id":null,"evidence_quote":"This supplies the HoVer and FEVEROUS-S evaluation splits and the ProgramFC baseline that ZeFaV is compared against."},{"cited_title":"In: Proceedings of the 2019 Conference on Empirical MethodsinNatural LanguageProcessing andthe9th InternationalJointConferenceon Natural Language Processing (EMNLP-IJCNLP)","cited_arxiv_id":null,"evidence_quote":"This provides the FewRel dataset and relation types used to fine-tune the relation extraction model."},{"cited_title":"In: Proceedings of the 2023 Conference on Empirical Methods in Nat- ural Language Processing: System Demonstrations","cited_arxiv_id":null,"evidence_quote":"This supplies the QACheck baseline and the recommendation to use locally run open-source LLMs."},{"cited_title":"In: Proceedings of the Fourth Workshop on Fact Ex- traction and VERification (FEVER)","cited_arxiv_id":null,"evidence_quote":"This supplies the FEVEROUS-S dataset used in the zero-shot evaluation."}],"review_version":1}