{"id":"3a264897-8b1c-4623-a25d-721e8a0dc2e2","arxiv_id":"2507.03671","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Removing annotator cues from fact-checking evidence lowers LLM scores substantially, and a three-agent question-answering pipeline, RAV, outperforms several published fact-checking baselines.","lead":"The authors built PFO, a fact-checking benchmark that strips verdict hints out of PolitiFact evidence, and found that large language models score lower when those hints are removed. They also present RAV, a three-agent question-answering pipeline that reports gains over several published fact-checking baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PFO's core premise is not met: evidence comes from post-hoc fact-checking articles with verdict cues only partially deleted, so the 22% drop does not isolate pre-claim information.","rationale":"Good-faith reading: the authors are not claiming RAV is state of the art on FEVEROUS; they explicitly report losing to ProgramFC there. The manual annotation process, Fleiss' kappa of 0.7092, and the ablation and iteration analyses are real evidence of careful work. The RAV relative gains on RAWFC and HOVER are arithmetically consistent with the cited baselines (the reader verified 25.28% and 1.54/4.94/1.78%). The load-bearing issue is the PFO construction, because the paper's novelty and its headline drop both depend on the claim that PFO contains only pre-claim information. The reader's weakest_assumption points at the same place; I partially agree because my concern is broader: even flawless deletion from a post-hoc article does not recreate a pre-claim evidence text, and the residual examples in Figures 11-12 and the Limitation show deletion is not flawless. The 22% figure also does not match Table 2's macro-F1 column, which adds reporting uncertainty to the conceptual problem. A temporal-provenance audit would settle whether PFO really approximates pre-claim evidence. If it does not, the abstract's guarantee must be weakened and the 22% drop re-framed. Since the dataset may still be useful as a harder, partially scrubbed benchmark and RAV has independent value, conditional acceptance remains appropriate; the reader already reached CONDITIONAL, so I recommend no change to the verdict.","tokens_in":18727,"tokens_out":6960,"duration_ms":86884,"concrete_test":"Audit a random sample of 100 PFO instances with a temporal-provenance test. For each retained evidence sentence, check whether the information (not just the underlying fact) appears in a document timestamped before claim_date, using the factcheck_analysis_link, the article's cited sources, and the Internet Archive. Record (a) the fraction of PFO evidence that is verbatim or near-verbatim pre-claim text versus post-claim article prose, and (b) the number of retained sentences containing explicit judgments, directives, or arithmetic (e.g., 'Do the math', 'no ordinary collision'). If post-claim prose exceeds roughly 20% of sentences, or if any such cue sentence remains, PFO does not satisfy the abstract's guarantee and the 22% drop cannot be attributed to cue removal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"PFO's central guarantee (Abstract; Section 4) is that models are evaluated 'using only the information that would have been available prior to the claim's verification.' The load-bearing premise is that manually deleting post-claim analysis and annotator cues from PolitiFact articles produces evidence equivalent to pre-claim information. That premise is not met. PFO evidence is extracted from fact-checking articles written after the claim; even if deletion were perfect, the retained string is post-hoc editorial prose, not a contemporaneous source. Deletion is also demonstrably imperfect: Figure 11 (half-true) retains the fact-checker's arithmetic, 'Do the math... That produces a total of $363 billion, well below the lowest estimate,' which is post-claim reasoning pointing at the verdict; Figure 12 (mostly-false) retains 'But this was no ordinary collision; it involved multiple vehicles that were not all pictured,' an interpretive refutation of the claim. The Limitation concedes 'some sentences could not be eliminated without compromising the context necessary to support or refute the claim.' Hence PFO is a partially scrubbed post-hoc corpus, not the pre-claim-only benchmark the abstract claims. If residual cues and post-hoc framing remain, the headline 22% drop (which is not reproducible from the macro-F1 columns of Table 2; it matches the average absolute micro-F1 drop) measures the gap between unfiltered and partially filtered articles, not the cost of removing annotator cues. RAV's comparative 'least drop' numbers inherit this confound.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper makes two main contributions. First, it introduces Politi-Fact-Only (PFO), a 5-class fact-checking dataset of 2,982 political claims derived from PolitiFact, in which the authors manually removed post-claim analysis and annotator cues. The stated aim is to evaluate models using only information available before the claim was verified, and the paper reports that zero-shot LLMs drop by 22% in macro-F1 on PFO relative to its unfiltered version. Second, it proposes RAV (Recon-Answer-Verify), an agentic pipeline with a question generator, an answer generator, and a label generator that iteratively decomposes claims into sub-questions; RAV is evaluated on RAWFC, LIAR-RAW, PFO, HOVER, and FEVEROUS, with reported improvements over published baselines on RAWFC and HOVER. The paper also includes ablations of question-generation strategies, question types, prompt sensitivity, and a human error analysis of RAV outputs.","tokens_in":18814,"tokens_out":4405,"duration_ms":49905,"significance":"If the PFO premise is accepted, the resource would be valuable: measuring the effect of annotator cues on LLM fact-checking is an important and understudied problem, and the manual curation effort, the inter-annotator agreement (Fleiss' kappa of 0.71), and the explicit documentation of annotation guidelines are strengths. The RAV pipeline is a reasonable agentic design, and the ablation study comparing iterative versus all-at-once question generation and verification versus inquiry questions is informative. The prompt-sensitivity analysis in Appendix A and the human error attribution in Appendix D are also useful practical contributions. However, the central quantitative claims, especially the 22% macro-F1 drop and the unconditional \"outperforms state of the art\" statements, are not fully supported by the tables as presented, and the PFO construction's core premise requires additional evidence before the headline results can be interpreted as measuring real-time verification.","major_comments":[{"comment":"The claim that PFO contains \"only the information that would have been available prior to the claim's verification\" is not established. The evidence is extracted from fact-checking articles written after the claim, and even after manual deletion the retained text includes post-claim reasoning and verdict-relevant arithmetic. For example, Figure 11 keeps \"Do the math: the sum exceeds $490.2 billion, much higher than even the highest estimate\" and \"That produces a total of $363 billion, well below the lowest estimate,\" and Figure 12 keeps \"But this was no ordinary collision; it involved multiple vehicles that were not all pictured,\" which is an interpretive rebuttal of the claim. The Limitation section explicitly concedes that \"some sentences could not be eliminated without compromising the context necessary to support or refute the claim.\" Therefore the 22% drop reported in the abstract is not demonstrably the cost of removing annotator cues; it may reflect partially scrubbed post-hoc articles. To make the PFO claim load-bearing, the authors should either re-scope the dataset description as \"manually scrubbed PolitiFact articles\" or provide a direct test of residual cue leakage, such as a blind annotator study measuring whether retained evidence still reveals the verdict, or a comparison against genuinely pre-claim corpora (e.g., WatClaimCheck or CofCED-style documents).","section":"Section 4, Figures 11-12, Limitation"},{"comment":"The headline \"average performance drop of 22% in terms of macro-f1\" is not reproducible from Table 2. The macro-F1 drops for the three models are: Mistral-7B-v0.3, 0.37 to 0.26 (absolute 0.11); LLaMA-3.1-8b, 0.48 to 0.21 (absolute 0.27); Gemma-2-9b, 0.59 to 0.20 (absolute 0.39). The average absolute macro-F1 drop is approximately 0.257, and the relative drops are much larger (roughly 30%, 56%, and 66%). The value 0.22 matches the average absolute micro-F1 drop (0.45 to 0.34, 0.51 to 0.28, and 0.60 to 0.28). The paper should correct this claim, report both relative and absolute drops, and state explicitly which macro-F1 columns support the abstract's numbers. This is a load-bearing error because the 22% figure is prominently presented in the abstract and contribution list.","section":"Abstract, Section 8, Table 2"},{"comment":"The claim that RAV \"outperforms state-of-the-art approaches on RAWFC by 25.28%\" is only true for the phi-4 backbone: RAV(phi-4) on RAWFC is 0.6753 versus HiSS's 0.5390, a 25.3% relative improvement. The abstract does not specify the backbone, and Table 3 shows that RAV(LLaMA-3.1-70b) improves over HiSS by only about 9.8% on RAWFC. Similarly, on HOVER 2-hop, RAV(phi-4) at 0.7558 is slightly below ProgramFC's 0.7565, so the \"1.54%\" improvement is only from the 70B model. These statements need qualification by backbone. In addition, the baselines ProgramFC and HiSS use text-davinci-003 (175B), so the comparison conflates model choice with method effectiveness; the paper should acknowledge this confound and, ideally, report variance or significance across multiple runs, since all reported numbers appear to come from single executions.","section":"Section 8, Table 3, Table 4"}],"minor_comments":[{"comment":"Table 9 reports a total of 2,981 instances for the unfiltered set, while Section 4 and Table 1 state 2,982; this inconsistency should be fixed.","section":"Table 9"},{"comment":"The sentence \"On this contains around 21k instances\" is grammatically broken and should be rewritten.","section":"Section 4"},{"comment":"The table header labels all columns \"Macro-F1,\" but the ProgramFC baselines for HOVER are likely accuracy values from the original paper; the authors should clarify which metric is being reported for each baseline.","section":"Table 3"},{"comment":"The notation r*_QG and r*_LG is used in Algorithm 1 but not defined in the text; please define these symbols when the agents are introduced.","section":"Section 5, Algorithm 1"},{"comment":"Instruction 5 says to write \"yes\" or \"no\" in the \"leaked\" field, but it is unclear whether \"yes\" indicates that the evidence required changes; please clarify the intended semantics.","section":"Appendix B.3"},{"comment":"For RAWFC and LIAR-RAW, the authors follow prior work in treating author-written explanations as gold evidence; since those explanations may themselves contain verdict cues, the cross-dataset comparison is not apples-to-apples, and this should be stated as a limitation.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a real problem and contains a substantial amount of useful experimental work, but the central quantitative claims need correction and the PFO premise needs stronger validation before publication. I would encourage the editors to request a revision rather than reject, because the underlying contribution is potentially valuable if the claims are re-scoped and the numerical reporting is fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about arXiv:2507.03671. First, the RAV pipeline is a legitimate, reasonably well-evaluated contribution: the three-agent iterative QA design generalizes across three datasets and five label sets, and the gains on RAWFC (25.28%) and HOVER are real relative improvements over published baselines. Second, the paper's central empirical claim—that LLMs drop 22% in macro-F1 when annotator cues are removed—does not hold up as stated. PFO is a partially scrubbed post-hoc corpus, not a pre-claim-only benchmark, and the abstract's 22% figure is actually the average absolute micro-F1 drop, not macro.\n\nWhat's good: The paper identifies a genuine problem (leakage in fact-checking benchmarks) and builds a useful artifact, PFO, by manually deleting verdict-cue sentences from PolitiFact articles. The deletion is careful and documented, with annotator instructions and inter-annotator agreement (Fleiss' kappa 0.71). RAV is a sensible decomposition of fact-checking into question generation, grounded answering, and label prediction, with a stop signal; the ablations show that iterative mixed-type questioning helps. The human evaluation of misclassified instances (Appendix D) is a nice touch and shows where the pipeline fails. The paper honestly reports losing to ProgramFC on FEVEROUS.\n\nWhere it's soft: The load-bearing premise of PFO is that deleting post-hoc sentences yields evidence equivalent to what existed before the claim was made. That's not met. Even the paper's own examples keep interpretive judgments (\"But this was no ordinary collision...\") and fact-checker arithmetic (\"That produces a total of $363 billion, well below the lowest estimate\"), which point at the verdict. The Limitation concedes \"some sentences could not be eliminated.\" So the 22% drop measures the gap between unfiltered and partially filtered articles, not the cost of truly pre-claim information. The numbers also don't add up: the abstract's 22% macro-F1 matches the average absolute micro-F1 drop across the three zero-shot models (0.11, 0.23, 0.32 average 0.22), not any macro-F1 column; and the 16.3% \"least drop\" is LLaMA-70B's relative macro drop, while Section 8 reports the same quantity as 7.36% absolute. The appendix also lists 2,982 instances in Table 8 and 2,981 in Table 9. These inconsistencies suggest the paper's quantitative claims were not checked carefully. Also, the best RAV variant and the k=10 iteration cap were chosen using the test sets themselves, with no variance or significance testing, so the reported gains should be treated as provisional.\n\nWho this is for: researchers working on fact-checking benchmarks and agentic verification pipelines. The RAV architecture is worth replicating, and PFO, despite its flaws, is a useful resource for studying cue sensitivity. But the paper needs a serious revision to fix the overclaiming and the arithmetic before it's citable as-is.\n\nRecommendation: Send it to peer review. The core idea is sound, the pipeline is interesting, and the problems are fixable. Reviewers should push for a corrected benchmark with a clearer statement of what PFO actually is, and for consistency between abstract and body.","headline":"Useful pipeline, overstated benchmark: the RAV agentic design is solid, but PFO's cleanup doesn't isolate pre-claim information and the headline 22% drop is a mislabeled micro-F1 number.","tokens_in":19581,"tokens_out":3714,"would_cite":false,"duration_ms":36831,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A cue-free fact-checking benchmark drops LLM macro-F1 by about 22 percent, showing how much models rely on leaked verdicts.","keywords":["fact checking","large language models","benchmark dataset","information leakage","agentic pipeline","claim verification","political claims","question decomposition"],"falsifier":"Take a random sample of PFO claims, retrieve the actual documents, transcripts, or data releases that existed on or before the claim date, and compare zero-shot macro-F1 on those true pre-claim documents with macro-F1 on PFO's scrubbed evidence; a large gap would show that PFO's manual deletion does not reproduce pre-claim information conditions.","tokens_in":1507,"feed_emoji":"🔍","tokens_out":1397,"duration_ms":73601,"temperature":0.7,"pith_summary":"The paper argues that existing fact-checking benchmarks overstate model ability because their evidence passages contain the fact-checker's verdict and interpretive commentary. To match real-time verification, it introduces PFO, a five-class benchmark of 2,982 political claims whose evidence has been manually stripped of all post-claim analysis. On PFO, zero-shot LLMs lose about 22% macro-F1 on average compared with the unfiltered version, quantifying how much of their apparent skill came from annotator cues. The paper then proposes RAV, a three-agent pipeline that decomposes a claim into subquestions, answers them from evidence, and generates a label, and reports that it outperforms published baselines on RAWFC and HOVER while degrading less than baselines on PFO.","feed_headline":"Cue-free benchmark cuts LLM fact-check scores by 22 percent","feed_subtitle":"Removing annotator hints from PolitiFact evidence reveals how much models lean on leaked verdicts.","key_machinery":"The argument is carried by two constructed objects: PFO, a manually scrubbed five-class benchmark, and RAV, a three-agent agentic pipeline. PFO operationalizes 'no leakage' by removing post-claim verdict sentences and annotator commentary from PolitiFact evidence, leaving only facts that existed before the claim was published. RAV breaks a claim into a sequence of subquestions generated without access to evidence, answers each subquestion from the gold evidence, and then labels based on the question-answer history; its iterative question generation, mixing true/false verification questions with open inquiry questions, is what allows it to generalize across domains and label granularities.","core_discovery":"The central claim is that leakage in fact-checking datasets materially inflates LLM evaluation: when evidence is reduced to factual content that existed before the claim was published, models perform far worse than when the evidence includes the fact-checker's post-publication analysis. PFO operationalizes this by manually deleting verdict sentences, label definitions, and interpretive commentary from PolitiFact articles, keeping only pre-claim facts. The paper's pipeline RAV—Recon, Answer, Verify—uses a question generator that iteratively asks subquestions without seeing the evidence, an answer generator that answers each subquestion from the evidence, and a label generator that predicts the final veracity label from the question-answer history. RAV outperforms state-of-the-art baselines on RAWFC and HOVER, and on FEVEROUS it trails ProgramFC; it also shows a smaller performance drop on PFO than the zero-shot baselines, with the best backbone losing only 7.36% macro-F1.","pith_inferences":["Editorial inference: the same manual scrubbing procedure could be applied to other fact-checking corpora, and PFO's measured 17.79% average length reduction offers a rough bound on how much leakage those datasets may contain.","Editorial inference: if PFO's filtering faithfully simulates pre-claim evidence, then the 22% gap is a direct measure of model reliance on leaked cues; a natural validation would compare PFO against true contemporaneous documents from the claim date.","Editorial inference: RAV's subquestion counts could serve as an automatic difficulty score, letting dataset curators stratify benchmarks by reasoning complexity without additional human annotation."],"forward_implications":["Benchmark scores on unfiltered fact-checking datasets overstate real-world performance by roughly a fifth in macro-F1, because models exploit verdict cues that would not exist during real-time verification.","Fact-checking systems should be evaluated on evidence that predates the claim, and PFO provides one such test set for political claims with five label granularities.","An iterative question-answer decomposition improves veracity prediction across 2-class, 3-class, and 5-class settings, outperforming program-guided and hierarchical baselines on most tested benchmarks.","Allowing the question generator to mix verification and inquiry questions helps more than using either type alone, and iterative questioning beats generating all questions at once.","The number of subquestions RAV generates tracks dataset difficulty classes such as HOVER's hop count, suggesting a measurable notion of claim reasoning complexity."],"supporting_citations":[{"why":"Supplies the PolitiFact dataset from which PFO is curated.","marker":"(Misra, 2022)"},{"why":"LIAR is the early benchmark whose leakage motivates the creation of PFO.","marker":"(Wang, 2017)"},{"why":"ProgramFC is the program-guided baseline RAV is compared against on HOVER and FEVEROUS.","marker":"(Pan et al., 2023)"},{"why":"CofCED is a supervised baseline, and its treatment of author-written explanations as gold evidence is reused for LIAR-RAW and RAWFC.","marker":"(Yang et al., 2022)"},{"why":"HiSS is the hierarchical step-by-step prior work that RAV extends with iterative question generation.","marker":"(Zhang and Gao, 2023)"},{"why":"HOVER provides the many-hop encyclopedia benchmark used to evaluate RAV across 2-hop, 3-hop, and 4-hop claims.","marker":"(Jiang et al., 2020)"},{"why":"MultiFC is one of the leaked-evidence datasets used in the zero-shot comparison against PFO.","marker":"(Augenstein et al., 2019)"}],"fun_headline_variants":["Cue-free PolitiFact: LLM fact-check scores drop 22%","Without post-hoc hints, LLM fact-checking falls 22%","RAV: agentic fact-checker resists leaked verdicts","Fact-check benchmark strips annotator cues, LLMs lose 22%","Agentic RAV outperforms baselines on leakage-free claims"],"cache_read_input_tokens":21376,"weakest_assumption_plain":"The load-bearing premise is that manually deleting verdict-like sentences and annotator commentary from fact-checking articles leaves evidence equivalent to what a fact-checker would have had before the claim was published.","fun_headline_variants_meta":{"raw":{"variants":["Cue-free PolitiFact: LLM fact-check scores drop 22%","Without post-hoc hints, LLM fact-checking falls 22%","RAV: agentic fact-checker resists leaked verdicts","Fact-check benchmark strips annotator cues, LLMs lose 22%","Agentic RAV outperforms baselines on leakage-free claims"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000412,"raw_usage":{"total_tokens":2189,"prompt_tokens":1059,"completion_tokens":1130,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":675,"completion_tokens_details":{"reasoning_tokens":1035}},"tokens_in":675,"tokens_out":1130,"duration_ms":13788,"temperature":1.0,"reasoning_tokens":1035,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:07:53.412704+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of PFO claims, retrieve the actual documents, transcripts, or data releases that existed on or before the claim date, and compare zero-shot macro-F1 on those true pre-claim documents with macro-F1 on PFO's scrubbed evidence; a large gap would show that PFO's manual deletion does not reproduce pre-claim information conditions.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CofCED is a supervised baseline, and its treatment of author-written explanations as gold evidence is reused for LIAR-RAW and RAWFC."}],"review_version":1}