{"id":"270fc567-640d-4984-9521-4eea8ddde4a7","arxiv_id":"2502.08909","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In a benchmark of 17,856 PolitiFact claims, larger Llama-3 models and retrieved web evidence improve automated fact-checking accuracy and justification quality, though fine-grained labels remain difficult.","lead":"This paper compares how well different sizes of Llama-3 language models can verify real-world political claims from PolitiFact, with and without web search evidence. It finds that larger models and added evidence both improve accuracy, but distinguishing fine-grained truth labels remains hard.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evidence-retrieval condition is vulnerable to temporal label leakage: current web snippets for historical PolitiFact claims can contain post-hoc verdicts, so the observed evidence benefit may largely reflect leaked answers, especially for larger models.","rationale":"The paper's central claim is plausible and the experimental effort is substantial, but the condition that must hold for the evidence-integration conclusion is that retrieved snippets are evidence rather than answers. Section 4.5 explicitly states the filter is intended to reduce, not eliminate, information leakage, and the temporal design makes leakage likely because current search results for historical claims inevitably include post-hoc reporting. The reader's weakest assumption identified this same area; I sharpen it as a temporal leakage problem rather than a mere keyword-bypass problem. The proposed Wayback Machine test is decisive: if the evidence benefit persists with pre-publication snapshots, the central claim survives; if it collapses, the reported improvements are inflated by leaked verdicts and the paper should be revised to separate genuine evidence grounding from answer extraction. Because the concern is concrete and testable, and the paper already flags part of the risk, the appropriate verdict remains CONDITIONAL rather than REJECT.","tokens_in":13108,"tokens_out":4865,"duration_ms":54293,"concrete_test":"Use the Internet Archive CDX API to retrieve contemporaneous web snapshots from before each PolitiFact article's publication date for a random sample of at least 300 claims; rerun the evidence-augmented conditions in Tables 4–6 on this temporally clean evidence instead of current Serper snippets. Compare accuracy and TIGERScore to the reported with-evidence numbers. If the evidence benefit shrinks by more than a few F1 points, or the larger-model advantage narrows, temporal leakage is confirmed and the central claim requires qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is in §4.5: top-10 snippets are retrieved via Serper today for claims published 2007–2024. The web index is retrospective: a query for a historical claim returns articles written after the fact-check, which often state the verdict even when they are not fact-checking sites (e.g., news coverage of the ruling or correction). The keyword filter ('PolitiFact', 'fact-check', 'debunk') only removes a subset; verdict words like 'false', 'no evidence', 'rating', or 'pants on fire' and indirect descriptions are not filtered. This means the with-evidence condition is not purely evidence-based verification—it may include reading the answer from the search snippet. The observed pattern in Tables 4–6, where evidence helps every model and helps larger models most, is exactly what one would expect if leakage were being exploited: larger LLMs are better at resolving a snippet that contains the verdict. The authors acknowledge the risk in §4.5 ('reduce information leaking in') but do not quantify it. If leakage is substantial, hypotheses H3 and H4, and the central claim about retrieval-augmented AFC, are not established by these experiments.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a one-shot LLM-based framework for automated fact-checking that jointly produces a step-by-step reasoning trace, a veracity verdict, and a natural-language explanation. Using 17,856 PolitiFact claims (2007–2024), it compares Llama-3.2-3B, Llama-3.1-8B, Llama-3.1-70B, and Llama-3.3-70B under binary, three-class, and five-class label schemes, in settings with and without web evidence retrieved via the Serper API, and additionally fine-tunes ModernBERT-large as an upper-bound classifier. The main reported findings are that evidence integration improves performance across all models, larger models benefit most from evidence, larger LLMs surpass fine-tuned SLMs, and smaller LLMs are comparable to fine-tuned SLMs in some settings.","tokens_in":13346,"tokens_out":6742,"duration_ms":63279,"significance":"If the findings hold, the paper provides a useful baseline for retrieval-augmented automated fact-checking with open-weight LLMs, together with a structured output format for transparent justifications. The study's strengths include its large corpus, multiple label granularities, comparison across model scales from the same architecture family, structured decoding via vLLM/outlines, the use of TIGERScore for justification evaluation, and statistical testing of the hypotheses. The central evidence-integration claim, however, is endangered by temporal label leakage in the retrieval protocol, and Table 6 contains internally inconsistent classification metrics. These issues must be resolved before the headline conclusions can be accepted.","major_comments":[{"comment":"The evidence-retrieval design is vulnerable to temporal label leakage. The claims span 2007–2024, but snippets are retrieved at experiment time via Serper, so the current web index can return post-hoc articles, corrections, and news coverage that disclose the PolitiFact verdict. The filter removes known fact-checking sites and keywords such as 'PolitiFact', 'fact-check', and 'debunk', but it does not remove general news reports, court documents, or commentary that restate the verdict; even the example in Table 3 includes an Aug 16, 2017 snippet stating that 'The Times subsequently issued a correction'. Because hypotheses H3 and H4 rest on the size of the evidence benefit, the reported improvements may partly reflect the model reading the answer from the snippet rather than verifying the claim. I ask for a leakage audit: sample a few hundred claims, annotate whether the retrieved snippets contain verdict-bearing information (explicit labels, 'false', 'corrected', 'no evidence', 'pants on fire', etc.), and report the evidence-condition results on the subset free of such leakage; alternatively, use temporally constrained retrieval with snapshots dated before the fact-checking ruling and rerun the main comparisons.","section":"§4.5"},{"comment":"The binary classification metrics in Table 6 are mutually inconsistent as labeled. For single-label predictions, micro-F1 equals accuracy, yet entries such as F1_micro=0.624 with Accuracy=0.504 for 3.2-3B-Instruct without evidence cannot both be correct as labeled; the same pattern occurs in every binary row. This makes the binary results and the abstract's claim about smaller LLMs being comparable to fine-tuned SLMs difficult to interpret: with the accuracy column, the 3.2-3B model with evidence (0.557) is far below ModernBERT with evidence (0.718), whereas with the F1 columns the gap is only 0.647 vs 0.696. Please recompute and clearly define all metrics, and state which numbers support the claims in the abstract and §5.3.","section":"Table 6"},{"comment":"The fine-tuning setup for ModernBERT-large is not specified. The manuscript does not report the train/validation/test split, hyperparameters (learning rate, batch size, number of epochs, early stopping, seed averaging), or whether the split is performed by claim or by speaker/context to prevent near-duplicate leakage. It also does not state how evidence snippets are concatenated and truncated to 8,192 tokens, nor whether the 667 claims without retrieved evidence are dropped from all conditions. Without these details, the claimed upper-bound comparison between one-shot LLMs and fine-tuned SLMs is not reproducible, and the conclusion that larger LLMs 'consistently surpass' fine-tuned SLMs is not fully established.","section":"§5.3, Table 7"},{"comment":"The statistical tests are underspecified. The paired t-tests for the evidence benefit do not state the pairing unit (per claim, per model, or per label scheme) or the number of comparisons, and pooling across models and label schemes would violate independence and inflate significance. The Friedman tests similarly mix model size and evidence conditions without a clear repeated-measures design. Please report the exact test design, sample sizes, effect sizes, and multiple-comparison corrections for the large number of comparisons in Tables 4–6.","section":"§5.3"}],"minor_comments":[{"comment":"The statement that smaller LLMs provide comparable task performance to fine-tuned SLMs should be qualified by label scheme: the 3-class and 5-class results are indeed comparable, but the binary accuracy gap in Table 6 (0.557 vs 0.718 for 3.2-3B and ModernBERT with evidence) is substantial.","section":"Abstract and §5.3"},{"comment":"The description of label aggregation is confusing: 'the original labels true and false are grouped into mostly true and mostly false, respectively' should read 'true is merged into mostly true, and false is merged into mostly false'. The binary merging of half-true into mostly true is described correctly.","section":"§4.4"},{"comment":"There are cross-reference errors: the claim example appears in Figure 1 and the snippets in Table 3, but the text refers to 'Table 1' and 'Table 2' for these; 'The output in Table 2 demonstrates' should be 'The output in Figure 2 demonstrates'.","section":"§5.2"},{"comment":"The list of excluded fact-checking sites and the exact keyword filter are not provided; please include them so that the leakage mitigation can be assessed and reproduced.","section":"§4.5"},{"comment":"The paper relies on TIGERScore as a reference-free metric for justification quality, but it does not validate TIGERScore against human judgments for fact-checking justifications. Since H2 and parts of H3/H4 depend on this metric, a small human-correlation study or sensitivity analysis would strengthen the conclusions.","section":"§4.6"},{"comment":"No code, prompts, retrieved evidence, or model outputs are released, which hinders reproducibility of this otherwise empirical study. Please consider making these available.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical study with a potentially useful baseline, but the two main issues—temporal label leakage in evidence retrieval and the inconsistent binary metrics in Table 6—strike at the central claims. The leakage concern is especially important because the evidence benefit is the main contribution. I would be willing to reconsider after a revision that includes a leakage audit and corrected, fully specified experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper's central claim—that evidence retrieval helps LLMs fact-check—is built on a retrieval setup that likely leaks the answer. Web snippets are retrieved today for claims from 2007–2024, and news coverage written after the fact-check often states the verdict. The authors filter out \"PolitiFact\" and \"fact-check\" but not \"false\", \"correction\", \"no evidence\", or indirect descriptions. Table 3's own example shows a snippet saying the Times issued a correction—that's the ground truth sitting in the evidence. So the evidence benefit in Tables 4–6, and the claim that larger models benefit most, is exactly what you'd expect from models reading the answer out of the snippets. The authors acknowledge the risk in §4.5 but don't quantify it. This is a real weak spot, not a nicety.\n\nSecond, the abstract overstates the small-model result. The 3B one-shot model is comparable to fine-tuned ModernBERT in the 5-class and 3-class schemes, but in binary classification it's 0.557 vs 0.696 micro-F1 with evidence (Table 6 vs Table 7). That's a large gap, and the abstract says \"comparable task performance\" without qualification.\n\nWhat's genuinely useful: this is a systematic baseline on 17,856 real PolitiFact claims, covering three label granularities, four Llama sizes, with and without evidence, and it adds a fine-tuned ModernBERT upper-bound comparison. The structured output (reasoning, verdict, explanation) is a sensible task formulation, and TIGERScore is applied consistently. The paper is clearly written and the statistical tests are appropriate for what they test.\n\nThere are smaller issues: fine-tuning details for ModernBERT are missing (no hyperparameters, no data split description), and no code or data artifacts are released. Using an LLM as a judge (TIGERScore) to score LLM justifications is a known circularity, but it's not the main result.\n\nWho gets value: anyone building an AFC baseline or comparing open-weight Llama models on this task. The paper is a reasonable starting point, but the evidence-condition results should not be trusted as evidence about RAG until the leakage is addressed (e.g., retrieve documents timestamped before the fact-check, or probe how often snippets contain verdict words). It deserves a serious referee, and if I were reviewing it, I'd ask for major revision on the leakage point and a corrected abstract.","headline":"Solid baseline study for LLM fact-checking, but the evidence condition leaks post-hoc verdicts into the retrieval snippets, and the abstract oversells small-model parity.","tokens_in":13850,"tokens_out":3134,"would_cite":false,"duration_ms":30717,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large language models, given web search snippets and a one-shot prompt, can fact-check real-world claims as accurately as fine-tuned small classifiers, and larger models benefit most from the evidence.","keywords":["automated fact-checking","large language models","retrieval-augmented generation","PolitiFact","TIGERScore","label schemes","one-shot prompting","evidence integration"],"falsifier":"Compute a held-out correlation between the words in the retrieved snippets and the model's predicted verdict: if, for example, snippets for claims labeled 'False' frequently contain the word 'false' or 'debunk' despite the filter, and models that see those snippets predict 'False' more often than models that do not, the evidence-benefit result would be an artifact of label leakage rather than genuine evidence use. A cleaner test would compare evidence-augmented accuracy against a control where snippets are replaced by unrelated web pages.","tokens_in":12943,"feed_emoji":"🔎","tokens_out":5444,"duration_ms":48618,"temperature":0.7,"pith_summary":"This paper tries to establish a baseline for automated fact-checking with large language models: given a real-world claim and a handful of web search snippets, a one-shot prompt that asks the model to reason, classify, and explain can yield accurate verdicts and useful justifications without fine-tuning. Using 17,856 PolitiFact claims and Llama-3 models of 3B, 8B, and 70B parameters, the authors find that larger models consistently outperform smaller ones on both classification accuracy and justification quality. They also find that adding retrieved evidence improves performance for every model, and that the largest models benefit the most, contrary to the expectation that smaller models would gain more from external knowledge. The result matters because it suggests a feasible path for practical fact-checking assistance: open-weight LLMs with retrieval, rather than costly fine-tuned systems, can provide transparent, explainable verdicts on real-world claims.","feed_headline":"On 17,856 claims, larger LLMs outperform smaller and fine-tuned models","feed_subtitle":"A one-shot Llama-3 setup with search snippets beats fine-tuned small classifiers on PolitiFact verdicts and justifications.","key_machinery":"The central object is a one-shot, retrieval-augmented prompting framework with constrained decoding. For each claim, the model receives a system prompt that defines the labeling scheme, a user message containing the speaker, context, claim, and (when used) the top-10 web search snippets, and a single assistant example; the model must output a JSON object with 'reasoning', 'verdict', and 'explanation'. The framework's work is carried by the combination of chain-of-thought reasoning, structured output enforcement via constrained generation, and the reference-free TIGERScore metric (which assigns penalty scores to errors in the generated justification) for evaluation.","core_discovery":"The central claim is that retrieval-augmented, few-shot inference with LLMs provides a viable and transparent alternative to fine-tuned small models for automated fact-checking of real-world claims. On a dataset of 17,856 PolitiFact claims, the authors evaluate Llama-3 instruction-tuned models at three sizes under binary, three-class, and five-class labeling schemes, prompting each model to produce a step-by-step analysis, a verdict, and a natural-language explanation in a structured JSON format. Across all schemes, the 70B models beat the 8B and 3B models, and the 3.3-70B model reaches a binary F1 of 0.747 with evidence, surpassing the fine-tuned ModernBERT classifier's 0.718. The paper also reports that evidence retrieval (top-10 web snippets with fact-checking sites filtered out) significantly improves both classification and justification quality for all models, with the largest models showing the largest gains. Distinguishing fine-grained labels remains hard, and the authors reject the hypothesis that justification quality degrades with label complexity.","pith_inferences":["The paper's evidence-leakage filter is imperfect; an extension that audits snippets for verdict-related wording could determine how much of the evidence benefit is genuine grounding rather than label leakage.","The framework's structured reasoning-plus-verdict-plus-explanation output could be adopted as an evaluation protocol for other real-world claim datasets, not just PolitiFact, since it does not rely on dataset-specific fine-tuning.","Because the authors find evidence helps most when labels are coarse, a natural next step is adaptive label granularity: use fine labels when evidence is strong and fall back to coarse labels when it is weak.","The 667 claims for which no evidence was retrieved could serve as a natural stress test for whether models know when to abstain, since the framework currently forces a verdict even without evidence."],"forward_implications":["If the central claim holds, automated fact-checking systems can be built on open-weight LLMs with no task-specific fine-tuning, reducing development cost and allowing rapid adaptation to new claims.","Evidence integration is the most reliable performance lever: adding top-10 web snippets improves classification and justification quality across every model size and label scheme, so future systems should invest in retrieval quality.","Larger models are the right target for retrieval-augmented fact-checking: the 70B models benefit most from evidence and surpass fine-tuned small classifiers, so compute spent on scale pays off in this task.","Fine-grained five-class verdicts are near the limit of what current LLMs can reliably distinguish, so practical deployments may need coarser schemes or additional evidence types.","The ModernBERT fine-tuning results confirm that evidence is the most informative input component, ahead of speaker identity and context, so claim-evidence alignment should be a design focus."],"supporting_citations":[{"why":"Defines the task formulation of automated fact-checking as verdict prediction plus justification production, which the framework implements.","marker":"[4]"},{"why":"Provides the prior result that fine-tuned small language models outperform LLMs on fact-checking classification, the baseline this paper's findings counter.","marker":"[9]"},{"why":"Introduces retrieval-augmented generation, the mechanism used to ground verdicts in retrieved web evidence.","marker":"[19]"},{"why":"Establishes PolitiFact as a benchmark dataset for fake news detection, anchoring the dataset lineage used here.","marker":"[29]"},{"why":"Supplies chain-of-thought prompting, the basis for the step-by-step reasoning component in the prompt.","marker":"[31]"},{"why":"Shows that generating fact-checking explanations alongside veracity predictions improves both, motivating the joint generation setup.","marker":"[34]"},{"why":"Provides TIGERScore, the reference-free metric used to evaluate justification quality.","marker":"[38]"},{"why":"Introduces ModernBERT, the fine-tuned long-context classifier used as the small-model upper bound for comparison.","marker":"[39]"}],"fun_headline_variants":["Larger LLMs beat fine-tuned models on fact-checking","70B Llama outperforms fine-tuned fact-check classifiers","Retrieval-augmented LLMs excel at automated fact-checking","Bigger LLMs, better verdicts: fact-checking study","One-shot 70B Llama tops fine-tuned fact-checkers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the top-10 web snippets retrieved from a single search API provide sufficient, unbiased evidence for each claim and do not indirectly reveal the official fact-check verdict; if snippets leak the answer, the measured benefit of evidence integration is inflated.","fun_headline_variants_meta":{"raw":{"variants":["Larger LLMs beat fine-tuned models on fact-checking","70B Llama outperforms fine-tuned fact-check classifiers","Retrieval-augmented LLMs excel at automated fact-checking","Bigger LLMs, better verdicts: fact-checking study","One-shot 70B Llama tops fine-tuned fact-checkers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001053,"raw_usage":{"total_tokens":4465,"prompt_tokens":1029,"completion_tokens":3436,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":645,"completion_tokens_details":{"reasoning_tokens":3344}},"tokens_in":645,"tokens_out":3436,"duration_ms":24395,"temperature":1.0,"reasoning_tokens":3344,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T23:14:29.435431+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute a held-out correlation between the words in the retrieved snippets and the model's predicted verdict: if, for example, snippets for claims labeled 'False' frequently contain the word 'false' or 'debunk' despite the filter, and models that see those snippets predict 'False' more often than models that do not, the evidence-benefit result would be an artifact of label leakage rather than genuine evidence use. A cleaner test would compare evidence-augmented accuracy against a control where snippets are replaced by unrelated web pages.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the prior result that fine-tuned small language models outperform LLMs on fact-checking classification, the baseline this paper's findings counter."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies chain-of-thought prompting, the basis for the step-by-step reasoning component in the prompt."},{"cited_title":"Atanasova, J","cited_arxiv_id":null,"evidence_quote":"Shows that generating fact-checking explanations alongside veracity predictions improves both, motivating the joint generation setup."},{"cited_title":"Jiang, Y","cited_arxiv_id":null,"evidence_quote":"Provides TIGERScore, the reference-free metric used to evaluate justification quality."}],"review_version":1}