{"id":"6ee79ae7-5162-4d12-95b7-8354ebaec6c1","arxiv_id":"2506.04583","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"SUCEA improves adversarial fact-checking by decomposing claims into atomic sub-claims, editing each sub-claim toward retrieved evidence, and re-retrieving before predicting the final label.","lead":"SUCEA is a new fact-checking pipeline that breaks hard, adversarial claims into smaller pieces and rewrites each piece using the evidence it first finds, which makes the second round of searching much more successful. The framework is training-free, uses ordinary search engines and language models, and improved fact-checking accuracy by several points on two Wikipedia-based challenge sets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline gains are reported without variance or significance tests on small samples; the central 'significant improvement' claim is not statistically anchored.","rationale":"The reader's conditional verdict already notes missing error bars in its rationale, but its formal weakest assumption concerns first-round retrieval quality. That failure mode is real and acknowledged in Section 5.3, yet it is a robustness limitation: SUCEA could still deliver its headline gains even if first-round evidence is sometimes wrong. The statistical concern directly targets the strongest claim that SUCEA 'significantly outperforms' baselines. The linked code, coherent ablations, and manual error analysis are genuine supporting evidence, and I see no obvious circularity or data leakage that would independently sink the method. However, a 7-8 point accuracy difference on 200 examples is not demonstrated to be statistically reliable, so the central empirical claim is currently over-stated. Since the reader's verdict is already CONDITIONAL and this concern reinforces that condition rather than overturning the paper, I recommend keeping the verdict unchanged.","tokens_in":19158,"tokens_out":4244,"duration_ms":54708,"concrete_test":"Obtain per-claim predictions for SUCEA and RALM on the exact FOOL METWICE 200 samples and WICE 358 samples, then run McNemar's exact test for each Table 2 row and bootstrap 95% confidence intervals for the accuracy and RAcc differences. Also rerun the headline configuration (GPT-4o-mini with Contriever) under 5 different seeds or API temperatures and report mean and standard deviation. If the confidence interval for the FOOL METWICE accuracy difference includes 0, or McNemar's p is above 0.05, the 'significantly outperforms' wording is unsupported and the abstract and Section 4.2 should be revised to describe numerical improvements rather than significance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing gap is statistical anchoring of the headline numbers. SUCEA's central claim is that it 'significantly improves' fact-checking and retrieval over RALM, with 75.0 vs 67.5 and 73.5 vs 65.5 accuracy on FOOL METWICE, and 11.0-point RAcc gains under TFIDF. The evaluation uses 200 FOOL METWICE claims and 358 WICE claims (Table 1), with no error bars, no repeated runs, no confidence intervals, and no significance test anywhere in the paper. On n=200, a raw 8-point accuracy difference is only about 16 claims; a two-proportion z-test gives z approximately 1.74 (p about 0.08), not conventionally significant. WICE gains are even smaller in many cells (1-3 points). The word 'significant' appears in the abstract, in Section 4.2, and in the ablation discussion, and footnote 4 explicitly refers to 'significance' without reporting a test. Consistency across models and retrievers is suggestive but not independent evidence, because configurations share the same claims and many quoted gains are single-digit point differences. If the 7-8 point FOOL METWICE accuracy gap is within sampling noise, the paper's central quantitative claim is not established, even though the pipeline itself may be reasonable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SUCEA, a training-free modular framework for adversarial fact-checking. The pipeline decomposes each claim into decontextualized sub-claims, performs a first round of retrieval, uses the retrieved passages as hints to rewrite each sub-claim, retrieves again with the edited queries, reranks the aggregated evidence with an LLM, and finally predicts an entailment label. Experiments on FOOL METWICE (200 sampled claims) and WICE (358 claims) with two backbone LLMs (GPT-4o-mini and Llama-3.1-70B) and two retrievers (Contriever and TFIDF) report consistent improvements in retrieval accuracy and end-to-end fact-checking accuracy over RALM and four decomposition-based baselines. The paper also includes ablations, qualitative case studies, an error analysis, and an appendix with full results and prompts.","tokens_in":19347,"tokens_out":9428,"duration_ms":95354,"significance":"If the improvements are reliable, the paper makes a useful contribution: it shows that evidence-grounded, iterative claim editing, without any fine-tuning, can make lexical and dense retrievers substantially more effective on claims deliberately written to evade lexical overlap, and it provides a modular decomposition that is easy to reuse. The paper is honest about its limitations, includes a thoughtful error analysis, and releases code. The main weakness is that the central \"significant improvement\" claims are not backed by any statistical inference: all numbers are single point estimates on 200 or 358 claims, with no confidence intervals, significance tests, or repeated runs. This is fixable and does not invalidate the approach, but it must be addressed before the headline claims are acceptable.","major_comments":[{"comment":"The abstract and Section 4.2 use the word \"significantly\" to describe SUCEA's improvements, but no significance test, confidence interval, or variance estimate is reported anywhere in the paper. On the 200-claim FOOL METWICE subset, the headline fact-checking gap of 75.0 vs. 67.5 corresponds to only about 15 claims; a two-proportion z-test on these counts gives z ≈ 1.74 (p ≈ 0.08), which is not significant at the conventional 95% level. Several WICE differences are even smaller; for example, Table 2 shows SUCEA at 39.0% versus PROGRAMFC at 38.4% under GPT-4o-mini with Contriever. The paper also does not report the random seed or stratification used to draw the 200-claim subset, so the error cannot be reconstructed. I recommend adding paired significance tests (e.g., McNemar for label accuracy, bootstrap intervals for retrieval metrics) and reporting them for all tables, and adjusting the \"significant\" wording to what the data actually support.","section":"Abstract, §4.2, Tables 2 and 3"},{"comment":"The ablation section claims that \"all components significantly contribute\" and that removing any component leads to \"notable reduction,\" but several ablation drops under Contriever are within one or two points on the same 200-claim set; for example, Table 4 shows wo. claim editing at 49.5 vs. 51.0 for Llama-3.1-70B and Paraphrase wo. Evidence at 50.5 vs. 51.0, and wo. claim editing at 53.0 vs. 54.0 for GPT-4o-mini. These differences (1-2 claims) are within sampling noise, so the universal claim of significant contribution is not supported by the reported numbers. The TFIDF columns show larger drops and are more convincing. The authors should either provide significance evidence for all ablation cells or restrict the conclusion to the configurations where the effect is clearly larger than noise.","section":"§5.1, Table 4"}],"minor_comments":[{"comment":"The header \"F OOLMETWICE\" contains an extra space; change to \"FOOL METWICE.\"","section":"Table 1"},{"comment":"The phrase \"the need of guidence with evidence\" contains a typo; it should read \"guidance.\"","section":"§5.1"},{"comment":"The word \"contriver\" is used instead of \"Contriever\" in the sentence \"when using contriver\"; please fix for consistency.","section":"§4.2"},{"comment":"Kamoi et al. is listed twice (2023a and 2023b) with the same WiCE paper; please consolidate into a single entry.","section":"References"},{"comment":"The paper should report the random seed and sampling procedure used to select the 200-claim FOOL METWICE subset, to make the evaluation reproducible.","section":"§4.1 / Experiments"},{"comment":"The statement that SUCEA \"outperforms all baseline approaches\" on WICE is contradicted in the full results in Table 5: at Top@3 with Llama-3.1-70B and Contriever, MINICHECK reaches 39.1% while SUCEA reaches 38.2%. Please qualify the claim to \"in most configurations\" or refer to the primary setting.","section":"§4.2, Table 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for the journal and the approach is credible. The main barrier is the absence of statistical inference for the headline numbers; the authors can likely address this with paired tests and bootstrap intervals. I also noted the duplicate reference and a small overstatement about WICE at Top@3; those are minor. I do not see a reason to reject on scientific grounds."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know about this paper: SUCEA is a clean modular recipe—segment the claim, retrieve, edit each sub-claim using first-round evidence, retrieve again—and that combination is genuinely new relative to the decomposition baselines it compares against. ClaimDecomp, QABriefs, ProgramFC, and MiniCheck decompose claims but don't rewrite sub-claims toward evidence; RARR edits generated text, not claims. The contribution is the composition, and the paper earns it.\n\nWhat it does well: results are consistent across two datasets, two main backbone LLMs plus three smaller ones in the appendix, and two retrievers. The ablation study is informative—removing evidence-grounded editing hurts more under TFIDF than Contriever, which matches the intuition that lexical retrieval needs the rewrite most. The error analysis is honest, quantifying over-segmentation and parametric-knowledge leakage, and the limitations section says plainly what isn't covered. No fitted parameters, no definitional circularity; the evidence-feedback loop is standard retrieval-augmented design, and the 'paraphrase without evidence' ablation is the right control.\n\nSoft spots, in proportion: the headline numbers are unanchored. The FOOL METWICE test set here is 200 sampled claims; WICE is 358. Every configuration appears to be a single run, no error bars, no significance test, yet the abstract and Section 4.2 say 'significantly.' On n=200, a 7.5-point accuracy gap is roughly 15 claims; a two-proportion test gives p around 0.08. Suggestive, not established. The WICE gains are smaller (1–7 points), and the retrieval numbers there are low in absolute terms (RAcc around 16 at Top@10), so the 'general applicability' claim is weaker than the FOOL METWICE one. The method's dependency on first-round evidence being at least topically relevant is acknowledged, but it is the load-bearing risk: if Round 1 returns junk, the edit is guided by junk. The code link exists but no commit hash, so reproducibility is plausible, not pinned.\n\nWho this is for: anyone working on retrieval for adversarial or complex claims. It's a solid empirical recipe paper, not a definitive benchmark result. I'd send it to peer review—the method is novel enough and the analysis careful enough that a good referee could push for variance estimates, multi-seed runs, and a clearer separation of retrieval gains from label-prediction gains. With those additions the central claim would be much stronger.\n\nRecommendation: engage. It deserves review, with the statistical anchoring as the main revision target.","headline":"SUCEA's decompose-retrieve-edit-retrieve recipe is genuinely new and worth peer review, but its headline 'significant' gains are not statistically anchored on samples of 200.","tokens_in":19927,"tokens_out":2393,"would_cite":true,"duration_ms":28452,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adversarial fact-checking improves when first-round retrieval guides rewriting of decomposed sub-claims before a second retrieval pass, lifting accuracy by 7.5 points on FOOL METWICE.","keywords":["adversarial fact-checking","claim decomposition","decontextualization","evidence-guided claim editing","iterative retrieval","reasoning-intensive retrieval","FOOL METWICE","WICE"],"falsifier":"Take any FOOL METWICE claim, run SUCEA as described, then rerun it with the first-round evidence replaced by random unrelated Wikipedia passages before claim editing; if accuracy stays near 73.5%, the evidence-guidance is not load-bearing, and if it collapses, the stated dependence on first-round hints is confirmed.","tokens_in":18875,"feed_emoji":"🔎","tokens_out":8138,"duration_ms":66379,"temperature":0.7,"pith_summary":"Adversarial fact-checking claims are deliberately written to avoid lexical overlap with the evidence that would verify them, so retrieval-augmented systems fail before the reasoning stage. The paper claims this can be fixed without fine-tuning by making the query itself more evidence-like: split the claim into atomic, context-free sub-claims; retrieve once purely for hints; rewrite each sub-claim with names, numbers, and corrections taken from that evidence; then retrieve again and predict the label from the combined evidence. On FOOL METWICE the framework lifts fact-checking accuracy by 7.5 points over a retrieval-augmented baseline with Llama-3.1-70B and improves top-10 retrieval accuracy by 11.0 points under TFIDF, with smaller consistent gains on WICE. A sympathetic reading of the results is that evidence-grounded claim editing, not decomposition alone, is what makes reasoning-intensive retrieval tractable for adversarial claims.","feed_headline":"One claim-edit pass lifts fact-checking by 7.5 points","feed_subtitle":"SUCEA splits claims, edits each piece using first-round retrieval hints, then re-retrieves; accuracy rises on two benchmarks.","key_machinery":"The load-bearing mechanism is the evidence-augmented claim editing loop. After claim segmentation and decontextualization turn the input into atomic standalone sub-claims, a first retrieval round returns top-k passages that are used only as hints; an LLM then rewrites each sub-claim under explicit constraints to add missing named entities, numerical values, and locations and to replace vague or adversarial wording with evidence-backed details, while being told not to add parametric knowledge. The edited sub-claim is submitted for a second retrieval round, which is what recovers the ground-truth evidence that the original adversarial phrasing hid. Segmentation reduces distraction for the retriever, and a final reranking step selects the most relevant evidence before the entailment label is predicted.","core_discovery":"The central claim is that the bottleneck in adversarial fact-checking is retrievability, not reasoning: the evidence exists in the corpus, but the claim has been paraphrased, stripped of names, or reworded so the retriever cannot find it. SUCEA turns the retrieval query into the object of repair by first decomposing the claim into decontextualized sub-claims, then running a provisional retrieval whose top passages act as hints, then instructing an LLM to rewrite each sub-claim to fill in missing entities, numbers, and locations and to correct counterfactual phrasing using only information present in those passages. A second retrieval on the edited sub-claims, followed by LLM reranking and label prediction, produces the reported gains: 65.5% to 73.5% accuracy on FOOL METWICE with Llama-3.1-70B, a 7.5-point improvement, and an 11.0-point gain in top-10 retrieval accuracy under TFIDF, with consistent improvements on WICE as well.","pith_inferences":["Beyond the paper: if evidence-grounded query editing is the active ingredient, the same pattern should transfer to other reasoning-intensive retrieval tasks, such as retrieving code from a natural-language description or finding documents for multi-hop questions, where the query is also deliberately abstract.","Beyond the paper: the method has no guardrail for a failed first round, so a cheap extension is to detect low-confidence first-round retrieval and widen k, switch retrievers, or abstain from editing; the paper's error analysis, which finds 15 of 50 failures tied to parametric knowledge, suggests a second check comparing the edited sub-claim against the first-round passages could reduce contaminati","Beyond the paper: because gains are larger for TFIDF than for Contriever, the framework's benefit may shrink as dense retrievers become more paraphrase-robust; a testable prediction is that SUCEA's advantage over a stronger dense retriever will be smaller."],"forward_implications":["On FOOL METWICE, SUCEA improves fact-checking accuracy from 65.5% to 73.5% with Llama-3.1-70B, and the gain is largest for supported claims, which rise from 52.4% to 64.1%.","Retrieval accuracy at Top-10 under TFIDF rises from 23.5% to 34.5%, showing that edited sub-claims are substantially easier for a purely lexical retriever to match.","Ablations show that editing without evidence guidance loses up to 8 points of retrieval accuracy under TFIDF, and removing segmentation costs up to 7 points, so both modules carry the result.","The framework works without fine-tuning across backbone LLMs of different sizes and also improves accuracy on WICE, indicating the mechanism transfers to long-form, non-adversarial claims.","Retrieval quality keeps improving with additional retrieve-and-edit rounds, with the largest gains in the first two iterations."],"supporting_citations":[{"why":"Supplies the FOOL METWICE adversarial fact-checking test set on which the main accuracy and retrieval gains are measured.","marker":"Eisenschlos et al., 2021b"},{"why":"Supplies the WICE long-form entailment dataset used to show the framework generalizes beyond adversarial claims.","marker":"Kamoi et al., 2023b"},{"why":"Provides the atomic fact decomposition approach that Claim Segmentation adapts to split claims into independently checkable sub-claims.","marker":"Min et al., 2023a"},{"why":"Motivates the decontextualization step that rewrites each sub-claim into a standalone statement with entity names restored.","marker":"Gunjal and Durrett, 2024"},{"why":"Supplies Contriever, one of the two off-the-shelf retrievers used in both rounds of evidence retrieval.","marker":"Izacard et al., 2022"},{"why":"Supplies the TFIDF lexical retriever whose large gains after claim editing demonstrate the method's effect on surface-form matching.","marker":"Schütze et al., 2008"},{"why":"Provides the LLM-as-reranker prompting pattern used to select top-k evidence before label prediction.","marker":"Sun et al., 2023"}],"fun_headline_variants":["Rewriting claims to help retrievers find facts","Claim editing boosts adversarial fact-checking","SUCEA fixes retrieval by rephrasing claims first","Decompose, retrieve, edit: 7.5-point accuracy gain","Making fact-checking work by fixing the query"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework's second-round retrieval and final verdict depend on the first-round passages being topically close enough to the truth that the LLM's rewrite moves toward the evidence rather than away from it.","fun_headline_variants_meta":{"raw":{"variants":["Rewriting claims to help retrievers find facts","Claim editing boosts adversarial fact-checking","SUCEA fixes retrieval by rephrasing claims first","Decompose, retrieve, edit: 7.5-point accuracy gain","Making fact-checking work by fixing the query"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1353,"prompt_tokens":946,"completion_tokens":407,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":329}},"tokens_in":562,"tokens_out":407,"duration_ms":11566,"temperature":1.0,"reasoning_tokens":329,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:39:19.264105+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any FOOL METWICE claim, run SUCEA as described, then rerun it with the first-round evidence replaced by random unrelated Wikipedia passages before claim editing; if accuracy stays near 73.5%, the evidence-guidance is not load-bearing, and if it collapses, the stated dependence on first-round hints is confirmed.","supporting_citations":[],"review_version":1}