{"id":"278b2616-57b8-4595-998e-4b65c9e5716d","arxiv_id":"2504.14905","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CRAVE combines LLM-generated true and false rationales with a fine-tuned small language model judge, reporting state-of-the-art accuracy on HOVER and FEVEROUS.","lead":"This paper presents CRAVE, a fact-checking system that asks a large language model to argue a claim is true and also to argue it is false, then uses a smaller fine-tuned model to weigh both arguments and pick a verdict. It reports higher accuracy than seven earlier systems on two complex claim-verification benchmarks, with an explanation attached to each verdict.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FEVEROUS subset and binary label space are undefined, so the 'state-of-the-art on FEVEROUS' claim is not auditable.","rationale":"The reader's weakest_assumption concerns the ordering signal derived from the claim that the rationale consistent with the truth is typically more comprehensive. That is a genuine theoretical soft spot, but it is not the most load-bearing issue for the paper's headline claim: even if the asymmetry is imperfect, CRAVE could still be empirically effective. The headline claim is that CRAVE achieves state-of-the-art accuracy on HOVER and FEVEROUS, and that claim is currently unauditable because the FEVEROUS evaluation protocol is not defined. The paper literally reports an ad hoc subset without giving claim IDs or class balance, uses binary labels in a dataset whose official label space is three-way, and does not state how the seven baselines were adapted to this protocol. The reader's own rationale identifies the label space and FEVEROUS subset as critical open questions, which is why I mark agreement as partial; my stress-test shifts the emphasized weakest point from the ordering assumption to the empirical comparison itself. The verdict CONDITIONAL remains appropriate because the concern can be resolved by a concrete reproducibility check rather than by any demonstrated error in the current numbers.","tokens_in":13287,"tokens_out":7840,"duration_ms":75597,"concrete_test":"Inspect the released code and the FEVEROUS dev set, identify the exact 2,962 claims by matching the paper's filtering criterion, and record the label distribution including whether NEI was excluded. Then rerun DeBERTaV3-NLI, MULTIVERS, and ProgramFC on this exact subset under both gold and open settings and compare against the same subset with full three-way FEVEROUS labels. If the subset differs from a standard FEVEROUS split, or if any baseline exceeds CRAVE on this subset, the state-of-the-art claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section VII that CRAVE achieves state-of-the-art results on FEVEROUS rests on Table I, but the FEVEROUS evaluation protocol is underspecified. Section IV-A2 says the authors 'only use claims that require exclusively structured evidence, containing 2962 claims in total.' This is internally inconsistent with the stated focus on textual claim verification and with the Wikipedia-paragraph retrieval pipeline, because structured evidence in FEVEROUS refers to tables. The paper never specifies which 2,962 claims were used, how the official three-way FEVEROUS labels (SUPPORT/REFUTE/NEI) map to the binary y in {true,false} used throughout Section III, or whether the same filtering was applied to all baselines. If NEI was dropped, or if baselines were not re-run on this exact subset, then the reported accuracy gaps such as 74.95 versus 67.80 in open-book FEVEROUS do not establish superiority on FEVEROUS as a benchmark. This is a missing-support issue rather than a minor omission: without the protocol, the headline empirical claim cannot be checked.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CRAVE, a three-module framework for explainable claim verification. The first module performs ambiguity elimination and entity-based retrieval to gather evidence from Wikipedia; the second uses an LLM to generate conflicting rationales (for the claim being true versus false) across four reasoning dimensions and to produce a preliminary judgment; the third fine-tunes a small language model (SLM) judge that receives the claim, the two rationales, and the LLM's preliminary judgment to make the final veracity decision. The authors evaluate CRAVE on HOVER and a subset of FEVEROUS in both gold-evidence and open-book settings, reporting accuracy improvements over seven baselines, and they provide ablation studies, evidence-quality metrics, LLM reasoning-perspective experiments, and SLM input/training-method comparisons. The central claim is that CRAVE achieves state-of-the-art accuracy on both datasets while also producing higher-quality explanations.","tokens_in":13488,"tokens_out":2959,"duration_ms":27132,"significance":"If the reported results are reproducible and the evaluation protocol is fully specified, the paper makes a useful contribution to explainable claim verification by combining LLM-generated conflicting rationales with a fine-tuned SLM judge. The ambiguity-elimination retrieval strategy and the four-dimension reasoning prompt are reasonable design ideas, and the ablation studies generally support the contribution of each module. The code is provided, which helps reproducibility. However, the FEVEROUS evaluation is not currently auditable: the subset definition, label mapping, and baseline comparison protocol are underspecified, and this directly affects the headline claim of state-of-the-art performance on FEVEROUS. The paper also lacks statistical significance reporting, which weakens the strength of the comparative claims.","major_comments":[{"comment":"The FEVEROUS evaluation protocol is underspecified in a way that blocks verification of the central claim. The paper says 'we only use claims that require exclusively structured evidence, containing 2962 claims in total,' but this is internally inconsistent with the stated focus on textual claim verification and with the Wikipedia-paragraph retrieval pipeline, because structured evidence in FEVEROUS refers to tables. The authors do not specify which 2,962 claims were selected, how the official three-way FEVEROUS labels (SUPPORT/REFUTE/NEI) are mapped to the binary label y used in Section III, whether NEI claims were dropped, or whether all baselines were re-run on exactly this filtered subset. Without this information, the reported FEVEROUS accuracy gaps (e.g., 74.95 versus 67.80 in the open-book setting) cannot be interpreted as a benchmark-level result.","section":"IV-A2 and Table I"},{"comment":"The input ordering to the SLM judge rests on an unverified empirical assumption. The paper states that the rationale generated from the stance consistent with the claim's truthfulness is 'typically more comprehensive and reasonable' than the opposing rationale, and Eq. (1) fixes the order of the two rationale segments based on the LLM's preliminary judgment yLLM. This assumption is load-bearing: if the asymmetry does not hold, yLLM carries no reliable confidence signal and the fixed ordering has no justification. The paper provides no quantitative evidence for this asymmetry, such as the accuracy of yLLM on each dataset or an analysis of confidence calibration across HOVER hop counts and FEVEROUS. Please add such an analysis or relax the order-dependence of the model.","section":"III-C and Eq. (1)-(3)"},{"comment":"There is a direct inconsistency between Table VI and the main results. The row 'training on gold+CFT(open:gold=100:0)' reports FEVEROUS open accuracy 84.68, which is 9.73 points higher than the main CRAVE open accuracy 74.95 in Table I. Unless the Table VI configuration differs in some unstated way (e.g., model size, training data, or evaluation subset), this contradicts the paper's conclusion in Section VI that the 80:20 open:gold ratio is optimal and also calls into question why the main reported model is not the best configuration found. The authors should clarify the relation between the Table VI configurations and the main CRAVE model, and report the configuration that yields the best open-setting accuracy.","section":"Table VI"},{"comment":"No variance or significance information is reported for any of the main accuracy comparisons. The accuracy differences in Table I are presented as single numbers, although some gaps are small (e.g., 0.44% on HOVER 2-hop open, 0.67% on HOVER 2-hop gold). The claim that 'CRAVE outperforms all baselines across all evaluation tasks' would be much stronger with confidence intervals, standard deviations over multiple seeds, or significance tests. Please add such statistics at least for the headline results.","section":"Section IV-C"}],"minor_comments":[{"comment":"The word 'enchanced' should be 'enhanced' in the title, abstract, and Section III-B heading.","section":"Title, Abstract, III-B"},{"comment":"In Section III-A, 'donated as D' should be 'denoted as D'. In Section III-D, 'rf lase' should be 'r_false'.","section":"III-A and III-D"},{"comment":"The word 'decesion' should be 'decision' in Section IV-D, and 'FERVEROUS' in Section IV-F should be 'FEVEROUS'.","section":"IV-D and IV-F"},{"comment":"The future-work sentence 'extend the detection tasks from textual claims to unstructured claim detection' appears to state the opposite of the intended scope, since the paper works on textual claims; please clarify.","section":"VII"},{"comment":"The word 'Evluation' should be 'Evaluation' in the caption of Figure 3.","section":"Figure 3 caption"},{"comment":"The notation in Eq. (4), specifically '∑_D log pver[y*=y]', is not fully defined. Please specify that D is the training dataset and use a clearer notation for the indicator of the correct class.","section":"Eq. (4)"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a reasonable applied paper with a clean pipeline, but the central empirical claim is under-specified to the point of being unverifiable on FEVEROUS. The method is a sensible combination: ambiguity elimination, entity-based retrieval, LLM reasoning from both truth stances across four dimensions, then a fine-tuned SLM judge that uses the LLM's preliminary judgment to order the two rationales. The ablations hang together, and the code is linked. Gains of ~9 points on 3-hop HOVER are not trivial.\n\nThe weak spots are in the evaluation protocol. Section IV-A2 says the FEVEROUS subset is 'claims that require exclusively structured evidence'—which contradicts their stated focus on textual claims and their Wikipedia paragraph retrieval. I suspect they meant the opposite, but as written it is a red flag. More importantly, they never say which 2,962 claims were used, whether the official three-way labels (SUPPORT/REFUTE/NEI) were collapsed to binary, or whether all baselines were re-run on this exact subset. Without that, the reported 74.95 vs 67.80 on open-book FEVEROUS does not establish SOTA. The same lack of protocol detail affects the label space on HOVER (binary only?).\n\nAlso, the ordering assumption in III-C—that the rationale matching the truth is 'typically more comprehensive'—is load-bearing for the SLM weighting, but it's supported only by a passing observation, not a measurement. That's a softer concern than the protocol issue, but it should be addressed.\n\nThe baselines end in 2023. That is a limitation, not a fatal one, but it weakens the 'state-of-the-art' phrasing.\n\nBottom line: the system is plausible and the paper deserves review, but the authors need to release an exact evaluation protocol, including label mapping and subset definition, before the results can be taken at face value. I'd send it to peer review with that requirement.","headline":"CRAVE is a coherent LLM+SLM claim-verification pipeline with real gains on multi-hop claims, but the paper's SOTA claim on FEVEROUS is not auditable because the evaluation subset and label space are never specified.","tokens_in":14023,"tokens_out":2540,"would_cite":false,"duration_ms":21996,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A conflicting-reasoning pipeline claims the highest reported accuracy on complex fact-checking benchmarks by pairing LLM arguments with a small judge model.","keywords":["claim verification","fact-checking","large language models","small language model judge","conflicting reasoning","explainability","evidence retrieval","misinformation"],"falsifier":"Take a random sample of claims with known labels, have annotators rate the two generated rationales for comprehensiveness without knowing the truth, and compare those ratings with the LLM's preliminary verdict; then feed the judge's inputs in randomized order instead of the fixed preferred-opposed order. If accuracy does not fall measurably under randomization, or if the LLM's preferred rationale is not the one rated higher, the ordering signal carries no real confidence information.","tokens_in":13087,"feed_emoji":"⚖️","tokens_out":9533,"duration_ms":80252,"temperature":0.7,"pith_summary":"CRAVE is a pipeline for verifying complex factual claims that splits the work between a large language model (LLM) and a small, fine-tuned judge. The LLM first disambiguates entities in the claim and gathers evidence from Wikipedia, then writes two rationales for the claim—one arguing it is true and one arguing it is false—covering direct evidence, semantic relationships, linguistic patterns, and logical reasoning, and it adds a preliminary verdict. The judge reads both rationales, with the LLM's preferred side placed first, and issues the final label; the winning rationale doubles as the explanation. The paper's central claim is that this division of labor gives the highest reported accuracy on the HOVER and FEVEROUS benchmarks under both gold-evidence and open-book retrieval, because the LLM's rationale-quality asymmetry and the judge's fine-tuning cover each other's weaknesses.","feed_headline":"Conflicting LLM rationales lift fact-check accuracy on complex claims","feed_subtitle":"CRAVE pairs an LLM's opposing arguments with a fine-tuned judge, topping seven baselines on HOVER and FEVEROUS.","key_machinery":"The central object is the ordered pair of conflicting rationales $(r_{\\mathrm{true}}, r_{\\mathrm{false}})$ produced by the LLM under a prompt that fixes four analytical dimensions: direct evidence, semantic features and relationships, linguistic patterns and connections, and logical reasoning. The pair is what makes the truth asymmetry visible: the paper observes that the rationale aligned with the claim's actual truth is typically more comprehensive, so the LLM's preliminary label $y_{\\mathrm{LLM}}$ is converted into an input-order prior for the judge. The judge encodes the claim with the preferred rationale first and the opposing rationale second, computes a verdict for each stream through a shared Transformer encoder and MLP, and combines them as $p^{\\mathrm{ver}} = w_1 p^{\\mathrm{ver}}_1 + w_2 p^{\\mathrm{ver}}_2$, with $w_1,w_2$ learned and normalized to sum to one. This ordered two-stream weighting is what lets the small model exploit the LLM's reasoning while staying protected from the LLM's tendency to hallucinate.","core_discovery":"On the paper's terms, the discovery is that a verifier need not construct a single correct reasoning chain; it can be built from a deliberate argument for each side. For any claim and its retrieved evidence, CRAVE generates reasons for 'true' and reasons for 'false' from the same LLM, and the paper argues that the rationale written from the side that is actually correct comes out more comprehensive and reasonable. That asymmetry turns the LLM's preliminary verdict into a signal about which rationale to trust, although not a signal reliable enough to be the final answer. The fine-tuned judge takes the claim and both rationales in the order fixed by that signal, weights the two encoded reads through learned parameters, and returns the final label together with the chosen rationale as the explanation.","pith_inferences":["The input-ordering trick is a general recipe for LLM-plus-small-model systems: whenever an LLM can argue both sides of a question, its preference can be passed to the judge as a positional prior instead of as a hard answer.","Replacing the binary preliminary label with a numeric confidence score would test whether the asymmetry is genuinely about rationale quality and could make the learned weights interpretable.","The four-part rationale decomposition could be reused to generate sentence-level explanations, letting readers check each step against the cited source pages."],"forward_implications":["Open-book fact-checking becomes feasible without gold evidence: entity-based retrieval plus a standard paragraph ranking step supplies evidence that the judge can reason over.","The largest reported gains appear on three-hop and four-hop HOVER claims, where the combination of conflicting rationales and the small judge matters most.","Every final verdict carries an explanation: the rationale matched to the predicted label is returned with source URLs for readers to check.","The four reasoning dimensions are load-bearing, since removing any one of them lowers the LLM judge's accuracy."],"supporting_citations":[{"why":"Supplies the HOVER dataset of multi-hop claims and their gold evidence, which defines the harder part of the benchmark.","marker":"[11]"},{"why":"Supplies the FEVEROUS dataset of complex claims with gold evidence, the second benchmark used in all comparisons.","marker":"[2]"},{"why":"The strongest LLM-based baseline for complex claims; its open-book setting defines the retrieval comparison CRAVE must match.","marker":"[18]"},{"why":"The fully open-book LLM baseline; comparing against it isolates the value of adding the small-model judge.","marker":"[26]"},{"why":"Supports the LLM-generates and SLM-judges design and documents why LLM judgments are unreliable on their own.","marker":"[8]"},{"why":"Motivates the ambiguity-elimination step by showing that ambiguous claims require disambiguated evidence.","marker":"[4]"},{"why":"Motivates the four-part rationale design through its analysis of fake-news argumentation.","marker":"[20]"},{"why":"Grounds the linguistic-patterns reasoning dimension in research on linguistic characteristics of misinformation.","marker":"[28]"}],"fun_headline_variants":["Conflicting LLM reasons improve fact-check accuracy","CRAVE uses both-sided LLM arguments to verify claims","LLM rationale conflict helps complex claim verification","Fine-tuned judge weighs conflicting LLM rationales","Two-sided LLM reasoning beats baselines on claim verification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that when an LLM argues both for and against a claim, the argument aligned with the claim's true status is typically the more comprehensive and reasonable one, so the LLM's preliminary verdict can be read as a ranking of the two rationales by quality.","fun_headline_variants_meta":{"raw":{"variants":["Conflicting LLM reasons improve fact-check accuracy","CRAVE uses both-sided LLM arguments to verify claims","LLM rationale conflict helps complex claim verification","Fine-tuned judge weighs conflicting LLM rationales","Two-sided LLM reasoning beats baselines on claim verification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001047,"raw_usage":{"total_tokens":4428,"prompt_tokens":1001,"completion_tokens":3427,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":3351}},"tokens_in":617,"tokens_out":3427,"duration_ms":24507,"temperature":1.0,"reasoning_tokens":3351,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:37:10.387498+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of claims with known labels, have annotators rate the two generated rationales for comprehensiveness without knowing the truth, and compare those ratings with the LLM's preliminary verdict; then feed the judge's inputs in randomized order instead of the fixed preferred-opposed order. If accuracy does not fall measurably under randomization, or if the LLM's preferred rationale is not the one rated higher, the ordering signal carries no real confidence information.","supporting_citations":[{"cited_title":"HoVer: A dataset for many-hop fact extraction and claim verification,","cited_arxiv_id":null,"evidence_quote":"Supplies the HOVER dataset of multi-hop claims and their gold evidence, which defines the harder part of the benchmark."},{"cited_title":"Feverous: Fact extraction and verification over unstructured and structured information,","cited_arxiv_id":null,"evidence_quote":"Supplies the FEVEROUS dataset of complex claims with gold evidence, the second benchmark used in all comparisons."},{"cited_title":"Fact-checking complex claims with program-guided reasoning,","cited_arxiv_id":null,"evidence_quote":"The strongest LLM-based baseline for complex claims; its open-book setting defines the retrieval comparison CRAVE must match."},{"cited_title":"AmbiFC: Fact-checking ambiguous claims with evidence,","cited_arxiv_id":null,"evidence_quote":"Motivates the ambiguity-elimination step by showing that ambiguous claims require disambiguated evidence."},{"cited_title":"Dialog in the echo chamber: Fake news framing predicts emotion, argumentation and dialogic social knowledge building in subsequent online discussions,","cited_arxiv_id":null,"evidence_quote":"Motivates the four-part rationale design through its analysis of fake-news argumentation."},{"cited_title":"Linguistic characteristics and the dissemination of misinformation in social media: The moderating effect of information richness,","cited_arxiv_id":null,"evidence_quote":"Grounds the linguistic-patterns reasoning dimension in research on linguistic characteristics of misinformation."}],"review_version":1}