REVIEW 4 major objections 6 minor 31 references
CRAVE: A Conflicting Reasoning Approach for Explainable Claim Verification Using LLMs
T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read A conflicting-reasoning pipeline claims the highest reported accuracy on complex fact-checking benchmarks by pairing LLM arguments with a small judge model.
desk verdict CRAVE is a coherent LLM+SLM claim-verification pipeline with real gains on multi-hop claims, but the paper's SOTA claim on FEVEROUS is not auditable because the evaluation subset and label space are never specified. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the ordered pair of conflicting rationales $(r_{\mathrm{true}}, r_{\mathrm{false}})$ produced by the LLM under a prompt that fixes four analytical dimensions: direct evidence, semantic features and relationships, linguistic patterns and connections, and logical reasoning. The pair is what makes the truth asymmetry visible: the paper observes that the rationale aligned with the claim's actual truth is typically more comprehensive, so the LLM's preliminary label $y_{\mathrm{LLM}}$ is converted into an input-order prior for the judge. The judge encodes the claim with the preferred rationale first and the opposing rationale second, computes a verdict for each stream through a shared Transformer encoder and MLP, and combines them as $p^{\mathrm{ver}} = w_1 p^{\mathrm{ver}}_1 + w_2 p^{\mathrm{ver}}_2$, with $w_1,w_2$ learned and normalized to sum to one. This ordered two-stream weighting is what lets the small model exploit the LLM's reasoning while staying protected from the LLM's tendency to hallucinate.
What would settle it
Take a random sample of claims with known labels, have annotators rate the two generated rationales for comprehensiveness without knowing the truth, and compare those ratings with the LLM's preliminary verdict; then feed the judge's inputs in randomized order instead of the fixed preferred-opposed order. If accuracy does not fall measurably under randomization, or if the LLM's preferred rationale is not the one rated higher, the ordering signal carries no real confidence information.
Extended reading notes
Core claim
On the paper's terms, the discovery is that a verifier need not construct a single correct reasoning chain; it can be built from a deliberate argument for each side. For any claim and its retrieved evidence, CRAVE generates reasons for 'true' and reasons for 'false' from the same LLM, and the paper argues that the rationale written from the side that is actually correct comes out more comprehensive and reasonable. That asymmetry turns the LLM's preliminary verdict into a signal about which rationale to trust, although not a signal reliable enough to be the final answer. The fine-tuned judge takes the claim and both rationales in the order fixed by that signal, weights the two encoded reads through learned parameters, and returns the final label together with the chosen rationale as the explanation.
Load-bearing premise
The load-bearing premise is that when an LLM argues both for and against a claim, the argument aligned with the claim's true status is typically the more comprehensive and reasonable one, so the LLM's preliminary verdict can be read as a ranking of the two rationales by quality.
Editorial extensions
If this is right
- Open-book fact-checking becomes feasible without gold evidence: entity-based retrieval plus a standard paragraph ranking step supplies evidence that the judge can reason over.
- The largest reported gains appear on three-hop and four-hop HOVER claims, where the combination of conflicting rationales and the small judge matters most.
- Every final verdict carries an explanation: the rationale matched to the predicted label is returned with source URLs for readers to check.
- The four reasoning dimensions are load-bearing, since removing any one of them lowers the LLM judge's accuracy.
Reading between the lines
- The input-ordering trick is a general recipe for LLM-plus-small-model systems: whenever an LLM can argue both sides of a question, its preference can be passed to the judge as a positional prior instead of as a hard answer.
- Replacing the binary preliminary label with a numeric confidence score would test whether the asymmetry is genuinely about rationale quality and could make the learned weights interpretable.
- The four-part rationale decomposition could be reused to generate sentence-level explanations, letting readers check each step against the cited source pages.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CRAVE, a three-module framework for explainable claim verification. The first module performs ambiguity elimination and entity-based retrieval to gather evidence from Wikipedia; the second uses an LLM to generate conflicting rationales (for the claim being true versus false) across four reasoning dimensions and to produce a preliminary judgment; the third fine-tunes a small language model (SLM) judge that receives the claim, the two rationales, and the LLM's preliminary judgment to make the final veracity decision. The authors evaluate CRAVE on HOVER and a subset of FEVEROUS in both gold-evidence and open-book settings, reporting accuracy improvements over seven baselines, and they provide ablation studies, evidence-quality metrics, LLM reasoning-perspective experiments, and SLM input/training-method comparisons. The central claim is that CRAVE achieves state-of-the-art accuracy on both datasets while also producing higher-quality explanations.
Significance. If the reported results are reproducible and the evaluation protocol is fully specified, the paper makes a useful contribution to explainable claim verification by combining LLM-generated conflicting rationales with a fine-tuned SLM judge. The ambiguity-elimination retrieval strategy and the four-dimension reasoning prompt are reasonable design ideas, and the ablation studies generally support the contribution of each module. The code is provided, which helps reproducibility. However, the FEVEROUS evaluation is not currently auditable: the subset definition, label mapping, and baseline comparison protocol are underspecified, and this directly affects the headline claim of state-of-the-art performance on FEVEROUS. The paper also lacks statistical significance reporting, which weakens the strength of the comparative claims.
major comments (4)
- [IV-A2 and Table I] The FEVEROUS evaluation protocol is underspecified in a way that blocks verification of the central claim. The paper says 'we only use claims that require exclusively structured evidence, containing 2962 claims in total,' but this is internally inconsistent with the stated focus on textual claim verification and with the Wikipedia-paragraph retrieval pipeline, because structured evidence in FEVEROUS refers to tables. The authors do not specify which 2,962 claims were selected, how the official three-way FEVEROUS labels (SUPPORT/REFUTE/NEI) are mapped to the binary label y used in Section III, whether NEI claims were dropped, or whether all baselines were re-run on exactly this filtered subset. Without this information, the reported FEVEROUS accuracy gaps (e.g., 74.95 versus 67.80 in the open-book setting) cannot be interpreted as a benchmark-level result.
- [III-C and Eq. (1)-(3)] The input ordering to the SLM judge rests on an unverified empirical assumption. The paper states that the rationale generated from the stance consistent with the claim's truthfulness is 'typically more comprehensive and reasonable' than the opposing rationale, and Eq. (1) fixes the order of the two rationale segments based on the LLM's preliminary judgment yLLM. This assumption is load-bearing: if the asymmetry does not hold, yLLM carries no reliable confidence signal and the fixed ordering has no justification. The paper provides no quantitative evidence for this asymmetry, such as the accuracy of yLLM on each dataset or an analysis of confidence calibration across HOVER hop counts and FEVEROUS. Please add such an analysis or relax the order-dependence of the model.
- [Table VI] There is a direct inconsistency between Table VI and the main results. The row 'training on gold+CFT(open:gold=100:0)' reports FEVEROUS open accuracy 84.68, which is 9.73 points higher than the main CRAVE open accuracy 74.95 in Table I. Unless the Table VI configuration differs in some unstated way (e.g., model size, training data, or evaluation subset), this contradicts the paper's conclusion in Section VI that the 80:20 open:gold ratio is optimal and also calls into question why the main reported model is not the best configuration found. The authors should clarify the relation between the Table VI configurations and the main CRAVE model, and report the configuration that yields the best open-setting accuracy.
- [Section IV-C] No variance or significance information is reported for any of the main accuracy comparisons. The accuracy differences in Table I are presented as single numbers, although some gaps are small (e.g., 0.44% on HOVER 2-hop open, 0.67% on HOVER 2-hop gold). The claim that 'CRAVE outperforms all baselines across all evaluation tasks' would be much stronger with confidence intervals, standard deviations over multiple seeds, or significance tests. Please add such statistics at least for the headline results.
minor comments (6)
- [Title, Abstract, III-B] The word 'enchanced' should be 'enhanced' in the title, abstract, and Section III-B heading.
- [III-A and III-D] In Section III-A, 'donated as D' should be 'denoted as D'. In Section III-D, 'rf lase' should be 'r_false'.
- [IV-D and IV-F] The word 'decesion' should be 'decision' in Section IV-D, and 'FERVEROUS' in Section IV-F should be 'FEVEROUS'.
- [VII] The future-work sentence 'extend the detection tasks from textual claims to unstructured claim detection' appears to state the opposite of the intended scope, since the paper works on textual claims; please clarify.
- [Figure 3 caption] The word 'Evluation' should be 'Evaluation' in the caption of Figure 3.
- [Eq. (4)] The notation in Eq. (4), specifically '∑_D log pver[y*=y]', is not fully defined. Please specify that D is the training dataset and use a clearer notation for the indicator of the correct class.
Circularity Check
No significant circularity: CRAVE's central claim is supported by held-out benchmark labels and component ablations, not by construction from its inputs.
full rationale
CRAVE's derivation chain is not circular. The final label is produced by an SLM judge (Section III-D) fine-tuned on the same external HOVER/FEVEROUS labels used for all baselines, so the accuracy comparison is not a fitted quantity renamed as a prediction. The LLM's preliminary judgment yLLM and the conflicting rationales are generated from evidence and the claim, and the order-dependent weighting in Equations (1)-(3) is trained with cross-entropy against ground-truth labels; no parameter is defined in terms of the target accuracy. The paper's assertion that the truth-consistent rationale is 'typically more comprehensive and reasonable' (Section III-C) is an empirical heuristic motivating input order, and it is directly tested in the ablation and input-setting experiments (Tables IV and V), not assumed as an output. There is no reliance on a self-citation chain or uniqueness theorem: prior LLM work is cited for background (Section II-B), and the baselines are external systems. The underspecified FEVEROUS subset and binary label mapping raised by reviewers would be a reproducibility/correctness problem, not a circularity one, because even if the protocol were fixed, the model's verdicts would still be measured against external labels. Score 0 reflects the absence of any step where a 'prediction' reduces to its input by construction.
Assumptions & free parameters
free parameters (3)
- w1, w2 (learned combination weights)
- Evidence block composition =
first 2 paragraphs + BM25-selected paragraphs
- Open:gold training data ratio =
80:20
assumptions (3)
- domain assumption Rationales from the truthful stance are more comprehensive than from the false stance
- domain assumption First two paragraphs of an entity page summarize the entity well
- domain assumption LLM-generated rationales are informative enough for SLM training despite hallucination risks
Cite this review
Pith. "Pith review of CRAVE: A Conflicting Reasoning Approach for Explainable Claim Verification Using LLMs." pith.science (2026). https://pith.science/paper/YRCDW77H
@misc{pith2026250414905,
author = {Pith},
title = {Pith review of: CRAVE: A Conflicting Reasoning Approach for Explainable Claim Verification Using LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/YRCDW77H}},
note = {Machine review of arXiv:2504.14905}
}
read the original abstract
The rapid spread of misinformation, driven by digital media and AI-generated content, has made automatic claim verification essential. Traditional methods, which depend on expert-annotated evidence, are labor-intensive and not scalable. Although recent automated systems have improved, they still struggle with complex claims that require nuanced reasoning. To address this, we propose CRAVE, a Conflicting Reasoning Approach for explainable claim VErification, that verify the complex claims based on the conflicting rationales reasoned by large language models (LLMs). Specifically, CRAVE introduces a three-module framework. Ambiguity Elimination enchanced Evidence Retrieval module performs ambiguity elimination and entity-based search to gather relevant evidence related to claim verification from external sources like Wikipedia. Conflicting Perspective Reasoning and Preliminary Judgment module with LLMs adopts LLMs to reason rationales with conflicting stances about claim verification from retrieved evidence across four dimensions, i.e., direct evidence, semantic relationships, linguistic patterns, and logical reasoning and make a preliminary judgment. Finally, Small Language Model (SLM) based Judge module is fine-tuned to make use of preliminary judgment from LLMs to assess the confidence of the conflicting rationales and make a final authenticity judgment. This methodology allows CRAVE to capture subtle inconsistencies in complex claims, improving both the accuracy and transparency of claim verification. Extensive experiments on two public claim verification datasets demonstrate that our CRAVE model achieves much better performance than state-of-the-art methods and exhibits a superior capacity for finding relevant evidence and explaining the model predictions. The code is provided at https://github.com/8zym/CRAVE.
Figures
Reference graph
Works this paper leans on
-
[1]
Towards covid-19 fake news detection using transformer-based models,
J. Alghamdi, Y . Lin, and S. Luo, “Towards covid-19 fake news detection using transformer-based models,” Knowledge-Based Systems , vol. 274, pp. 110 642 – 110 642, 2023. [Online]. Available: https: //api.semanticscholar.org/CorpusID:258807287
work page 2023
-
[2]
Feverous: Fact extraction and verification over unstructured and structured information,
R. Aly, Z. Guo, M. Schlichtkrull, J. Thorne, A. Vlachos, C. Christodoulopoulos, O. Cocarascu, and A. Mittal, “Feverous: Fact extraction and verification over unstructured and structured information,”
-
[3]
Y . Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung, Q. V . Do, Y . Xu, and P. Fung, “A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity,” 2023. [Online]. Available: https://arxiv.org/abs/2302.04023
arXiv 2023
-
[4]
AmbiFC: Fact-checking ambiguous claims with evidence,
M. Glockner, I. Stali ¯unait˙e, J. Thorne, G. Vallejo, A. Vlachos, and I. Gurevych, “AmbiFC: Fact-checking ambiguous claims with evidence,” Transactions of the Association for Computational Linguistics , vol. 12, pp. 1–18, 2024. [Online]. Available: https://aclanthology.org/2024. tacl-1.1/
work page 2024
-
[5]
Language models hallucinate, but may excel at fact verification,
J. Guan, J. Dodge, D. Wadden, M. Huang, and H. Peng, “Language models hallucinate, but may excel at fact verification,” 2024. [Online]. Available: https://arxiv.org/abs/2310.14564
arXiv 2024
-
[6]
A survey on automated fact-checking,
Z. Guo, M. Schlichtkrull, and A. Vlachos, “A survey on automated fact-checking,” Transactions of the Association for Computational Linguistics, vol. 10, pp. 178–206, 2022. [Online]. Available: https: //aclanthology.org/2022.tacl-1.11/
work page 2022
-
[7]
P. He, J. Gao, and W. Chen, “Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing,” 2023. [Online]. Available: https://arxiv.org/abs/2111.09543
arXiv 2023
-
[8]
Bad actor, good advisor: Exploring the role of large language models in fake news detection,
B. Hu, Q. Sheng, J. Cao, Y . Shi, Y . Li, D. Wang, and P. Qi, “Bad actor, good advisor: Exploring the role of large language models in fake news detection,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 20, p. 22105–22113, Mar. 2024. [Online]. Available: http://dx.doi.org/10.1609/aaai.v38i20.30214 TABLE VI: Evaluation of diff...
Show all 31 references
-
[9]
Fakegpt: Fake news generation, explanation and detection of large language models,
Y . Huang and L. Sun, “Fakegpt: Fake news generation, explanation and detection of large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2310.05046
2024 arXiv
-
[10]
Exploring listwise evidence reasoning with t5 for fact verification,
K. Jiang, R. Pradeep, and J. Lin, “Exploring listwise evidence reasoning with t5 for fact verification,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volum...
2021
-
[11]
HoVer: A dataset for many-hop fact extraction and claim verification,
Y . Jiang, S. Bordia, Z. Zhong, C. Dognin, M. Singh, and M. Bansal, “HoVer: A dataset for many-hop fact extraction and claim verification,” in Findings of the Association for Computational Linguistics: EMNLP 2020, T. Cohn, Y . He, and Y . Liu, Eds. Online: Association for Comp...
2020
-
[12]
Predicting information pathways across online communities,
Y . Jin, Y .-C. Lee, K. Sharma, M. Ye, K. Sikka, A. Divakaran, and S. Kumar, “Predicting information pathways across online communities,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , ser. KDD ’23. New York, NY , USA: Association for...
2023
-
[13]
FactKG: Fact verification via reasoning on knowledge graphs,
J. Kim, S. Park, Y . Kwon, Y . Jo, J. Thorne, and E. Choi, “FactKG: Fact verification via reasoning on knowledge graphs,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , A. Rogers, J. Boyd-Graber, and N. Okaz...
2023
-
[14]
Can llms produce faithful explanations for fact-checking? towards faithful explainable fact-checking via multi-agent debate,
K. Kim, S. Lee, K.-H. Huang, H. P. Chan, M. Li, and H. Ji, “Can llms produce faithful explanations for fact-checking? towards faithful explainable fact-checking via multi-agent debate,” 2024. [Online]. Available: https://arxiv.org/abs/2402.07401
2024 arXiv
-
[15]
Muser: A multi-step evidence retrieval enhancement framework for fake news detection,
H. Liao, J. Peng, Z. Huang, W. Zhang, G. Li, K. Shu, and X. Xie, “Muser: A multi-step evidence retrieval enhancement framework for fake news detection,” Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2023. [Online]. Available: https://ap...
2023
-
[16]
Towards explainable harmful meme detection through multimodal debate between large language models,
H. Lin, Z. Luo, W. Gao, J. Ma, B. Wang, and R. Yang, “Towards explainable harmful meme detection through multimodal debate between large language models,” Proceedings of the ACM on Web Conference 2024, 2024. [Online]. Available: https://api.semanticscholar. org/CorpusID:267199823
2024
-
[17]
Adversarial nli: A new benchmark for natural language understanding,
Y . Nie, A. Williams, E. Dinan, M. Bansal, J. Weston, and D. Kiela, “Adversarial nli: A new benchmark for natural language understanding,”
-
[18]
Fact-checking complex claims with program-guided reasoning,
L. Pan, X. Wu, X. Lu, A. T. Luu, W. Y . Wang, M.-Y . Kan, and P. Nakov, “Fact-checking complex claims with program-guided reasoning,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , A. Rogers, J. Boyd-Graber,...
2023
-
[19]
FACTIFY-5WQA: 5W aspect- based fact verification through question answering,
A. Rani, S. T. I. Tonmoy, D. Dalal, S. Gautam, M. Chakraborty, A. Chadha, A. Sheth, and A. Das, “FACTIFY-5WQA: 5W aspect- based fact verification through question answering,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: ...
2023
-
[20]
Dialog in the echo chamber: Fake news framing predicts emotion, argumentation and dialogic social knowledge building in subsequent online discussions,
C. Scheibenzuber, L.-M. Neagu, S. Ruseti, B. Artmann, C. Bartsch, M. Kubik, M. Dascalu, S. Trausan-Matu, and N. Nistor, “Dialog in the echo chamber: Fake news framing predicts emotion, argumentation and dialogic social knowledge building in subsequent online discussions,” Comp...
2023
-
[21]
Bert for evidence retrieval and claim verification,
A. Soleimani, C. Monz, and M. Worring, “Bert for evidence retrieval and claim verification,” 2019. [Online]. Available: https: //arxiv.org/abs/1910.02655
2019 arXiv
-
[22]
MultiVerS: Improving scientific claim verification with weak supervision and full-document context,
D. Wadden, K. Lo, L. L. Wang, A. Cohan, I. Beltagy, and H. Hajishirzi, “MultiVerS: Improving scientific claim verification with weak supervision and full-document context,” in Findings of the Association for Computational Linguistics: NAACL 2022 , M. Carpuat, M.-C. de Marneffe...
2022
-
[23]
MultiVerS: Improving scientific claim verification with weak supervision and full-document context,
D. Wadden, K. Lo, and L. L. e. a. Wang, “MultiVerS: Improving scientific claim verification with weak supervision and full-document context,” in Findings of the Association for Computational Linguistics: NAACL 2022 , M. Carpuat, M.-C. de Marneffe, and I. V . Meza Ruiz, Eds. Se...
2022
-
[24]
Dell: Generating reactions and explanations for llm-based misinformation detection,
H. Wan, S. Feng, Z. Tan, H. Wang, Y . Tsvetkov, and M. Luo, “Dell: Generating reactions and explanations for llm-based misinformation detection,” 2024. [Online]. Available: https://arxiv.org/abs/2402.10426
2024 arXiv
-
[25]
Explainable fake news detection with large language model via defense among competing wisdom,
B. Wang, J. Ma, H. Lin, Z. Yang, R. Yang, Y . Tian, and Y . Chang, “Explainable fake news detection with large language model via defense among competing wisdom,” 2024. [Online]. Available: https://arxiv.org/abs/2405.03371
2024 arXiv
-
[26]
Explainable claim verification via knowledge- grounded reasoning with large language models,
H. Wang and K. Shu, “Explainable claim verification via knowledge- grounded reasoning with large language models,” in Findings of the Association for Computational Linguistics: EMNLP 2023 , H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguis...
2023
-
[27]
Siren’s song in the ai ocean: A survey on hallucination in large language models,
Y . Zhang, Y . Li, L. Cui, D. Cai, L. Liu, T. Fu, X. Huang, E. Zhao, Y . Zhang, Y . Chen, L. Wang, A. T. Luu, W. Bi, F. Shi, and S. Shi, “Siren’s song in the ai ocean: A survey on hallucination in large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2309.01219
2023 arXiv
-
[28]
Linguistic characteristics and the dissemination of misinformation in social media: The moderating effect of information richness,
C. Zhou, K. Li, and Y . Lu, “Linguistic characteristics and the dissemination of misinformation in social media: The moderating effect of information richness,” Information Processing & Management, vol. 58, no. 6, p. 102679, 2021. [Online]. Available: https: //www.sciencedirec...
2021
-
[410]
Available: https://aclanthology.org/2021.acl-short.51/
[Online]. Available: https://aclanthology.org/2021.acl-short.51/
2021
-
[2020]
Available: https://arxiv.org/abs/1910.14599
[Online]. Available: https://arxiv.org/abs/1910.14599
1910 arXiv
-
[2021]
Available: https://arxiv.org/abs/2106.05707
[Online]. Available: https://arxiv.org/abs/2106.05707
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.