Pith. sign in

REVIEW 5 major objections 5 minor 65 references

Towards Robust Fact-Checking: A Multi-Agent System with Advanced Evidence Retrieval

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A four-agent LLM pipeline that decomposes claims into logic predicates and reads full, credibility-filtered pages claims a 12.3% Macro F1 gain over baseline fact-checkers.

desk verdict Sensible multi-agent fact-checking integration with an overclaimed headline: the 12.3% improvement doesn't match the table and the temporal-leakage guard is weaker than stated. read the letter →

arxiv 2506.17878 v1 pith:TKZ5LKBE submitted 2025-06-22 cs.AI

Tam Trinh , Manh Nguyen , Truong-Son Hy This is my paper
classification cs.AI
keywords fact-checkingmulti-agentsystemslargelanguagemodelsevidenceretrievalfirst-orderlogicdecompositionsourcecredibilityfilteringexplainableAImisinformationdetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that automated fact-checking gets markedly better when claim decomposition, search, and verdict writing are split into specialized agents rather than run as one prompt. Its central proposal is a four-agent pipeline: an ingestion agent breaks a claim into atomic predicates in the style of first-order logic and discards pieces that cannot be verified, a query agent writes several search questions per predicate, an evidence agent retrieves the full text of pages from a credibility-filtered shortlist instead of shallow search snippets, and a verdict agent votes over the collected evidence and issues a supported or not-supported label with an explanation. On 100 stratified claims from FEVEROUS, HoVER, and SciFact-Open, the system reports a 12.3% relative Macro F1 improvement over the Direct, chain-of-thought, Self-Ask, and FOLK baselines, winning six of seven evaluation tasks and posting its largest margins on HoVER 3-hop and 4-hop claims. The paper also reports that its explanations outrank the baselines' on coverage, soundness, and readability under an LLM judge. If correct, the system is evidence that source-credibility filtering plus full-document reading, guided by logical decomposition, is what pushes verification accuracy up on complex claims.

What carries the argument

The load-bearing object is the retrieval loop inside the Evidence Seeking Agent, combined with the decomposition rule of the Input Ingestion Agent. The decomposition rule treats a claim as a conjunction of first-order-logic predicates, $C = p_1 \land p_2 \land \dots \land p_n$, so a supported verdict requires every atomic predicate to verify as true. The retrieval loop then works in stages: it turns each predicate into several deliberately varied search questions, applies a dataset-specific end date to the search (October 2021 for FEVEROUS, November 2020 for HoVer, October 2020 for SciFact) so that pages published after the benchmark's creation are excluded, rejects any source whose publisher falls below a credibility threshold (with a fallback heuristic for unrated domains), renders and reads the full text of the top surviving page, and uses a long-context language model to extract only passages relevant to the subclaim. These passages feed a weighted voting step that emits the final supported or not_supported verdict along with an explanation.

What would settle it

Re-run the HoVER 3-hop and SciFact-Open evaluations while logging, for every page the system reads, when the text that decides each verdict was actually written; if any meaningful share of the deciding passages post-dates the dataset cutoff (late 2020 for HoVer and SciFact, late 2021 for FEVEROUS), the temporal filter is leaky and the reported 12.3% gain is not a like-for-like comparison. A cleaner variant retrieves an archived copy of each page as of its cutoff date and checks whether the verdicts change; the explanation-quality claim is separately testable by having human annotators apply the paper's own coverage, soundness, and readability criteria to the same outputs.

Watch

Extended reading notes

Core claim

The paper's central claim is that its multi-agent system (MAS) beats the four representative baseline paradigms — direct closed-book prompting, chain-of-thought, self-ask with search, and the FOLK first-order-logic framework — on the benchmark tasks because of two design choices the alternatives lack: reading the full text of retrieved pages rather than search snippets, and filtering sources through a professional publisher-credibility rating before any evidence reaches the verdict stage. With GPT-4o-mini as the foundation model and three search queries per subclaim, the system reports Macro F1 scores of 0.600, 0.617, and 0.507 on HoVER 2-, 3-, and 4-hop claims against best-baseline scores of 0.595, 0.501, and 0.466, and 0.770 on SciFact-Open against 0.737, for a 12.3% relative improvement overall and wins in six of seven settings. The authors attribute the advantage to the structured pipeline: decomposing a claim into a conjunction of predicates makes each retrieval step narrower, and the weighted voting over consistent credible evidence reduces the risk that a single unreliable snippet decides the verdict. A secondary claim, backed by an LLM-as-judge ranking on the coverage, soundness, and readability criteria, is that MAS explanations are the most preferred in five of seven tasks.

Load-bearing premise

The system's reported gains assume that setting the search engine's time filter to each dataset's cutoff date actually quarantines the pipeline from post-publication information, even though the pages themselves are fetched live and read in full and could have been updated after those cutoffs.

Editorial extensions

If this is right

  • Full-document reading plus credibility filtering, not stronger reasoning prompts alone, is where the accuracy gains come from; the largest margins appear in the multi-hop settings where the snippet-based baselines are weakest.
  • The same zero-shot pipeline transfers across three benchmark domains — Wikipedia-style multi-hop, table-plus-text, and open-domain scientific claims — with one foundation model and no fine-tuning.
  • Verdict accuracy improves as the query agent generates more questions per subclaim — HoVER 2-hop rises from 0.472 at one question to 0.600 at three, and SciFact-Open from 0.462 to 0.770 — before plateauing or declining as questions grow past three or four, since redundant queries add noise.
  • Every verdict ships with an evidence-referencing explanation, and on the paper's LLM-judge evaluation those explanations rank first in five of seven settings across the coverage, soundness, and readability criteria.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An ablation the paper does not run — switching the credibility filter off or downgrading full-text reading to snippets within MAS itself — would isolate whether the multi-hop gains come from the logical decomposition or from the retrieval upgrades; the paper's own link-survival data suggest the two effects concentrate in different datasets.
  • Because the verdict stage is a weighted vote over visible evidence, a natural extension is confidence-aware abstention: when surviving credible sources conflict, the system could report 'unverifiable' instead of forcing a binary label, which the paper's non-verifiable-subclaim filter already gestures toward.
  • The architecture's transfer to non-English or culturally different claims is bounded mainly by the US-centric search configuration and the limited reach of the credibility database, both of which the paper names as limitations; the decomposition and verdict stages are language-agnostic in principle.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a multi-agent fact-checking system that decomposes claims into FOL-style predicates, generates multiple search queries per subclaim, retrieves full-text evidence from credible web sources via SerperAPI and MBFC filtering, and produces supported/not_supported verdicts with explanations. The system is evaluated on 100-example stratified samples from HoVER, FEVEROUS, and SciFact with GPT-4o-mini, Llama-3.2-1B, and Qwen-2.5-3B, and compared with Direct, CoT, Self-Ask with search, and FOLK baselines. The authors report a 12.3% relative Macro F1 improvement over baselines and superior performance in six of seven evaluation tasks, with the largest gains on HoVER 3-hop and 4-hop.

Significance. Strong fact-checking pipelines that combine decomposable reasoning with credibility-aware full-text retrieval would be a valuable contribution to the AI-for-social-good area. The proposed architecture is plausible and the authors have made the code available, which supports reproducibility. However, the quantitative evidence for the claimed advantage is currently weak: the evaluation is small, lacks significance testing, and is subject to tuning on the same samples. If the temporal-leakage concern is resolved and the evaluation is performed on a proper held-out set, the system's architecture could still be of interest to the community.

major comments (5)
  1. [IV.D, Table II] The evaluation uses 100 examples per dataset with no confidence intervals or significance tests. In Table II, the margin over the best baseline is as small as 0.005 (HoVER 2-hop) and 0.033 (SciFact), which with n=100 corresponds to one or a few examples. Without variance estimates, the claimed superiority is not established.
  2. [IV.C and IV.E] The number of queries per subclaim (k=3) was selected by evaluating k=1..5 on the same 100-example samples used to report the main results. This is a form of model selection on the test set and inflates the reported scores. A separate development set or nested validation is required for an unbiased estimate.
  3. [III-C1 and III-C3] The temporal boundary parameter tbs restricts search results, not the content of the pages fetched by Selenium. A page updated after the dataset cutoff can contain post-publication information that the system reads, which could leak future knowledge into the verdict. The Limitations section (Section VII) does not mention this risk. A direct leakage audit is needed, and the authors should either filter content by page revision date or use a snapshot service.
  4. [IV.D] The claim of 'superior performance in six of seven evaluation tasks' is contradicted by Table II. MAS is below the best baseline on FEVEROUS Numerical (0.548 vs 0.553 for SA+SE) and FEVEROUS Multi-hop (0.601 vs 0.630 for FOLK). The abstract's 12.3% relative improvement is not derived from the per-task results; the authors need to define the baseline and aggregation used.
  5. [III and IV] The paper does not isolate the contributions of the individual components (claim decomposition, multi-query generation, credibility filtering, full-text extraction) through ablations. Without ablations, the advantage of the multi-agent architecture over simpler retrieval-augmented baselines is not convincingly demonstrated.
minor comments (5)
  1. [III-A1] The sentence 'C = p1 ∧ p2 ∧ ...' conflates predicates with propositions; this is formally imprecise and may confuse readers.
  2. [Figure 2] Figure 2 is referenced in the text but does not appear in the manuscript; please include the figure or remove the reference.
  3. [Table I] Table I lists 'pro-science' under political bias, which is an unusual category; the authors should clarify whether this is a factuality rating.
  4. [VII] The paper has several typos, e.g., 'does not in-corporate human evaluation' in Section VII and the awkward phrasing in Section III-C; the manuscript would benefit from proofreading.
  5. [III-C] The novelty claim 'To the best of our knowledge, our work is the first...' should be supported by a more thorough related-work comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported gains rest on an independent empirical comparison, not on self-defined targets, fitted outputs, or load-bearing self-citations.

full rationale

Reviewing the full derivation chain (claim decomposition, query generation, evidence retrieval, credibility filtering, verdict synthesis, and evaluation), I find no step in which a predicted quantity is defined in terms of the target quantity, no fitted parameter that is relabeled as a prediction, and no load-bearing premise resting on a self-citation. The system's supported/not_supported verdict is produced by an LLM from retrieved evidence using the prompts in the appendix; the baselines (Direct, CoT, SA+SE, FOLK) are run under the same GPT-4o-mini foundation, and the comparison is empirical rather than definitional. The choice k=3 is a hyperparameter selected from a 1-5 sweep on the same evaluation sample, which may risk selection bias and should be flagged as a methodological concern, but it does not make the verdicts true by construction. The paper's temporal-boundary mechanism may be leaky (tbs filters search results while Selenium renders full page content), but that is a data-contamination/evaluation-validity issue, not circularity. The reported 12.3% relative improvement is not straightforwardly reproducible from the column means in Table II, but an arithmetic inconsistency is a correctness issue rather than a circular reduction. No self-citations by the present authors are load-bearing; the cited prior work (e.g., FOLK [33] and Self-Ask [36]) is external. Therefore the central empirical claim has independent content and is not forced by the paper's own definitions or fitted inputs.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on two hand-tuned parameters (query count and credibility thresholds) and on domain assumptions about FOL decomposition, MBFC reliability, and temporal leakage prevention. No new physical or formal entities are introduced.

free parameters (2)
  • number_of_search_queries_per_subclaim = 3
    Tuned on the validation subsamples (values 1 to 5 tested, Figure 3 shows F1 varying from 0.472 with 1 query to 0.600 with 3 queries on HoVER 2-Hop). The headline results use k=3, so the score depends on this hand-picked value.
  • credibility_filter_thresholds = factuality: very high, high, mostly factual; bias: least biased, left-center, right-center, pro-science
    Manually chosen in Table I. These thresholds determine which sources survive for evidence extraction, directly changing the information available to the verdict agent and hence the final labels.
assumptions (3)
  • domain assumption A claim is equivalent to the conjunction of its FOL predicates (C = p1 ^ ... ^ pn), and the claim is supported iff every predicate is true.
    Section III-A1: the Input Ingestion Agent decomposes claims this way, so if the FOL decomposition misses a predicate or the conjunction rule is wrong, the final verdict is invalid.
  • domain assumption MBFC credibility ratings are accurate and complete enough that filtering by the chosen thresholds removes unreliable evidence without discarding relevant reliable evidence.
    Section III-C2 and Table I: the Evidence Seeking Agent discards all sources failing the credibility thresholds, so any errors or gaps in the MBFC database propagate into the evidence set and the verdict.
  • domain assumption The search engine and full-text extraction respect the dataset-specific temporal boundaries, preventing temporal leakage.
    Section III-C1: tbs end dates are set to benchmark creation dates, but the system also reads full page content that may have been updated after those dates, so the assumption is fragile.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Robust Fact-Checking: A Multi-Agent System with Advanced Evidence Retrieval." pith.science (2026). https://pith.science/paper/TKZ5LKBE

@misc{pith2026250617878,
  author       = {Pith},
  title        = {Pith review of: Towards Robust Fact-Checking: A Multi-Agent System with Advanced Evidence Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TKZ5LKBE}},
  note         = {Machine review of arXiv:2506.17878}
}
read the original abstract

The rapid spread of misinformation in the digital era poses significant challenges to public discourse, necessitating robust and scalable fact-checking solutions. Traditional human-led fact-checking methods, while credible, struggle with the volume and velocity of online content, prompting the integration of automated systems powered by Large Language Models (LLMs). However, existing automated approaches often face limitations, such as handling complex claims, ensuring source credibility, and maintaining transparency. This paper proposes a novel multi-agent system for automated fact-checking that enhances accuracy, efficiency, and explainability. The system comprises four specialized agents: an Input Ingestion Agent for claim decomposition, a Query Generation Agent for formulating targeted subqueries, an Evidence Retrieval Agent for sourcing credible evidence, and a Verdict Prediction Agent for synthesizing veracity judgments with human-interpretable explanations. Evaluated on benchmark datasets (FEVEROUS, HOVER, SciFact), the proposed system achieves a 12.3% improvement in Macro F1-score over baseline methods. The system effectively decomposes complex claims, retrieves reliable evidence from trusted sources, and generates transparent explanations for verification decisions. Our approach contributes to the growing field of automated fact-checking by providing a more accurate, efficient, and transparent verification methodology that aligns with human fact-checking practices while maintaining scalability for real-world applications. Our source code is available at https://github.com/HySonLab/FactAgent

Figures

Figures reproduced from arXiv: 2506.17878 by the authors.

Figure 1
Figure 1. Overview of the multi-agent system. The system [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed architecture. The system takes input data and decomposes claims using Input Ingestion Agent. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Macro F1-score of varying the number of generated questions per claim from 1 to 5 for Query Generation Agent. The [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Comparison of total retrieved links and credible links [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

65 extracted references · 52 canonical work pages

  1. [1]

    A survey on fake news and rumour detection techniques,

    A. Bondielli and F. Marcelloni, “A survey on fake news and rumour detection techniques,” Information Sciences, vol. 497, pp. 38–55, 2020

    work page 2020

  2. [2]

    Understanding the promise and limits of automated fact- checking,

    L. Graves, “Understanding the promise and limits of automated fact- checking,” Factsheet, 2018

    work page 2018

  3. [3]

    The science of fake news,

    D. M. J. Lazer, M. A. Baum, Y . Benkler, A. J. Berinsky, K. M. Greenhill, F. Menczer, M. J. Metzger, B. Nyhan, G. Pennycook, D. Rothschild, M. Schudson, S. A. Sloman, C. R. Sunstein, E. A. Thorson, D. J. Watts, and J. L. Zittrain, “The science of fake news,” Science, vol. 359, no. 6380, pp. 1094–1096, 2018

    work page 2018

  4. [4]

    The covid-19 social media infodemic,

    M. Cinelli, W. Quattrociocchi, A. Galeazzi, C. M. Valensise, E. Brugnoli, A. L. Schmidt, P. Zola, F. Zollo, and A. Scala, “The covid-19 social media infodemic,” Scientific Reports, vol. 10, no. 1, p. 16598, 2020

    work page 2020

  5. [5]

    Fake news detection on social media: A data mining perspective,

    K. Shu, A. Sliva, S. Wang, J. Tang, and H. Liu, “Fake news detection on social media: A data mining perspective,” ACM SIGKDD Explorations Newsletter, vol. 19, no. 1, pp. 22–36, 2017

    work page 2017

  6. [6]

    A survey of large language models,

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, Z. Dong et al. , “A survey of large language models,” arXiv preprint arXiv:2303.18223, 2023

    arXiv 2023

  7. [7]

    Survey of hallucination in natural language generation,

    Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y . Xu, E. Ishii, Y . Bang, A. Madotto, and P. Fung, “Survey of hallucination in natural language generation,” ACM Computing Surveys , vol. 55, no. 12, pp. 1–38, 2023

    work page 2023

  8. [8]

    Knowledge staleness in large language models,

    S. Feng, W. Jiang, C. Graber, K. Vafa, N. Mathur, Y . Zhou, B. I. P. Rubinstein, and J. Leskovec, “Knowledge staleness in large language models,” arXiv preprint arXiv:2310.19215 , 2023

    arXiv 2023

Show all 65 references
  1. [9]

    Retrieval- augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschel et al. , “Retrieval- augmented generation for knowledge-intensive nlp tasks,” inAdvances in Neural Information Processing Systems , vol. 33, 2020, pp. 9459–9474

    2020

  2. [10]

    Chain of thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain of thought prompting elicits reasoning in large language models,” arXiv preprint arXiv:2201.11903 , 2022. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 10

    2022 arXiv

  3. [11]

    Factool: Factuality detection in generative ai,

    B. Chern, S. Peng, A. Verma, X. Yin, A. Ranade, N. Jindal, R. Joshi, A. Saxena, and R. Sarikaya, “Factool: Factuality detection in generative ai,” arXiv preprint arXiv:2307.13528 , 2023

    2023 arXiv

  4. [12]

    Where the truth lies: Explaining the credibility of emerging claims on the web and social media,

    K. Popat, S. Mukherjee, J. Str ¨otgen, and G. Weikum, “Where the truth lies: Explaining the credibility of emerging claims on the web and social media,” Proceedings of the 26th International Conference on World Wide Web Companion, pp. 1003–1012, 2017

    2017

  5. [13]

    FEVEROUS: Fact extraction and VERification over unstructured and structured information,

    R. Aly, Z. Guo, M. S. Schlichtkrull, J. Thorne, A. Vlachos, C. Christodoulopoulos, O. Cocarascu, and A. Mittal, “FEVEROUS: Fact extraction and VERification over unstructured and structured information,” in Thirty-fifth Conference on Neural Information Processing Systems Datase...

    2021

  6. [14]

    HOVER: A dataset for many-hop fact extraction and claim verification,

    Y . Jiang, S. Bordia, Z. Zhong, C. Dognin, M. Singh, and M. Bansal, “HOVER: A dataset for many-hop fact extraction and claim verification,” in Findings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 3441–3460

    2020

  7. [15]

    Fact or fiction: Verifying scientific claims,

    D. Wadden, S. Lin, K. Lo, L. L. Wang, M. van Zuylen, A. Cohan, and H. Hajishirzi, “Fact or fiction: Verifying scientific claims,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2020, pp. 7534–7550

    2020

  8. [16]

    Politifact,

    PolitiFact, “Politifact,” https://www.politifact.com/, accessed: 2025-03- 20

    2025

  9. [17]

    Snopes, “Snopes,” https://www.snopes.com/, accessed: 2025-03-20

    2025

  10. [18]

    Factcheck,

    FactCheck, “Factcheck,” https://www.factcheck.org/, accessed: 2025-03- 20

    2025

  11. [19]

    Revisiting the epistemology of fact-checking,

    M. A. Amazeen, “Revisiting the epistemology of fact-checking,” Critical Review, vol. 27, no. 1, pp. 1–22, 2015

    2015

  12. [20]

    Fact checking: Task definition and dataset construction,

    A. Vlachos and S. Riedel, “Fact checking: Task definition and dataset construction,” in Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science , 2014, pp. 18–22

    2014

  13. [21]

    Toward automated fact-checking: Detecting check-worthy factual claims by claimbuster,

    N. Hassan, G. Zhang, F. Arslan, J. Caraballo, D. Jimenez, S. Gawsane, S. Hasan, M. Joseph, A. Kulkarni, A. K. Nayak et al. , “Toward automated fact-checking: Detecting check-worthy factual claims by claimbuster,” in Proceedings of the 23rd ACM SIGKDD International Conference o...

    2017

  14. [22]

    Declare: Debunking fake news and false claims using evidence-aware deep learning,

    K. Popat, S. Mukherjee, A. Yates, and G. Weikum, “Declare: Debunking fake news and false claims using evidence-aware deep learning,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 22–32

    2018

  15. [23]

    MultiFC: A real-world multi-domain dataset for evidence-based fact checking of claims,

    I. Augenstein, C. Lioma, D. Wang, L. C. Lima, C. Hansen, C. Hansen, and J. G. Pedersen, “MultiFC: A real-world multi-domain dataset for evidence-based fact checking of claims,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9t...

    2019

  16. [24]

    GEAR: Graph-based evidence aggregating and reasoning for fact verification,

    J. Zhou, X. Han, C. Yang, Z. Liu, L. Wang, C. Li, and M. Sun, “GEAR: Graph-based evidence aggregating and reasoning for fact verification,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 892–901

    2019

  17. [25]

    Detecting covid-19 misinformation with veracity assessment of social media posts via deep learning,

    C. Stiff and F. Johansson, “Detecting covid-19 misinformation with veracity assessment of social media posts via deep learning,” Journal of Medical Internet Research , vol. 23, no. 9, p. e30315, 2021

    2021

  18. [26]

    Towards few-shot fact-checking via perplexity,

    N. Lee, B. Z. Li, S. Wang, W.-t. Yih, H. Ma, and M. Khabsa, “Towards few-shot fact-checking via perplexity,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 2021, pp. 1971–1981

    2021

  19. [27]

    Quantifying and analyzing hal- lucinations in large language models,

    Y . Du, S. Ding, Z. Zhao, Y . Lin, R. Nallapati, B. Xiang, B. Zhou, D. Roth, L. Zettlemoyer, X. Liang et al., “Quantifying and analyzing hal- lucinations in large language models,” arXiv preprint arXiv:2310.00905, 2023

    2023 arXiv

  20. [28]

    The rise and potential of large language model based agents: A survey,

    Z. Xi, W. Chen, X. Guo, W. He, Y . Ding, X. Hong, Y . Luo, W. Liang, L. Bing, L. Si et al. , “The rise and potential of large language model based agents: A survey,” arXiv preprint arXiv:2309.07864 , 2023

    2023 arXiv

  21. [29]

    Can large language models provide faithful explana- tions for fake news detection?

    Y . Li and K. Shu, “Can large language models provide faithful explana- tions for fake news detection?” arXiv preprint arXiv:2305.15005, 2023

    2023 arXiv

  22. [30]

    Web agents: Evaluating the reliability of web search engines for ai agents,

    X. Zhang, H. Jiang, W. Yin, X. Ren, and J. Han, “Web agents: Evaluating the reliability of web search engines for ai agents,” arXiv preprint arXiv:2312.09254, 2023

    2023 arXiv

  23. [31]

    Self-checker: Plug-and-play modules for fact-checking with large language models,

    M. Wang, Z. Yin, M. Guo, X. Jiang, X. Ren, and J. Han, “Self-checker: Plug-and-play modules for fact-checking with large language models,” arXiv preprint arXiv:2305.14623 , 2023

    2023 arXiv

  24. [32]

    H. B. Enderton, A mathematical introduction to logic . Elsevier, 2001

    2001

  25. [33]

    Explainable claim verification via knowledge- grounded reasoning with large language models,

    H. Wang and K. Shu, “Explainable claim verification via knowledge- grounded reasoning with large language models,” in Findings of the Association for Computational Linguistics: EMNLP 2023 , H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguis...

    2023

  26. [34]

    Claim matching beyond english to scale global fact-checking,

    D. Micallef, K. Kakaes, A. Haghighi, S. Lightseed, L. Gu, Q. Liao, I. Liskovich, Y . W. Tay, and E. Kamar, “Claim matching beyond english to scale global fact-checking,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long ...

    2022

  27. [35]

    Claim de- tection in biomedical twitter posts,

    L. Konstantinovskiy, O. Price, M. Babakar, and A. Zubiaga, “Claim de- tection in biomedical twitter posts,” in Proceedings of the 20th Workshop on Biomedical Language Processing . Association for Computational Linguistics, 2021, pp. 131–142

    2021

  28. [36]

    Measuring and narrowing the compositionality gap in language models,

    O. Press, M. Zhang, S. Min, L. Schmidt, N. Smith, and M. Lewis, “Measuring and narrowing the compositionality gap in language models,” in Findings of the Association for Computational Linguistics: EMNLP 2023 , H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for C...

    2023

  29. [37]

    Search engine optimization: what drives organic traffic to retail sites?

    M. R. Baye, B. De los Santos, and M. R. Wildenbeest, “Search engine optimization: what drives organic traffic to retail sites?” Journal of Economics & Management Strategy , vol. 25, no. 1, pp. 6–31, 2016

    2016

  30. [38]

    Investigating search engine optimization techniques for effective ranking: A case study of an educational site,

    S. Krrabaj, F. Baxhaku, and D. Sadrijaj, “Investigating search engine optimization techniques for effective ranking: A case study of an educational site,” in 2017 6th Mediterranean Conference on Embedded Computing (MECO), 2017, pp. 1–4

    2017

  31. [39]

    Serper: Real-time google search api,

    Serper, “Serper: Real-time google search api,” https://serper.dev, 2024, accessed: 2025-04-05

    2024

  32. [40]

    Media bias/fact check api,

    “Media bias/fact check api,” https://mediabiasfactcheck.com/, accessed: 2025-04-18

    2025

  33. [41]

    Credibility and trust of information in online environments: The use of cognitive heuristics,

    M. J. Metzger and A. J. Flanagin, “Credibility and trust of information in online environments: The use of cognitive heuristics,” Journal of Pragmatics , vol. 59, pp. 210–220, 2013, biases and constraints in communication: Argumentation, persuasion and manipulation. [Online]. ...

    2013

  34. [42]

    Knowledge-based trust: Estimating the trustwor- thiness of web sources,

    X. L. Dong, E. Gabrilovich, K. Murphy, V . Dang, W. Horn, C. Lugaresi, S. Sun, and W. Zhang, “Knowledge-based trust: Estimating the trustwor- thiness of web sources,” in Proceedings of the VLDB Endowment, vol. 8, no. 9. VLDB Endowment, 2015, pp. 938–949

    2015

  35. [43]

    Correcting misinformation on social media with a large language model,

    X. Zhou, A. Sharma, A. X. Zhang, and T. Althoff, “Correcting misinformation on social media with a large language model,” ArXiv, vol. abs/2403.11169, 2024. [Online]. Available: https://api. semanticscholar.org/CorpusID:268513555

    2024

  36. [44]

    Towards LLM-based fact verification on news claims with a hierarchical step-by-step prompting method,

    X. Zhang and W. Gao, “Towards LLM-based fact verification on news claims with a hierarchical step-by-step prompting method,” in Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Assoc...

    2023

  37. [45]

    Selenium: Web browser automation,

    Selenium Contributors, “Selenium: Web browser automation,” https:// www.selenium.dev/, 2023, accessed: 2025-04-24

    2023

  38. [46]

    Beautiful soup documentation,

    L. Richardson, “Beautiful soup documentation,” 2007

    2007

  39. [47]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,

    G. Team, P. Georgiev, V . I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang et al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  40. [48]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” in Proceedings of the 36th International Conference on Neural Information Processing Systems , ser. NIPS ’22...

    2022

  41. [49]

    Language models are few-shot learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B...

    2020

  42. [50]

    Judging JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 11 llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 11 llm-as-a-judge with mt-bench and chatbot arena,” in Proceedings of ...

    2021

  43. [51]

    Generating fact checking explanations,

    P. Atanasova, J. G. Simonsen, C. Lioma, and I. Augenstein, “Generating fact checking explanations,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, Eds. Online: Association for Co...

    2020

  44. [52]

    Managing multi-regional and multilingual sites,

    Google, “Managing multi-regional and multilingual sites,” 2024, accessed 15 June 2025. [Online]. Available: https://developers.google. com/search/docs/specialty/international/managing-multi-regional-sites

    2024

  45. [53]

    Developing a unifying framework of credibility assessment: Construct, heuristics, and interaction in context,

    B. Hilligoss and S. Y . Rieh, “Developing a unifying framework of credibility assessment: Construct, heuristics, and interaction in context,” Information Processing & Management , vol. 44, no. 4, pp. 1467–1484, 2008. [Online]. Available: https://www.sciencedirect.com/ science/...

    2008

  46. [54]

    Multimodal fake news detection with textual, visual and semantic information,

    A. Giachanou, G. Zhang, and P. Rosso, “Multimodal fake news detection with textual, visual and semantic information,” in Text, Speech, and Dialogue: 23rd International Conference, TSD 2020, Brno, Czech Republic, September 8–11, 2020, Proceedings . Berlin, Heidelberg: Springer-...

    2020 doi

  47. [55]

    Use Specific Keywords: Include precise terms related to entities and relationships in the claim

  48. [56]

    Incorporate Synonyms and Related Terms: Use alternative phrasings to overcome vocabulary mismatches

  49. [57]

    Vary Specificity: Generate both specific queries targeting exact details and broader queries that may capture contextual information

  50. [58]

    Consider Different Angles: Approach the claim from multiple perspectives to ensure comprehensive evidence gathering

  51. [59]

    claim":

    Maintain Simplicity: Keep questions straightforward and directly relevant to the claim. Return the output in JSON format like this: [{ "claim": "Location(Howard Hospital, Washington D.C.) ::: Verify Howard University Hospital is located in Washington, D.C.", "questions": ["Whe...

    2021

  52. [60]

    - Assess the credibility, consistency, and reliability of each piece of evidence

    Analyze the Retrieved Evidence - Review all provided evidence relevant to the subclaim. - Assess the credibility, consistency, and reliability of each piece of evidence

  53. [61]

    supported

    Apply a Voting System for Classification - If multiple sources strongly support the subclaim, classify it as "supported". - If multiple sources contradict the subclaim, classify it as "not_supported". - If the evidence is mixed, insufficient, or inconclusive, classify it as "n...

  54. [62]

    supported

    Provide a Justification - Clearly explain why the subclaim is classified as "supported" or "not_supported". - Reference key pieces of evidence that influenced your decision. - If the evidence is inconclusive, explain the limitations or uncertainties. - Remember to adjust not t...

  55. [63]

    Coverage: To what extent the explanation includes all the salient and relevant information necessary to verify the claim

  56. [64]

    Soundness: The logical consistency of the explanation; whether it supports or contradicts its own label and the original claim

  57. [65]

    original_claim

    Readability: The clarity and coherence of the explanation; how easily a human can follow and understand it. - Provide a **ranking (1 for best, 4 for worst) ** for each criterion. Here is the input: { "original_claim": "", "explanations": { "CoT": { "label": "<label>", "explana...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.