Pith. sign in

REVIEW 10 cited by

Resolving Conflicting Evidence in Automated Fact-Checking: A Study on Retrieval-Augmented LLMs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2505.17762 v1 pith:2OSCSDAR submitted 2025-05-23 cs.CL cs.IR

Resolving Conflicting Evidence in Automated Fact-Checking: A Study on Retrieval-Augmented LLMs

Ziyu Ge , Yuhao Wu , Daniel Wai Kit Chin , Roy Ka-Wei Lee , Rui Cao This is my paper
classification cs.CL cs.IR
keywords conflictingevidencefact-checkingcredibilitymodelstextbfconfactdataset
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Large Language Models (LLMs) augmented with retrieval mechanisms have demonstrated significant potential in fact-checking tasks by integrating external knowledge. However, their reliability decreases when confronted with conflicting evidence from sources of varying credibility. This paper presents the first systematic evaluation of Retrieval-Augmented Generation (RAG) models for fact-checking in the presence of conflicting evidence. To support this study, we introduce \textbf{CONFACT} (\textbf{Con}flicting Evidence for \textbf{Fact}-Checking) (Dataset available at https://github.com/zoeyyes/CONFACT), a novel dataset comprising questions paired with conflicting information from various sources. Extensive experiments reveal critical vulnerabilities in state-of-the-art RAG methods, particularly in resolving conflicts stemming from differences in media source credibility. To address these challenges, we investigate strategies to integrate media background information into both the retrieval and generation stages. Our results show that effectively incorporating source credibility significantly enhances the ability of RAG models to resolve conflicting evidence and improve fact-checking performance.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ArbGraph: Conflict-Aware Evidence Arbitration for Reliable Long-Form Retrieval-Augmented Generation

    cs.CL 2026-04 unverdicted novelty 7.0

    ArbGraph resolves conflicts in RAG evidence by constructing a conflict-aware graph of atomic claims and applying intensity-driven iterative arbitration to suppress unreliable claims prior to generation.

  2. When Specifications Conflict: A Symmetry-Based Framework for Measuring LLM Preferences

    cs.AI 2026-07 conditional novelty 6.0

    Under controlled conflicting specifications, LLMs systematically prefer formal and naturalized-formal task descriptions over pure natural language and input–output examples, with domain-specific exceptions.

  3. MEDIAREF: A Public Knowledge Store for Media Background Checks

    cs.CL 2026-07 unverdicted novelty 6.0

    MEDIAREF is a publicly available knowledge store of documents from 200 media sources that enables low-cost, reproducible evaluation of media background check generation for fact-checking systems.

  4. Courtroom-Style Multi-Agent Debate with Progressive RAG and Role-Switching for Controversial Claim Verification

    cs.CL 2026-03 unverdicted novelty 6.0

    PROClaim achieves 81.7% accuracy on Check-COVID claim verification by combining courtroom roles, progressive RAG, and multi-judge aggregation, outperforming standard multi-agent debate by 10 percentage points.

  5. MEDIAREF: A Public Knowledge Store for Media Background Checks

    cs.CL 2026-07 conditional novelty 5.5

    MEDIAREF is a public, updatable web-document store that lets LLMs generate media background checks more reproducibly and with higher fact recall than zero-shot generation alone.

  6. MEDIAREF: A Public Knowledge Store for Media Background Checks

    cs.CL 2026-07 conditional novelty 5.0

    MEDIAREF is a 21,921-document knowledge store covering 200 media outlets that enables cheaper, reproducible media background check generation; adding its evidence to LLM prompts modestly improves fact recall without r...

  7. ConflictRAG: Detecting and Resolving Knowledge Conflicts in Retrieval Augmented Generation

    cs.CL 2026-05 unverdicted novelty 5.0

    ConflictRAG introduces a conflict-aware RAG pipeline with two-stage detection (MLP + selective LLM), Entropy-TOPSIS credibility assessment, and a new CARS metric, reporting 88.7% F1 and 5.3-6.1% gains on benchmarks.

  8. ConflictRAG: Detecting and Resolving Knowledge Conflicts in Retrieval Augmented Generation

    cs.CL 2026-05 unverdicted novelty 5.0

    ConflictRAG adds conflict detection, source credibility assessment via Entropy-TOPSIS, and a CARS diagnostic score to RAG pipelines, reporting 88.7% F1 detection and 5.3-6.1% correctness gains on three benchmarks.

  9. Courtroom-Style Multi-Agent Debate with Progressive RAG and Role-Switching for Controversial Claim Verification

    cs.CL 2026-03 conditional novelty 4.0

    A courtroom-style multi-agent debate with progressive retrieval reaches 81.7% accuracy on Check-COVID binary claims, 10 points above a simple MAD baseline, though a single-call RAG baseline outperforms it.

  10. HaluNet: Learning Hallucination Risk from Internal Signals in LLM Question Answering

    cs.CL 2025-12 conditional novelty 4.0

    HaluNet fuses token-level log-likelihood, entropy, and hidden-state embeddings via a multi-branch neural network to rank hallucinated LLM QA answers with reported AUROC up to 0.922 in-domain.