REVIEW 14 cited by
Retrieval-Augmented Generation with Conflicting Evidence
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Retrieval-Augmented Generation with Conflicting Evidence
read the original abstract
Large language model (LLM) agents are increasingly employing retrieval-augmented generation (RAG) to improve the factuality of their responses. However, in practice, these systems often need to handle ambiguous user queries and potentially conflicting information from multiple sources while also suppressing inaccurate information from noisy or irrelevant documents. Prior work has generally studied and addressed these challenges in isolation, considering only one aspect at a time, such as handling ambiguity or robustness to noise and misinformation. We instead consider multiple factors simultaneously, proposing (i) RAMDocs (Retrieval with Ambiguity and Misinformation in Documents), a new dataset that simulates complex and realistic scenarios for conflicting evidence for a user query, including ambiguity, misinformation, and noise; and (ii) MADAM-RAG, a multi-agent approach in which LLM agents debate over the merits of an answer over multiple rounds, allowing an aggregator to collate responses corresponding to disambiguated entities while discarding misinformation and noise, thereby handling diverse sources of conflict jointly. We demonstrate the effectiveness of MADAM-RAG using both closed and open-source models on AmbigDocs -- which requires presenting all valid answers for ambiguous queries -- improving over strong RAG baselines by up to 11.40% and on FaithEval -- which requires suppressing misinformation -- where we improve by up to 15.80% (absolute) with Llama3.3-70B-Instruct. Furthermore, we find that RAMDocs poses a challenge for existing RAG baselines (Llama3.3-70B-Instruct only obtains 32.60 exact match score). While MADAM-RAG begins to address these conflicting factors, our analysis indicates that a substantial gap remains especially when increasing the level of imbalance in supporting evidence and misinformation.
Forward citations
Cited by 14 Pith papers
-
EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding
EgoMemReason is a new benchmark showing that even the best multimodal models achieve only 39.6% accuracy on reasoning tasks that require integrating sparse evidence across days in egocentric video.
-
CounterRefine: Answer-Conditioned Counterevidence Retrieval for Inference-Time Knowledge Repair in Factual Question Answering
CounterRefine improves factual QA by retrieving answer-conditioned counterevidence and deterministically refining draft answers, lifting a GPT-5 RAG baseline by 5.8 points to 73.1% on SimpleQA.
-
CounterRefine: Answer-Conditioned Counterevidence Retrieval for Inference-Time Knowledge Repair in Factual Question Answering
CounterRefine improves factual QA accuracy by up to 5.8 points on SimpleQA through answer-conditioned counterevidence retrieval and validated refinement with minimal output changes.
-
To Isolate or to Score? Model-Adaptive Assessment for Cost-Efficient Multi-Agent RAG
Empirical study finds isolation drives gains for weak models in multi-agent RAG while scoring matters for strong ones, enabling MADARA for cost-efficient adaptive assessment.
-
X-MADAM-RAG: Diagnosing and Handling Chinese-English Evidence Conflict in Retrieval-Augmented Generation
Introduces controlled benchmark X-RAMDocs-ZHEN and X-MADAM-RAG pipeline for Chinese-English evidence conflict in RAG, with high accuracy on templated data but sharp drop on naturalized stress test revealing extraction...
-
Detecting Is Not Resolving: The Monitoring Control Gap in Retrieval Augmented LLMs
RAG models exhibit a monitoring-control gap: they acknowledge epistemic conflicts in accumulating documents yet fail to constrain unsafe recommendations, with single-turn tests overestimating safety.
-
Remembering More, Risking More: Longitudinal Safety Risks in Memory-Equipped LLM Agents
Memory-equipped LLM agents exhibit increasing safety violation rates as memory accumulates across independent tasks, termed temporal memory contamination, detected via a new trigger-probe protocol.
-
Can It Reach the Generator? Investigating the Survival of Prompt-Injection Attacks in Realistic RAG Settings
Gradient-based and instruction-override prompt injections largely fail to survive retrieval and reranking in realistic RAG systems, while only LLM-driven injections remain effective end-to-end, and all attacks are det...
-
ConflictRAG: Detecting and Resolving Knowledge Conflicts in Retrieval Augmented Generation
ConflictRAG adds conflict detection, source credibility assessment via Entropy-TOPSIS, and a CARS diagnostic score to RAG pipelines, reporting 88.7% F1 detection and 5.3-6.1% correctness gains on three benchmarks.
-
ConflictRAG: Detecting and Resolving Knowledge Conflicts in Retrieval Augmented Generation
ConflictRAG introduces a conflict-aware RAG pipeline with two-stage detection (MLP + selective LLM), Entropy-TOPSIS credibility assessment, and a new CARS metric, reporting 88.7% F1 and 5.3-6.1% gains on benchmarks.
-
Do LLMs have core beliefs?
LLMs generally fail to maintain stable worldviews under adversarial conversational pressure, indicating they lack core beliefs akin to those in human cognition.
-
LLM-Oriented Information Retrieval: A Denoising-First Perspective
Denoising to maximize usable evidence density and verifiability is becoming the primary bottleneck in LLM-oriented information retrieval, conceptualized via a four-stage framework and addressed through a pipeline taxo...
-
Improving Factuality in LLMs via Inference-Time Knowledge Graph Construction
A framework for inference-time knowledge graph construction and expansion improves factual accuracy in LLMs on three QA benchmarks by combining internal LLM knowledge with selective external retrieval.
-
LLM-Oriented Information Retrieval: A Denoising-First Perspective
Argues for a denoising-first paradigm in LLM-oriented information retrieval, framing challenges via a four-stage progression and providing a taxonomy of signal-to-noise optimization techniques across the pipeline.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.