Pith. sign in

REVIEW 17 cited by

Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.07176 v2 pith:CEZU4Y6H submitted 2024-10-09 cs.CL cs.AIcs.LG

Fei Wang , Xingchen Wan , Ruoxi Sun , Jiefeng Chen , Sercan Ö. Ar{i}k This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords knowledgeretrievalimperfectastuteconflictsexternalllmsaugmentation
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Retrieval augmented generation (RAG), while effectively integrating external knowledge to address the inherent limitations of large language models (LLMs), can be hindered by imperfect retrieval that contain irrelevant, misleading, or even malicious information. Previous studies have rarely connected the behavior of RAG through joint analysis, particularly regarding error propagation coming from imperfect retrieval and potential conflicts between LLMs' internal knowledge and external sources. Through comprehensive and controlled analyses under realistic conditions, we find that imperfect retrieval augmentation is inevitable, common, and harmful. We identify the knowledge conflicts between LLM-internal and external knowledge from retrieval as a bottleneck to overcome imperfect retrieval in the post-retrieval stage of RAG. To address this, we propose Astute RAG, a novel RAG approach designed to be resilient to imperfect retrieval augmentation. It adaptively elicits essential information from LLMs' internal knowledge, iteratively consolidates internal and external knowledge with source-awareness, and finalizes the answer according to information reliability. Our experiments with Gemini and Claude demonstrate the superior performance of Astute RAG compared to previous robustness-enhanced RAG approaches. Specifically, Astute RAG is the only RAG method that achieves performance comparable to or even surpassing conventional use of LLMs under the worst-case scenario. Further analysis reveals the effectiveness of Astute RAG in resolving knowledge conflicts, thereby improving the trustworthiness of RAG.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Information Discernment in Large Language Models

    cs.AI 2026-05 conditional novelty 7.0 of 10

    LLMs update their stated numeric beliefs almost regardless of source reliability or whether a claim moves them closer to the truth, performing near chance on both dimensions.

  2. RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment

    cs.CL 2024-12 conditional novelty 7.0 of 10

    RAG-RewardBench is the first benchmark for evaluating reward models in retrieval-augmented generation, and it shows top reward models score only 78.3% accuracy on RAG-specific preference pairs.

  3. HierSearch: A Hierarchical Enterprise Deep Search Framework Integrating Local and Web Searches

    cs.IR 2025-08 conditional novelty 6.0 of 10

    Hierarchical reinforcement learning with separate local and web deep search agents, plus an evidence refiner, beats flat RL and single-source deep search baselines on multi-source QA benchmarks.

  4. Query-driven Document-level Scientific Evidence Extraction from Biomedical Studies

    cs.CL 2025-05 conditional novelty 6.0 of 10

    CochraneForest is a new benchmark for document-level evidence extraction, and URCA, which combines uniform retrieval and clustering, achieves up to 10.3% higher F1 than prior RAG baselines on it.

  5. Retrieval-Augmented Generation with Conflicting Evidence

    cs.CL 2025-04 conditional novelty 6.0 of 10

    MADAM-RAG, a per-document multi-agent debate method for RAG, beats standard RAG on ambiguity and misinformation benchmarks while a new RAMDocs benchmark shows combined conflicts remain hard.

  6. RbFT: Robust Fine-tuning for Retrieval-Augmented Generation against Retrieval Defects

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Fine-tuning an LLM with defect detection and utility extraction tasks makes it more robust to noisy, irrelevant, and counterfactual documents in retrieval-augmented generation.

  7. Towards More Robust Retrieval-Augmented Generation: Evaluating RAG Under Adversarial Poisoning Attacks

    cs.IR 2024-12 conditional novelty 6.0 of 10

    A controlled evaluation shows skeptical prompting and robust retrievers soften the impact of adversarial corpus poisoning on RAG, with gains tied to model reasoning ability.

  8. RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Verification and Refinement

    cs.CL 2024-12 conditional novelty 6.0 of 10

    A Monte Carlo tree search reasoning framework where each node is a sub-question and its answer, and retrieval-augmented rewards guide the search toward verifiable answers.

  9. Generating Privacy Stories From Software Documentation

    cs.SE 2025-06 conditional novelty 5.0 of 10

    LLMs can extract privacy behaviors from software documents and draft privacy stories, but the best overall F1 is 0.766, not the abstract's 0.8+.

  10. CrEst: Credibility Estimation for Contexts in LLMs via Weak Supervision

    cs.CL 2025-06 conditional novelty 5.0 of 10

    A label-free method that scores retrieved documents by their agreement with the majority in embedding space and uses those scores to filter context in LLM question answering.

  11. Benchmarking Poisoning Attacks against Retrieval-Augmented Generation

    cs.CR 2025-05 conditional novelty 5.0 of 10

    A unified benchmark evaluation finds that existing RAG poisoning attacks remain effective on standard QA datasets, drop on expanded knowledge bases, and are only partially mitigated by current defenses.

  12. Chain-of-Defensive-Thought: Structured Reasoning Elicits Robustness in Large Language Models against Reference Corruption

    cs.CL 2025-04 conditional novelty 5.0 of 10

    A few-shot prompt that instructs LLMs to reason about which retrieved references are relevant and reliable (majority-supported) substantially improves robustness against prompt injection and knowledge corruption in RA...

  13. PaRT: Enhancing Proactive Social Chatbots with Personalized Real-Time Retrieval

    cs.AI 2025-04 conditional novelty 5.0 of 10

    PaRT combines user profiling, intent-guided query rewriting, and retrieval-augmented generation to make social chatbots proactive, and reports a 21.77% increase in average dialogue duration online.

  14. RAGtifier: Evaluating RAG Generation Approaches of State-of-the-Art RAG Systems for the SIGIR LiveRAG Competition

    cs.IR 2025-06 conditional novelty 4.0 of 10

    A RAG pipeline using InstructRAG, Pinecone, and BGE placed third in the 2025 LiveRAG Challenge, though internal evaluation only weakly predicted official scores.

  15. Position: Towards a Responsible LLM-empowered Multi-Agent Systems

    cs.MA 2025-02 unverdicted novelty 4.0 of 10

    This position paper argues that responsible LLM-based multi-agent systems require quantifiable agreement metrics, system-level uncertainty quantification, and a human-centered dynamic moderator.

  16. Context Awareness Gate For Retrieval Augmented Generation

    cs.LG 2024-11 conditional novelty 4.0 of 10

    A statistical gate named Vector Candidates decides when to skip retrieval in RAG systems, accompanied by the new CRSB benchmark.

  17. Automatic Dataset Generation for Knowledge Intensive Question Answering Tasks

    cs.CL 2025-05 reject novelty 3.0 of 10

    Fine-tuning Llama-3-8b and Mistral-7b-v0.3 on LLM-generated QA pairs from IBM Technotes can improve no-context QA scores over training on human-annotated TechQA data, but the evaluation may be inflated by test-documen...

Pith tools