REVIEW 6 cited by
Seven Failure Points When Engineering a Retrieval Augmented Generation System
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Software engineers are increasingly adding semantic search capabilities to applications using a strategy known as Retrieval Augmented Generation (RAG). A RAG system involves finding documents that semantically match a query and then passing the documents to a large language model (LLM) such as ChatGPT to extract the right answer using an LLM. RAG systems aim to: a) reduce the problem of hallucinated responses from LLMs, b) link sources/references to generated responses, and c) remove the need for annotating documents with meta-data. However, RAG systems suffer from limitations inherent to information retrieval systems and from reliance on LLMs. In this paper, we present an experience report on the failure points of RAG systems from three case studies from separate domains: research, education, and biomedical. We share the lessons learned and present 7 failure points to consider when designing a RAG system. The two key takeaways arising from our work are: 1) validation of a RAG system is only feasible during operation, and 2) the robustness of a RAG system evolves rather than designed in at the start. We conclude with a list of potential research directions on RAG systems for the software engineering community.
Forward citations
Cited by 6 Pith papers
-
Three Sides of Retrieval: Factorial Evidence for Document-Side, Query-Side, and Answer-Side Complementarity in RAG
A layout-inferred, zero-LLM-cost heading index that loads full pages improves RAG answer completeness and usefulness, and complements answer-side verification, though the evaluation is biased by a ToC-inclusive reference.
-
RAG Strategies for Natural Language-Based SQL Query and REST API Call Generation
In a 631-case enterprise benchmark, iterative retrieval (CoRAG) achieved 10.29% exact match vs 7.45% for standard RAG on combined SQL/API generation with hybrid documentation, while no-RAG gives 0% exact match.
-
Schema Lineage Extraction at Scale: Multilingual Pipelines, Composite Evaluation, and Language-Model Benchmarks
A new benchmark and composite metric for multilingual schema lineage extraction, with evidence that a 32B open model under chain-of-thought prompting approaches GPT-level accuracy.
-
Enhancing LLMs through human feedback: a journey towards self-improvement
An auxiliary feedback RAG continuously ingests classified human feedback to iteratively raise a primary RAG system’s answer accuracy and relevance.
-
A Comprehensive Graph Framework for Question Answering with Mode-Seeking Preference Alignment
GraphMPA combines an embedding-similarity hierarchical graph with mode-seeking preference optimization to improve RAG question answering on six datasets.
-
Automatic Dataset Generation for Knowledge Intensive Question Answering Tasks
Fine-tuning Llama-3-8b and Mistral-7b-v0.3 on LLM-generated QA pairs from IBM Technotes can improve no-context QA scores over training on human-annotated TechQA data, but the evaluation may be inflated by test-documen...
Discussion (0). Sign in to comment.