REVIEW 15 cited by
CRAG -- Comprehensive RAG Benchmark
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
CRAG -- Comprehensive RAG Benchmark
read the original abstract
Retrieval-Augmented Generation (RAG) has recently emerged as a promising solution to alleviate Large Language Model (LLM)'s deficiency in lack of knowledge. Existing RAG datasets, however, do not adequately represent the diverse and dynamic nature of real-world Question Answering (QA) tasks. To bridge this gap, we introduce the Comprehensive RAG Benchmark (CRAG), a factual question answering benchmark of 4,409 question-answer pairs and mock APIs to simulate web and Knowledge Graph (KG) search. CRAG is designed to encapsulate a diverse array of questions across five domains and eight question categories, reflecting varied entity popularity from popular to long-tail, and temporal dynamisms ranging from years to seconds. Our evaluation of this benchmark highlights the gap to fully trustworthy QA. Whereas most advanced LLMs achieve <=34% accuracy on CRAG, adding RAG in a straightforward manner improves the accuracy only to 44%. State-of-the-art industry RAG solutions only answer 63% of questions without any hallucination. CRAG also reveals much lower accuracy in answering questions regarding facts with higher dynamism, lower popularity, or higher complexity, suggesting future research directions. The CRAG benchmark laid the groundwork for a KDD Cup 2024 challenge and attracted thousands of participants and submissions. We commit to maintaining CRAG to serve research communities in advancing RAG solutions and general QA solutions. CRAG is available at https://github.com/facebookresearch/CRAG/.
Forward citations
Cited by 15 Pith papers
-
BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms
On a 28-tier nested enterprise corpus, BM25 becomes the most accurate scalable RAG retriever after ~10M tokens, beating an agentic file-search system by ~20 points at 601M tokens and remaining low-cost.
-
Illocutionary Explanation Planning for Source-Faithful Explanations in Retrieval-Augmented Language Models
Chain-of-illocution prompting improves source adherence in RAG explanations for programming education by up to 63% over baselines.
-
BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms
On an enterprise corpus scaled from 1.7M to 601M tokens, BM25 beats raw-file agentic search, dense retrieval, and graph RAG at large sizes, crossing near 10M tokens.
-
EvidentialRAG: Quantifying and Mitigating Information Conflict in Multi-Source Retrieval-Augmented Generation via Evidential Deep Learning
Mapping multi-source RAG chunks to Dirichlet evidence and fusing them with conflict-preserving Dempster-Shafer mass reduces hallucination and raises conflict resolution without large standard-QA loss.
-
CacheRAG: A Semantic Caching System for Retrieval-Augmented Generation in Knowledge Graph Question Answering
CacheRAG turns stateless LLM planners for KGQA into continual learners via schema-agnostic parsing, diversity-optimized hierarchical caching, and bounded subgraph expansion, yielding up to 13.2% accuracy gains on benchmarks.
-
CacheRAG: A Semantic Caching System for Retrieval-Augmented Generation in Knowledge Graph Question Answering
CacheRAG is a cache-augmented architecture for LLM KGQA using ISR parsing, hierarchical MMR-based retrieval, and bounded subgraph expansion, claiming +13.2% accuracy gains on CRAG.
-
Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook
DOVE constructs a value codebook via rate-distortion variational optimization from 10K documents and measures LLM-human cultural alignment through unbalanced optimal transport, showing 31.56% correlation with downstre...
-
Association Is Not Similarity: Learning Corpus-Specific Associations for Multi-Hop Retrieval
AAR trains a small MLP with contrastive learning on co-occurrence annotations to rerank dense retrieval results using bi-directional association scores, improving multi-hop passage recall on HotpotQA and MuSiQue.
-
Always-OnAgents:A Survey of Persistent Memory, State, and Governance in LLMAgents
Survey mapping persistent state in LLM agents along six axes and proposing the AOEP-v0 protocol to evaluate governance and recovery obligations.
-
Executable Schema Contracts: From Automatic Ingestion to Multi-Source Retrieval
A system for automatic schema discovery from multi-source data builds provenance-aware knowledge graphs and conditions multi-tool agents for improved QA retrieval over baselines.
-
Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook
DOVE measures LLM cultural value alignment via a rate-distortion value codebook and unbalanced optimal transport between human and model open-ended text distributions.
-
RAG-DIVE: A Dynamic Approach for Multi-Turn Dialogue Evaluation in Retrieval-Augmented Generation
RAG-DIVE uses an LLM to dynamically generate, validate, and evaluate multi-turn dialogues for assessing RAG system performance in interactive settings.
-
Predict the Retrieval! Test time adaptation for Retrieval Augmented Generation
Training the language model for a few gradient steps to predict the end of retrieved passages improves specialized-domain RAG accuracy in the paper's tests, by up to 25 points on medical QA.
-
From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review
A survey consolidating benchmarks, agent frameworks, real-world applications, and protocols for LLM-based autonomous agents into a proposed taxonomy with recommendations for future research.
-
Little Brains, Big Feats: Exploring Compact Language Models
Small language models can run RAG generation on-device without GPUs in reasonable time.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.