REVIEW 21 cited by
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems
read the original abstract
Retrieval-Augmented Generation (RAG) has become a standard architectural pattern for incorporating domain-specific knowledge into user-facing chat applications powered by Large Language Models (LLMs). RAG systems are characterized by (1) a document retriever that queries a domain-specific corpus for context information relevant to an input query, and (2) an LLM that generates a response based on the provided query and context. However, comprehensive evaluation of RAG systems remains a challenge due to the lack of unified evaluation criteria and annotated datasets. In response, we introduce RAGBench: the first comprehensive, large-scale RAG benchmark dataset of 100k examples. It covers five unique industry-specific domains and various RAG task types. RAGBench examples are sourced from industry corpora such as user manuals, making it particularly relevant for industry applications. Further, we formalize the TRACe evaluation framework: a set of explainable and actionable RAG evaluation metrics applicable across all RAG domains. We release the labeled dataset at https://huggingface.co/datasets/rungalileo/ragbench. RAGBench explainable labels facilitate holistic evaluation of RAG systems, enabling actionable feedback for continuous improvement of production applications. Thorough extensive benchmarking, we find that LLM-based RAG evaluation methods struggle to compete with a finetuned RoBERTa model on the RAG evaluation task. We identify areas where existing approaches fall short and propose the adoption of RAGBench with TRACe towards advancing the state of RAG evaluation systems.
Forward citations
Cited by 21 Pith papers
-
QO-Bench: Diagnosing Query-Operator-Preserving Retrieval over Typed Event Tuples
QO-Bench shows RAG systems retrieve relevant text but often discard typed values required for query operators, with paradigm performance inverting across operators and execution remaining a bottleneck even with gold evidence.
-
ACL-Verbatim: hallucination-free question answering for research
The work creates a new ground truth dataset for mapping queries to verbatim text spans in research papers and shows a 150M-parameter ModernBERT token classifier achieving 53.6 word-level F1, outperforming LLM extracto...
-
Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
MemoryAgentBench is a new multi-turn benchmark assessing four memory competencies in LLM agents—accurate retrieval, test-time learning, long-range understanding, and selective forgetting—showing that existing methods ...
-
PolyWorkBench: Benchmarking Multilingual Long-Horizon LLM Agents
A 67-task multilingual long-horizon agent benchmark across five workplace domains reports large drops versus monolingual settings and attributes them to compounding language-reasoning failures.
-
OPD-Evolver: Cultivating Holistic Agent Evolver via On-Policy Distillation
OPD-Evolver uses on-policy self-distillation in fast interaction and slow attribution loops to build agents with holistic memory competence, outperforming prior systems by up to 11.5% and allowing a 9B model to compet...
-
Grounded Cache Routing for Retrieval-Augmented Generation: When Is It Safe to Reuse an Answer?
GroundedCache reduces unsafe-served rate in RAG answer caching to 0-1.5% (vs 15-51.5% naive) via four validation gates while keeping p50 latency within 1.07x of no-cache baseline.
-
Query Symbolically or Retrieve Semantically? A Dataset and Method for Semi-Structured Question Answering
DualGraph combines semantic textual KGs with symbolic KGs for semi-structured QA and introduces the SpecsQA benchmark, outperforming baselines on both open and specification questions.
-
Evaluating Multi-Hop Reasoning in RAG Systems: A Comparison of LLM-Based Retriever Evaluation Strategies
CARE, a context-aware LLM judge, outperforms standard methods when evaluating multi-hop retrieval quality in RAG systems.
-
Facet-Level Tracing of Evidence Uncertainty and Hallucination in RAG
Introduces a facet-level diagnostics framework using Facet x Chunk matrices and controlled inference modes to show that RAG hallucinations arise mainly from evidence integration failures rather than retrieval errors.
-
Facet-Level Tracing of Evidence Uncertainty and Hallucination in RAG
Facet-level analysis of RAG systems on medical QA and HotpotQA shows hallucinations stem primarily from evidence integration and override failures during generation, not from retrieval inaccuracy.
-
UnWeaving the knots of GraphRAG -- turns out VectorRAG is almost enough
UnWeaver disentangles documents into entities via LLM to retrieve original chunks, yielding a simpler alternative to GraphRAG that still reduces noise and preserves source fidelity.
-
Retrieval-Augmented Generation for Natural Language Processing: A Survey
The survey organizes RAG methods via a taxonomy of query-based, logits-based, latent, and parametric fusion with comparisons on accessibility, efficiency, applications, and challenges.
-
Always-OnAgents:A Survey of Persistent Memory, State, and Governance in LLMAgents
Survey mapping persistent state in LLM agents along six axes and proposing the AOEP-v0 protocol to evaluate governance and recovery obligations.
-
RAIDS: Rethinking Data Systems as Responsible Intelligent Infrastructure
RAIDS proposes making responsibility an execution-level property in data systems via composable operator contracts and a preservation objective.
-
Towards Dependable Retrieval-Augmented Generation Using Factual Confidence Prediction
A conformal prediction filter for retrieval chunks plus an attention-based factuality classifier can raise RAG answer quality by up to 6% and detect inconsistent generations up to 77% of the time.
-
Retrieval-Augmented Generation Must Move Beyond Factual Grounding to Represent Diverse Opinions
Opinion-aware RAG with LLM opinion extraction and entity-linked graphs improves retrieval diversity by 26-42% over factual baselines on e-commerce forum data.
-
Empirical Evaluation of PDF Parsing and Chunking for Financial Question Answering with RAG
Systematic tests show that specific PDF parsers combined with overlapping chunking strategies better preserve structure and improve RAG answer correctness on financial QA benchmarks including the new TableQuest dataset.
-
LLMs as Assessors: Right for the Right Reason?
LLMs judge document relevance at a level comparable to humans but frequently highlight different passages, indicating they are often not right for the right reasons and cannot fully replace human assessors.
-
Reducing Hallucinations in Complex Question Answering using Simple Graph-based Retrieval-Augmented Generation (long version)
A graph-augmented RAG system with vector and graph query tools halves hallucinations and raises factual correctness scores on the MoNaCo complex QA benchmark.
-
Seeking Information with RAG-Assistants: Does Model Size Matter in Human-AI Collaborations?
Human-AI teams with RAG assistants outperform AI-only systems in information-seeking tasks independent of model size, with similar perceived usability across 3B to 70B models.
-
Little Brains, Big Feats: Exploring Compact Language Models
Small language models can run RAG generation on-device without GPUs in reasonable time.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.