REVIEW 6 cited by
RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Retrieval-Augmented Generation (RAG) is a powerful approach that enables large language models (LLMs) to incorporate external knowledge. However, evaluating the effectiveness of RAG systems in specialized scenarios remains challenging due to the high costs of data construction and the lack of suitable evaluation metrics. This paper introduces RAGEval, a framework designed to assess RAG systems across diverse scenarios by generating high-quality documents, questions, answers, and references through a schema-based pipeline. With a focus on factual accuracy, we propose three novel metrics: Completeness, Hallucination, and Irrelevance to evaluate LLM generated responses rigorously. Experimental results show that RAGEval outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples. Furthermore, the use of LLMs for scoring the proposed metrics demonstrates a high level of consistency with human evaluations. RAGEval establishes a new paradigm for evaluating RAG systems in real-world applications. The code and dataset are released at https://github.com/OpenBMB/RAGEval.
Forward citations
Cited by 6 Pith papers
-
RAG-Stack: Co-Optimizing RAG Serving Performance and Quality
RAG-Stack jointly optimizes RAG algorithm choices and serving-system settings via sub-metric-aware multi-objective Bayesian optimization plus an analytical performance model, reporting Pareto frontiers covering 52.5% ...
-
DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidence
An audit framework and empirical study showing that generative search engines and deep research agents frequently produce one-sided answers and weakly supported citations, with citation accuracy between 40 and 80%.
-
RAG-TESTER: Automated End-to-End Testing of Retrieval-Augmented Large Language Models
A coverage-oriented test generator that adds negative-rejection and complex-passage queries to RAG evaluation, reported to detect 6.6% more failures than a strategy-free baseline in 24 model configurations.
-
Benchmarking Vector, Graph and Hybrid Retrieval Augmented Generation (RAG) Pipelines for Open Radio Access Networks (ORAN)
On a 600-question subset of ORAN-Bench-13K, GraphRAG and Hybrid GraphRAG beat plain vector RAG on factual accuracy, but Hybrid GraphRAG scored below vector RAG on context relevance.
-
AI-Driven Climate Policy Scenario Generation for Sub-Saharan Africa
A RAG pipeline using llama3.2-3B and UN COP documents generated 34 policy scenarios for Sub-Saharan Africa, 30 passed author validation, but automated evaluation showed mixed agreement with human judgment.
-
Unlocking the Potential of Large Language Models in the Nuclear Industry with Synthetic Data
A pipeline converts CANDU textbook chapters into synthetic QA pairs using LLMs, embedding clustering, and similarity metrics, with no downstream validation.
Discussion (0). Sign in to comment.