Pith. sign in

REVIEW 5 cited by

RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2408.01262 v5 pith:VLGDQ4HJ submitted 2024-08-02 cs.CL cs.IR

Kunlun Zhu , Yifan Luo , Dingling Xu , Yukun Yan , Zhenghao Liu , Shi Yu , Ruobing Wang , Shuo Wang
show 5 more authors
Yishan Li Nan Zhang Xu Han Zhiyuan Liu Maosong Sun
This is my paper
classification cs.CLcs.IR
keywords ragevalmetricssystemsdatasetevaluatingevaluationframeworkgenerated
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Retrieval-Augmented Generation (RAG) is a powerful approach that enables large language models (LLMs) to incorporate external knowledge. However, evaluating the effectiveness of RAG systems in specialized scenarios remains challenging due to the high costs of data construction and the lack of suitable evaluation metrics. This paper introduces RAGEval, a framework designed to assess RAG systems across diverse scenarios by generating high-quality documents, questions, answers, and references through a schema-based pipeline. With a focus on factual accuracy, we propose three novel metrics: Completeness, Hallucination, and Irrelevance to evaluate LLM generated responses rigorously. Experimental results show that RAGEval outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples. Furthermore, the use of LLMs for scoring the proposed metrics demonstrates a high level of consistency with human evaluations. RAGEval establishes a new paradigm for evaluating RAG systems in real-world applications. The code and dataset are released at https://github.com/OpenBMB/RAGEval.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RAG-Stack: Co-Optimizing RAG Serving Performance and Quality

    cs.DB 2026-08 conditional novelty 7.0 of 10

    RAG-Stack jointly optimizes RAG algorithm choices and serving-system settings via sub-metric-aware multi-objective Bayesian optimization plus an analytical performance model, reporting Pareto frontiers covering 52.5% ...

  2. DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidence

    cs.CL 2025-09 conditional novelty 6.0 of 10

    An audit framework and empirical study showing that generative search engines and deep research agents frequently produce one-sided answers and weakly supported citations, with citation accuracy between 40 and 80%.

  3. RAG-TESTER: Automated End-to-End Testing of Retrieval-Augmented Large Language Models

    cs.AI 2026-07 conditional novelty 4.0 of 10

    A coverage-oriented test generator that adds negative-rejection and complex-passage queries to RAG evaluation, reported to detect 6.6% more failures than a strategy-free baseline in 24 model configurations.

  4. Benchmarking Vector, Graph and Hybrid Retrieval Augmented Generation (RAG) Pipelines for Open Radio Access Networks (ORAN)

    cs.AI 2025-07 conditional novelty 4.0 of 10

    On a 600-question subset of ORAN-Bench-13K, GraphRAG and Hybrid GraphRAG beat plain vector RAG on factual accuracy, but Hybrid GraphRAG scored below vector RAG on context relevance.

  5. Unlocking the Potential of Large Language Models in the Nuclear Industry with Synthetic Data

    cs.CL 2025-06 conditional novelty 2.0 of 10

    A pipeline converts CANDU textbook chapters into synthetic QA pairs using LLMs, embedding clustering, and similarity metrics, with no downstream validation.

Pith tools