A systematic review of 63 RAG evaluation papers concludes that LLM-based automation is feasible across dataset generation, retrieval scoring, and answer evaluation, but only six studies directly compare LLM judges with human judges.
Retrieval Augmented Generation Systems: Automatic Dataset Creation, Evaluation and Boolean Agent Setup
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Retrieval Augmented Generation (RAG) systems have seen huge popularity in augmenting Large-Language Model (LLM) outputs with domain specific and time sensitive data. Very recently a shift is happening from simple RAG setups that query a vector database for additional information with every user input to more sophisticated forms of RAG. However, different concrete approaches compete on mostly anecdotal evidence at the moment. In this paper we present a rigorous dataset creation and evaluation workflow to quantitatively compare different RAG strategies. We use a dataset created this way for the development and evaluation of a boolean agent RAG setup: A system in which a LLM can decide whether to query a vector database or not, thus saving tokens on questions that can be answered with internal knowledge. We publish our code and generated dataset online.
fields
cs.IR 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasets
A systematic review of 63 RAG evaluation papers concludes that LLM-based automation is feasible across dataset generation, retrieval scoring, and answer evaluation, but only six studies directly compare LLM judges with human judges.