Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasets

T0 review · 4 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This survey of 63 papers argues that almost every stage of RAG evaluation—dataset creation, retrieval scoring, and answer judging—can be automated with large language models, with humans retained for domain-specific oversight.

desk verdict Useful map of RAG evaluation practices, but the 'feasibility' claim holds per component, not as an end-to-end pipeline. read the letter →

arxiv 2504.20119 v2 pith:DQ5F34VK submitted 2025-04-28 cs.IR cs.AI

Lorenz Brehme , Thomas Ströhle , Ruth Breu This is my paper · ORCID
classification cs.IRcs.AI
keywords Retrieval-AugmentedGenerationRAGevaluationLLMasjudgesystematicliteraturereviewdatasetsfaithfulnessmetricsretrieverhuman
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This systematic review of 63 papers asks whether large language models can be trusted to evaluate retrieval-augmented generation (RAG) systems, which answer questions by pulling in external documents. It concludes that nearly every stage of evaluation could potentially be automated: LLMs can generate the question-answer-context datasets used as benchmarks, judge whether retrieved chunks are relevant, and score generated answers on faithfulness, truthfulness, and completeness. The authors organize current practice into four areas—datasets, indexing and databases, retrievers, and generators—and extract best practices for each. If the claim is right, teams can run end-to-end evaluation pipelines with far less manual labeling, reserving human experts for domain-specific oversight and final validation. The survey also notes that the evidence base is thin: only six of the studies directly compared LLM judges with human judges.

What carries the argument

The paper's organizing machinery is a component-wise taxonomy of RAG evaluation, classifying methods by four components—datasets, indexing and databases, retriever, and generator—and by who or what does the judging: exact match, human experts, classical embedding or n-gram metrics, and LLM-as-judge. Within this taxonomy, the load-bearing mechanism is the LLM-as-dataset-generator-and-judge loop: the same model that writes evaluation questions from domain contexts can also score retrieved chunks and generated answers, which is what makes the automation-feasibility claim concrete. The taxonomy is also what lets the authors identify gaps, such as the scarcity of human-comparison studies and the lack of standardized prompts across LLM-based evaluations.

What would settle it

Run a controlled study in a high-stakes domain, such as legal or medical, where an LLM generates the evaluation dataset, retrieves contexts, and scores answers, and compare those scores against blind expert human scores: if the LLM scores diverge from expert judgments more than human annotators diverge from each other, the claim that almost every evaluation step can be trusted to LLMs would be refuted.

Watch

Extended reading notes

Core claim

The central discovery is that the same type of model that powers a RAG system can also power its evaluation at every level. Across the 63 reviewed papers, the survey finds LLMs being used to create domain-specific datasets by prompting from contexts, to determine context relevance for retriever scoring through direct classification or indirect question-generation checks, and to compute generator metrics such as faithfulness, truthfulness, relevance, completeness, and noise sensitivity. The review identifies five question types, seven context-relevance determination methods, and five LLM-judge computation methods, and concludes that almost every aspect of evaluation can potentially be automated. It stops short of full endorsement: the six studies that compared LLM judges with human judges showed positive correlation, but the survey calls the validity of LLM-based evaluation something that remains to be thoroughly established and flags unresolved questions about an LLM evaluating its own output.

Load-bearing premise

The conclusion rests on the assumption that the 63 papers found by the November 2024 search fairly represent the whole field of RAG evaluation; a search that missed substantial work could make the field look more ready for full LLM automation than it is.

Editorial extensions

If this is right

  • Teams can assemble end-to-end automated evaluation pipelines for RAG systems, with LLMs generating domain datasets, scoring retrieval relevance, and grading answers.
  • Human evaluators can shift from routine scoring to domain-specific curation, error analysis, and final oversight of LLM-judged results.
  • Because only six studies compare LLM judges with human judges, the evidence for trusting LLM evaluation is promising but not yet comprehensive; more human-comparison studies are needed.
  • Evaluation datasets for RAG must be designed so the LLM cannot answer from parametric memory, for example by using post-training or proprietary knowledge, or scores will overstate performance.
  • LLM-version drift and the lack of standardized prompts make current evaluation results hard to compare across studies, so a version-independent evaluation standard is needed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the six human-comparison studies all show positive but not perfect correlation, a natural next experiment is to test whether agreement depends on the component being judged: LLM judges may match humans on factual short answers while diverging on long-form reasoning or domain nuance.
  • The survey leaves open the self-evaluation circularity; a direct test would compare an LLM judging its own RAG outputs against a different LLM judging the same outputs, isolating model-specific bias.
  • If the automation-feasibility claim holds, evaluation cost drops dramatically, but so does the cost of silent regressions; organizations should freeze a judge-model version and keep a small human-scored holdout for spot checks.
  • The taxonomy implies that indexing quality is currently measured mainly by speed, not by downstream effect; a testable extension is to correlate specific chunking and embedding choices with retriever and generator scores to give indexing a real quality metric.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. This paper presents a systematic literature review of 63 papers on the evaluation of Retrieval-Augmented Generation (RAG) systems. It organizes the landscape into four components—evaluation datasets, indexing and databases, retrievers, and generators—and catalogs the metrics and methods used for each. A central observation is that LLMs can generate evaluation datasets, score retrieved chunks, and judge generated answers, so that most components of the evaluation workflow can in principle be automated. The paper also discusses the interplay between LLM-based automation and human judgment, notes the need for domain-specific datasets, and concludes that although LLMs can be trusted "to some extent," their validity as evaluators remains to be established.

Significance. If the synthesis is accurate, this is a useful reference for practitioners: it provides a component-wise taxonomy, an inventory of metrics, and a clear map of where automated LLM-based evaluation has been demonstrated. The paper is at its strongest when it stays close to the reviewed evidence, such as the concrete descriptions of LLM-generated datasets in Section III, LLM-based context relevance scoring in Section V, and LLM-as-judge approaches in Section VI. It also explicitly flags self-evaluation bias, the lack of standardized prompts, and the shortage of human-comparison studies. The significance is currently limited by audibility problems: the paper reports precise counts and categories without a full list of included papers or a paper-to-category mapping, and the strongest conclusion extrapolates from component-level evidence to an end-to-end automation claim. These issues are fixable and do not invalidate the component-level synthesis, but they need to be addressed before the survey can serve as a reliable reference.

major comments (4)
  1. [Section II and Tables I–V] The survey reports exact counts—63 papers, 87 datasets, 24 retriever papers, 56 generator papers, 41 LLM-judge papers—but it does not provide a full list of the included papers or a mapping from each paper to the categories and tables. For example, Section V states that "24 different papers" address retriever evaluation, and Table I lists a set of references, but I could not verify whether those references are exactly the 24 papers or whether some are contextual. The reader cannot audit the taxonomy or reproduce the counts. Please provide a supplementary table listing the 63 included papers, their assignment to the four component categories, and the tables or metrics in which they appear.
  2. [Section II] The description of the systematic literature review is too sparse to support the label "SLR." The paper gives one final search query, the date of search, and the number of papers from each source, but it does not report the per-database search strings, explicit inclusion and exclusion criteria, screening steps, or a PRISMA-style flow from 71 initial papers to 48, then to 63 after forward/backward search. It also does not discuss how duplicate records or multi-database hits were handled. Given that the central observation rests on the representativeness of the 63 selected papers, the authors should either provide the complete protocol and screening decisions or weaken the claim of systematicity.
  3. [Section VI-B and Section VII] There is a numeric inconsistency in the human-comparison evidence. Section VI-B says "eight approaches utilized human judgment as a benchmark to compare ... methods where an LLM acted as a judge" and cites [58], [59], [26], [42], [20], [44], while the conclusion says "Only six studies compared LLM judges with human judges." Additionally, one of the cited sources, [58], is a vendor blog post rather than a peer-reviewed study. The conclusion "LLMs can be trusted to some extent" depends on the strength and quantity of these comparisons, so the authors should reconcile the count, state which of the cited works are actually comparative studies, and report the quantitative nature of the "positive correlation" for each comparison (e.g., correlation coefficients or agreement rates).
  4. [Section VII] The conclusion "almost every aspect of the evaluation process could potentially be automated using LLMs" is a composition inference that goes beyond the component-level evidence in Sections III–VI. The paper demonstrates that individual LLM-based components exist—dataset generation, retriever relevance scoring, and answer judging—but it does not report any reviewed study that validates an end-to-end pipeline in which all three stages are composed and the final judgment is compared with human labels. The paper itself concedes that "validity remains to be thoroughly established" and flags self-evaluation bias. Please either scope the feasibility claim to individual components or explicitly identify composed-pipeline validation as an open problem rather than presenting it as a demonstrated capability.
minor comments (7)
  1. [Table IV and Section VI-C] "ROGUE-n" and "ROGUE-L" are typographical errors and should be "ROUGE-n" and "ROUGE-L."
  2. [Section V] The metric abbreviation NDCG stands for "Normalized Discounted Cumulative Gain," not "Discounted Cumulative Gain," as currently written.
  3. [Table V] The formula "Kendall/acute.ts1s tau" appears to be a rendering error; it should read "Kendall's tau."
  4. [Section III] The paper states that it found "87 existing question and answer (QA) datasets" but provides no enumeration or per-dataset references. An appendix table listing the datasets and their domains would make this inventory auditable.
  5. [Section IV] The paragraph says "Two studies specifically focused on evaluating these components [9], [5]" and then immediately cites a third study [37] evaluating embeddings, chunking, and databases; please clarify whether [37] is the third study or whether the count should be revised.
  6. [References] The reference list mixes peer-reviewed papers, preprints, blog posts, documentation pages, and a GitHub repository without distinguishing source types. For a systematic review, it would be helpful to label the source type or at least mark which references are gray literature.
  7. [Section III-c] The phrase "post-training articles were used for question generation" is ambiguous; please specify whether this means documents published after a model's training cutoff and explain how contamination was checked.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the survey's conclusions are syntheses of 63 external papers, not derivations from the paper's own definitions or fitted values.

full rationale

This is a systematic literature review; its claims are inductive syntheses of the reviewed literature rather than derivations from author-defined quantities. The search query is seeded by RAGAS and MultiHop-RAG, but those are external frameworks examined as part of the corpus, and the query merely retrieves candidate papers; the conclusions are not defined in terms of those frameworks' outputs. The authors report counts such as 'only six studies compared LLM judges with human judges' from the selected papers, and they explicitly flag limitations: 'their validity remains to be thoroughly established' and question 'whether the quality of evaluation is compromised when an LLM generates questions, answers them, and ultimately evaluates its own output.' These admissions show the central feasibility claim is presented as an open empirical observation, not as a result forced by the paper's own construction. There are no fitted parameters, no self-citations by the authors, and no uniqueness theorem imported from prior work. The reader's strongest inference about composing components into a validated end-to-end pipeline is an extrapolation beyond the survey's evidence, which is a correctness/scope risk, not circularity. No circular step can be exhibited with a quote and reduction, so the score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. Its conclusions rest on the representativeness of the literature sample and on the accuracy of the authors' qualitative categorization.

assumptions (3)
  • domain assumption The 63 included papers are representative of the broader RAG evaluation literature.
    The SLR relies on a single search date and specific databases; the authors do not provide a full list of exclusions or a PRISMA-style flow diagram.
  • domain assumption The manual categorization of papers into question types, retriever methods, and generator metrics is accurate and complete.
    The survey does not provide a table mapping every one of the 63 papers to its assigned category, so readers cannot verify the taxonomy.
  • domain assumption The reported correlations between LLM and human judges in the six cited studies are reliable and correctly interpreted.
    The survey uses these six studies to conclude LLMs can be trusted to some extent, but the studies vary in design and the survey does not report effect sizes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasets." pith.science (2026). https://pith.science/paper/DQ5F34VK

@misc{pith2026250420119,
  author       = {Pith},
  title        = {Pith review of: Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DQ5F34VK}},
  note         = {Machine review of arXiv:2504.20119}
}
read the original abstract

Retrieval-Augmented Generation (RAG) has advanced significantly in recent years. The complexity of RAG systems, which involve multiple components-such as indexing, retrieval, and generation-along with numerous other parameters, poses substantial challenges for systematic evaluation and quality enhancement. Previous research highlights that evaluating RAG systems is essential for documenting advancements, comparing configurations, and identifying effective approaches for domain-specific applications. This study systematically reviews 63 academic articles to provide a comprehensive overview of state-of-the-art RAG evaluation methodologies, focusing on four key areas: datasets, retrievers, indexing and databases, and the generator component. We observe the feasibility of an automated evaluation approach for each component of a RAG system, leveraging an LLM capable of both generating evaluation datasets and conducting evaluations. In addition, we found that further practical research is essential to provide companies with clear guidance on the do's and don'ts of implementing and evaluating RAG systems. By synthesizing evaluation approaches for key RAG components and emphasizing the creation and adaptation of domain-specific datasets for benchmarking, we contribute to the advancement of systematic evaluation methods and the improvement of evaluation rigor for RAG systems. Furthermore, by examining the interplay between automated approaches leveraging LLMs and human judgment, we contribute to the ongoing discourse on balancing automation and human input, clarifying their respective contributions, limitations, and challenges in achieving robust and reliable evaluations.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Rewrite-to-Rank: Optimizing Ad Visibility via Retrieval-Aware Text Rewriting

    cs.CL 2025-07 conditional novelty 5.0 of 10

    PPO-trained rewriting of ads improves retrieval rank and LLM inclusion on a custom ad dataset, measured by two new delta metrics.

Reference graph

Works this paper leans on

72 extracted references · 26 canonical work pages · cited by 1 Pith paper

  1. [58]

    Q. Leng. Best practices for LLM evalua- tion of RAG applications. [Online]. Available: https://www.databricks.com/blog/LLM-auto-eval-best-practices-RAG

    work page

  2. [59]

    Ev aluating quality of answers for retrieval-augmented generation: A s trong LLM is all you need

    Y . Wang, A. G. Hernandez, R. Kyslyi, and N. Kersting, “Ev aluating quality of answers for retrieval-augmented generation: A s trong LLM is all you need.” [Online]. Available: http://arxiv.org/a bs/2406.18064

    arXiv

  3. [26]

    RAG-QA arena: Evaluating domain robustne ss for long-form retrieval augmented question answering

    R. Han, Y . Zhang, P . Qi, Y . Xu, J. Wang, L. Liu, W. Y . Wang, B . Min, and V . Castelli, “RAG-QA arena: Evaluating domain robustne ss for long-form retrieval augmented question answering.” [Onli ne]. Available: http://arxiv.org/abs/2407.13998

    arXiv

  4. [42]

    Towards op timizing and evaluating a retrieval augmented QA chatbot using LLMs with human in the loop

    A. Afzal, A. Kowsik, R. Fani, and F. Matthes, “Towards op timizing and evaluating a retrieval augmented QA chatbot using LLMs with human in the loop.” [Online]. Available: http://arxiv.org/abs/ 2407.05925

    arXiv

  5. [20]

    RAGChecker: A fine-grained framewor k for diagnosing retrieval-augmented generation

    D. Ru, L. Qiu, X. Hu, T. Zhang, P . Shi, S. Chang, C. Jiayang , C. Wang, S. Sun, H. Li, Z. Zhang, B. Wang, J. Jiang, T. He, Z. Wang, P . Liu , Y . Zhang, and Z. Zhang, “RAGChecker: A fine-grained framewor k for diagnosing retrieval-augmented generation.” [Online ]. Available: http://arxiv.org/abs/2408.08067

    arXiv

  6. [44]

    Evaluating R AG-fusion with RAGElo: an automated elo-based framework

    Z. Rackauckas, A. Cˆ amara, and J. Zavrel, “Evaluating R AG-fusion with RAGElo: an automated elo-based framework.” [Online]. Available: http://arxiv.org/abs/2406.14783

    arXiv

  7. [1]

    Retrieval-augmented generation for large lan guage models: A survey

    Y . Gao, Y . Xiong, X. Gao, K. Jia, J. Pan, Y . Bi, Y . Dai, J. Sun , M. Wang, and H. Wang, “Retrieval-augmented generation for large lan guage models: A survey.” [Online]. Available: http://arxiv.org /abs/2312.10997

    arXiv

  8. [2]

    Si ren’s Song in the AI Ocean: A Survey on Hallucination in Large Langu age Models,

    Y . Zhang, Y . Li, L. Cui, D. Cai, L. Liu, T. Fu, X. Huang, E. Zh ao, Y . Zhang, Y . Chen, L. Wang, A. Luu, W. Bi, F. Shi, and S. Shi, “Si ren’s Song in the AI Ocean: A Survey on Hallucination in Large Langu age Models,” ArXiv, Sep. 2023

    work page 2023

Show all 72 references
  1. [3]

    Retrieval-Augmented Generation for Knowledge- Intensive NLP Tasks,

    P . Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N . Goyal, H. K¨ uttler, M. Lewis, W.-t. Yih, T. Rockt¨ aschel, S. Riedel , and D. Kiela, “Retrieval-Augmented Generation for Knowledge- Intensive NLP Tasks,” Apr. 2021, arXiv:2005.11401. [Online]. Availa ble: http://a...

    2021 arXiv

  2. [4]

    CRUD-RAG: A comprehensive chines e benchmark for retrieval-augmented generation of large lan guage models

    Y . Lyu, Z. Li, S. Niu, F. Xiong, B. Tang, W. Wang, H. Wu, H. Liu, T. Xu, and E. Chen, “CRUD-RAG: A comprehensive chines e benchmark for retrieval-augmented generation of large lan guage models.” [Online]. Available: http://arxiv.org/abs/240 1.17043

  3. [5]

    Performance evaluation of vector embeddings with retrieval- augmented generation,

    S. Kukreja, T. Kumar, V . Bharate, A. Purohit, A. Dasgupta , and D. Guha, “Performance evaluation of vector embeddings with retrieval- augmented generation,” in 2024 9th International Conference on Computer and Communication Systems (ICCCS) , 2024, pp. 333–340. [Online]. Avail...

    2024

  4. [6]

    RA GAS: Automated evaluation of retrieval augmented generation

    S. Es, J. James, L. Espinosa-Anke, and S. Schockaert, “RA GAS: Automated evaluation of retrieval augmented generation.” [Online]. Available: http://arxiv.org/abs/2309.15217

    arXiv

  5. [7]

    AR ES: An automated evaluation framework for retrieval-augmented g eneration systems

    J. Saad-Falcon, O. Khattab, C. Potts, and M. Zaharia, “AR ES: An automated evaluation framework for retrieval-augmented g eneration systems.” [Online]. Available: http://arxiv.org/abs/23 11.09476

  6. [8]

    Evaluating retrieval qual- ity in retrieval-augmented generation

    A. Salemi and H. Zamani, “Evaluating retrieval qual- ity in retrieval-augmented generation.” [Online]. Availa ble: http://arxiv.org/abs/2404.13781

    arXiv

  7. [9]

    Beyond benchmarks: Evaluating embedding mo del similarity for retrieval augmented generation systems

    L. Caspari, K. G. Dastidar, S. Zerhoudi, J. Mitrovic, and M. Granitzer, “Beyond benchmarks: Evaluating embedding mo del similarity for retrieval augmented generation systems.” [ Online]. Available: http://arxiv.org/abs/2407.08275

    arXiv

  8. [10]

    MultiHop-RAG: Benchmarking retri eval- augmented generation for multi-hop queries

    Y . Tang and Y . Y ang, “MultiHop-RAG: Benchmarking retri eval- augmented generation for multi-hop queries.” [Online]. Av ailable: http://arxiv.org/abs/2401.15391

    arXiv

  9. [11]

    DomainRAG: A chinese benchmark for evaluating domain-specific retrieval-augmented generation

    S. Wang, J. Liu, S. Song, J. Cheng, Y . Fu, P . Guo, K. Fang, Y . Zhu, and Z. Dou, “DomainRAG: A chinese benchmark for evaluating domain-specific retrieval-augmented generation.” [Onlin e]. Available: http://arxiv.org/abs/2406.05654

    arXiv

  10. [12]

    Customized retriev al augmented generation and benchmarking for EDA tool documen tation QA

    Y . Pu, Z. He, T. Qiu, H. Wu, and B. Y u, “Customized retriev al augmented generation and benchmarking for EDA tool documen tation QA.” [Online]. Available: http://arxiv.org/abs/2407.15 353

  11. [13]

    Evalu ation of retrieval-augmented generation: A survey,

    H. Y u, A. Gan, K. Zhang, S. Tong, Q. Liu, and Z. Liu, “Evalu ation of retrieval-augmented generation: A survey,” 2024. [Online ]. Available: http://arxiv.org/abs/2405.07437

    2024 arXiv

  12. [14]

    Benchmarking of retrieval augmented gene ration: A comprehensive systematic literature review on evaluatio n dimensions, evaluation metrics and datasets,

    S. Knollmeyer, O. Caymazer, L. Koval, M. Akmal, S. Asif, S. Mathias, and D. Großmann, “Benchmarking of retrieval augmented gene ration: A comprehensive systematic literature review on evaluatio n dimensions, evaluation metrics and datasets,” in Proceedings of the 16th Internat...

    2024

  13. [15]

    Guidelines for perform ing systematic literature reviews in software engineering,

    B. Kitchenham and S. Charters, “Guidelines for perform ing systematic literature reviews in software engineering,” vol. 2, 2007

    2007

  14. [16]

    LegalBench-RAG: A benchma rk for retrieval-augmented generation in the legal domain

    N. Pipitone and G. H. Alami, “LegalBench-RAG: A benchma rk for retrieval-augmented generation in the legal domain.” [ Online]. Available: http://arxiv.org/abs/2408.10343

    arXiv

  15. [17]

    Benchmarking retrieval-augmented generation for medicine

    G. Xiong, Q. Jin, Z. Lu, and A. Zhang, “Benchmarking retrieval-augmented generation for medicine.” [Online]. Available: http://arxiv.org/abs/2402.13178

    arXiv

  16. [18]

    WeQA: A benchmark for retrieval augmented generation in wind energ y domain

    R. Meyur, H. Phan, S. Wagle, J. Strube, M. Halappanavar, S. Horawalavithana, A. Acharya, and S. Munikoti, “WeQA: A benchmark for retrieval augmented generation in wind energ y domain.” [Online]. Available: http://arxiv.org/abs/2408.11800

    arXiv

  17. [19]

    D. Oberst. How to evaluate LLMs for RAG? [Online]. Avail able: https://medium.com/@darrenoberst/how-accurate-is-rag-8f0706281fd9

  18. [21]

    Lynx: An open source hallucination evaluation model

    S. S. Ravi, B. Mielczarek, A. Kannappan, D. Kiela, and R. Qian, “Lynx: An open source hallucination evaluation model.” [On line]. Available: http://arxiv.org/abs/2407.08488

    arXiv

  19. [22]

    Face4rag: Factual c onsistency evaluation for retrieval augmented generation in chinese

    Y . Xu, T. Cai, J. Jiang, and X. Song, “Face4rag: Factual c onsistency evaluation for retrieval augmented generation in chinese. ” [Online]. Available: http://arxiv.org/abs/2407.01080

    arXiv

  20. [23]

    RAG-Confu sionQA: A benchmark for evaluating LLMs on confusing questions

    Z. Peng, J. Nian, A. Evfimievski, and Y . Fang, “RAG-Confu sionQA: A benchmark for evaluating LLMs on confusing questions.” [O nline]. Available: http://arxiv.org/abs/2410.14567

    arXiv

  21. [24]

    Retrieval aug mented generation systems: Automatic dataset creation, evaluati on and boolean agent setup

    T. Kenneweg, P . Kenneweg, and B. Hammer, “Retrieval aug mented generation systems: Automatic dataset creation, evaluati on and boolean agent setup.” [Online]. Available: http://arxiv.org/abs /2403.00820

    arXiv

  22. [25]

    ClimRe- trieve: A benchmarking dataset for information retrieval f rom corporate climate disclosures

    T. Schimanski, J. Ni, R. Spacey, N. Ranger, and M. Leippo ld, “ClimRe- trieve: A benchmarking dataset for information retrieval f rom corporate climate disclosures.” [Online]. Available: http://arxiv.org/abs/2406.09818

    arXiv

  23. [27]

    Benchmarking multimodal retrieval augmented generation with dynamic VQ A dataset and self-adaptive planning agent

    Y . Li, Y . Li, X. Wang, Y . Jiang, Z. Zhang, X. Zheng, H. Wang , H.-T. Zheng, P . S. Y u, F. Huang, and J. Zhou, “Benchmarking multimodal retrieval augmented generation with dynamic VQ A dataset and self-adaptive planning agent.” [Online]. Avai lable: http://arxiv.org/abs/2411.02937

    arXiv

  24. [28]

    Do RAG systems cover what matters? evaluating and optimizing responses with sub-question coverage

    K. Xie, P . Laban, P . K. Choubey, C. Xiong, and C.-S. Wu, “Do RAG systems cover what matters? evaluating and optimizing responses with sub-question coverage.” [Onlin e]. Available: http://arxiv.org/abs/2410.15531

    arXiv

  25. [29]

    FeB4r ag: Evaluating federated search in the context of retrieval aug mented generation

    S. Wang, E. Khramtsova, S. Zhuang, and G. Zuccon, “FeB4r ag: Evaluating federated search in the context of retrieval aug mented generation.” [Online]. Available: http://arxiv.org/abs /2402.11891

    arXiv

  26. [30]

    Long$ˆ2$RAG: Evaluating long-context & long-form retrie val- augmented generation with key point recall

    Z. Qi, R. Xu, Z. Guo, C. Wang, H. Zhang, and W. Xu, “Long$ˆ2$RAG: Evaluating long-context & long-form retrie val- augmented generation with key point recall.” [Online]. Ava ilable: http://arxiv.org/abs/2410.23000

    arXiv

  27. [31]

    CoFE-RAG : A comprehensive full-chain evaluation framework for retri eval- augmented generation with enhanced data diversity

    J. Liu, R. Ding, L. Zhang, P . Xie, and F. Huang, “CoFE-RAG : A comprehensive full-chain evaluation framework for retri eval- augmented generation with enhanced data diversity.” [Onli ne]. Available: http://arxiv.org/abs/2410.12248

    arXiv

  28. [32]

    Fact, fetch, and reason: A unifi ed evaluation of retrieval-augmented generation

    S. Krishna, K. Krishna, A. Mohananey, S. Schwarcz, A. St ambler, S. Upadhyay, and M. Faruqui, “Fact, fetch, and reason: A unifi ed evaluation of retrieval-augmented generation.” [Online] . Available: http://arxiv.org/abs/2409.12941

    arXiv

  29. [33]

    RAGEval: Scenario specific RAG evaluation dataset generation framework

    K. Zhu, Y . Luo, D. Xu, R. Wang, S. Y u, S. Wang, Y . Y an, Z. Liu, X. Han, Z. Liu, and M. Sun, “RAGEval: Scenario specific RAG evaluation dataset generation framework.” [Online]. A vailable: http://arxiv.org/abs/2408.01262

    arXiv

  30. [34]

    Multimodal re- trieval augmented generation evaluation benchmark,

    T. Sun, A. Somalwar, and H. Chan, “Multimodal re- trieval augmented generation evaluation benchmark,” in 2024 IEEE 99th V ehicular Technology Conference (VTC2024-Sprin g), 2024, pp. 1–5, ISSN: 2577-2465. [Online]. Available: https://ieeexplore.ieee.org/document/10683437/?arnum...

    2024

  31. [35]

    Benchmarking large l anguage models in retrieval-augmented generation

    J. Chen, H. Lin, X. Han, and L. Sun, “Benchmarking large l anguage models in retrieval-augmented generation.” [Online]. Ava ilable: http://arxiv.org/abs/2309.01431

    arXiv

  32. [36]

    FaithEval: Can your language model stay faithf ul to context, even if

    Y . Ming, S. Purushwalkam, S. Pandit, Z. Ke, X.-P . Nguyen , C. Xiong, and S. Joty, “FaithEval: Can your language model stay faithf ul to context, even if ”the moon is made of marshmallows”.” [Onlin e]. Available: http://arxiv.org/abs/2410.03727

    arXiv

  33. [37]

    A knowledge-cen tric benchmarking framework and empirical study for retrieval- augmented generation

    S. Y u, M. Cheng, J. Y ang, and J. Ouyang, “A knowledge-cen tric benchmarking framework and empirical study for retrieval- augmented generation.” [Online]. Available: http://arxiv.org/abs /2409.13694

  34. [38]

    Does RAG intro duce unfairness in LLMs? evaluating fairness in retrieval-augm ented generation systems

    X. Wu, S. Li, H.-T. Wu, Z. Tao, and Y . Fang, “Does RAG intro duce unfairness in LLMs? evaluating fairness in retrieval-augm ented generation systems.” [Online]. Available: http://arxiv.org/abs/2409.19804

    arXiv

  35. [39]

    Enhancing q&a text retrieval with ranking models: Benchmarking, fine-tuning and deploying rerankers for RAG

    G. d. S. P . Moreira, R. Ak, B. Schifferer, M. Xu, R. Osmuls ki, and E. Oldridge, “Enhancing q&a text retrieval with ranking models: Benchmarking, fine-tuning and deploying rerankers for RAG. ” [Online]. Available: http://arxiv.org/abs/2409.07691

    arXiv

  36. [40]

    CORAL: Benchmarking multi- turn conversational retrieval-augmentation generation

    Y . Cheng, K. Mao, Z. Zhao, G. Dong, H. Qian, Y . Wu, T. Sakai, J.-R. Wen, and Z. Dou, “CORAL: Benchmarking multi- turn conversational retrieval-augmentation generation.” [On line]. Available: http://arxiv.org/abs/2410.23090

    arXiv

  37. [41]

    IRSC: A zero - shot evaluation benchmark for information retrieval throu gh semantic comprehension in retrieval-augmented generation scenari os

    H. Lin, S. Zhan, J. Su, H. Zheng, and H. Wang, “IRSC: A zero - shot evaluation benchmark for information retrieval throu gh semantic comprehension in retrieval-augmented generation scenari os.” [Online]. Available: http://arxiv.org/abs/2409.15763

    arXiv

  38. [43]

    UDA: A benchmark suite for re trieval augmented generation in real-world document analysis

    Y . Hui, Y . Lu, and H. Zhang, “UDA: A benchmark suite for re trieval augmented generation in real-world document analysis.” [O nline]. Available: http://arxiv.org/abs/2406.15187

    arXiv

  39. [45]

    RAGBench: Explainab le benchmark for retrieval-augmented generation systems

    R. Friel, M. Belyi, and A. Sanyal, “RAGBench: Explainab le benchmark for retrieval-augmented generation systems.” [Online]. A vailable: http://arxiv.org/abs/2407.11005

    arXiv

  40. [46]

    VERA: V alidation and evaluation of retrieval-augmented systems

    T. Ding, A. Banerjee, L. Mombaerts, Y . Li, T. Borogovac, and J. P . D. l. C. Weinstein, “VERA: V alidation and evaluation of retrieval-augmented systems.” [Online]. Av ailable: http://arxiv.org/abs/2409.03759

    arXiv

  41. [47]

    Evaluating the re trieval component in LLM-based question answering systems

    A. Alinejad, K. Kumar, and A. V ahdat, “Evaluating the re trieval component in LLM-based question answering systems.” [Onli ne]. Available: http://arxiv.org/abs/2406.06458

    arXiv

  42. [48]

    FaaF: Facts as a function f or the evaluation of generated text

    V . Katranidis and G. Barany, “FaaF: Facts as a function f or the evaluation of generated text.” [Online]. Available: http://arxiv.or g/abs/2403.03888

    arXiv

  43. [49]

    A method ology for evaluating RAG systems: A case study on configuration depend ency validation

    S. Simon, A. Mailach, J. Dorn, and N. Siegmund, “A method ology for evaluating RAG systems: A case study on configuration depend ency validation.” [Online]. Available: http://arxiv.org/abs /2410.08801

    arXiv

  44. [50]

    BERGEN: A benchmarking library for retrieval-augmented generation

    D. Rau, H. D´ ejean, N. Chirkova, T. Formal, S. Wang, V . Nikoulina, and S. Clinchant, “BERGEN: A benchmarking library for retrieval-augmented generation.” [Online]. A vailable: http://arxiv.org/abs/2407.01102

    arXiv

  45. [51]

    Evaluating lar ge language models for arabic sentiment analysis: A comparati ve study using retrieval-augmented generation,

    S. Khaled, E. H. Mohamed, and W. Medhat, “Evaluating lar ge language models for arabic sentiment analysis: A comparati ve study using retrieval-augmented generation,” Procedia Computer Science, vol. 244, pp. 363–370, 2024. [Online]. Available: https://www.sciencedirect.com/sci...

    2024

  46. [52]

    ReEval: Automatic hallucination evaluation for retrieval-augmen ted large language models via transferable adversarial attacks,

    X. Y u, H. Cheng, X. Liu, D. Roth, and J. Gao, “ReEval: Automatic hallucination evaluation for retrieval-augmen ted large language models via transferable adversarial attacks,” in Findings of the Association for Computational Linguistics: NAACL 20 24, K. Duh, H. Gomez, and S. ...

    2024

  47. [53]

    Automated evaluation of retrieval-augmented language models with ta sk-specific exam generation

    G. Guinet, B. Omidvar-Tehrani, A. Deoras, and L. Callot , “Automated evaluation of retrieval-augmented language models with ta sk-specific exam generation.” [Online]. Available: http://arxiv.org /abs/2405.13622

    arXiv

  48. [54]

    CRAG – comprehensive RAG benchmark

    X. Y ang, K. Sun, H. Xin, Y . Sun, N. Bhalla, X. Chen, S. Chou dhary, R. D. Gui, Z. W. Jiang, Z. Jiang, L. Kong, B. Moran, J. Wang, Y . E. Xu, A. Y an, C. Y ang, E. Y uan, H. Zha, N. Tang, L. Chen, N. Scheffer, Y . Liu, N. Shah, R. Wanga, A. Kumar, W.-t. Yih, an d X. L. Dong, ...

    arXiv

  49. [55]

    Automatic questi on answering for the linguistic domain – an evaluation of LLM knowledge ba se extension with RAG,

    C. Lang, R. Schneider, and N. D. T. Tu, “Automatic questi on answering for the linguistic domain – an evaluation of LLM knowledge ba se extension with RAG,” in Natural Language Processing and Information Systems, A. Rapp, L. Di Caro, F. Meziane, and V . Sugumaran, Eds. Springe...

    2024

  50. [56]

    Should we fine-tune or RAG? evaluating different technique s to adapt LLMs for dialogue

    S. Alghisi, M. Rizzoli, G. Roccabruna, S. M. Mousavi, an d G. Riccardi, “Should we fine-tune or RAG? evaluating different technique s to adapt LLMs for dialogue.” [Online]. Available: http://arxiv.org/abs/2406.06399

    arXiv

  51. [57]

    Benchmarking retrieval augme nted genera- tion in quantitative finance,

    A. Onan and E. D. Dursun, “Benchmarking retrieval augme nted genera- tion in quantitative finance,” in Intelligent and Fuzzy Systems , C. Kahra- man, S. Cevik Onar, S. Cebi, B. Oztaysi, A. C. Tolga, and I. Uca l Sari, Eds. Springer Nature Switzerland, 2024, pp. 64–74

    2024

  52. [60]

    MIRAGE- bench: Automatic multilingual benchmark arena for retriev al-augmented generation systems

    N. Thakur, S. Kazi, G. Luo, J. Lin, and A. Ahmad, “MIRAGE- bench: Automatic multilingual benchmark arena for retriev al-augmented generation systems.” [Online]. Available: http://arxiv.org/abs/2410.13716

    arXiv

  53. [61]

    Evaluating the efficacy of open-sour ce LLMs in enterprise-specific RAG systems: A comparative study of per formance and scalability

    G. B and A. Purwar, “Evaluating the efficacy of open-sour ce LLMs in enterprise-specific RAG systems: A comparative study of per formance and scalability.” [Online]. Available: http://arxiv.org /abs/2406.11424

    arXiv

  54. [62]

    BERT: Pre-training of deep bidirectional transformers for language understan ding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understan ding,” version:

  55. [63]

    Available: http://arxiv.org/abs/1810.0480 5

    [Online]. Available: http://arxiv.org/abs/1810.0480 5

  56. [64]

    Sentence-BERT: Sentence embeddings using siamese BERT-networks

    N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using siamese BERT-networks.” [Online]. Avail able: http://arxiv.org/abs/1908.10084

    1908 arXiv

  57. [65]

    BERTScore: Evaluating text generation with BERT

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artz i, “BERTScore: Evaluating text generation with BERT.” [Onlin e]. Available: http://arxiv.org/abs/1904.09675

    1904 arXiv

  58. [66]

    Towards a unified multi-dimensional evaluator f or text generation

    M. Zhong, Y . Liu, D. Yin, Y . Mao, Y . Jiao, P . Liu, C. Zhu, H. Ji, and J. Han, “Towards a unified multi-dimensional evaluator f or text generation.” [Online]. Available: http://arxiv.org/abs /2210.07197

    arXiv

  59. [67]

    [Online]

    Evaluation concepts | LangSmith. [Online]. Available: https://docs.smith.langchain.com/evaluation/concepts

  60. [68]

    Evaluation of orca 2 against other LLMs for retrieval augmented generation,

    D. Huang and Z. Wang, “Evaluation of orca 2 against other LLMs for retrieval augmented generation,” in Trends and Applications in Knowledge Discovery and Data Mining , Z. Wang and C. W. Tan, Eds. Springer Nature, 2024, pp. 3–19

    2024

  61. [69]

    RAD-bench: Evaluating large language mode ls capabilities in retrieval augmented dialogues

    T.-L. Kuo, F.-T. Liao, M.-W. Hsieh, F.-C. Chang, P .-C. H su, and D.-S. Shiu, “RAD-bench: Evaluating large language mode ls capabilities in retrieval augmented dialogues.” [Online] . Available: http://arxiv.org/abs/2409.12558

    arXiv

  62. [70]

    RAGProbe: An automated approach for evaluating RAG applic ations

    S. Sivasothy, S. Barnett, S. Kurniawan, Z. Rasool, and R . V asa, “RAGProbe: An automated approach for evaluating RAG applic ations.” [Online]. Available: http://arxiv.org/abs/2409.19019

    arXiv

  63. [71]

    Intrinsic evaluation of RAG systems for deep-logic questions

    J. Hu, Y . Zhou, and J. Wang, “Intrinsic evaluation of RAG systems for deep-logic questions.” [Online]. Availabl e: http://arxiv.org/abs/2410.02932

    arXiv

  64. [72]

    [Online]

    Zeta Alpha V ector, “Ragelo,” 2023. [Online]. Availabl e: https://github.com/zetaalphavector/RAGElo

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.