Pith. sign in

REVIEW 4 major objections 6 minor 68 references

Automated coverage-guided tests expose 6.6% more RAG failures than naive queries.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 01:28 UTC pith:7ZNAC3NW

load-bearing objection Useful tooling paper whose headline comparison is confounded by the baseline omitting negative-rejection queries, so the 6.6% improvement should be read as a property of the test-suite composition, not of coverage strategy. the 4 major comments →

arxiv 2608.00054 v1 pith:7ZNAC3NW submitted 2026-07-26 cs.AI cs.SE

RAG-TESTER: Automated End-to-End Testing of Retrieval-Augmented Large Language Models

Ange Maiztegi , Jon Ayerdi , Miren Illarramendi , Aitor Arrieta This is my paper
classification cs.AI cs.SE
keywords RAG testingretrieval-augmented generationtest generationLLM-as-a-judgenegative testingcoverage criteriahallucination detectionend-to-end testing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the reliability of a retrieval-augmented generation system cannot be inferred from its components in isolation, because failures arise from how retrieval and generation interact. To test that claim, the authors build RAG-TESTER, an end-to-end pipeline that generates its own retrieval documents, test queries and expected answers, runs them against a RAG system, and judges the answers with a separate LLM acting as an oracle. The test-generation strategy deliberately targets hard-to-read passages, queries whose answers are absent from the corpus, and document regions not yet covered by earlier tests. Across 72,000 executions spanning 24 combinations of eight LLMs and six embedding models, the approach detected 21,633 failures—6.6% more than a baseline generator that picks queries without any strategy—and outperformed the baseline in 20 of 24 configurations. The evident point of caring: systematic, automated test generation can find RAG weaknesses, including hallucinated answers on unsupported queries, that simplistic benchmarks miss.

Core claim

RAG-TESTER's central discovery is that a coverage-oriented test-input generator exposes substantially more RAG failures than a naive random one, and that the extra failures are qualitatively different: hallucinated responses to negative-rejection queries, incomplete use of retrieved context, retrieval inaccuracies, and misinterpretation of complex passages. The claim is that these failures stem from the interaction between the embedding model, retriever, prompt, and generative model, so they can only be found by end-to-end testing. The paper reports that even the strongest configuration failed about 17% of the time, that real-world PDFs provoked 64.4% more failures than AI-generated ones, an

What carries the argument

The load-bearing mechanism is the test-input generator's three-pronged strategy: (1) selecting the most complex chunks by a standard readability score, under the assumption they are hardest for the model to interpret; (2) synthesizing negative-rejection queries about information absent from the corpus, to test whether the system admits ignorance; and (3) a coverage pass that generates queries from document regions not yet touched, maximizing exercise of the corpus. The other key piece is the oracle: a separate LLM judges each answer on faithfulness, relevance, completeness, clarity, and keyword presence, with an automatic fail if faithfulness is false. This machinery is what converts a corpu

Load-bearing premise

The comparison rests on the assumption that failure counts produced by an LLM-as-a-judge oracle are a valid, unbiased metric for ranking test-generation strategies; if the oracle's verdicts are skewed by which passage the ground truth was built from, or by its own preferences, the 6.6% improvement could be an artifact of measurement rather than a real difference in test power.

What would settle it

Re-run the comparison with a baseline that also generates negative-rejection queries, and have humans label a stratified sample of the flagged failures; if the failure gap shrinks below statistical significance, or if most of the extra failures vanish under human review, the coverage strategy's advantage is not what the paper claims.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Developers can compare LLM–embedding configurations before deployment; even the best configuration still failed roughly 17% of RAG-TESTER's tests.
  • Negative-rejection queries surface hallucination failures that a strategy-free baseline cannot detect, since the baseline never asks for absent information.
  • Real-world documents are stronger test material than auto-generated ones: they triggered 64.4% more failures, so synthetic corpora should be supplemented with real documents for high-stakes evaluation.
  • Oracle verdicts must be read with caution at the individual-test level, since roughly one in five reported failures is a false positive at 80.6% precision.
  • The approach generalizes beyond the tested models: because any RAG pipeline can be executed and judged this way, the same four-stage flow can assess other retrievers, chunking schemes, and prompt templates.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reported 6.6% gain is partly a category effect: the baseline deliberately omits negative-rejection queries, so part of RAG-TESTER's advantage may simply be asking a new kind of question rather than asking better questions. A baseline that also includes unsupported queries would isolate the value of complexity and coverage selection.
  • Several 'failures' in the tables compare the model's answer against a ground truth derived from a different retrieved passage than the one the system actually used; some flagged failures may therefore be artifacts of passage mismatch rather than genuine system defects. Human labeling of a stratified sample would settle this.
  • A testable extension: measure failure-detection yield per query type (complex, negative, coverage). If negative queries account for nearly all of the surplus, then the complexity and coverage mechanisms should be re-weighted or replaced.
  • Chunk-coverage adequacy could be merged into RAG-TESTER's loop, using actual retrieval feedback rather than source-side selection, to verify that coverage of the corpus translates into coverage of the retrieval space.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents RAG-TESTER, an automated end-to-end testing framework for RAG-enabled LLMs. The framework generates retrieval documents (PDFs), test inputs (positive and negative queries, including complexity-targeted and coverage-based selection), expected outputs, and uses an LLM-as-a-judge to classify system responses as pass or fail. The authors evaluate on 24 LLM-embedding configurations, executing 72,000 test cases, and claim that RAG-TESTER detects 21,633 failures versus 20,293 for a baseline generator (a 6.6% increase), outperforming the baseline in 20 of 24 configurations. Additional claims include a test-oracle precision of 80.6% and that real-world PDFs expose more failures than automatically generated ones.

Significance. If the comparative claim could be supported, RAG-TESTER would be a useful practical contribution: it is open-source, has a replication package, is modular, and addresses a real need for systematic, end-to-end testing of RAG pipelines. The idea of combining negative rejection queries, complexity-based selection, and document-coverage criteria is sensible and worth evaluating. However, the headline result currently rests on a confounded baseline and on an oracle whose precision is estimated by LLM majority vote rather than human labels. The paper's contribution is therefore promising but not yet established at the level claimed.

major comments (4)
  1. [§5.3, §6.2] The baseline comparison is confounded. The baseline is described as removing coverage and paragraph-complexity features, but §6.2 explicitly states: "No different test query types were used in the baseline, so no hallucination failures were detected." This means the baseline generates only supported positive queries, while RAG-TESTER adds NEG_REJ queries. The reported 6.6% overall improvement (21,633 vs 20,293) and the +19.68% for OpenAI models (8,880 vs 7,132) therefore conflate the addition of a query category with the coverage/complexity strategy. An ablation baseline that also generates NEG_REJ queries at the same proportion but without Flesch sorting or coverage selection is required. Without such an ablation, the central claim that coverage-oriented generation is more effective is not supported.
  2. [§5.2, §6.4] Oracle precision is estimated by majority vote of four LLMs, not by human annotation. The 80.6% precision and 19.4% false-discovery rate therefore measure agreement with an LLM panel, not verified correctness against ground truth. Since the central outcome—failure counts—is produced by this oracle, the comparison inherits any systematic bias of the judge LLM. A human-annotated sample (the 382 cases, or a fresh sample) is needed, along with inter-annotator agreement and per-query-type precision estimates.
  3. [Table 3 (LOW_FLESCH example)] The oracle evaluation may be comparing answers against a ground truth generated from a different retrieval result. In the LOW_FLESCH example, the ground-truth answer and its retrieved text concern the $6.2 million/EQIP figures, while the system under test retrieved a different passage with $5,098,856/$6,632,204. If the retriever returns a different but related passage, the model may correctly answer from the retrieved evidence yet be marked as failing because the oracle uses the original ground truth. This is a test-generation artifact rather than a system failure. The oracle should either compare against the evidence actually retrieved, or separately flag cases where the ground-truth passage was not retrieved, rather than treating them as answer failures.
  4. [§6.1–§6.2, Tables 6 and 7] The 20-of-24 superiority claim lacks statistical support. The same 3,000 generated test inputs are reused across all 24 configurations, so the 72,000 executions are not independent observations; no confidence intervals or significance tests are reported. Per-configuration differences (e.g., the baseline outperforming RAG-TESTER on Llama 3.2 across all embeddings) are large, and aggregate counts may be dominated by a few weak models. A matched analysis (e.g., McNemar's test on per-document or per-query matching) or at least bootstrap confidence intervals over documents/models is needed to support the comparative claim.
minor comments (6)
  1. [§3.4] Typo: "specifies the the role" should read "specifies the role".
  2. [Appendix F] Typo: "permorfance" should be "performance"; also "NEG_REG" appears twice where "NEG_REJ" is meant.
  3. [§6.1] Formatting: "16,93%" should be "16.93%".
  4. [§7] Typo: "heaviliy" should be "heavily".
  5. [Table 5] The generated answer column contains "srael's", apparently missing the leading "I".
  6. [Tables 6 and 7] The first references to Tables 6 and 7 appear in §6.1, but the tables are placed later, after the RQ3 discussion. Please reorder or renumber the tables so that the appearance matches the reference order.

Circularity Check

0 steps flagged

No circularity: the central claim is an empirical comparison, not a derivation that reduces to its inputs.

full rationale

RAG-TESTER's core result is an empirical measurement: under an LLM-as-a-judge oracle, its test-generation strategy flagged 21,633 failures versus 20,293 for a baseline. There is no fitted parameter, no equation that equates output with input, and no load-bearing self-citation. The fact that GPT-4.1 Nano generates expected answers and also serves as judge is a measurement-validity/bias concern, not a constructional circularity; the oracle could in principle pass or fail a given answer independently of how the expected answer was generated. Oracle precision is estimated by majority vote of four different, larger LLMs; although this is not human labeling, it is a proxy validation rather than a definitional equivalence. The baseline confound is real and correctly identified in the paper: the paper states 'No different test query types were used in the baseline, so no hallucination failures were detected' (Section 6.2), so the 6.6% improvement may partly reflect adding the NEG_REJ query category rather than the coverage/complexity strategy. However, a confounded experimental comparison is a correctness/threat-to-validity issue, not circularity. The self-citations present (e.g., refs. 19 and 20) are minor and are not used to justify the central claim. No uniqueness theorem, ansatz-by-citation, or renamed-known-result pattern appears. The paper even acknowledges oracle limitations ('its verdicts should be interpreted with caution', Section 6.4). A non-finding is therefore appropriate: the derivation chain is not circular, though the empirical claims carry validity risks that are outside the circularity construct.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central empirical claim rests on LLM-generated tests, LLM-generated ground truth, and LLM-based scoring, with hand-chosen thresholds and model choices. No external human-labeled benchmark anchors the failure counts.

free parameters (4)
  • Pass/fail threshold = 0.5 (mean of five criteria)
    Section 3.5: answers with average score <0.5 are Failed. All failure counts in the paper depend on this threshold.
  • Number of tests per document (N_test) = 50
    Section 5.4: 50 test inputs per PDF; determines total executions (3,000) and all reported counts.
  • Judge LLM (GPT-4.1 Nano)
    Section 3.5: selected based on 'preliminary analyses' not reported; oracle verdicts and all failure counts depend on this choice.
  • Number of complex passages (N_complex)
    Section 3.3: N_complex < N_test; value not reported, but determines how many LOW_FLESCH tests are generated.
axioms (4)
  • domain assumption LLM-as-a-judge verdicts accurately reflect ground-truth correctness
    Section 3.5 and RQ4: the oracle is the sole arbiter of pass/fail; precision estimated via majority vote of 4 LLMs, not human labels.
  • domain assumption LLM-generated expected outputs (ground truth) are correct
    Section 3.3: GPT-4.1 Nano generates expected answers from document chunks; errors in these propagate directly to failure classification.
  • domain assumption Flesch Reading Ease ranks passage comprehension difficulty for LLMs
    Section 3.3: low-Flesch passages are assumed harder for MUTs, motivating prioritized test generation.
  • domain assumption Coverage-based query selection exposes more real failures than random selection
    Answer to RQ2: this is the paper's central causal claim; it is tested only against an ablated baseline, so it remains an assumption.

pith-pipeline@v1.3.0-alltime-deepseek · 24567 in / 12712 out tokens · 121647 ms · 2026-08-04T01:28:36.045101+00:00 · methodology

0 comments
read the original abstract

Retrieval-Augmented Generation (RAG) enables Large Language Models (LLMs) to use external and domain-specific knowledge, but its reliability depends on the interaction between the generative model, embedding model, retrieval mechanism, and prompt construction strategy. We present RagTester, an automated end-to-end testing approach for RAG systems. RagTester generates retrieval documents, test inputs, and expected outputs; executes the tests; and evaluates the resulting answers using an LLM as a judge. Its test-generation strategy targets complex passages, unsupported queries, and document-coverage criteria. We evaluate RagTester using eight LLMs and six embedding models, yielding 24 compatible configurations, and compare it with a baseline test-input generator. Across 72,000 test executions, RagTester detected 21,633 failures, 6.6% more than the baseline, and outperformed it in 20 of the 24 configurations. The detected failures include inaccurate retrieval, unsupported answers, incomplete use of retrieved context, and difficulties interpreting complex passages. These results show that coverage-oriented test generation can effectively expose failures caused by the interaction between retrieval and generation components and support the assessment of RAG configurations before deployment.

Figures

Figures reproduced from arXiv: 2608.00054 by Aitor Arrieta, Ange Maiztegi, Jon Ayerdi, Miren Illarramendi.

Figure 1
Figure 1. Figure 1: RAG-Tester execution flow architecture 3. RAG-TESTER 3.1. Overview RAG-TESTER consists of four main components that together form an end-to-end automated test execution frame￾work for Retrieval-Augmented LLMs. Algorithm 1 and Fig￾ure 1 outline the main steps of RAG-TESTER and illustrate how these four components interact. As input, RAG-TESTER receives a text document and some instructions, producing a set … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

68 extracted references · 1 canonical work pages

  1. [1]

    Retrieval-augmented generation for knowledge-intensivenlptasks,

    P.Lewis,E.Perez,A.Piktus,F.Petroni,V.Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rock- täschelet al., “Retrieval-augmented generation for knowledge-intensivenlptasks,”inAdvancesinNeural Information Processing Systems (NeurIPS), vol. 33, 2020, pp. 9459–9474

    2020

  2. [2]

    Rageval: Scenario specific rag evaluation dataset generation framework,

    K. Zhu, Y. Luo, D. Xu, R. Wang, S. Yu, S. Wang, Y. Yan, Z. Liu, X. Han, Z. Liu, and M. Sun, “Rageval: Scenario specific rag evaluation dataset generation framework,” 2024. [Online]. Available: https://arxiv.org/abs/2408.01262

    Pith/arXiv arXiv 2024

  3. [3]

    Multihop-rag: Benchmarking retrieval-augmentedgenerationformulti-hopqueries,

    Y. Tang and Y. Yang, “Multihop-rag: Benchmarking retrieval-augmentedgenerationformulti-hopqueries,” arXiv preprint arXiv:2401.15391, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    Legalbench-rag: A benchmark for retrieval-augmented generation in the legal domain,

    N. Pipitone and G. H. Alami, “Legalbench-rag: A benchmark for retrieval-augmented generation in the legal domain,”arXiv preprint arXiv:2408.10343, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Ragbench: Ex- plainable benchmark for retrieval-augmented genera- tionsystems,

    R. Friel, M. Belyi, and A. Sanyal, “Ragbench: Ex- plainable benchmark for retrieval-augmented genera- tionsystems,”arXivpreprintarXiv:2407.11005,2024

    Pith/arXiv arXiv 2024

  6. [6]

    Crag-comprehensive rag benchmark,

    X. Yang, K. Sun, H. Xin, Y. Sun, N. Bhalla, X. Chen, S. Choudhary, R. D. Gui, Z. W. Jiang, Z. Jianget al., “Crag-comprehensive rag benchmark,”Advances in Neural Information Processing Systems, vol. 37, pp. 10470–10490, 2024. Ange Maiztegui, Jon Ayerdi, Miren Illarramendi, Aitor Arrieta:Preprint submitted to ElsevierPage 13 of 21 Automated End-to-End Testi...

    2024

  7. [7]

    Retrieval augmentation reduces hallucination in con- versation,

    K. Shuster, S. Poff, M. Chen, D. Kiela, and J. Weston, “Retrieval augmentation reduces hallucination in con- versation,”arXiv preprint arXiv:2104.07567, 2021

    Pith/arXiv arXiv 2021

  8. [9]

    Dense pas- sageretrievalforopen-domainquestionanswering

    V. Karpukhin, B. Oguz, S. Min, P. S. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih, “Dense pas- sageretrievalforopen-domainquestionanswering.”in EMNLP (1), 2020, pp. 6769–6781

    2020

  9. [10]

    Sentence-bert: Sen- tenceembeddingsusingsiamesebert-networks,

    N. Reimers and I. Gurevych, “Sentence-bert: Sen- tenceembeddingsusingsiamesebert-networks,”arXiv preprint arXiv:1908.10084, 2019

    Pith/arXiv arXiv 1908

  10. [11]

    Billion-scale similaritysearchwithgpus,

    J. Johnson, M. Douze, and H. Jégou, “Billion-scale similaritysearchwithgpus,”IEEETransactionsonBig Data, vol. 7, no. 3, pp. 535–547, 2019

    2019

  11. [12]

    The faiss library,

    M. Douze, A. Guzhva, C. Deng, J. Johnson, G. Szil- vasy,P.-E.Mazaré,M.Lomeli,L.Hosseini,andH.Jé- gou, “The faiss library,”IEEE Transactions on Big Data, 2025

    2025

  12. [14]

    Atlas: Few-shot learning withretrievalaugmentedlanguagemodels,

    G. Izacard, P. Lewis, M. Lomeli, L. Hosseini, F. Petroni, T. Schick, J. Dwivedi-Yu, A. Joulin, S. Riedel, and E. Grave, “Atlas: Few-shot learning withretrievalaugmentedlanguagemodels,”Journalof MachineLearningResearch,vol.24,no.251,pp.1–43, 2023

    2023

  13. [15]

    Retrieval-augmented generationforlargelanguagemodels:Asurvey,

    Y.Gao,Y.Xiong,X.Gao,K.Jia,J.Pan,Y.Bi,Y.Dai, J. Sun, H. Wang, and H. Wang, “Retrieval-augmented generationforlargelanguagemodels:Asurvey,”arXiv preprint arXiv:2312.10997, 2023

    Pith/arXiv arXiv 2023

  14. [16]

    Teaching language models to support answers with verified quotes,

    J. Menick, M. Trebacz, V. Mikulik, J. Aslanides, F. Song, M. Chadwick, M. Glaese, S. Young, L. Campbell-Gillingham, G. Irvinget al., “Teaching language models to support answers with verified quotes,”arXiv preprint arXiv:2203.11147, 2022

    Pith/arXiv arXiv 2022

  15. [17]

    A new readability yardstick

    R. Flesch, “A new readability yardstick.”Journal of applied psychology, vol. 32, no. 3, p. 221, 1948

    1948

  16. [18]

    Chase, Oct 2022

    H. Chase, Oct 2022. [Online]. Available: https: //www.langchain.com/

    2022

  17. [19]

    Meta-fair: Ai- assisted fairness testing of large language models,

    M. Romero-Arjona, J. A. Parejo, J. C. Alonso, A. B. Sánchez, A. Arrieta, and S. Segura, “Meta-fair: Ai- assisted fairness testing of large language models,”In- formation and Software Technology, p. 108075, 2026

    2026

  18. [20]

    Rag-tester demo,

    A.Maiztegi,J.Ayerdi,M.Illarramendi,andA.Arrieta, “Rag-tester demo,” https://youtu.be/jNxVVm-c-mc, 2026, youTube video

    2026

  19. [21]

    Dataset of pdf files,

    Manisha717, “Dataset of pdf files,” https://www. kaggle.com/datasets/manisha717/dataset-of-pdf-files/ data, 2024, kaggle dataset, accessed April 14, 2026

    2024

  20. [22]

    designed for modularity and ease-of-use, backed by research.” [Online].Available:https://github.com/AnswerDotAI/ RAGatouille

    AnswerDotAI, “Answerdotai/ragatouille: Easily use and train state of the art late-interaction retrieval methods (colbert) in any rag pipeline. designed for modularity and ease-of-use, backed by research.” [Online].Available:https://github.com/AnswerDotAI/ RAGatouille

  21. [23]

    ranx: A blazing-fast python library forrankingevaluationandcomparison,

    E. Bassani, “ranx: A blazing-fast python library forrankingevaluationandcomparison,”inAdvancesin InformationRetrieval,M.Hagen,S.Verberne,C.Mac- donald, C. Seifert, K. Balog, K. Nørvåg, and V. Setty, Eds. Cham:SpringerInternationalPublishing,2022, pp. 259–264

    2022

  22. [24]

    G- eval: NLG evaluation using gpt-4 with better human alignment,

    Y.Liu,D.Iter,Y.Xu,S.Wang,R.Xu,andC.Zhu,“G- eval: NLG evaluation using gpt-4 with better human alignment,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore:AssociationforComputationalLinguistics, Dec. 2023, pp. 2511–2522. [Online]. Available: https://aclanthology.org/...

    2023

  23. [25]

    Uptrain-Ai, “Uptrain-ai/uptrain: Uptrain is an open- source unified platform to evaluate and improve generative ai applications. we provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.” [Online]. Available: https://github.com/uptra...

  24. [26]

    RAGAs: Automated evaluation of retrieval augmented generation,

    S. Es, J. James, L. Espinosa Anke, and S. Schockaert, “RAGAs: Automated evaluation of retrieval augmented generation,” inProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, N. Aletras and O. De Clercq, Eds. St. Julians, Malta: Association for Computational Linguistics, Mar. 20...

    2024

  25. [27]

    Recall@k surrogate loss with large batches and similarity mixup,

    Y. Patel, G. Tolias, and J. Matas, “Recall@k surrogate loss with large batches and similarity mixup,” 2022. [Online]. Available: https://arxiv.org/abs/2108.11179

    Pith/arXiv arXiv 2022

  26. [28]

    Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection,

    K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection,” in Proceedings of the 16th ACM Workshop on Ange Maiztegui, Jon Ayerdi, Miren Illarramendi, Aitor Arrieta:Preprint submitted to ElsevierPage 14 of 21 Automated End-to-En...

    arXiv 2023

  27. [29]

    Trustllm: Trustworthiness in large language models,

    Y. Huang, L. Sun, H. Wang, S. Wu, Q. Zhang, Y. Li, C. Gao, Y. Huang, W. Lyu, Y. Zhang, X. Li, Z. Liu, Y. Liu, Y. Wang, Z. Zhang, B. Vidgen, B. Kailkhura, C. Xiong, C. Xiao, C. Li, E. Xing, F. Huang, H. Liu, H.Ji,H.Wang,H.Zhang,H.Yao,M.Kellis,M.Zitnik, M. Jiang, M. Bansal, J. Zou, J. Pei, J. Liu, J. Gao, J. Han, J. Zhao, J. Tang, J. Wang, J. Vanschoren, J....

    Pith/arXiv arXiv 2024

  28. [30]

    Benchmarking large language models in retrieval-augmented generation,

    J. Chen, H. Lin, X. Han, and L. Sun, “Benchmarking large language models in retrieval-augmented generation,”Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 16, pp. 17754–17762, Mar. 2024. [Online]. Available: https: //ojs.aaai.org/index.php/AAAI/article/view/29728

    2024

  29. [31]

    Crud-rag: A comprehensive chinese benchmark for retrieval- augmented generation of large language models,

    Y. Lyu, Z. Li, S. Niu, F. Xiong, B. Tang, W. Wang, H. Wu, H. Liu, T. Xu, and E. Chen, “Crud-rag: A comprehensive chinese benchmark for retrieval- augmented generation of large language models,” ACM Trans. Inf. Syst., vol. 43, no. 2, Jan. 2025. [Online]. Available: https://doi.org/10.1145/3701228

    work page doi:10.1145/3701228 2025

  30. [32]

    HaluEval: A large-scale hallucination evaluation benchmarkforlargelanguagemodels,

    J. Li, X. Cheng, X. Zhao, J.-Y. Nie, and J.-R. Wen, “HaluEval: A large-scale hallucination evaluation benchmarkforlargelanguagemodels,”inProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 6449–

    2023

  31. [33]

    Towards global retrieval augmented generation: A benchmark forcorpus-levelreasoning,

    Q. Luo, X. Li, T. Fan, X. Chen, and X. Qiu, “Towards global retrieval augmented generation: A benchmark forcorpus-levelreasoning,”2025.[Online].Available: https://arxiv.org/abs/2510.26205

    arXiv 2025

  32. [34]

    mtrag: A multi-turn conversational benchmark for evaluating retrieval-augmentedgenerationsystems,

    Y. Katsis, S. Rosenthal, K. Fadnis, C. Gunasekara, Y.-S. Lee, L. Popa, V. Shah, H. Zhu, D. Contractor, and M. Danilevsky, “mtrag: A multi-turn conversational benchmark for evaluating retrieval-augmentedgenerationsystems,”Transactions of the Association for Computational Linguistics, vol. 13, pp. 784–808, 07 2025. [Online]. Available: https://doi.org/10.11...

    doi:10.1162/tacl.a.19 2025

  33. [35]

    Worse than zero-shot? a fact-checking dataset for evaluating the robustness of rag against misleading retrievals,

    L. Zeng, R. Gupta, D. Motwani, Y. Zhang, and D. Yang, “Worse than zero-shot? a fact-checking dataset for evaluating the robustness of rag against misleading retrievals,” 2026. [Online]. Available: https://arxiv.org/abs/2502.16101

    arXiv 2026

  34. [36]

    Rare: Retrieval- aware robustness evaluation for retrieval-augmented generation systems,

    Y. Zeng, T. Cao, D. Wang, X. Zhao, Z. Qiu, M. Ziyadi, T. Wu, and L. Li, “Rare: Retrieval- aware robustness evaluation for retrieval-augmented generation systems,” 2025. [Online]. Available: https://arxiv.org/abs/2506.00789

    arXiv 2025

  35. [37]

    Redefining retrieval evaluation in the era of LLMs,

    G. Trappolini, F. Cuconasu, S. Filice, Y. Maarek, and F. Silvestri, “Redefining retrieval evaluation in the era of LLMs,” inProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), V. Demberg, K. Inui, and L. Marquez, Eds. Rabat, Morocco: Association for Computational Linguistics,...

    2026

  36. [38]

    Harmbench: A standardized evaluation framework for automated red teamingandrobustrefusal,

    M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li, D. Forsyth, and D. Hendrycks, “Harmbench: A standardized evaluation framework for automated red teamingandrobustrefusal,”2024.[Online].Available: https://arxiv.org/abs/2402.04249

    Pith/arXiv arXiv 2024

  37. [39]

    Jailbreakbench: An open robustness benchmark for jailbreaking large language models,

    P. Chao, E. Debenedetti, A. Robey, M. Andriushchenko, F. Croce, V. Sehwag, E. Dobriban, N. Flammarion, G. J. Pappas, F. Tramèr, H. Hassani, and E. Wong, “Jailbreakbench: An open robustness benchmark for jailbreaking large language models,” inAdvances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomc...

    2024

  38. [40]

    ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection,

    T. Hartvigsen, S. Gabriel, H. Palangi, M. Sap, D. Ray, and E. Kamar, “ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection,” inProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), S. Muresan, P. Nakov, and A. Villavicencio, Eds. Dublin, Ireland: Asso...

    2022

  39. [41]

    BBQ: A hand-built bias benchmark for question answering,

    A. Parrish, A. Chen, N. Nangia, V. Padmakumar, J. Phang, J. Thompson, P. M. Htut, and S. Bowman, “BBQ: A hand-built bias benchmark for question answering,” inFindings of the Association for Computational Linguistics: ACL 2022, S. Muresan, Ange Maiztegui, Jon Ayerdi, Miren Illarramendi, Aitor Arrieta:Preprint submitted to ElsevierPage 15 of 21 Automated En...

    2022

  40. [42]

    Retromorphic testing with hierarchical verification for hallucination detection in rag,

    B. Yu, Y. Zhang, L. Lin, L. Briand, and E. Muñoz, “Retromorphic testing with hierarchical verification for hallucination detection in rag,” 2026. [Online]. Available: https://arxiv.org/abs/2603.27752

    Pith/arXiv arXiv 2026

  41. [43]

    AdvERSEM: Adversarial robustness testing and training of LLM-based groundedness evaluators via semantic structure manipulation,

    K. Dhole, R. Chandradevan, and E. Agichtein, “AdvERSEM: Adversarial robustness testing and training of LLM-based groundedness evaluators via semantic structure manipulation,” inProceedings of the 14th Joint Conference on Lexical and Computational Semantics (*SEM 2025), L. Frermann and M. Stevenson, Eds. Suzhou, China: Association for Computational Linguis...

    2025

  42. [44]

    Artificial intelligence risk management framework (ai rmf 1.0),

    E. Tabassi, “Artificial intelligence risk management framework (ai rmf 1.0),” 2023-01-26 05:01:00

    2023

  43. [45]

    Testing retrieval- augmented generation systems with chunk coverage,

    J. Kim, S. Pasini, and P. Tonella, “Testing retrieval- augmented generation systems with chunk coverage,” arXiv preprint arXiv:2607.18155, 2026

    Pith/arXiv arXiv 2026

  44. [46]

    (2025) Deepeval: An open-source evaluation framework for large language models

    Confident AI. (2025) Deepeval: An open-source evaluation framework for large language models. Framework for evaluating LLM performance, including metrics for accuracy, robustness, and hallucination detection. [Online]. Available: https: //github.com/AI-App/DeepEval

    2025

  45. [47]

    Platform for simulation, evaluation, and monitoring of AI agents across the full development lifecycle

    MaximAI.(2026)Maximai:End-to-endaievaluation and observability platform. Platform for simulation, evaluation, and monitoring of AI agents across the full development lifecycle. [Online]. Available: https://www.getmaxim.ai/

    2026

  46. [51]

    - Include a detailed table of contents with section numbers

    Appendix A. PDF Generation System Prompt @staticmethod def g et _s ys tem _p ro mp t ( length : str = None , detail_level : str = None , tone : str = None , include_toc : bool = None , in cl ud e_ cit at io ns : bool = None ) -> str : length = length or P D F G e n e r a t o r P r o m p t s . DEFAULT_LENGTH detail_level = detail_level or P D F G e n e r a...

  47. [52]

    ** id **: Sequential number (1 to { n u m _ q u e s t i o n s _ p e r _ s e c t i o n })

  48. [53]

    ** question **: Clear , unambiguous question matching the { spec ['label']} type

  49. [54]

    ** g r o u n d _ t r u t h _ a n s w e r **: Accurate , concise answer based ONLY on document content Ange Maiztegui, Jon Ayerdi, Miren Illarramendi, Aitor Arrieta:Preprint submitted to ElsevierPage 18 of 21 Automated End-to-End Testing of Retrieval-Augmented Large Language Models

  50. [55]

    ** source_document **: Name / path of the source document , visible in section metadata as' source'

  51. [56]

    Specific section / page where answer is found . Visible in section metadata as'page_number'

    ** source_location **: {" Specific section / page where answer is found . Visible in section metadata as'page_number'" if question_type != " negative rejection " else " unknown "}

  52. [57]

    ** r e le v a nt _ t ex t _s p a n **: Exact quote from document containing the answer

  53. [58]

    { spec [' label']}

    ** question_type **: MUST be exactly "{ spec [' label']}" for ALL questions

  54. [59]

    easy " ,

    ** difficulty **: " easy " , " medium " , or " hard "

  55. [60]

    ** r e as o n in g _ re q ui r e d **: Brief description of reasoning needed

  56. [61]

    - For negative rejection questions , answer MUST be :'Insufficient information to answer this question'

    ** keywords **: Key terms from the answer for evaluation QUALITY STANDARDS Question Quality : - Avoid yes / no questions ; prefer questions requiring specific information - Ensure questions are clear and have a single correct answer - Create questions that test different aspects of the document - Questions must align with the { spec ['label']} category re...

  57. [62]

    Cover all major topics in the document

  58. [63]

    Include a mix of difficulty levels

  59. [64]

    Are answerable solely from the provided content

  60. [65]

    question

    Include exact source locations for verification Generate the test cases now in the specified JSON format .""" Ange Maiztegui, Jon Ayerdi, Miren Illarramendi, Aitor Arrieta:Preprint submitted to ElsevierPage 19 of 21 Automated End-to-End Testing of Retrieval-Augmented Large Language Models return user_prompt E. Test Execution Prompt def create_prompt ( sel...

  61. [66]

    Faithfulness ( true - partially - false ) : Is the model answer grounded compared to the correct answer ?

  62. [67]

    Relevance ( true - partially - false ) : Does the model answer directly address the query ?

  63. [68]

    Completeness ( true - partially - false ) : Is the Model Under Test answer as complete as the correct one ? If the Model Under Test answer is more complete than the correct answer , evaluate it positively

  64. [69]

    Clarity ( true - partially - false ) : Is the generated answer clear and well - structured ?

  65. [70]

    "" , input_variables =[

    Keywords match ( true - partially - false ) : Do the keywords appear in the Model Under Test's answer ? True if all keywords appear , partially if some appear and false if none appear . Write two lines summarizing the permorfance of the model under test . Compare the Correct Source Location with the Model Source Location , Correct Text Span with the Model...

  66. [408]

    Available: https://aclanthology.org/ 2025.starsem-1.32/

    [Online]. Available: https://aclanthology.org/ 2025.starsem-1.32/

    2025

  67. [2023]

    Available: https://tsapps.nist.gov/ publication/get_pdf.cfm?pub_id=936225

    [Online]. Available: https://tsapps.nist.gov/ publication/get_pdf.cfm?pub_id=936225

  68. [6464]

    Available: https://aclanthology.org/ 2023.emnlp-main.397/

    [Online]. Available: https://aclanthology.org/ 2023.emnlp-main.397/

    2023