Pith. sign in

REVIEW 3 major objections 4 minor 28 references

This paper introduces Chunk Coverage, the fraction of corpus chunks retrieved at least once, as an oracle-free adequacy criterion for RAG retrieval, and shows that guiding test selection by it exposes distinct faults earlier.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 15:48 UTC pith:CKOXTH7E

load-bearing objection Chunk Coverage is a clean, practical adequacy metric for RAG retrieval and the coverage-speedup results hold up, but the fault-detection claim in RQ3 is partly built into the fault definition, so treat the 10–25% APFD gain with caution. the 3 major comments →

arxiv 2607.18155 v1 pith:CKOXTH7E submitted 2026-07-20 cs.SE

Testing Retrieval-Augmented Generation Systems with Chunk Coverage

Jinhan Kim , Samuele Pasini , Paolo Tonella This is my paper
classification cs.SE
keywords Retrieval-Augmented Generationtest adequacycoverage criteriaChunk Coveragecoverage-guided test generationretrieval diversityfault detectionLLM testing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

RAG systems behave correctly only if the right documents are retrieved, but existing evaluation metrics judge each query on its own and usually need reference answers or relevance labels. This paper introduces Chunk Coverage (CC), the fraction of corpus chunks retrieved at least once across a suite, as a structural, oracle-free view of how thoroughly a test suite explores the retriever's behaviour. It then shows that using CC as a feedback signal to select or generate queries accelerates exploration—reaching 50% of attainable coverage 1.7x faster than random and 4.2x faster than redundancy-biased strategies—and that coverage-guided suites expose distinct retrieval faults earlier, improving APFD by 10–25% over random. The central payoff is a practical, per-query-independent yardstick for constructing and curating test suites in high-stakes, retrieval-essential RAG deployments.

Core claim

Chunk Coverage (CC) measures, for a given test suite, the fraction of the document corpus's chunks that the retriever returns at least once: CC(T)=|∪_{q∈T} R_k(q)| / |C|. The paper's claim is that CC is a meaningful test adequacy criterion for the retrieval component of RAG systems, analogous to code coverage: it abstracts away from output correctness and requires no oracle, yet it tracks established diversity measures such as entropy, Gini concentration, and Jaccard overlap. The empirical claim is that greedily prioritising queries that touch previously uncovered chunks reaches 50% of attainable coverage 1.7x faster than uniform random selection and 4.2x faster than an overlap-biased strate

What carries the argument

The central object is Chunk Coverage (CC), the set-based ratio of unioned retrieved chunks to corpus chunks. It does the work of a coverage criterion: it is cheap to compute from retrieval traces alone, it only increases as tests are added, and it gives an explicit list of uncovered chunks that serve as targets. The algorithmic machinery is a feedback loop (Algorithm 1) that executes tests, accumulates exercised chunks, identifies uncovered chunks, and directs an auxiliary LLM (or a query pool) to synthesize or select queries targeting those chunks, repeating until budget or target coverage is reached. Fault definitions and APFD then connect this structural metric to testing effectiveness.

Load-bearing premise

The fault-detection advantage is measured under an operational definition of 'fault' as clusters of failing queries with highly overlapping retrieved chunk sets; this definition is aligned with CC's goal of diversifying retrieved sets, so the measured benefit may partly reflect the metric's own design.

What would settle it

Re-run the prioritization experiments with faults defined by external ground-truth relevance violations or by injected retrieval mutations (e.g., corrupted embeddings or a broken index entry): if coverage-guided selection no longer detects these faults earlier than random selection, the claim that CC captures fault-relevant diversity is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Test suites for RAG systems can be built or pruned without ground-truth relevance: CC alone tells practitioners which regions of the corpus remain untouched.
  • Coverage-guided generation provides a budgeted exploration strategy: with a fixed query-execution budget, CC reaches the same coverage with fewer queries than random or redundancy-biased selection.
  • Earlier detection of distinct retrieval faults: suites ordered to maximise CC expose new fault behaviours sooner, which is valuable for debugging retrieval configuration before deployment.
  • CC makes suite-level blind spots visible: a suite with high average per-query quality can still have low CC, alerting developers that most of the corpus is never exercised.
  • CC complements, rather than replaces, per-query RAG metrics: it attaches to the retrieval component only, so it can run in settings where oracle-free structural testing is the only option.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to use CC as a regression-test prioritisation signal for RAG components in continuous integration, where coverage deltas between versions could flag retrieval regressions.
  • The plateaus the paper observes imply a testable hypothesis: some corpus regions are unreachable by any query the generation model can produce, making CC a diagnostic for query-pool adequacy rather than only retriever adequacy.
  • Because the fault-detection result relies on Jaccard-based fault clustering, re-running the experiments with faults defined by injected retrieval mutations or external relevance annotations would test whether the 10–25% APFD gain is an artefact of the clustering choice.
  • CC could be combined with per-query relevance metrics into a two-dimensional view (exploration plus correctness), a possibility the paper does not develop.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces Chunk Coverage (CC), an oracle-independent test adequacy criterion for RAG retrieval components, defined as the fraction of corpus chunks retrieved at least once across a test suite. It argues that per-query RAG metrics (e.g., context precision, faithfulness) do not capture suite-level retrieval exploration, and proposes CC as a structural analog of code coverage. The authors present three studies: (RQ1) CC correlates with existing diversity metrics (entropy, Gini, Jaccard similarity) on randomly sampled suites; (RQ2) coverage-guided test selection/generation reaches 50% of attainable coverage 1.7x faster than random and 4.2x faster than overlap-biased selection; (RQ3) coverage-guided ordering improves APFD by 10%–25% over random across five datasets. The evaluation covers MIMIC-IV clinical decision-making and four financial QA datasets from T2-RAGBench.

Significance. If the central claim holds, CC would be a useful practical tool: it gives testers a cheap, oracle-free signal for measuring and guiding suite-level exploration of retrieval behavior, complementing per-query quality metrics. The RQ2 speedup results are concrete and reproducible, and the paper includes a public artifact. However, the RQ3 fault-detection claim is weakened by a definitional circularity: the operational fault model in Section 4.3 defines faults through Jaccard overlap of retrieved chunk sets, which is exactly the signal CC-guided selection optimizes. Thus the empirical evidence does not independently establish that CC-guided testing improves detection of externally defined retrieval faults. The paper's contribution is still valuable, but the fault-detection claim needs reframing or complementary validation.

major comments (3)
  1. The APFD gains reported in Table 5 are largely a consequence of the fault definition. Section 4.3 defines two failing queries as the same fault if Jaccard(R_k(q1), R_k(q2)) > 0.8, i.e., faults are clusters of failures with highly overlapping retrieved chunk sets. CC-guided selection (Section 4.4) explicitly prioritizes queries whose retrieved chunks are novel, thereby minimizing overlap with previously executed queries. Under the Jaccard-based fault model, the first failure in a newly explored retrieval region automatically becomes a new fault, so CC-guided ordering is expected to achieve higher APFD almost by construction. The sensitivity analysis over tau=0.6–1.0 does not break this circularity, since all thresholds still partition failures by retrieved-chunk overlap. To support the claim that CC improves fault detection effectiveness, the authors should evaluate with an independent fa
  2. The RQ1 correlations are partly tautological. CC is the fraction of distinct chunks retrieved; entropy, Gini, and average pairwise Jaccard are all summary statistics of the same underlying retrieval distribution over chunks. For a fixed suite size, a suite with more distinct chunks must generally have higher entropy, lower concentration, and lower pairwise overlap, so the observed strong correlations (e.g., r=-0.881 with Gini, r=-0.862 with Jaccard) do not strongly validate CC as an independent construct. The paper's claim that 'CC aligns closely with established diversity measures' is technically correct but the interpretative weight placed on RQ1 should be reduced, or the analysis should include a comparison with a non-overlapping diversity metric (e.g., novelty of embedded chunk vectors) to show that CC carries information beyond the distribution of retrieval frequencies.
  3. The paper reports that on MIMIC-IV, the random strategy slightly outperforms CC-guided selection for Response Groundedness (APFD 0.615 vs 0.597, Table 5). The authors explain this as a limitation of the GPT-4o-mini query generator and state that re-running with GPT-5-mini resolves the exception. However, the revised results are not shown in the paper; they are only said to be available in the online artifact. Since this exception directly contradicts the universal 'CC improves fault detection' claim, the supplementary results should be included or summarized in the paper (at least as an appendix) so that the reader can verify that the failure is indeed a generation-model artifact rather than a counterexample to the central RQ3 claim.
minor comments (4)
  1. Typographical issue: 'it removed or anonymised diagnostic mentions' — the subject 'it' is unclear; should be 'the dataset creators removed...'.
  2. Minor phrasing: 'can be attributable to' should be 'can be attributed to'.
  3. The sensitivity analysis is reported in prose without a table. Since threshold choice is a free parameter, a small table listing APFD values for each tau would improve transparency.
  4. For MIMIC-IV, the query pool itself is generated by CC-guided target chunking. While the three strategies operate on the same pool, the pool construction is not independent of CC; this should be stated more prominently as a potential source of bias affecting all strategies similarly.

Circularity Check

2 steps flagged

RQ3's fault definition clusters failures by Jaccard similarity of retrieved chunk sets — the very signal CC-guided selection optimizes — so the 10–25% APFD gain is partly definitional rather than evidence of better fault detection.

specific steps
  1. other [Section 4.4.1 (coverage-guided scenario) and RQ2 (Section 4.1)]
    "First, in the coverage-guided (i.e., CC-guided) scenario, test queries are generated/selected to maximise incremental CC. At each step, queries that retrieve previously unexercised chunks are prioritised, with the explicit goal of systematically expanding coverage of the induced retrieval space."

    RQ2 evaluates strategies in terms of coverage growth and achieved CC under a fixed budget — precisely the quantity that the CC-guided strategy is defined to maximize. Comparing this strategy against random and overlap-biased selection therefore measures the optimizer against its own objective; the direction 'CC-guided reaches 50% of attainable coverage faster' is fixed by construction. Only the speedup magnitudes (1.7x/4.2x) are empirical properties of the query pools, not an independent validation that CC is an effective adequacy criterion.

  2. self definitional [Section 4.3 (fault definition), Section 4.4.1 (CC-guided selection), RQ3 (Section 5.3)]
    "If the Jaccard similarity between R_k(q1) and R_k(q2) exceeds a threshold τ (0.8 in our study), the corresponding failures are attributed to the same fault and counted once. By grouping failures in this way, each identified fault corresponds to a distinct region of the retrieval space that is systematically mishandled."

    A fault is operationally defined as a cluster of failing queries whose retrieved chunk sets overlap strongly (Jaccard > 0.8), so distinct faults correspond to low-overlap retrieval regions. The CC-guided strategy (Section 4.4.1) is explicitly designed to minimize retrieved-set overlap by prioritizing previously unexercised chunks. Thus, when a failing query appears in a new retrieval region, it is automatically counted as a new fault under the Section 4.3 definition. Random or overlap-biased orderings revisit overlapping regions and cluster more failures into the same fault, lowering APFD. The observed 10–25% APFD improvement is therefore largely a consequence of defining faults in terms of the same chunk-overlap signal that CC maximizes, rather than an independent demonstration that CC de

full rationale

Chunk Coverage itself is oracle-independent, and RQ1 provides genuinely independent evidence: on randomly sampled suites, CC correlates with entropy, Gini, and pairwise Jaccard diversity in the expected directions. The criterion is therefore not circular. However, the two headline validations are partially self-referential. RQ2's coverage-growth speedup is largely a restatement of the greedy selection rule: the strategy maximizes incremental CC and is then measured on CC growth, so the direction of the result is fixed by construction; only the magnitudes are empirical. More importantly, RQ3's APFD advantage is coupled to the fault model. Section 4.3 defines a fault as a cluster of failing queries with Jaccard similarity of retrieved chunk sets above τ=0.8, i.e., distinct faults are low-overlap retrieval regions. The CC-guided strategy in Section 4.4.1 is explicitly designed to minimize overlap by prioritizing previously unexercised chunks. Hence a failing query from a new retrieval region is automatically a new fault, and CC-guided ordering will tend to accumulate distinct faults earlier than random or overlap-biased ordering essentially by construction. The tau sensitivity analysis (0.6–1.0) preserves this structural coupling because all thresholds still partition failures by retrieved-chunk overlap. The APFD gain is thus not an independent validation that CC improves detection of externally defined faults; it largely quantifies how quickly the strategy reaches failure-prone, low-overlap retrieval regions. Failures themselves are defined by established RAG metrics, and the ranking degrades for Response Groundedness on MIMIC-IV (Random slightly outperforms CC), so the advantage is not fully forced. Nevertheless, the load-bearing fault-detection claim is substantially self-validating, warranting a score of 6 rather than a clean bill of health.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The paper's central claims rest on a small set of domain assumptions about retrieval-essential RAG systems and, more importantly, on an ad hoc fault model that aligns with the CC objective. The free parameters (tau=0.8, k=3, zero-metric failure threshold) are chosen by hand; sensitivity analysis is provided for tau but not for the failure threshold or k.

free parameters (3)
  • Jaccard fault clustering threshold tau = 0.8
    Chosen by authors to group failing queries into 'faults' in RQ3; sensitivity analysis across 0.6-1.0 shows ranking stability, but the definition still drives the APFD result.
  • Top-k retrieval parameter k = 3
    Retrieval configuration used for all datasets; CC is computed over top-3 chunk sets, and the coverage and fault numbers depend on this choice.
  • Zero-metric failure threshold = 0
    Failures are defined as the lowest attainable metric value (0) for RAG evaluation metrics; any other threshold would change failure and fault counts in RQ3.
axioms (5)
  • domain assumption Retrieval behaviour of a fixed RAG configuration is adequately characterized by the set of chunks retrieved across a test suite.
    This is the premise of CC; it abstracts away from relevance, ordering, and scores. If two queries retrieve the same chunks they are treated as equivalent.
  • domain assumption In the evaluated scenarios, correct system behaviour depends essentially on retrieved documents.
    Authors restrict to retrieval-essential domains and state CC is not informative in open-knowledge settings; this limits generalization of the adequacy claim.
  • ad hoc to paper Distinct retrieval faults can be inferred by clustering failing queries with Jaccard similarity > 0.8 of retrieved chunks, and failures can be recognized by metric value 0.
    This fault model is introduced in Section 4.3; it is the main source of circularity for RQ3 because CC optimizes the same diversity that defines a fault.
  • domain assumption An auxiliary LLM can synthesize queries that target a given uncovered chunk.
    Algorithm 1's GenerateQueries step and the MIMIC-IV pool construction assume this; the observed plateau in Figure 4 suggests it fails for some chunks.
  • domain assumption Existing RAG evaluation metrics (RAGAS-style and page-level relevance) capture observable failure symptoms without ground truth.
    Used to operationalize failures in RQ3; relies on LLM-as-a-judge and page-level relevance annotations from T2-RAGBench.

pith-pipeline@v1.3.0-alltime-deepseek · 18135 in / 14387 out tokens · 131111 ms · 2026-08-01T15:48:25.729006+00:00 · methodology

0 comments
read the original abstract

Retrieval-Augmented Generation (RAG)-based systems\footnote{For brevity, RAG-based systems are referred to as RAG systems throughout this paper.} are increasingly deployed in high-stakes settings where correct behaviour depends not only on the language model but also on the retrieval component that selects external documents at inference time. While existing RAG evaluation metrics assess retrieval and generation quality on a per-query basis, typically relying on query-level test oracles such as reference answers or relevance annotations, they provide limited insight into whether a test suite adequately exercises the retrieval behaviour of the system as a whole. In this paper, we introduce Chunk Coverage (CC), an oracle-independent test adequacy criterion for testing the retrieval component of RAG systems. CC measures the fraction of corpus chunks that are retrieved at least once across a test suite, providing a structural view of which parts of the retrieval space have been exercised. We further show how CC can be used to guide test selection and generation by prioritising queries that expand coverage of previously unexercised retrieval regions. We evaluate CC on clinical and financial RAG system scenarios. CC-guided testing reaches 50% of attainable coverage 1.7x faster than random selection and 4.2x faster than redundancy-biased strategies. Moreover, CC improves fault detection effectiveness (APFD) by 10% to 25% over random, indicating earlier discovery of distinct retrieval faults. These results show that CC captures retrieval diversity relevant to effective testing without requiring test oracles.

Figures

Figures reproduced from arXiv: 2607.18155 by Jinhan Kim, Paolo Tonella, Samuele Pasini.

Figure 1
Figure 1. Figure 1: Overview of RAG Procedure Output generation. Figure 1b depicts the online inference phase. Given an input query, the query text is embedded using the same embedding model and used to retrieve the top-𝑘 most similar chunks from the vector database according to a distance or similarity measure (e.g., cosine similarity). This retrieval step determines which parts of the corpus are selected for use during gene… view at source ↗
Figure 2
Figure 2. Figure 2: Test Generation with Chunk Coverage metrics that characterise how thoroughly a RAG system’s retrieval behaviour is exercised by a test set, independently of per-query retrieval quality or answer correctness. 3 Chunk Coverage for Testing RAG Systems This work focuses on testing the retrieval component of RAG systems. Concretely, we consider retrieval behaviour as determined by two core elements: (1) the emb… view at source ↗
Figure 3
Figure 3. Figure 3: Relationship between Chunk Coverage and diversity metrics for randomly sampled test suites ( [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Coverage growth under different test selection strategies. Coverage-guided selection consistently [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 1 canonical work pages

  1. [1]

    2017.Introduction to software testing

    Paul Ammann and Jeff Offutt. 2017.Introduction to software testing. Cambridge University Press

    2017

  2. [2]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.Advances in neural information processing systems33 (2020), 1877–1901

    2020

  3. [3]

    Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, et al . 2021. Finqa: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 3697–3711

    2021

  4. [4]

    Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. Ragas: Automated evaluation of retrieval augmented generation. InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations. 150–158

    2024

  5. [5]

    Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024. Bias and fairness in large language models: A survey.Computational Linguistics 50, 3 (2024), 1097–1179. Proc. ACM Softw. Eng., Vol. 3, No. ISSTA, Article ISSTA117. Publication date: October 2026. ISSTA117:20 Kim et al

    2024

  6. [6]

    Aoran Gan, Hao Yu, Kai Zhang, Qi Liu, Wenyu Yan, Zhenya Huang, Shiwei Tong, and Guoping Hu. 2025. Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey.arXiv preprint arXiv:2504.14891(2025)

    Pith/arXiv arXiv 2025

  7. [7]

    Paul Hager, Friederike Jungmann, and Daniel Rueckert. [n. d.]. MIMIC-IV-Ext Clinical Decision Making: A MIMIC-IV Derived Dataset for Evaluation of Large Language Models on the Task of Clinical Decision Making for Abdominal Pathologies. Version Number: 1.0 Type: dataset. doi:10.13026/2PFQ-5B68

    work page doi:10.13026/2pfq-5b68

  8. [8]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation.ACM computing surveys55, 12 (2023), 1–38

    2023

  9. [9]

    Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al. 2023. MIMIC-IV, a freely accessible electronic health record dataset.Scientific data10, 1 (2023), 1

    2023

  10. [10]

    Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. 2021. Dynabench: Rethinking benchmarking in NLP. InProceedings of the 2021 conference of the North American chapter of the Association for Computational Linguistics: human language technologies. 4110–4124

    2021

  11. [11]

    Jinhan Kim, Robert Feldt, and Shin Yoo. 2019. Guiding deep learning system testing using surprise adequacy. In2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 1039–1049

    2019

  12. [12]

    Jinhan Kim, Robert Feldt, and Shin Yoo. 2023. Evaluating surprise adequacy for deep learning system testing.ACM Transactions on Software Engineering and Methodology32, 2 (2023), 1–29

    2023

  13. [13]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks.Advances in neural information processing systems33 (2020), 9459–9474

    2020

  14. [14]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers). 3214–3252

    2022

  15. [15]

    Songyang Liu, Chaozhuo Li, Jiameng Qiu, Xi Zhang, Feiran Huang, Litian Zhang, Yiming Hei, and Philip S Yu. 2025. The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs.arXiv preprint arXiv:2506.11094(2025)

    Pith/arXiv arXiv 2025

  16. [16]

    Lei Ma, Felix Juefei-Xu, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Chunyang Chen, Ting Su, Li Li, Yang Liu, et al

  17. [17]

    Bo Ni, Zheyuan Liu, Leyao Wang, Yongjia Lei, Yuying Zhao, Xueqi Cheng, Qingkai Zeng, Luna Dong, Yinglong Xia, Krishnaram Kenthapadi, et al. 2025. Towards trustworthy retrieval augmented generation for large language models: A survey.arXiv preprint arXiv:2502.06872(2025)

    Pith/arXiv arXiv 2025

  18. [18]

    Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. InProceedings of the 58th annual meeting of the association for computational linguistics. 4885–4901

    2020

  19. [19]

    Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Mutation testing advances: an analysis and survey. InAdvances in computers. Vol. 112. Elsevier, 275–378

    2019

  20. [20]

    Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2017. Deepxplore: Automated whitebox testing of deep learning systems. Inproceedings of the 26th Symposium on Operating Systems Principles. 1–18

    2017

  21. [21]

    Sachin Raja, Ajoy Mondal, and CV Jawahar. 2023. Icdar 2023 competition on visual question answering on business document images. InInternational Conference on Document Analysis and Recognition. Springer, 454–470

    2023

  22. [22]

    Marco Tulio Ribeiro and Scott Lundberg. 2022. Adaptive testing and debugging of NLP models. InProceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers). 3253–3267

    2022

  23. [23]

    Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList.arXiv preprint arXiv:2005.04118(2020)

    Pith/arXiv arXiv 2020

  24. [24]

    Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2023. ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems. arXiv:2311.09476 [cs.CL]

    Pith/arXiv arXiv 2023

  25. [25]

    Jan Strich, Enes Kutay Isgorur, Maximilian Trescher, Chris Biemann, and Martin Semmann. 2025. T2-RAGBench: Text-and-Table Benchmark for Evaluating Retrieval-Augmented Generation.arXiv preprint arXiv:2506.12071(2025)

    arXiv 2025

  26. [26]

    Fengbin Zhu, Wenqiang Lei, Fuli Feng, Chao Wang, Haozhou Zhang, and Tat-Seng Chua. 2022. Towards complex document understanding by discrete reasoning. InProceedings of the 30th ACM International Conference on Multimedia. 4857–4866

    2022

  27. [27]

    Hong Zhu, Patrick AV Hall, and John HR May. 1997. Software unit test coverage and adequacy.Acm computing surveys (csur)29, 4 (1997), 366–427. Received 2026-01-30; accepted 2026-04-16 Proc. ACM Softw. Eng., Vol. 3, No. ISSTA, Article ISSTA117. Publication date: October 2026

    1997

  28. [2018]

    InProceedings of the 33rd ACM/IEEE international conference on automated software engineering

    Deepgauge: Multi-granularity testing criteria for deep learning systems. InProceedings of the 33rd ACM/IEEE international conference on automated software engineering. 120–131