Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

mmRAG: A Modular Benchmark for Retrieval-Augmented Generation over Text, Tables, and Knowledge Graphs

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read mmRAG is a modular benchmark that annotates chunk-level and dataset-level relevance across text, tables, and knowledge graphs, so retrieval accuracy and query routing can be measured directly instead of inferred from end-to-end generation…

desk verdict Useful multi-modal RAG benchmark with real gaps in routing-label validation and a metric inconsistency; deserves revision, not rejection. read the letter →

arxiv 2505.11180 v1 pith:5AOOMNMM submitted 2025-05-16 cs.IR

Chuan Xu , Qiaosheng Chen , Yutong Feng , Gong Cheng This is my paper · ORCID
classification cs.IR
keywords retrieval-augmentedgenerationRAGbenchmarkmulti-modalretrievalqueryroutingrelevanceannotationknowledgegraphtablequestionansweringinformationevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Retrieval-augmented generation (RAG) systems answer questions by first retrieving relevant material and then asking a language model to read it, but existing benchmarks usually judge only the final answer, so a wrong answer cannot be traced to a retrieval failure or a generation failure. This paper introduces mmRAG, a benchmark that separates the two steps. It converts six question-answering datasets—covering plain text, tables, and knowledge graphs—into a unified corpus of documents split into chunks, and annotates which chunks are relevant to each of 5,124 queries with three-level graded labels. It then derives per-dataset relevance scores from the chunk labels, so query routing—deciding which source to search—can be evaluated directly. The paper argues that this modular, multi-modal design fills a gap in existing RAG benchmarks, which are mostly single-modal and evaluate only end-to-end quality.

What carries the argument

The machinery that carries mmRAG is its two-tier relevance annotation. At the chunk level, a pooled-annotation protocol combines two complementary retrievers—BM25 (lexical) and BGE-large-en-v1.5 (semantic)—to gather up to 19 candidate chunks per query, ensuring global top-10 chunks, the top-3 chunks from the query's original relevant document, and the top-1 chunk from each of the five datasets; an ensemble of two large language models with a third as tiebreaker then labels every pooled chunk as irrelevant (0), partially relevant (1), or highly relevant (2). At the dataset level, the aggregated label is $S_{q,D} = \sum_{d \in D} \max_{c \in d} L_{q,c}$, which sums, over all documents in a dataset, the score of each document's most relevant chunk; this is the quantity used as ground truth for query routing. The unified corpus itself is also part of the machinery: KG entities are expanded into verbalized one-hop subgraphs, and every document is cut into non-overlapping 512-token chunks so that lexical and dense retrievers can be compared on equal footing.

What would settle it

Take a random sample of mmRAG queries, ask human judges whether each dataset actually contains the answer, and compare their judgments against the rankings produced by Eq. (1); if the formula systematically over-ranks datasets that contain many partially relevant documents but no truly answering document, the router evaluation in Section 5 rests on an artifact. A complementary test is to run a strong retriever not used in pooling (e.g., a cross-encoder) and measure how often it finds relevant chunks outside the pooled set; a substantial miss rate would violate the 'unpooled chunks are irrelevant' assumption underlying the retrieval labels.

Watch

Extended reading notes

Core claim

The central claim is that a modular, multi-modal benchmark can be built by converting heterogeneous data into a uniform document collection and annotating relevance at two levels. mmRAG takes six QA datasets (NQ and TriviaQA for text, OTT and TAT for tables plus text, CWQ and WebQSP for knowledge graphs), converts the KGs into verbalized one-hop subgraphs, segments every document into 512-token chunks, and keeps only queries with at least one relevant chunk, yielding 5,124 queries. A pooled annotation procedure—BM25 and BGE-large-en-v1.5 retrieve up to 19 candidate chunks per query, and an LLM ensemble assigns each chunk a label $L_{q,c} \in \{0,1,2\}$ for irrelevant, partially relevant, or highly relevant—produces 88,751 annotated query-chunk pairs. Dataset-level relevance is then defined as $S_{q,D} = \sum_{d \in D} \max_{c \in d} L_{q,c}$, which the paper uses as the ground truth for query routing. The paper evaluates seven retrievers and two routers on this benchmark; the best retriever, BGE, reaches NDCG@1 of 0.617, the LLM router beats the semantic router at small top-k, and direct accuracy metrics track but do not perfectly coincide with generation quality, which the authors interpret as evidence that direct component evaluation is valuable and complementary.

Load-bearing premise

The load-bearing premise is that summing, over each document in a dataset, the relevance score of its single most relevant chunk tells you how useful that dataset is for answering the query, but this aggregation is never checked against human judgments of dataset usefulness.

Editorial extensions

If this is right

  • With mmRAG's chunk-level labels, retrieval accuracy can be benchmarked directly with standard ranking metrics such as NDCG, MAP, and Hits, without running a generator, which is cheaper and more reproducible than end-to-end evaluation.
  • Query routing can be measured directly using the derived dataset-level relevance labels, enabling systematic comparison of routers (the paper finds an LLM router beats a semantic router at small top-k).
  • The benchmark's cross-dataset relevance statistics (e.g., up to about 68% of WebQSP queries have relevant chunks outside their original dataset) show that routing is practically necessary, not just a luxury.
  • Retrieving only from the query's original dataset often outperforms retrieving from the full multi-modal corpus (e.g., BM25 drops from 0.5210 to 0.4162 on CWQ with GLM), demonstrating that multi-modal retrieval and routing remain open challenges.
  • Direct routing accuracy trends align with generation quality trends, so mmRAG's routing labels provide a cheaper proxy for system-level performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The validity of the routing labels hinges on Eq. (1); a natural extension is to collect human dataset-level usefulness judgments for a sample of queries and test whether $S_{q,D}$ agrees with them, or to train a router directly on the provided labels and compare it against the two baseline routers.
  • The pooling-based annotation could be stress-tested by running a third, stronger retriever and checking whether it surfaces relevant chunks outside the pooled set; the fraction of such misses would quantify how much the 'everything else is irrelevant' assumption may bias retrieval evaluation.
  • The benchmark's structure enables a failure-attribution protocol: compare generation with the Oracle retriever against generation with each real retriever to separate retrieval errors from generation errors, which the paper does not explicitly report per system.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces mmRAG, a modular benchmark for retrieval-augmented generation that integrates six question-answering datasets spanning text, tables, and knowledge graphs. The datasets are converted into a unified corpus of 90,998 documents and 3.2 million chunks, and relevance labels are produced through pooling with BM25 and BGE followed by an LLM-based annotation ensemble. From the chunk-level labels the authors derive dataset-level relevance scores using Eq. (1) and use these to evaluate query routing. The central claim is that mmRAG enables direct, component-level evaluation of retrieval and routing, rather than relying only on opaque end-to-end generation quality. The paper reports baseline retrieval and routing results for several retrievers and routers.

Significance. If the relevance labels are reliable, mmRAG fills a genuine gap: most existing RAG benchmarks under-annotate retrieval and provide no query-routing labels. The resource is substantial in scale, includes cross-dataset relevance annotation that most benchmarks lack, and is publicly released with a DOI and code, which supports reproducibility and community reuse. The authors also provide a useful baseline comparison of retrievers and routers. However, the significance of the contribution depends on the validity of the annotation protocol and of the dataset-level aggregation, both of which have issues that currently undermine the reported retrieval and routing numbers.

major comments (5)
  1. [Section 4.1, Table 8] Hits@k is defined as the proportion of queries that have at least one relevant chunk in the top-k, which is bounded above by 1. However, Table 8 reports Hits@3 values such as 1.726 for BM25, 1.971 for BGE, and 1.481 for fine-tuned GTE, and Hits@5 values up to 3.107. These numbers are inconsistent with the stated definition and suggest that a different quantity, such as the average number of relevant chunks in the top-k, was computed. This invalidates the retrieval baseline results as currently reported and must be corrected.
  2. [Section 3.4, Eq. (1)] The dataset-level relevance label S_q,D is an unnormalized sum over documents of the maximum chunk-level label in each document. This makes S_q,D depend on dataset size and on the number of partially relevant documents, so a dataset with many marginal matches can outrank a dataset that contains one document with the exact answer. The paper does not validate Eq. (1) against human routing judgments or any independent measure of dataset usefulness, yet Section 5 uses these labels as ground truth for NDCG@k, MAP@k, and Hits@k routing metrics. Because the central contribution is direct measurement of routing accuracy, the unvalidated aggregation formula means the router comparisons in Section 5 may be measuring an artifact of Eq. (1).
  3. [Section 3.4, pooling design] For datasets other than the query's original dataset, only the top-1 chunk per retriever enters the pooling set. Consequently, a cross-dataset relevant chunk that is not ranked first by either BM25 or BGE is never annotated and is implicitly treated as irrelevant. This biases the cross-dataset relevance statistics in Table 6 and the derived S_q,D values in Table 7, which are then used as routing ground truth in Section 5. The paper should either broaden the per-dataset pooling depth or explicitly analyze how much cross-dataset relevance may have been missed.
  4. [Section 3.4, annotation quality] All relevance labels are produced by an LLM ensemble (DeepSeek-V3 and GLM-4-Plus, with GPT-4o as tiebreaker) with no human verification. The reported 85% pairwise agreement between the two primary LLM annotators demonstrates consistency but not accuracy; systematic LLM errors would propagate into every retrieval and routing evaluation built on the benchmark. A human-annotated sample of query-chunk pairs, with inter-annotator agreement against the LLM labels, is needed to substantiate the claim of high-quality annotations.
  5. [Section 4.1 vs. Section 3.4, pool-contribution bias] BGE-large-en-v1.5 is one of the two retrievers used to build the annotation pool and is also one of the evaluated retrievers. Because BGE's top-ranked chunks are preferentially included in the pool, relevant-chunk labels are more likely to be found for BGE than for retrievers that did not contribute to the pool, potentially inflating BGE's retrieval scores relative to the others. The paper does not acknowledge or correct for this pool-contribution bias, which is a standard concern in pooled evaluation.
minor comments (5)
  1. [Abstract and Section 3.4] The abstract states there are 88,751 annotated query-chunk pairs, while Section 3.4 reports 90,846 pairs in the agreement analysis. The difference is not explained and should be reconciled.
  2. [Section 3.4, pooling] The text says the pool consists of 'up to 19 top-ranked chunks' from each retriever, but the described components (globally top-10, top-3 from the original relevant document, top-1 from each of five datasets) sum to at most 18 before deduplication. The arithmetic should be clarified.
  3. [Section 5.1, semantic router] The semantic router uses BGE to encode queries and dataset descriptions, but the dataset descriptions are only referenced as 'collected from the dataset homepages' and are not provided in the paper or appendix, which hampers reproducibility of the routing baselines.
  4. [Section 3.3, query selection] The LLM-based filtering of queries is described only as 'inspired by [34]'; the prompt or filtering criterion is not given, so the representativeness step is not reproducible as described.
  5. [Section 3.3, wording] The phrase 'that—latter in data annotation—are not associated' appears to contain a typo and should read 'later in data annotation.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: mmRAG is a self-contained benchmark whose labels are produced by explicit annotation and aggregation, not derived from the claims they support.

full rationale

The paper's central deliverable is a benchmark resource, not a derivation of a prediction from first principles. Chunk-level relevance labels are produced by a stated LLM-based annotation protocol over a pooled candidate set (Section 3.4), following standard IR pooling; these labels are external to the paper's own evaluation claims. Dataset-level relevance labels are explicitly defined by aggregation Eq. (1), S_q,D = sum_{d in D} max_{c in d} L_q,c, as a stated design choice to support query-routing evaluation. This is a definition of a benchmark label, not a fitted parameter masquerading as a prediction, and no experimental result is then used to justify the definition itself. The only self-citation is reference [29], the Hugging Face release of the benchmark itself, used for availability; this is not load-bearing for any derivation. The paper does not invoke a uniqueness theorem, import an ansatz from prior work, or rename a known result. The skeptically flagged concern about Eq. (1) — that it is an unvalidated aggregation that might misrank datasets for routing — is a validity and correctness risk, not a circularity: S_q,D does not reduce to the routing accuracy it is used to measure. Accordingly, no circular step is identified and the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters or invented entities are introduced. The benchmark rests on standard IR pooling, LLM-based annotation, and one constructed aggregation formula (Eq. 1).

assumptions (4)
  • domain assumption LLM relevance judgments (DeepSeek-V3, GLM-4-Plus, GPT-4o) are treated as ground truth for chunk-level relevance without human verification.
    Section 3.4 defines the annotation protocol and reports agreement between LLMs only; the central evaluation metrics inherit this assumption.
  • domain assumption Unpooled chunks are assumed irrelevant, per standard IR pooling practice.
    Section 3.4 states 'Unpooled chunks are assumed to be irrelevant to the query'; this is conventional but biases evaluation toward retrievers similar to the pooling retrievers.
  • domain assumption One-hop verbalized subgraphs around entities preserve enough information for the benchmark queries.
    Section 3.3 constructs KG documents from one-hop edges ordered by gold SPARQL results, rdf:type, literals, and random edges; there is no check that answers remain recoverable from the verbalized text.
  • ad hoc to paper Eq. (1) defines a valid dataset-level relevance measure.
    The sum-of-max-chunk-labels formula is introduced solely for this benchmark's routing labels, without comparison to human routing judgments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of mmRAG: A Modular Benchmark for Retrieval-Augmented Generation over Text, Tables, and Knowledge Graphs." pith.science (2026). https://pith.science/paper/5AOOMNMM

@misc{pith2026250511180,
  author       = {Pith},
  title        = {Pith review of: mmRAG: A Modular Benchmark for Retrieval-Augmented Generation over Text, Tables, and Knowledge Graphs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5AOOMNMM}},
  note         = {Machine review of arXiv:2505.11180}
}
read the original abstract

Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm for enhancing the capabilities of large language models. However, existing RAG evaluation predominantly focuses on text retrieval and relies on opaque, end-to-end assessments of generated outputs. To address these limitations, we introduce mmRAG, a modular benchmark designed for evaluating multi-modal RAG systems. Our benchmark integrates queries from six diverse question-answering datasets spanning text, tables, and knowledge graphs, which we uniformly convert into retrievable documents. To enable direct, granular evaluation of individual RAG components -- such as the accuracy of retrieval and query routing -- beyond end-to-end generation quality, we follow standard information retrieval procedures to annotate document relevance and derive dataset relevance. We establish baseline performance by evaluating a wide range of RAG implementations on mmRAG.

Figures

Figures reproduced from arXiv: 2505.11180 by the authors.

Figure 1
Figure 1. Construction of mmRAG. ing (Section 3.2). Data Processing employs three core techniques. First, we select representative queries from these QA datasets using clustering techniques to capture various information needs and reduce redundancy. Next, we build a collection of textual documents converted from different data formats, including documents relevant to the queries and random documents as noise to simulate real-… view at source ↗
Figure 2
Figure 2. Evaluation of query routers (routing accuracy). [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. Evaluation of query routers (generation quality). [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. WorkSurface-Bench: Benchmarking Enterprise Agents on Multi-Surface Knowledge Routing

    cs.CL 2026-07 conditional novelty 6.0 of 10

    WorkSurface-Bench measures surface routing separately from answer correctness and finds near-perfect routing still leaves 25–44% answer errors across four LLM backbones.

Reference graph

Works this paper leans on

38 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [1]

    In: Wooldridge, M.J., Dy, J.G., Natarajan, S

    Chen, J., Lin, H., Han, X., Sun, L.: Benchmarking large language models in retrieval-augmented generation. In: Wooldridge, M.J., Dy, J.G., Natarajan, S. (eds.) Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Ad...

    2024

  2. [2]

    CoRRabs/2010.10439(2020), https://arxiv.org/ abs/2010.10439

    Chen, W., Chang, M., Schlinger, E., Wang, W.Y., Cohen, W.W.: Open question answering over tables and text. CoRRabs/2010.10439(2020), https://arxiv.org/ abs/2010.10439

    arXiv 2020

  3. [3]

    In: Find- ings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020

    Chen, W., Zha, H., Chen, Z., Xiong, W., Wang, H., Wang, W.Y.: Hybridqa: A dataset of multi-hop question answering over tabular and textual data. In: Find- ings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020. Findings of ACL, vol. EMNLP 2020, pp. 1026–1036. Asso- ciation for Computational Linguistics (2020)....

    doi:10.18653/v1/2020 2020

  4. [4]

    In: Companion Proceedings of the ACM on Web Con- ference 2024, WWW 2024, Singapore, Singapore, May 13-17, 2024

    Christmann, P., Roy, R.S., Weikum, G.: Compmix: A benchmark for heteroge- neous question answering. In: Companion Proceedings of the ACM on Web Con- ference 2024, WWW 2024, Singapore, Singapore, May 13-17, 2024. pp. 1091–1094. ACM (2024). https://doi.org/10.1145/3589335.3651444, https://doi.org/10.1145/ 3589335.3651444

    arXiv 2024

  5. [5]

    DeepSeek-AI: Deepseek-v3 technical report (2024), https://arxiv.org/abs/2412. 19437

    work page 2024

  6. [6]

    In: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2024, Barcelona, Spain, August 25-29, 2024

    Fan, W., Ding, Y., Ning, L., Wang, S., Li, H., Yin, D., Chua, T., Li, Q.: A survey on RAG meeting llms: Towards retrieval-augmented large language models. In: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2024, Barcelona, Spain, August 25-29, 2024. pp. 6491–6501. ACM (2024). https://doi.org/10.1145/3637528.36714...

    arXiv 2024

  7. [7]

    CoRRabs/2407.11005(2024)

    Friel, R., Belyi, M., Sanyal, A.: Ragbench: Explainable benchmark for retrieval- augmented generation systems. CoRRabs/2407.11005(2024). https://doi.org/ 10.48550/ARXIV.2407.11005, https://doi.org/10.48550/arXiv.2407.11005

    arXiv doi:10.48550/arxiv.2407.11005 2024

  8. [8]

    GLM, T., Zeng, A., Xu, B., Wang, B., Zhang, C., Yin, D., Rojas, D., Feng, G., Zhao, H., Lai, H., Yu, H., Wang, H., Sun, J., Zhang, J., Cheng, J., Gui, J., Tang, J., Zhang, J., Li, J., Zhao, L., Wu, L., Zhong, L., Liu, M., Huang, M., Zhang, P., Zheng, Q., Lu, R., Duan, S., Zhang, S., Cao, S., Yang, S., Tam, W.L., Zhao, W., Liu, X., Xia, X., Zhang, X., Gu, ...

    2024

Show all 38 references
  1. [9]

    CoRR abs/2410.12837(2024)

    Gupta, S., Ranjan, R., Singh, S.N.: A comprehensive survey of retrieval-augmented generation (RAG): evolution, current landscape and future directions. CoRR abs/2410.12837(2024). https://doi.org/10.48550/ARXIV.2410.12837, https:// doi.org/10.48550/arXiv.2410.12837

    2024 doi arXiv

  2. [10]

    He, X., Tian, Y., Sun, Y., Chawla, N.V., Laurent, T., LeCun, Y., Bres- son, X., Hooi, B.: G-retriever: Retrieval-augmented generation for textual mmRAG: A Modular Benchmark for Retrieval-Augmented Generation 17 graph understanding and question answering. In: Advances in Neural...

    2024

  3. [11]

    https://doi.org/10.48550/ARXIV.2112.09118, https://arxiv.org/abs/2112.09118

    Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A., Grave, E.: Unsupervised dense information retrieval with contrastive learning (2021). https://doi.org/10.48550/ARXIV.2112.09118, https://arxiv.org/abs/2112.09118

    2021 doi arXiv

  4. [12]

    CoRRabs/1705.03551 (2017), http://arxiv.org/abs/1705.03551

    Joshi, M., Choi, E., Weld, D.S., Zettlemoyer, L.: Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. CoRRabs/1705.03551 (2017), http://arxiv.org/abs/1705.03551

    2017 arXiv

  5. [13]

    Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., Yih, W.: Dense passage retrieval for open-domain question answering. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020,Online,November16-20,2020.pp.6...

    2020 doi

  6. [14]

    Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A.P., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M., Dai, A.M., Uszkoreit, J., Le, Q., Petrov, S.: Natural questions: a benchmark for question ans...

    2019 doi

  7. [15]

    arXiv preprint arXiv:2308.03281 (2023)

    Li, Z., Zhang, X., Zhang, Y., Long, D., Xie, P., Zhang, M.: Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281 (2023)

    2023 arXiv

  8. [16]

    In: Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024

    Luo, H., E, H., Tang, Z., Peng, S., Guo, Y., Zhang, W., Ma, C., Dong, G., Song, M., Lin, W., Zhu, Y., Luu, A.T.: Chatkbqa: A generate-then-retrieve frame- work for knowledge base question answering with fine-tuned large language mod- els. In: Findings of the Association for Co...

    2024 doi

  9. [17]

    In: IEEE Conference on Com- puter Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019

    Marino, K., Rastegari, M., Farhadi, A., Mottaghi, R.: OK-VQA: A visual question answering benchmark requiring external knowledge. In: IEEE Conference on Com- puter Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. pp. 3195–3204. Computer Vision ...

    2019

  10. [18]

    Petroni, F., Piktus, A., Fan, A., Lewis, P., Yazdani, M., Cao, N.D., Thorne, J., Jernite, Y., Karpukhin, V., Maillard, J., Plachouras, V., Rocktäschel, T., Riedel, S.: KILT: a benchmark for knowledge intensive language tasks. In: Pro- ceedings of the 2021 Conference of the Nor...

    2021

  11. [19]

    In: Al- 18 C

    Rau, D., Déjean, H., Chirkova, N., Formal, T., Wang, S., Clinchant, S., Nikoulina, V.: BERGEN: A benchmarking library for retrieval-augmented generation. In: Al- 18 C. Xu et al. Onaizan, Y., Bansal, M., Chen, Y. (eds.) Findings of the Association for Com- putational Linguistic...

    2024

  12. [20]

    In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024

    Samarinas, C., Zamani, H.: Procis: A benchmark for proactive retrieval in con- versations. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024. pp. 830–840. ACM (2...

    2024

  13. [21]

    Shah, S., Mishra, A., Yadati, N., Talukdar, P.P.: KVQA: knowledge-aware visual question answering. In: The Thirty-Third AAAI Conference on Artificial Intelli- gence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelli- gence Conference, IAAI 2019, The Nin...

    2019

  14. [22]

    In: Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Process- ing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024

    Sun, W., Shi, Z., Long, W., Yan, L., Ma, X., Liu, Y., Cao, M., Yin, D., Ren, Z.: MAIR: A massive benchmark for evaluating instructed retrieval. In: Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Process- ing, EMNLP 2024, Miami, FL, USA, November ...

    2024

  15. [23]

    Talmor, A., Berant, J.: The web as a knowledge-base for answering complex ques- tions. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, NAACL-HLT 2018, New Orleans, Louisiana, ...

    2018 doi

  16. [24]

    In: 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021

    Talmor, A., Yoran, O., Catav, A., Lahav, D., Wang, Y., Asai, A., Ilharco, G., Hajishirzi, H., Berant, J.: Multimodalqa: complex question answering over text, tables and images. In: 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May...

    2021

  17. [25]

    Team, Q.: Qwen2.5: A party of foundation models (September 2024), https:// qwenlm.github.io/blog/qwen2.5/

    2024

  18. [26]

    CoRRabs/2406.17158(2024)

    V, V., Prabhu, D., Anand, A.: DEXTER: A benchmark for open-domain complex question answering using llms. CoRRabs/2406.17158(2024). https://doi.org/ 10.48550/ARXIV.2406.17158, https://doi.org/10.48550/arXiv.2406.17158

    2024 doi arXiv

  19. [27]

    IEEE Trans

    Wang, P., Wu, Q., Shen, C., Dick, A.R., van den Hengel, A.: FVQA: fact-based visual question answering. IEEE Trans. Pattern Anal. Mach. Intell.40(10), 2413– 2427 (2018). https://doi.org/10.1109/TPAMI.2017.2754246, https://doi.org/10. 1109/TPAMI.2017.2754246

    2018

  20. [28]

    Xiao, S., Liu, Z., Zhang, P., Muennighoff, N.: C-pack: Packaged resources to ad- vance general chinese embedding (2023)

    2023

  21. [29]

    https://doi.org/10.57967/hf/5475, https://huggingface.co/datasets/ Askio/mmrag_benchmark

    Xu, C., Chen, Q., Feng, Y., Cheng, G.: mmrag_benchmark (revision 72f010b) (2025). https://doi.org/10.57967/hf/5475, https://huggingface.co/datasets/ Askio/mmrag_benchmark

    2025 doi

  22. [30]

    arXiv preprint arXiv:2407.10671 (2024)

    Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, J., Ma, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., mmRAG: A Modular Benchmark...

    2024 arXiv

  23. [31]

    In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018

    Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W.W., Salakhutdinov, R., Manning, C.D.: Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, Oct...

    2018 doi

  24. [32]

    Yeo, W., Kim, K., Jeong, S., Baek, J., Hwang, S.J.: Universalrag: Retrieval- augmented generation over multiple corpora with diverse modalities and granu- larities (2025), https://arxiv.org/abs/2504.20734

    2025 arXiv

  25. [33]

    In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 2: Short Papers

    Yih, W., Richardson, M., Meek, C., Chang, M., Suh, J.: The value of seman- tic parse labeling for knowledge base question answering. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volum...

    2016 doi

  26. [34]

    CoRRabs/2410.10594(2024)

    Yu, S., Tang, C., Xu, B., Cui, J., Ran, J., Yan, Y., Liu, Z., Wang, S., Han, X., Liu, Z., Sun, M.: Visrag: Vision-based retrieval-augmented generation on multi- modality documents. CoRRabs/2410.10594(2024). https://doi.org/10.48550/ ARXIV.2410.10594, https://doi.org/10.48550/a...

    2024 doi arXiv

  27. [35]

    AI Open4, 1–12 (2023)

    Zhang, L., Zhang, J., Ke, X., Li, H., Huang, X., Shao, Z., Cao, S., Lv, X.: A survey on complex factual question answering. AI Open4, 1–12 (2023). https://doi.org/ 10.1016/J.AIOPEN.2022.12.003, https://doi.org/10.1016/j.aiopen.2022.12.003

    2023 doi

  28. [36]

    arXiv preprint arXiv:2407.19669 (2024)

    Zhang, X., Zhang, Y., Long, D., Xie, W., Dai, Z., Tang, J., Lin, H., Yang, B., Xie, P., Huang, F., et al.: mgte: Generalized long-context text representation and reranking models for multilingual text retrieval. arXiv preprint arXiv:2407.19669 (2024)

    2024 arXiv

  29. [37]

    Zhu, F., Lei, W., Huang, Y., Wang, C., Zhang, S., Lv, J., Feng, F., Chua, T.: TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance. In: Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11t...

    2021

  30. [2024]

    7640–7663

    pp. 7640–7663. Association for Computational Linguistics (2024), https: //aclanthology.org/2024.findings-emnlp.449

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.