{"id":"04745f44-3450-4281-b506-cfe7f12d7f3a","arxiv_id":"2505.11180","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new multi-modal RAG benchmark, mmRAG, adds chunk-level retrieval labels and dataset-level routing labels across six question-answering datasets.","lead":"This paper introduces mmRAG, a benchmark for evaluating retrieval-augmented generation systems across text, tables, and knowledge graphs. It provides relevance labels that let researchers measure retrieval and query routing separately from final answer quality.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (1) dataset-level relevance labels are unvalidated and can misrank datasets for routing, threatening the benchmark's central query-routing contribution.","rationale":"The central claim is that mmRAG enables direct, modular evaluation of retrieval and query routing. Retrieval accuracy depends on chunk-level labels, which have at least 85% LLM agreement and standard pooling, though human verification would strengthen them. However, the novel routing labels depend on an additional, arbitrary aggregation step (Eq. 1) with no empirical anchor. The paper's only check is a qualitative 'similar trends' comparison that is confounded by retriever behavior within selected datasets. A dataset with many partial hits can dominate a dataset with a single exact hit, which would misrank datasets for routing. This directly threatens the paper's unique selling point of supporting query-routing evaluation. The Hits@k table inconsistency (values >1) is real but is a reporting issue in the baseline experiments, not in the label validity; it lowers confidence in the evaluation code but does not change the fundamental concern. The LLM-only annotation is a secondary issue because agreement is reported and LLM annotation is now common, though human spot-checks would help. Thus the most load-bearing concern is the unvalidated Eq. (1), and the verdict should remain CONDITIONAL pending a human-validation study.","tokens_in":15409,"tokens_out":8720,"duration_ms":83882,"concrete_test":"Sample 100 queries from the mmRAG test split. For each query, have two independent human annotators rate each of the five datasets on a 0-2 usefulness scale (0=no useful content, 1=partial, 2=contains the answer) for answering the query. Compute Spearman rank correlation between mean human usefulness and S_q,D from Eq. (1), and Cohen's weighted kappa on binarized labels (relevant if S_q,D>=1 vs human>=1). If the correlation or kappa is low (e.g., <0.6), Eq. (1) does not faithfully represent dataset-level usefulness and the routing accuracy numbers in Section 5 need to be re-derived with validated labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The routing labels that support the paper's unique query-routing evaluation are derived by Eq. (1): S_q,D = sum_{d in D} max_{c in d} L_q,c. This makes S_q,D essentially a weighted count of documents in D that contain at least one relevant chunk. A dataset with many partially relevant documents will score higher than a dataset with one document containing the exact answer, even though the latter is more useful for answering the query. The paper never validates Eq. (1) against human routing judgments or any independent measure of dataset usefulness. Section 5.2's claim that routing accuracy and generation quality trends 'appear similar' is only qualitative and is confounded: generation quality after routing to top-k datasets also depends on the retriever's ability to find relevant chunks within those datasets. Because the central contribution is 'direct measurement' of routing accuracy, an unvalidated aggregation formula means the router comparisons in Section 5 may be measuring an artifact. The pooling design exacerbates this: for non-original datasets only the top-1 chunk per retriever enters the pool, so cross-dataset relevance can be missed entirely, further biasing S_q,D.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces mmRAG, a modular benchmark for retrieval-augmented generation that integrates six question-answering datasets spanning text, tables, and knowledge graphs. The datasets are converted into a unified corpus of 90,998 documents and 3.2 million chunks, and relevance labels are produced through pooling with BM25 and BGE followed by an LLM-based annotation ensemble. From the chunk-level labels the authors derive dataset-level relevance scores using Eq. (1) and use these to evaluate query routing. The central claim is that mmRAG enables direct, component-level evaluation of retrieval and routing, rather than relying only on opaque end-to-end generation quality. The paper reports baseline retrieval and routing results for several retrievers and routers.","tokens_in":15591,"tokens_out":5543,"duration_ms":54116,"significance":"If the relevance labels are reliable, mmRAG fills a genuine gap: most existing RAG benchmarks under-annotate retrieval and provide no query-routing labels. The resource is substantial in scale, includes cross-dataset relevance annotation that most benchmarks lack, and is publicly released with a DOI and code, which supports reproducibility and community reuse. The authors also provide a useful baseline comparison of retrievers and routers. However, the significance of the contribution depends on the validity of the annotation protocol and of the dataset-level aggregation, both of which have issues that currently undermine the reported retrieval and routing numbers.","major_comments":[{"comment":"Hits@k is defined as the proportion of queries that have at least one relevant chunk in the top-k, which is bounded above by 1. However, Table 8 reports Hits@3 values such as 1.726 for BM25, 1.971 for BGE, and 1.481 for fine-tuned GTE, and Hits@5 values up to 3.107. These numbers are inconsistent with the stated definition and suggest that a different quantity, such as the average number of relevant chunks in the top-k, was computed. This invalidates the retrieval baseline results as currently reported and must be corrected.","section":"Section 4.1, Table 8"},{"comment":"The dataset-level relevance label S_q,D is an unnormalized sum over documents of the maximum chunk-level label in each document. This makes S_q,D depend on dataset size and on the number of partially relevant documents, so a dataset with many marginal matches can outrank a dataset that contains one document with the exact answer. The paper does not validate Eq. (1) against human routing judgments or any independent measure of dataset usefulness, yet Section 5 uses these labels as ground truth for NDCG@k, MAP@k, and Hits@k routing metrics. Because the central contribution is direct measurement of routing accuracy, the unvalidated aggregation formula means the router comparisons in Section 5 may be measuring an artifact of Eq. (1).","section":"Section 3.4, Eq. (1)"},{"comment":"For datasets other than the query's original dataset, only the top-1 chunk per retriever enters the pooling set. Consequently, a cross-dataset relevant chunk that is not ranked first by either BM25 or BGE is never annotated and is implicitly treated as irrelevant. This biases the cross-dataset relevance statistics in Table 6 and the derived S_q,D values in Table 7, which are then used as routing ground truth in Section 5. The paper should either broaden the per-dataset pooling depth or explicitly analyze how much cross-dataset relevance may have been missed.","section":"Section 3.4, pooling design"},{"comment":"All relevance labels are produced by an LLM ensemble (DeepSeek-V3 and GLM-4-Plus, with GPT-4o as tiebreaker) with no human verification. The reported 85% pairwise agreement between the two primary LLM annotators demonstrates consistency but not accuracy; systematic LLM errors would propagate into every retrieval and routing evaluation built on the benchmark. A human-annotated sample of query-chunk pairs, with inter-annotator agreement against the LLM labels, is needed to substantiate the claim of high-quality annotations.","section":"Section 3.4, annotation quality"},{"comment":"BGE-large-en-v1.5 is one of the two retrievers used to build the annotation pool and is also one of the evaluated retrievers. Because BGE's top-ranked chunks are preferentially included in the pool, relevant-chunk labels are more likely to be found for BGE than for retrievers that did not contribute to the pool, potentially inflating BGE's retrieval scores relative to the others. The paper does not acknowledge or correct for this pool-contribution bias, which is a standard concern in pooled evaluation.","section":"Section 4.1 vs. Section 3.4, pool-contribution bias"}],"minor_comments":[{"comment":"The abstract states there are 88,751 annotated query-chunk pairs, while Section 3.4 reports 90,846 pairs in the agreement analysis. The difference is not explained and should be reconciled.","section":"Abstract and Section 3.4"},{"comment":"The text says the pool consists of 'up to 19 top-ranked chunks' from each retriever, but the described components (globally top-10, top-3 from the original relevant document, top-1 from each of five datasets) sum to at most 18 before deduplication. The arithmetic should be clarified.","section":"Section 3.4, pooling"},{"comment":"The semantic router uses BGE to encode queries and dataset descriptions, but the dataset descriptions are only referenced as 'collected from the dataset homepages' and are not provided in the paper or appendix, which hampers reproducibility of the routing baselines.","section":"Section 5.1, semantic router"},{"comment":"The LLM-based filtering of queries is described only as 'inspired by [34]'; the prompt or filtering criterion is not given, so the representativeness step is not reproducible as described.","section":"Section 3.3, query selection"},{"comment":"The phrase 'that—latter in data annotation—are not associated' appears to contain a typo and should read 'later in data annotation.'","section":"Section 3.3, wording"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nmmRAG is a serious piece of benchmark engineering. It gives the RAG community something it actually lacks: a single corpus spanning text, tables, and knowledge graphs with chunk-level relevance labels annotated across dataset boundaries, plus derived dataset-level labels for query routing. The construction is careful—pooling with BM25 and BGE, LLM ensemble with tiebreaker, stratified split, public release with DOI and code. If the labels hold up, this is a genuinely reusable diagnostic instrument.\n\nThat said, I wouldn't plug it into a project yet. Three things need attention.\n\nFirst, Table 8 reports Hits@3 of 1.726 and Hits@5 of 2.725 for BM25, with similar values for other retrievers. The paper defines Hits@k as the proportion of queries with at least one relevant chunk in the top-k. That number cannot exceed 1. Either the metric is computed differently (e.g., average count of relevant chunks, which is a different thing and should be called out) or there's a bug. Since the table is the main retrieval result, this needs a fix.\n\nSecond, the dataset-level relevance labels from Eq. (1) are the basis for the routing evaluation, and that equation is doing something odd. It sums over documents the maximum chunk label per document, so it's essentially a weighted count of relevant documents. A dataset with ten partially relevant documents scores higher than one with a single document containing the exact answer. The paper calls this 'the contribution of each dataset to answering a query,' but never validates that this aggregation matches human routing judgments or end-to-end generation quality. The Section 5.2 claim that routing accuracy and generation 'appear similar' is qualitative and confounded by the retriever. On top of that, the pooling design keeps only the top-1 chunk from non-original datasets per retriever, so genuinely relevant chunks that neither BM25 nor BGE surfaces can be missed, biasing those derived labels. This is the weakest link, because it's the unique contribution.\n\nThird, the LLM-only annotation pipeline has no human verification. Eighty-five percent agreement between two LLMs is not the same as accuracy. It's a common compromise, but for a benchmark meant to be a ground-truth source, I'd want at least a small human-verified subset reported.\n\nNone of these are fatal to the resource. The retrieval labels for text and tables are probably fine; the routing labels need validation or at least a transparent discussion of the aggregation's limitations. The paper deserves peer review—it's a well-scoped, reproducible contribution—but it should come back with the Hits@k issue fixed and the routing labels either validated or softened.","headline":"Useful multi-modal RAG benchmark with real gaps in routing-label validation and a metric inconsistency; deserves revision, not rejection.","tokens_in":16095,"tokens_out":3846,"would_cite":false,"duration_ms":34645,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"mmRAG is a modular benchmark that annotates chunk-level and dataset-level relevance across text, tables, and knowledge graphs, so retrieval accuracy and query routing can be measured directly instead of inferred from end-to-end generation…","keywords":["retrieval-augmented generation","RAG benchmark","multi-modal retrieval","query routing","relevance annotation","knowledge graph","table question answering","information retrieval evaluation"],"falsifier":"Take a random sample of mmRAG queries, ask human judges whether each dataset actually contains the answer, and compare their judgments against the rankings produced by Eq. (1); if the formula systematically over-ranks datasets that contain many partially relevant documents but no truly answering document, the router evaluation in Section 5 rests on an artifact. A complementary test is to run a strong retriever not used in pooling (e.g., a cross-encoder) and measure how often it finds relevant chunks outside the pooled set; a substantial miss rate would violate the 'unpooled chunks are irrelevant' assumption underlying the retrieval labels.","tokens_in":15198,"feed_emoji":"🔍","tokens_out":13067,"duration_ms":109264,"temperature":0.7,"pith_summary":"Retrieval-augmented generation (RAG) systems answer questions by first retrieving relevant material and then asking a language model to read it, but existing benchmarks usually judge only the final answer, so a wrong answer cannot be traced to a retrieval failure or a generation failure. This paper introduces mmRAG, a benchmark that separates the two steps. It converts six question-answering datasets—covering plain text, tables, and knowledge graphs—into a unified corpus of documents split into chunks, and annotates which chunks are relevant to each of 5,124 queries with three-level graded labels. It then derives per-dataset relevance scores from the chunk labels, so query routing—deciding which source to search—can be evaluated directly. The paper argues that this modular, multi-modal design fills a gap in existing RAG benchmarks, which are mostly single-modal and evaluate only end-to-end quality.","feed_headline":"New benchmark measures RAG retrieval and routing directly","feed_subtitle":"Graded relevance labels across text, tables, and knowledge graphs enable component-level RAG diagnosis.","key_machinery":"The machinery that carries mmRAG is its two-tier relevance annotation. At the chunk level, a pooled-annotation protocol combines two complementary retrievers—BM25 (lexical) and BGE-large-en-v1.5 (semantic)—to gather up to 19 candidate chunks per query, ensuring global top-10 chunks, the top-3 chunks from the query's original relevant document, and the top-1 chunk from each of the five datasets; an ensemble of two large language models with a third as tiebreaker then labels every pooled chunk as irrelevant (0), partially relevant (1), or highly relevant (2). At the dataset level, the aggregated label is $S_{q,D} = \\sum_{d \\in D} \\max_{c \\in d} L_{q,c}$, which sums, over all documents in a dataset, the score of each document's most relevant chunk; this is the quantity used as ground truth for query routing. The unified corpus itself is also part of the machinery: KG entities are expanded into verbalized one-hop subgraphs, and every document is cut into non-overlapping 512-token chunks so that lexical and dense retrievers can be compared on equal footing.","core_discovery":"The central claim is that a modular, multi-modal benchmark can be built by converting heterogeneous data into a uniform document collection and annotating relevance at two levels. mmRAG takes six QA datasets (NQ and TriviaQA for text, OTT and TAT for tables plus text, CWQ and WebQSP for knowledge graphs), converts the KGs into verbalized one-hop subgraphs, segments every document into 512-token chunks, and keeps only queries with at least one relevant chunk, yielding 5,124 queries. A pooled annotation procedure—BM25 and BGE-large-en-v1.5 retrieve up to 19 candidate chunks per query, and an LLM ensemble assigns each chunk a label $L_{q,c} \\in \\{0,1,2\\}$ for irrelevant, partially relevant, or highly relevant—produces 88,751 annotated query-chunk pairs. Dataset-level relevance is then defined as $S_{q,D} = \\sum_{d \\in D} \\max_{c \\in d} L_{q,c}$, which the paper uses as the ground truth for query routing. The paper evaluates seven retrievers and two routers on this benchmark; the best retriever, BGE, reaches NDCG@1 of 0.617, the LLM router beats the semantic router at small top-k, and direct accuracy metrics track but do not perfectly coincide with generation quality, which the authors interpret as evidence that direct component evaluation is valuable and complementary.","pith_inferences":["The validity of the routing labels hinges on Eq. (1); a natural extension is to collect human dataset-level usefulness judgments for a sample of queries and test whether $S_{q,D}$ agrees with them, or to train a router directly on the provided labels and compare it against the two baseline routers.","The pooling-based annotation could be stress-tested by running a third, stronger retriever and checking whether it surfaces relevant chunks outside the pooled set; the fraction of such misses would quantify how much the 'everything else is irrelevant' assumption may bias retrieval evaluation.","The benchmark's structure enables a failure-attribution protocol: compare generation with the Oracle retriever against generation with each real retriever to separate retrieval errors from generation errors, which the paper does not explicitly report per system."],"forward_implications":["With mmRAG's chunk-level labels, retrieval accuracy can be benchmarked directly with standard ranking metrics such as NDCG, MAP, and Hits, without running a generator, which is cheaper and more reproducible than end-to-end evaluation.","Query routing can be measured directly using the derived dataset-level relevance labels, enabling systematic comparison of routers (the paper finds an LLM router beats a semantic router at small top-k).","The benchmark's cross-dataset relevance statistics (e.g., up to about 68% of WebQSP queries have relevant chunks outside their original dataset) show that routing is practically necessary, not just a luxury.","Retrieving only from the query's original dataset often outperforms retrieving from the full multi-modal corpus (e.g., BM25 drops from 0.5210 to 0.4162 on CWQ with GLM), demonstrating that multi-modal retrieval and routing remain open challenges.","Direct routing accuracy trends align with generation quality trends, so mmRAG's routing labels provide a cheaper proxy for system-level performance."],"supporting_citations":[{"why":"RAGBench, the closest prior benchmark with sufficient retrieval labels but no query-routing support; mmRAG's contrast highlights the missing routing annotations.","marker":"[7]"},{"why":"Natural Questions (NQ), a source of text queries and documents that anchor the text modality.","marker":"[14]"},{"why":"TriviaQA, a source of text queries and documents adding single- and multi-hop reasoning variety.","marker":"[12]"},{"why":"OTT-QA, a source of table-plus-text documents and multi-hop queries.","marker":"[2]"},{"why":"TAT-QA, a source of financial table-plus-text documents with numerical reasoning queries.","marker":"[37]"},{"why":"CWQ, a source of knowledge-graph queries over Freebase.","marker":"[23]"},{"why":"WebQSP, a source of knowledge-graph queries over Freebase.","marker":"[33]"},{"why":"Procis, which supplies the three-level graded relevance labeling scheme used for chunks.","marker":"[20]"},{"why":"BGE-large-en-v1.5, one of the two pooling retrievers and a main baseline across the experiments.","marker":"[28]"},{"why":"UniversalRAG, the model the paper's LLM-based query router is inspired by.","marker":"[32]"}],"fun_headline_variants":["mmRAG: Drill-down tests for RAG retrieval and routing","Modular RAG benchmark zooms in on retrieval and routing","New benchmark dissects RAG components, not just output","RAG under the microscope: mmRAG tests each part","Component-level RAG scoring across text, tables, and KGs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that summing, over each document in a dataset, the relevance score of its single most relevant chunk tells you how useful that dataset is for answering the query, but this aggregation is never checked against human judgments of dataset usefulness.","fun_headline_variants_meta":{"raw":{"variants":["mmRAG: Drill-down tests for RAG retrieval and routing","Modular RAG benchmark zooms in on retrieval and routing","New benchmark dissects RAG components, not just output","RAG under the microscope: mmRAG tests each part","Component-level RAG scoring across text, tables, and KGs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000772,"raw_usage":{"total_tokens":3438,"prompt_tokens":983,"completion_tokens":2455,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":2368}},"tokens_in":599,"tokens_out":2455,"duration_ms":18090,"temperature":1.0,"reasoning_tokens":2368,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:55:29.966383+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of mmRAG queries, ask human judges whether each dataset actually contains the answer, and compare their judgments against the rankings produced by Eq. (1); if the formula systematically over-ranks datasets that contain many partially relevant documents but no truly answering document, the router evaluation in Section 5 rests on an artifact. A complementary test is to run a strong retriever not used in pooling (e.g., a cross-encoder) and measure how often it finds relevant chunks outside the pooled set; a substantial miss rate would violate the 'unpooled chunks are irrelevant' assumption underlying the retrieval labels.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BGE-large-en-v1.5, one of the two pooling retrievers and a main baseline across the experiments."}],"review_version":1}