{"id":"34d1a42a-e67c-41a9-a1eb-1c58cec156d2","arxiv_id":"2504.17137","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MIRAGE is a lightweight RAG evaluation benchmark with 7,560 QA pairs and four error-type metrics that partition outcomes across base, oracle, and mixed contexts.","lead":"This paper introduces MIRAGE, a compact benchmark of 7,560 question-answer pairs and a small retrieval pool for evaluating retrieval-augmented generation (RAG) systems. It also proposes four metrics that classify model errors based on performance with no context, correct context, and noisy context, and reports experiments across multiple retrievers and LLMs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MIRAGE's filtering pipeline (Sec. 3.4) selects queries that Llama-3.1-8B cannot answer without context, and that criterion was never human-validated, so the benchmark may be idiosyncratic to the filtering models.","rationale":"I agree with the reader's weakest_assumption: the filtering models are the least secure link in the argument. The paper presents the filter as a quality guarantee, but the only human check is on support labels, not on the no-context criterion that defines what 'base' means. Because the base rate is later used to decompose errors into the four metrics, any model-specific selection bias propagates into every reported number. I considered the alternative concern that the five-chunk, same-document retrieval pool weakens retriever evaluation; that is real but secondary, since the paper frames MIRAGE as a lightweight proxy and the retrieval task, while controlled, still measures ranking among distractors. The novelty and 'guaranteed insight' issues raised by the reader are presentation problems, not threats to the core methodology. The release-status contradiction and underspecified exact-match scoring also need fixing, but they are not the most load-bearing. The concrete test above would settle whether the filter bias actually changes results; until then I would keep the reader's CONDITIONAL verdict unchanged.","tokens_in":18307,"tokens_out":5253,"duration_ms":49309,"concrete_test":"Take a random sample of 500 MIRAGE queries that passed Step 2 because Llama-3.1-8B could not answer them without context. Run the same base-setting prompt (Appendix A, Table 4) with Qwen2-7B-Instruct, Mistral-7B-Instruct, and GPT-4o-mini, and also run the oracle-setting prompt. If any of these models answers a large fraction of the sample correctly in the base setting, or if its oracle-minus-base gain is far smaller than Llama-3.1-8B's, the dataset has been shaped by one model's limitations. Report the per-model base rates and recompute the top-k retriever rankings for the mixed context; if rankings change materially, the filtering bias is empirically confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MIRAGE is a valid benchmark for arbitrary RAG systems depends on Section 3.4's multi-layered filter. Step 2 keeps an instance only if Llama-3.1-8B answers correctly with the retrieved chunk and fails without it; Step 1 uses Command-R to label support. The human validation in Section 3.5 (100 queries, 500 chunks) checks only whether a chunk contains the answer; it does not test the 'unanswerable without context' condition. Since base accuracy in Table 3 ranges from 6.6% (Llama-2-7B) to 45.8% (GPT-4o), the no-context behavior is strongly model-dependent. If the filter has selected for Llama-3.1-8B's parametric blind spots, then for other LLMs the base setting is not a clean lower bound, and the reported noise vulnerability and context acceptability numbers, as well as retriever rankings computed from them, may not generalize. This is a validity risk rather than an internal inconsistency, but it is load-bearing because every metric and every model comparison flows through the filtered dataset.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MIRAGE, a QA benchmark built by re-filtering PopQA, NQ, TriviaQA, IfQA, and DROP into 7,560 query instances paired with a 37,800-chunk retrieval pool, and proposes four RAG adaptability metrics (noise vulnerability, context acceptability, context insensitivity, context misinterpretation) computed from base, oracle-context, and mixed-context answers. The authors report experiments across multiple retrievers and LLMs, claiming that the benchmark allows efficient, component-specific RAG evaluation and that the metrics reveal systematic LLM weaknesses and retriever effects.","tokens_in":18364,"tokens_out":4983,"duration_ms":45998,"significance":"If the dataset-construction validity issues are resolved, MIRAGE would be a useful community resource: the retrieval pool is compact, the data and code are released, the four metrics are simple to compute and provably sum to 1, and the 60-configuration experiment matrix provides a rich empirical snapshot of retriever-LLM interactions. The paper's main contributions are the curated dataset and the metric decomposition, both of which are potentially valuable for reproducible RAG evaluation; the weaknesses described below are validity and specification issues rather than unsound derivations.","major_comments":[{"comment":"The load-bearing filtering step is not fully validated. Step 2 of Section 3.4 retains a query only when Llama-3.1-8B answers correctly with the retrieved chunk and fails without it, but the human validation in Section 3.5 checks only whether a chunk contains the answer, not the 'unanswerable without context' condition. Table 3 shows base accuracy ranging from 6.6% (Llama-2-7B) to 45.8% (GPT-4o), so the no-context behavior is strongly model-dependent. If the filter selected for Llama-3.1-8B's parametric blind spots, then for other LLMs the base setting is not a clean lower bound, which would distort noise vulnerability and context acceptability and could change retriever rankings. Please report per-LLM base-response statistics on MIRAGE, or independently validate the unanswerable-without-context criterion across the evaluated models.","section":"3.4, 3.5, Table 3"},{"comment":"The exact-match evaluation is underspecified for the answer formats in the dataset. Answer labels are lists of surface forms (e.g., ['actor, actress, actors, actresses']), date expressions (['January 19, 2018']), and noun phrases, and no normalization or matching procedure is stated. Without a precise rule for case/punctuation normalization, multi-answer labels, and partial or paraphrased matches, the accuracy numbers in Tables 1 and 3 and all derived metrics cannot be reproduced or compared across systems.","section":"4.1"},{"comment":"The oracle-chunk selection procedure is not described. The text says 'One relevant chunk is selected from the top-5 chunks mapped to each query,' but Figure 4 shows queries can have 1 to 5 relevant chunks. When multiple chunks are support-labeled, which chunk is used as the oracle, and is the selection deterministic? Since AnsO determines every metric, this choice is load-bearing for the reported scores and for cross-retriever comparisons.","section":"4.1"},{"comment":"The reported constancy of context insensitivity and context misinterpretation across retrievers and top-k settings is an artifact of the metric definitions and evaluation setup, not an empirical discovery. Equations 4 and 5 depend only on b and o, not on the mixed answer m, and the oracle response does not depend on the retriever in the experimental protocol, so CI and CM are constant across retriever rows by construction. The claim in Section 5.3 that this 'indicates that the ability to utilize the context properly relies solely on the LLM's capabilities' should be recast as a design property of the metrics, or the paper should define a retriever-dependent oracle setting if this is meant to be an empirical result.","section":"4.2, 5.3, Equations 4-5"},{"comment":"The context misinterpretation metric includes the group G(1,0,1), where the base answer is correct, the oracle-context answer is wrong, but the mixed-context answer is correct. Such cases do not fit the prose description of 'hallucination' or 'over-relying on irrelevant information,' since the model succeeds in the noisy mixed setting while failing with the clean oracle chunk. Please clarify whether this group is intentionally included and what behavior it is intended to measure.","section":"4.2, Equation 5"}],"minor_comments":[{"comment":"There are several typographical errors: 'Context Acceptibility' in Figure 1, 'Misintepretation' in Tables 9-11, 'Contect' in Table 11, and 'poplation' in the Figure 1 example query.","section":"Figure 1, Tables 9-11"},{"comment":"The benchmark name is inconsistently rendered as 'MIRAGE' and 'MiRAGE'; please standardize.","section":"Throughout"},{"comment":"The model list includes 'Llama3-8B: 8B parameters,' but Llama-3.1-8B is not an evaluated generation model in Tables 1 or 3; clarify whether this entry refers to the filtering model and move it out of the experimental model list if so.","section":"Appendix D"},{"comment":"The description of the retrieval mapping is terse: 'top 5 document chunks by title match' is not enough to reproduce the chunk-to-query mapping, especially for source datasets like TriviaQA where passage-to-article linking is fragmented.","section":"3.3"},{"comment":"No stage-wise counts are reported for the multi-layered filter (e.g., how many of the 61,165 mapped pairs were removed by support labeling, inference validation, and title verification). Reporting these numbers would strengthen the reproducibility claims.","section":"3.4"}],"recommendation":"major_revision","confidential_remarks":"The filtering-model dependence is the main validity risk; I would ask the authors for a per-LLM base-response analysis and an explicit specification of oracle-chunk selection and exact-match normalization in the revision. The dataset itself is potentially valuable, and the issues appear addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: MIRAGE is a real, usable RAG evaluation benchmark, but its retriever component is much weaker than advertised, and the filtering pipeline makes the benchmark partially idiosyncratic to Llama-3.1-8B.\n\nWhat's new: a 7,560-query dataset with a fixed 37,800-chunk pool, built from five existing QA sets, plus four metrics that partition responses over base/oracle/mixed conditions. The construction is documented in unusual detail — prompts, filtering stages, human validation of 100 queries with 95% agreement. The metrics are cleanly defined and sum to 1. That's a genuine contribution.\n\nSoft spots: the per-query retrieval pool is five chunks from the same Wikipedia article. That makes the retriever evaluation a ranking task over near-duplicate chunks, not a realistic retrieval problem. F1/NDCG numbers on this pool don't transfer to full-corpus retrieval. The filtering pipeline keeps queries Llama-3.1-8B can't answer without context; human validation never checks that condition, so for other LLMs the \"base\" setting is not a clean lower bound. The paper also presents retriever-invariance of context insensitivity and misinterpretation as an empirical finding, but that's built into the metric definitions and the fixed oracle-in-mixed design — it's not surprising. Exact match for list answers is underspecified, oracle chunk selection among multiple relevant chunks isn't described, and the text contradicts itself on whether data is released now or after publication.\n\nThat said, the core artifact is usable, the writing is honest about limitations, and the metrics are a reasonable addition. The retriever-invariance point is a mislabeling, not a fatal flaw. I'd send this to a serious referee. The right outcome is a conditional accept: fix the release statement, specify the scoring, justify or shrink the retriever claims, and add a robustness check on how the benchmark behaves when filtered with a different model. If those are addressed, it's a solid benchmark paper.","headline":"MIRAGE is a genuinely useful compact RAG benchmark, but the retriever evaluation is artificially easy and the filtering pipeline gives it a partial model-specific bias.","tokens_in":19081,"tokens_out":3763,"would_cite":false,"duration_ms":33143,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MIRAGE: a 7,560-question benchmark with a 37,800-chunk retrieval pool, plus four metrics that partition any RAG system's exact-match outcomes into retriever-driven and LLM-driven failure modes.","keywords":["retrieval-augmented generation","RAG evaluation","question answering benchmark","retriever-LLM analysis","noise vulnerability","context acceptability","context insensitivity","context misinterpretation"],"falsifier":"Take a random sample of the 61,165 mapped query-chunk pairs, re-run the support-labeling and inference-validation steps with a different LLM pair or with human annotators, and compare the surviving instances to MIRAGE's current 7,560; if the four metric values or the retriever rankings on the two versions diverge materially, the reported results are artifacts of the filtering models rather than stable RAG properties.","tokens_in":17950,"feed_emoji":"🎯","tokens_out":8632,"duration_ms":72184,"temperature":0.7,"pith_summary":"This paper introduces MIRAGE, a retrieval-augmented generation benchmark built from 7,560 question-answer pairs, each mapped to five Wikipedia chunks for a fixed retrieval pool of 37,800. It claims that this compact pool is enough to evaluate both retriever and generator precisely, without indexing an entire document collection. The paper's diagnostic tool is a four-metric partition of exact-match scores under three conditions: no context, only the oracle chunk, and a mix of oracle plus noise. Because the four metrics cover every possible response pattern and sum to 100%, a single run shows whether a system loses accuracy to noisy chunks, fails to use good context, or simply lacks the needed knowledge. This matters because it would make component-level RAG evaluation cheap, reproducible, and able to separate retriever quality from LLM capability.","feed_headline":"7,560 questions isolate retriever noise from LLM context failures","feed_subtitle":"Four scores that always sum to 100% show where RAG accuracy goes: noisy chunks, missed context, or the LLM.","key_machinery":"The load-bearing device is the three-condition scoring scheme with the partition identity $$\\sum_{b,o,m\\in\\{0,1\\}} |G(b,o,m)|/|\\mathcal{D}| = 1,$$ where $\\mathcal{D}$ is the dataset and $G(b,o,m)$ is the set of queries whose base, oracle, and mixed answers are correct (1) or wrong (0). The eight cells are grouped into four named metrics—noise vulnerability, context acceptability, context insensitivity, context misinterpretation—each a sum of two cells divided by dataset size. Because context insensitivity and context misinterpretation are defined only from the base and oracle conditions, they are insensitive to retrieval, while noise vulnerability and context acceptability depend on the mixed condition and therefore on retriever output. This decomposition is what lets a single benchmark pass tell a user whether to improve retrieval, improve the LLM's reading, or accept a knowledge ceiling.","core_discovery":"Starting from 500K+ question-answer pairs in five existing Wikipedia-based QA datasets, the authors remap queries to a September 2024 Wikipedia dump, split articles into 330-token chunks, and keep only instances that survive two automatic checks: a RAG-oriented LLM labels a chunk as supporting the answer, and a second LLM can answer the query with that chunk but not without it. Human agreement with the automatic labels, checked on 100 queries and 500 chunks, is about 95%. The resulting benchmark, MIRAGE, consists of 7,560 queries and a 37,800-chunk retrieval pool. The paper's central evaluative claim is that scoring a system in base, oracle, and mixed contexts, then grouping the eight binary response patterns into the four named metrics, gives a complete and decomposable profile of where accuracy is lost. Experiments show the metrics separate retriever-driven effects (noise vulnerability and context acceptability move with retriever quality) from LLM-driven effects (context insensitivity and context misinterpretation stay nearly constant for a fixed LLM regardless of retriever or top-k).","pith_inferences":["If the filtering labels generalize, the benchmark rankings should survive validation with a different LLM pair; a cheap test would be to re-derive support labels on a random 1,000-query subset with an independent model and compare which instances survive.","The same base/oracle/mixed decomposition could be applied to non-QA RAG tasks such as summarization or tool use, using task-specific correctness instead of exact match, which would extend the diagnostic profile beyond factoid questions.","Because filtering deliberately keeps only instances where a model cannot answer without context, MIRAGE likely over-represents retrieval-dependent cases; a complementary benchmark with a base-answerable half could measure whether adding context helps or hurts when the model already knows the answer."],"forward_implications":["For a fixed LLM, switching to a stronger retriever should lower noise vulnerability and raise context acceptability while leaving context insensitivity and context misinterpretation unchanged, so retriever quality can be read directly from two of the four scores.","Users can map a system's four-number profile to an intervention: high noise vulnerability suggests better retrieval or denoising, high context insensitivity suggests the LLM cannot exploit the chunk, and high context misinterpretation suggests the context actively misleads the model.","The 37,800-chunk pool makes retriever evaluation (F1, NDCG) reproducible without re-indexing a full Wikipedia dump, lowering the cost of head-to-head retriever comparisons.","Every tested LLM scores below 100% in the oracle-only condition, so MIRAGE exposes a context-reading ceiling in current models rather than assuming retrieved evidence is always used."],"supporting_citations":[{"why":"Supplies PopQA, the entity-centric QA pair source that contributes a large share of MIRAGE.","marker":"Mallen et al., 2022"},{"why":"Supplies Natural Questions, the largest source of QA pairs in the benchmark.","marker":"Kwiatkowski et al., 2019"},{"why":"Supplies TriviaQA, used as one of the five source datasets.","marker":"Joshi et al., 2017"},{"why":"Supplies IfQA, the counterfactual-pressupposition QA source in MIRAGE.","marker":"Yu et al., 2023b"},{"why":"Supplies DROP, the reading-comprehension source contributing discrete-reasoning questions.","marker":"Dua et al., 2019"},{"why":"Provides the Command-R model whose support labels decide which chunks are kept as positive evidence.","marker":"Cohere, 2024"},{"why":"Provides Llama-3.1-8B, the model whose inference success and failure define which instances survive filtering.","marker":"Dubey et al., 2024"}],"fun_headline_variants":["MIRAGE: new metrics pin RAG errors to retriever or LLM","7,560 QA queries, four metrics: know where RAG fails","RAG evaluation reboot: MIRAGE splits noise from context","Retriever vs LLM: MIRAGE quantifies each RAG loss","Benchmark MIRAGE: 37,800 chunks, 4 error scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's validity rests on the assumption that the two LLM filters label chunks the way a human would; only 100 of the 7,560 queries were human-checked, and the filters are calibrated to one model family's ability to answer with and without context.","fun_headline_variants_meta":{"raw":{"variants":["MIRAGE: new metrics pin RAG errors to retriever or LLM","7,560 QA queries, four metrics: know where RAG fails","RAG evaluation reboot: MIRAGE splits noise from context","Retriever vs LLM: MIRAGE quantifies each RAG loss","Benchmark MIRAGE: 37,800 chunks, 4 error scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000257,"raw_usage":{"total_tokens":1605,"prompt_tokens":999,"completion_tokens":606,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":519}},"tokens_in":615,"tokens_out":606,"duration_ms":5784,"temperature":1.0,"reasoning_tokens":519,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:48:54.552635+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the 61,165 mapped query-chunk pairs, re-run the support-labeling and inference-validation steps with a different LLM pair or with human annotators, and compare the surviving instances to MIRAGE's current 7,560; if the four metric values or the retriever rankings on the two versions diverge materially, the reported results are artifacts of the filtering models rather than stable RAG properties.","supporting_citations":[],"review_version":1}