{"id":"457734aa-6df1-4dca-85f5-83aa1af22807","arxiv_id":"2507.06956","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Retrieval-augmented generation performance drops noticeably under minor query perturbations, with end-to-end results often tracking retriever behavior.","lead":"Small changes to a user's question, such as typos, added words, or a more formal tone, can significantly reduce the accuracy of retrieval-augmented question answering systems. This paper maps those failures across four retrievers and three language models, and offers a way to test which part of the system is most fragile.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Perturbation validity is the load-bearing risk: GPT-4o ambiguity/redundancy may alter the question's answer, so observed retriever/RAG degradation could reflect label shift rather than surface fragility; Section 3's 'same semantics' claim and Appendix A.2's checks do not establish answer…","rationale":"The paper is a broad empirical study; the best evidence is the large and consistent Recall@5 drops (Table 4), and the use of a model-free Match metric is sensible for reproducibility. However, the central interpretation—that RAG is fragile to minor surface variations and that retrieval dominates end-to-end behavior—rests on the perturbations preserving the original information need. The reader flagged this as the weakest assumption; I agree and sharpen it as an internal-validity threat, because the phrase 'unclear to answer' in the ambiguity prompt suggests the task changes, and the oracle-setting drops in Figure 3 are hard to explain if the answer is unchanged and the gold documents are in context. A human-annotation answer-preservation check is the decisive experiment. If the perturbations are answer-preserving, my objection is refuted; if not, the quantitative conclusions need re-estimation on answer-preserving subsets, making acceptance conditional on that evidence. I do not find an internal inconsistency in the experiments themselves; the retriever-predominance claim is qualified by the BioASQ results and the correlation tables, though the Pearson correlations are modest and would benefit from confidence intervals. No objection to the authors; this is a measurement-validity issue.","tokens_in":17016,"tokens_out":11106,"duration_ms":110350,"concrete_test":"Sample 100 perturbed queries per perturbation type per dataset. For each, give three annotators the original query, the perturbed query, the gold document, and the gold answer, and ask whether the gold answer is still a correct and complete answer to the perturbed query. Compute the annotator-agreement-adjusted fraction of answer-preserving perturbations. If that fraction is below ~90% for ambiguity or redundancy, regenerate those perturbations with stricter prompts (e.g., 'rephrase the question without changing what is being asked or the answer') and re-run the retriever Recall@5, oracle Match, and end-to-end Match experiments. If the large performance drops persist on the answer-preserving subset, the central claim stands; if they largely vanish, the original conclusions are confounded by label shift.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 defines Perturb(q) as retaining 'the same or very similar semantics as q,' and the abstract frames the findings as degradation under 'minor query variations.' The ambiguity prompt (Table 3) explicitly instructs GPT-4o to make the query 'unclear to answer by introducing ambiguity,' which can change the information need (e.g., 'when does X take place' vs 'when might X take place'). The redundancy prompt also adds long background clauses, changing query length and salience. Appendix A.2 validates perturbations using GPT-2 perplexity and e5-base cosine similarity only; neither metric verifies that the gold answer to the original query is still a correct answer to the perturbed query, and e5 embeddings are themselves surface-sensitive. Internal evidence raises the same concern: Section 5.3 reports that ambiguity insertion decreases performance in the oracle setting (gold documents provided) across datasets, and redundancy causes drastic oracle drops on BioASQ; if the answer were preserved and the gold documents were in context, a robust generator should still answer correctly, so these drops are consistent with the perturbed query having a different or underspecified answer. If the perturbation changes the answer, then retriever Recall@5 drops and end-to-end Match drops partly measure sensitivity to a changed task rather than to query surface variation, undermining the headline claim and the module-decoupling conclusions. This concern is more internal than the representativeness worry: it applies even if the perturbations occur in real user queries.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the robustness of retrieval-augmented generation (RAG) at the query level. It introduces five perturbation types: two levels of typographical errors inserted with TextAttack, and three LLM-generated semantic paraphrases (redundancy insertion, formal tone change, ambiguity introduction). Using four retrievers (BGE-base, Contriever, BM25 flat, BM25 multi-field) and three LLM generators (Llama-3.1-8B, Mistral-7B, Qwen2.5-7B), it measures retriever Recall@k, generator closed-book and oracle Match, and end-to-end RAG Match on NQ, HotpotQA, and BioASQ. The main findings are that retriever performance degrades under minor query variations, that dense and sparse retrievers differ in which perturbations hurt most, that end-to-end RAG trends on NQ follow retriever trends while BioASQ shows additional generator-driven drops for ambiguity and redundancy, and that module-level correlations can help practitioners identify the sensitive component. The paper also proposes an evaluation framework and gives recommendations for robustness testing.","tokens_in":17244,"tokens_out":6003,"duration_ms":62766,"significance":"If the central claims withstand scrutiny, this is a valuable empirical study: it systematically measures query-level robustness across a broad matrix of retrievers, generators, datasets, and perturbation types, and it proposes a module-decoupling evaluation protocol that practitioners can adapt to their own pipelines. The deliberate choice of a model-free Match metric aids reproducibility, and the promise to release data and code is a concrete asset. The work is not circular in the technical sense: the findings are empirical measurements rather than predictions obtained by fitting free parameters, and the recommendations are derived from the measured results. However, the validity of the perturbation battery is the load-bearing assumption, and the current evidence does not establish that all five perturbation types preserve the original answer; this limits the scope of the headline claim about 'minor query variations'.","major_comments":[{"comment":"The central claim that the tested perturbations are 'minor query variations' that preserve semantics is not established for the ambiguity and redundancy perturbations. The prompt in Table 3 instructs GPT-4o to make the query 'unclear to answer by introducing ambiguity,' which can change the information need; the example in Section 3 changes 'when does the cannes film festival take place' to 'when might the Cannes Film Festival be held?', a question whose answer need not coincide with the original gold answer. The validation in Appendix A.2 uses only perplexity and e5-base cosine similarity, neither of which verifies that the original gold answer is still a correct answer for the perturbed query. Internal evidence in Section 5.3 strengthens this concern: ambiguity insertion lowers performance in the oracle setting (gold documents provided) across all datasets, and redundancy causes drastic oracle drops on BioASQ; if the gold documents are in context and the question still asks for the same fact, a robust generator should not degrade this much. These drops are consistent with the perturbed queries having different or underspecified answers, so the end-to-end and retriever degradation may partly measure a changed task rather than surface fragility. I recommend adding a human or expert evaluation of answer preservation for each perturbed sample, filtering out or annotating samples where the gold answer is no longer valid, and clearly separating intent-changing perturbations (ambiguity, possibly redundancy) from pure surface variations in the claims.","section":"Section 3 and Appendix A.2; Table 3"},{"comment":"The reported averages over five perturbation samples per original query are presented without variance, confidence intervals, or significance tests. The qualitative ordering of retriever robustness across perturbation types (e.g., 'dense retrievers are more robust against the redundant information,' 'sparse methods performances are more robust against the typos,' and 'typo insertions are particularly impactful in the closed-book setting') could be driven by sampling noise, especially because the five samples are arbitrary outputs of GPT-4o or TextAttack rather than an independently drawn sample from a well-defined distribution. I request that the authors provide confidence intervals or bootstrap estimates for the average Recall@k and Match values, and report significance tests or at least effect sizes for the key pairwise comparisons that support the conclusions.","section":"Section 5.2-5.4 and Table 4"},{"comment":"The Pearson correlations between module-level and end-to-end performance discrepancies are computed on per-sample data that are clustered by original query (five perturbed samples per query), yet the analysis ignores this clustering and reports no confidence intervals or tests for the differences between correlations. The conclusion that 'different dominant factors exist within the pipeline for different perturbation types' rests on comparisons of correlations that are small in magnitude (e.g., 0.05 vs 0.35 in Table 2) and may be within statistical noise. The authors should either provide uncertainty estimates for these correlations (e.g., cluster-bootstrap confidence intervals) and test whether the differences are significant, or soften the claims accordingly.","section":"Section 5.4, Table 2 and Tables 5-6"}],"minor_comments":[{"comment":"The text contains a duplicated phrase: 'we recommend that practitioners use our analysis framework on their we recommend that practitioners use our analysis framework on their own data'; this should be corrected.","section":"Section 6"},{"comment":"The reference list contains duplicate entries: Kwiatkowski et al. 2019a and 2019b are the same paper, and Zhao et al. 2024b and 2024c have the same title and venue; please merge or remove duplicates.","section":"References"},{"comment":"There is a typo in 'langueg processing community'; it should be 'language processing community'.","section":"Appendix A.2"},{"comment":"The caption defines PERT as 'Number of perturbed samples for each perturbation type,' but the values appear to be the number of original queries used to generate perturbed samples; please clarify the definition.","section":"Table 1"},{"comment":"The conclusion that redundancy and ambiguity 'result in more scattered internal representations' is based on visual inspection of PCA plots; a quantitative dispersion measure (e.g., average pairwise distance or a scatter index) would make this claim verifiable.","section":"Section 5.4, Figure 5"},{"comment":"The phrase 'more redundant correpondents' should be 'more redundant counterparts'.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical study, but the 'ambiguity' perturbation is arguably not a 'minor query variation' but an intent-changing one; if the authors cannot provide answer-preservation evidence, they should consider removing ambiguity from the headline-claim set or redefining it as a separate stress test. The statistical reporting also needs strengthening before the comparative conclusions can be trusted. The contribution is closer to a benchmark or evaluation protocol than a new method, and the claim to be 'the first framework' to decouple module sensitivities may be overstated given existing robustness benchmarks such as RobustQA and PromptRobust; I recommend the editor weigh the novelty framing accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Take a look at this one if you are building or evaluating RAG systems. The paper runs a genuinely wide grid: four retrievers, three LLMs, three datasets, five perturbation types, with modular isolation (retriever, closed-book, oracle) plus end-to-end. That is a useful diagnostic template. The typo results are believable and consistent with prior work: dense retrievers lose more, sparse are more robust to typos, formal tone is the mildest, and redundancy hurts BM25 badly on BioASQ. The Pearson correlations between module-level and pipeline-level drops are a reasonable way to locate the weakest link, even if they are not conclusive. The soft spot that matters is the ambiguity perturbation. The prompt tells GPT-4o to make the query 'unclear to answer by introducing ambiguity,' which can change what the correct answer is or leave it under-specified. The oracle results in Section 5.3 show large drops under ambiguity even when gold documents are provided, and the Appendix A.2 validation uses perplexity and e5 embedding similarity, neither of which checks whether the gold answer is still a valid answer to the perturbed query. So part of the measured degradation could be a changed task, not surface fragility. The same worry applies to redundancy on BioASQ, which also drops sharply in the oracle setting. This does not sink the paper, but it undercuts the sweeping claim that retrievers degrade 'even under minor query variations.' The oracle drop could still reflect real generator sensitivity to phrasing, but the two explanations are not disentangled. Second, there is no variance or significance testing anywhere. Each perturbation has five samples per query, so per-query differences could be summarized, but the paper reports only averages. Many of the reported Pearson correlations are small (0.05 to 0.35) and no significance or confidence intervals are given, yet the 'retriever dominates' conclusion leans on those numbers. Third, code and data are promised but not released, so the 1092 experiments are not yet independently checkable. This is a solid empirical contribution for practitioners who need a quick robustness check, and the modular framework is worth adopting. But I would want the authors to validate answer preservation (human or automated, e.g. entailment of the gold answer in the perturbed query) and add basic statistics before relying on the quantitative rankings. It deserves a serious referee, with major revisions rather than a desk reject.","headline":"Useful empirical map of RAG query-perturbation robustness, but the ambiguity perturbation likely shifts the answer, so the headline 'minor variations' claim needs a validity check.","tokens_in":676,"tokens_out":682,"would_cite":false,"duration_ms":52428,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that RAG systems are most fragile at the retrieval step: common retrievers degrade substantially under minor query rewording and typos, and end-to-end answer quality largely follows retriever performance.","keywords":["RAG robustness","query perturbations","retriever robustness","retrieval-augmented generation","question answering","typographical errors","dense vs sparse retrieval","evaluation framework"],"falsifier":"Take a set of naturally occurring query variants for the same information need, such as real search-log paraphrases or crowd-written reformulations, run them through the same four retrievers, and compare Recall@k against the original queries; if naturally varied wording does not produce the clear recall drops reported here, the paper's quantitative robustness claim does not generalize. A second check would be having human raters judge whether the LLM-generated redundancy, formal tone, and ambiguity rewrites preserve meaning; if a large share change the question, the observed end-to-end drops conflate semantic change with surface fragility.","tokens_in":16809,"feed_emoji":"🔍","tokens_out":7588,"duration_ms":69049,"temperature":0.7,"pith_summary":"The paper sets out to show that retrieval-augmented generation (RAG) is much more fragile to how a user words a query than to what the generator does with the retrieved text. Across more than 1,092 experiments with four retrievers, three language models, three question-answering datasets, and five perturbation types, it finds that common retrievers lose substantial accuracy under minor rewording, typo insertion, added redundancy, formal tone shifts, and ambiguity. Running the same perturbations through the full pipeline shows that end-to-end answer quality usually follows the retriever's performance curve rather than the generator's. The paper's practical point is that robustness testing must decouple the modules, because a score from the generator alone, or from the retriever alone, can mislead about where the system will fail. A sympathetic reader would take away that query-level robustness deserves its own evaluation step before deploying a RAG system.","feed_headline":"Retrievers, not generators, decide RAG robustness","feed_subtitle":"1,092 experiments map how query rewording and typos hurt answer quality; the retriever is usually the weak link.","key_machinery":"The load-bearing object is the decoupling evaluation framework: the same perturbed queries are run through four isolated conditions, retriever-only Recall@k, closed-book generation, oracle-context generation, and end-to-end RAG, with five meaning-preserving perturbation types generated by an LLM prompt for redundancy, formal tone, and ambiguity, plus QWERTY-adjacent character replacement at 10% and 25% typo levels. Pearson correlations between each module's performance delta and the end-to-end delta are then used to attribute the pipeline's sensitivity, and a PCA projection of the LLM's internal representations checks whether perturbations scatter the model's understanding even when the gold documents are provided. This machinery is what lets the paper claim that end-to-end RAG trends are predominantly characterized by the performance of the retriever while still flagging the BioASQ cases where generation is the weak link.","core_discovery":"The central claim is that the query-level robustness of a RAG pipeline is set primarily by the retriever: when a query is lightly perturbed, retriever Recall@k drops markedly, and end-to-end Match performance tracks those retriever drops more closely than it tracks the generator's closed-book or oracle performance. The study isolates each module, using retriever alone, generator in closed-book mode, generator with an oracle retriever, and the full pipeline, across NQ, HotpotQA, and BioASQ with dense and sparse retrievers. It finds that dense retrievers absorb redundant wording better, sparse retrievers absorb typos better, formal tone changes hurt least, and 25% typos hurt most. The one notable exception is BioASQ, where ambiguity and redundancy cause end-to-end drops even when the retriever looks stable, showing that the generator can be the binding constraint on domain-specific biomedical questions. The authors conclude that practitioners should use a decoupling analysis on their own data to identify which module is sensitive to which perturbation type.","pith_inferences":["Because the synthetic perturbations were produced by an LLM and keyboard-noise rules, real user queries with more varied phrasing could produce different absolute drop sizes; collecting natural reformulations of the same information need would test how well these magnitudes transfer.","The decoupling logic extends naturally to modules the paper explicitly leaves out, such as rerankers, query expansion or disambiguation steps, and jointly trained retriever-generator systems; the same delta-correlation analysis could localize fragility inside those pipelines.","A human rating study of whether the perturbed queries really preserve meaning would strengthen or qualify the claim that the observed degradation is purely surface-level, since the paper's own checks rely on perplexity and embedding similarity rather than human judgment."],"forward_implications":["Retriever-level defenses, such as better typo handling, query normalization, or robust encoders, should move end-to-end RAG robustness more than generator hardening for general-domain question answering.","Closed-book LLM robustness results are a poor proxy for RAG performance; an oracle-context evaluation gives a more meaningful upper bound and separates retrieval failure from generation failure.","Retriever choice involves a robustness trade-off: dense retrievers fail more on typos, sparse retrievers fail more on redundant or ambiguous wording, so the expected user input noise should guide the choice.","The decoupling correlations give practitioners a diagnostic: compute retriever and generator deltas on their own data to know whether to fix retrieval or generation for each perturbation type.","Because end-to-end performance largely tracks retriever performance, improving retrieval robustness to query variation should be a higher priority than tuning the generator for query-level robustness."],"supporting_citations":[{"why":"Supplies the BEIR benchmark, the source of the NQ, HotpotQA, and BioASQ datasets and their document corpora.","marker":"Thakur et al. (2021)"},{"why":"Provides the HotpotQA multi-hop questions whose answer labels are used for retrieval and generation evaluation.","marker":"Yang et al. (2018)"},{"why":"Provides Natural Questions, built from real user queries, with the answer labels matched in Appendix A.4.","marker":"Kwiatkowski et al. (2019a)"},{"why":"Defines the BioASQ biomedical question answering task used as the domain-specific dataset.","marker":"Tsatsaronis et al. (2015)"},{"why":"Supplies BGE-base-en-v1.5, the dense retriever whose Recall@k is measured under all perturbations.","marker":"Xiao et al. (2024)"},{"why":"Supplies Contriever, the second dense retriever used in the pipeline comparisons.","marker":"Izacard et al. (2022)"},{"why":"Defines BM25, the sparse retriever baseline in its flat and multi-field variants.","marker":"Robertson et al. (1995)"},{"why":"Provides the TextAttack library used to generate the 10% and 25% QWERTY typo perturbations.","marker":"Morris et al. (2020)"},{"why":"Supplies the BERGEN benchmarking setup, prompts, and the surface Match metric used for end-to-end and generator evaluation.","marker":"Rau et al. (2024)"}],"fun_headline_variants":["Retriever, not generator, sets RAG query-level robustness","Retriever is the weak link in RAG under query edits","RAG robustness under query changes is retriever-driven","Query rewording hurts RAG retrieval hardest"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion rests on the assumption that the five synthetic perturbation types, LLM-generated redundancy, formal tone, and ambiguity plus two levels of QWERTY typos, are representative of naturally occurring user query variations and preserve the original query's meaning; no human evaluation verifies that.","fun_headline_variants_meta":{"raw":{"variants":["Retriever, not generator, sets RAG query-level robustness","Retriever is the weak link in RAG under query edits","RAG robustness under query changes is retriever-driven","Query rewording hurts RAG retrieval hardest"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1324,"prompt_tokens":925,"completion_tokens":399,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":331}},"tokens_in":541,"tokens_out":399,"duration_ms":4480,"temperature":1.0,"reasoning_tokens":331,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:50:26.366035+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of naturally occurring query variants for the same information need, such as real search-log paraphrases or crowd-written reformulations, run them through the same four retrievers, and compare Recall@k against the original queries; if naturally varied wording does not produce the clear recall drops reported here, the paper's quantitative robustness claim does not generalize. A second check would be having human raters judge whether the LLM-generated redundancy, formal tone, and ambiguity rewrites preserve meaning; if a large share change the question, the observed end-to-end drops conflate semantic change with surface fragility.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines BM25, the sparse retriever baseline in its flat and multi-field variants."}],"review_version":1}