Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

Investigating the Robustness of Retrieval-Augmented Generation at the Query Level

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that RAG systems are most fragile at the retrieval step: common retrievers degrade substantially under minor query rewording and typos, and end-to-end answer quality largely follows retriever performance.

desk verdict Useful empirical map of RAG query-perturbation robustness, but the ambiguity perturbation likely shifts the answer, so the headline 'minor variations' claim needs a validity check. read the letter →

arxiv 2507.06956 v1 pith:FU7P2BTT submitted 2025-07-09 cs.CL

Sezen Perçin , Xin Su , Qutub Sha Syed , Phillip Howard , Aleksei Kuvshinov , Leo Schwinn , Kay-Ulrich Scholl This is my paper
classification cs.CL
keywords RAGrobustnessqueryperturbationsretrieverretrieval-augmentedgenerationquestionansweringtypographicalerrorsdensevssparseretrievalevaluationframework
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that retrieval-augmented generation (RAG) is much more fragile to how a user words a query than to what the generator does with the retrieved text. Across more than 1,092 experiments with four retrievers, three language models, three question-answering datasets, and five perturbation types, it finds that common retrievers lose substantial accuracy under minor rewording, typo insertion, added redundancy, formal tone shifts, and ambiguity. Running the same perturbations through the full pipeline shows that end-to-end answer quality usually follows the retriever's performance curve rather than the generator's. The paper's practical point is that robustness testing must decouple the modules, because a score from the generator alone, or from the retriever alone, can mislead about where the system will fail. A sympathetic reader would take away that query-level robustness deserves its own evaluation step before deploying a RAG system.

What carries the argument

The load-bearing object is the decoupling evaluation framework: the same perturbed queries are run through four isolated conditions, retriever-only Recall@k, closed-book generation, oracle-context generation, and end-to-end RAG, with five meaning-preserving perturbation types generated by an LLM prompt for redundancy, formal tone, and ambiguity, plus QWERTY-adjacent character replacement at 10% and 25% typo levels. Pearson correlations between each module's performance delta and the end-to-end delta are then used to attribute the pipeline's sensitivity, and a PCA projection of the LLM's internal representations checks whether perturbations scatter the model's understanding even when the gold documents are provided. This machinery is what lets the paper claim that end-to-end RAG trends are predominantly characterized by the performance of the retriever while still flagging the BioASQ cases where generation is the weak link.

What would settle it

Take a set of naturally occurring query variants for the same information need, such as real search-log paraphrases or crowd-written reformulations, run them through the same four retrievers, and compare Recall@k against the original queries; if naturally varied wording does not produce the clear recall drops reported here, the paper's quantitative robustness claim does not generalize. A second check would be having human raters judge whether the LLM-generated redundancy, formal tone, and ambiguity rewrites preserve meaning; if a large share change the question, the observed end-to-end drops conflate semantic change with surface fragility.

Watch

Extended reading notes

Core claim

The central claim is that the query-level robustness of a RAG pipeline is set primarily by the retriever: when a query is lightly perturbed, retriever Recall@k drops markedly, and end-to-end Match performance tracks those retriever drops more closely than it tracks the generator's closed-book or oracle performance. The study isolates each module, using retriever alone, generator in closed-book mode, generator with an oracle retriever, and the full pipeline, across NQ, HotpotQA, and BioASQ with dense and sparse retrievers. It finds that dense retrievers absorb redundant wording better, sparse retrievers absorb typos better, formal tone changes hurt least, and 25% typos hurt most. The one notable exception is BioASQ, where ambiguity and redundancy cause end-to-end drops even when the retriever looks stable, showing that the generator can be the binding constraint on domain-specific biomedical questions. The authors conclude that practitioners should use a decoupling analysis on their own data to identify which module is sensitive to which perturbation type.

Load-bearing premise

The conclusion rests on the assumption that the five synthetic perturbation types, LLM-generated redundancy, formal tone, and ambiguity plus two levels of QWERTY typos, are representative of naturally occurring user query variations and preserve the original query's meaning; no human evaluation verifies that.

Editorial extensions

If this is right

  • Retriever-level defenses, such as better typo handling, query normalization, or robust encoders, should move end-to-end RAG robustness more than generator hardening for general-domain question answering.
  • Closed-book LLM robustness results are a poor proxy for RAG performance; an oracle-context evaluation gives a more meaningful upper bound and separates retrieval failure from generation failure.
  • Retriever choice involves a robustness trade-off: dense retrievers fail more on typos, sparse retrievers fail more on redundant or ambiguous wording, so the expected user input noise should guide the choice.
  • The decoupling correlations give practitioners a diagnostic: compute retriever and generator deltas on their own data to know whether to fix retrieval or generation for each perturbation type.
  • Because end-to-end performance largely tracks retriever performance, improving retrieval robustness to query variation should be a higher priority than tuning the generator for query-level robustness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the synthetic perturbations were produced by an LLM and keyboard-noise rules, real user queries with more varied phrasing could produce different absolute drop sizes; collecting natural reformulations of the same information need would test how well these magnitudes transfer.
  • The decoupling logic extends naturally to modules the paper explicitly leaves out, such as rerankers, query expansion or disambiguation steps, and jointly trained retriever-generator systems; the same delta-correlation analysis could localize fragility inside those pipelines.
  • A human rating study of whether the perturbed queries really preserve meaning would strengthen or qualify the claim that the observed degradation is purely surface-level, since the paper's own checks rely on perplexity and embedding similarity rather than human judgment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies the robustness of retrieval-augmented generation (RAG) at the query level. It introduces five perturbation types: two levels of typographical errors inserted with TextAttack, and three LLM-generated semantic paraphrases (redundancy insertion, formal tone change, ambiguity introduction). Using four retrievers (BGE-base, Contriever, BM25 flat, BM25 multi-field) and three LLM generators (Llama-3.1-8B, Mistral-7B, Qwen2.5-7B), it measures retriever Recall@k, generator closed-book and oracle Match, and end-to-end RAG Match on NQ, HotpotQA, and BioASQ. The main findings are that retriever performance degrades under minor query variations, that dense and sparse retrievers differ in which perturbations hurt most, that end-to-end RAG trends on NQ follow retriever trends while BioASQ shows additional generator-driven drops for ambiguity and redundancy, and that module-level correlations can help practitioners identify the sensitive component. The paper also proposes an evaluation framework and gives recommendations for robustness testing.

Significance. If the central claims withstand scrutiny, this is a valuable empirical study: it systematically measures query-level robustness across a broad matrix of retrievers, generators, datasets, and perturbation types, and it proposes a module-decoupling evaluation protocol that practitioners can adapt to their own pipelines. The deliberate choice of a model-free Match metric aids reproducibility, and the promise to release data and code is a concrete asset. The work is not circular in the technical sense: the findings are empirical measurements rather than predictions obtained by fitting free parameters, and the recommendations are derived from the measured results. However, the validity of the perturbation battery is the load-bearing assumption, and the current evidence does not establish that all five perturbation types preserve the original answer; this limits the scope of the headline claim about 'minor query variations'.

major comments (3)
  1. [Section 3 and Appendix A.2; Table 3] The central claim that the tested perturbations are 'minor query variations' that preserve semantics is not established for the ambiguity and redundancy perturbations. The prompt in Table 3 instructs GPT-4o to make the query 'unclear to answer by introducing ambiguity,' which can change the information need; the example in Section 3 changes 'when does the cannes film festival take place' to 'when might the Cannes Film Festival be held?', a question whose answer need not coincide with the original gold answer. The validation in Appendix A.2 uses only perplexity and e5-base cosine similarity, neither of which verifies that the original gold answer is still a correct answer for the perturbed query. Internal evidence in Section 5.3 strengthens this concern: ambiguity insertion lowers performance in the oracle setting (gold documents provided) across all datasets, and redundancy causes drastic oracle drops on BioASQ; if the gold documents are in context and the question still asks for the same fact, a robust generator should not degrade this much. These drops are consistent with the perturbed queries having different or underspecified answers, so the end-to-end and retriever degradation may partly measure a changed task rather than surface fragility. I recommend adding a human or expert evaluation of answer preservation for each perturbed sample, filtering out or annotating samples where the gold answer is no longer valid, and clearly separating intent-changing perturbations (ambiguity, possibly redundancy) from pure surface variations in the claims.
  2. [Section 5.2-5.4 and Table 4] The reported averages over five perturbation samples per original query are presented without variance, confidence intervals, or significance tests. The qualitative ordering of retriever robustness across perturbation types (e.g., 'dense retrievers are more robust against the redundant information,' 'sparse methods performances are more robust against the typos,' and 'typo insertions are particularly impactful in the closed-book setting') could be driven by sampling noise, especially because the five samples are arbitrary outputs of GPT-4o or TextAttack rather than an independently drawn sample from a well-defined distribution. I request that the authors provide confidence intervals or bootstrap estimates for the average Recall@k and Match values, and report significance tests or at least effect sizes for the key pairwise comparisons that support the conclusions.
  3. [Section 5.4, Table 2 and Tables 5-6] The Pearson correlations between module-level and end-to-end performance discrepancies are computed on per-sample data that are clustered by original query (five perturbed samples per query), yet the analysis ignores this clustering and reports no confidence intervals or tests for the differences between correlations. The conclusion that 'different dominant factors exist within the pipeline for different perturbation types' rests on comparisons of correlations that are small in magnitude (e.g., 0.05 vs 0.35 in Table 2) and may be within statistical noise. The authors should either provide uncertainty estimates for these correlations (e.g., cluster-bootstrap confidence intervals) and test whether the differences are significant, or soften the claims accordingly.
minor comments (6)
  1. [Section 6] The text contains a duplicated phrase: 'we recommend that practitioners use our analysis framework on their we recommend that practitioners use our analysis framework on their own data'; this should be corrected.
  2. [References] The reference list contains duplicate entries: Kwiatkowski et al. 2019a and 2019b are the same paper, and Zhao et al. 2024b and 2024c have the same title and venue; please merge or remove duplicates.
  3. [Appendix A.2] There is a typo in 'langueg processing community'; it should be 'language processing community'.
  4. [Table 1] The caption defines PERT as 'Number of perturbed samples for each perturbation type,' but the values appear to be the number of original queries used to generate perturbed samples; please clarify the definition.
  5. [Section 5.4, Figure 5] The conclusion that redundancy and ambiguity 'result in more scattered internal representations' is based on visual inspection of PCA plots; a quantitative dispersion measure (e.g., average pairwise distance or a scatter index) would make this claim verifiable.
  6. [Appendix A.2] The phrase 'more redundant correpondents' should be 'more redundant counterparts'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all claims are empirical measurements over fixed datasets and models; no parameter is fitted and then renamed as a prediction.

full rationale

The paper's central claims—retriever Recall@5 drops under query perturbations and end-to-end Match largely tracks retriever trends—are measurement results over fixed BEIR datasets (NQ, HotpotQA, BioASQ) and fixed retriever/generator combinations. There is no derivation in which an output quantity is defined in terms of, or fitted to, the quantity it is used to predict. The proposed evaluation framework is presented as a methodology and demonstrated on the same experiments, but this is normal practice; the recommendations are not presented as independent predictions validated by those same experiments. The only self-referential aspect is that the framework is proposed and then used, which does not constitute a logical circle. The assumption that GPT-4o-generated perturbations preserve query semantics is an empirical validity/correctness concern—the Appendix's perplexity and e5 cosine checks do not verify that gold answers are preserved—but it is not circularity, because the paper does not define success in terms of its perturbation-generation prompts. No load-bearing self-citations or imported uniqueness theorems appear.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on four domain assumptions: semantic preservation of the perturbations, the validity of the Match metric, correctness of the precomputed indices and corpus splits, and representativeness of synthetic query variations. No free parameters are fitted to data, and no new entities are postulated. The analysis itself is empirical, so the axioms are about measurement validity rather than mathematical derivation.

assumptions (4)
  • domain assumption The five perturbation types preserve the semantic content of the original queries, or change it only minimally.
    Section 3 states this requirement; Appendix A.2 checks perplexity and cosine similarity but there is no human validation of meaning preservation for the GPT-4o-generated variants.
  • domain assumption The Match metric, based on surface span overlap, is an adequate measure of answer correctness for these QA datasets.
    Section 4.3 states a model-free surface matching metric is chosen for reproducibility; this can misjudge correct answers phrased differently from the gold label, especially under formal or ambiguous perturbations.
  • domain assumption Precomputed Pyserini indices and the BEIR corpus splits correctly represent the intended knowledge bases for the three datasets.
    Section 4.2 relies on publicly available precomputed indices from Pyserini; any mismatch between index version and corpus would affect all retrieval numbers.
  • domain assumption GPT-4o-generated samples are representative of real user query variations.
    Section 3.1 generates the prompt-based perturbations with GPT-4o; no human study or real-world query log is used to validate representativeness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Investigating the Robustness of Retrieval-Augmented Generation at the Query Level." pith.science (2026). https://pith.science/paper/FU7P2BTT

@misc{pith2026250706956,
  author       = {Pith},
  title        = {Pith review of: Investigating the Robustness of Retrieval-Augmented Generation at the Query Level},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FU7P2BTT}},
  note         = {Machine review of arXiv:2507.06956}
}
read the original abstract

Large language models (LLMs) are very costly and inefficient to update with new information. To address this limitation, retrieval-augmented generation (RAG) has been proposed as a solution that dynamically incorporates external knowledge during inference, improving factual consistency and reducing hallucinations. Despite its promise, RAG systems face practical challenges-most notably, a strong dependence on the quality of the input query for accurate retrieval. In this paper, we investigate the sensitivity of different components in the RAG pipeline to various types of query perturbations. Our analysis reveals that the performance of commonly used retrievers can degrade significantly even under minor query variations. We study each module in isolation as well as their combined effect in an end-to-end question answering setting, using both general-domain and domain-specific datasets. Additionally, we propose an evaluation framework to systematically assess the query-level robustness of RAG pipelines and offer actionable recommendations for practitioners based on the results of more than 1092 experiments we performed.

Figures

Figures reproduced from arXiv: 2507.06956 by the authors.

Figure 1
Figure 1. Illustration of our approach to evaluating RAG robustness. (a) Perturbations are generated via prompting [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Recall@k results obtained with different retrievers on HotpotQA with respect to the changing "k" [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. LLM performances under different pertur [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The average end-to-end results on NQ dataset according to "Match" metric. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Representation of samples taken from Llama-3.1-8B-Instruct for the BioASQ dataset for the oracle (upper) [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Perplexity and semantic similarity of the generated samples for different perturbations and datasets. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Effect of Top@k choice on RAG performance [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Remaining retriever performances with Recall@k and Precision@5 metrics on all datasets with respect to [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: End-to-end RAG results with all combinations using the "Match" metric. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Paraphrase Brittleness in Production Retrieval-Augmented Commercial Recommendation: Reproducibility Below the Rerun-Stability Baseline

    cs.IR 2026-05 unverdicted novelty 6.0 of 10

    Paraphrase Jaccard similarity of 0.135-0.288 falls below the 0.50-0.61 same-prompt rerun baseline on OpenAI and Anthropic models, showing prompt wording dominates buyer intent in commercial recommendations.

  2. Evaluating the Robustness of Retrieval-Augmented Generation to Adversarial Evidence in the Health Domain

    cs.IR 2025-09 conditional novelty 6.0 of 10

    Misleading health documents in RAG context sharply lower LLM accuracy, and heavily helpful-biased retrieval pools restore it.

Reference graph

Works this paper leans on

44 extracted references · 10 canonical work pages · cited by 2 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Negar Arabzadeh, Radin Hamidi Rad, Maryam Khodabakhsh, and Ebrahim Bagheri. 2023. Noisy perturbations for estimating query difficulty in dense retrievers. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 3722--3727

    work page 2023

  4. [4]

    Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2023. https://arxiv.org/abs/2309.01431 Benchmarking large language models in retrieval-augmented generation . Preprint, arXiv:2309.01431

    arXiv 2023

  5. [5]

    Feiteng Fang, Yuelin Bai, Shiwen Ni, Min Yang, Xiaojun Chen, and Ruifeng Xu. 2024. https://arxiv.org/abs/2405.20978 Enhancing noise robustness of retrieval-augmented language models with adaptive adversarial training . Preprint, arXiv:2405.20978

    arXiv 2024

  6. [6]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. https://arxiv.org/abs/2407.21783 The llama 3...

    arXiv 2024

  7. [7]

    Rujun Han, Peng Qi, Yuhao Zhang, Lan Liu, Juliette Burger, William Yang Wang, Zhiheng Huang, Bing Xiang, and Dan Roth. 2023. Robustqa: Benchmarking the robustness of domain adaptation for open-domain question answering. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4294--4311

    work page 2023

  8. [8]

    Jennifer Hsia, Afreen Shaikh, Zhiruo Wang, and Graham Neubig. 2024. https://arxiv.org/abs/2403.09040 Ragged: Towards informed design of retrieval augmented generation systems . Preprint, arXiv:2403.09040

    arXiv 2024

Show all 44 references
  1. [9]

    Zhibo Hu, Chen Wang, Yanfeng Shu, Helen, Paik, and Liming Zhu. 2024. https://arxiv.org/abs/2402.07179 Prompt perturbation in retrieval-augmented generation based large language models . Preprint, arXiv:2402.07179

    2024 arXiv

  2. [10]

    Quzhe Huang, Mingxu Tao, Zhenwei An, Chen Zhang, Cong Jiang, Zhibin Chen, Zirui Wu, and Yansong Feng. 2023. https://doi.org/10.48550/ARXIV.2305.15062 Lawyer llama technical report . CoRR, abs/2305.15062

    2023 doi arXiv

  3. [11]

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. https://arxiv.org/abs/2112.09118 Unsupervised dense information retrieval with contrastive learning . Preprint, arXiv:2112.09118

    2022 arXiv

  4. [12]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  5. [14]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019 doi

  6. [15]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  7. [16]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Informati...

    2020

  8. [17]

    Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. https://arxiv.org/abs/2102.10073 Pyserini: An easy-to-use python toolkit to support replicable ir research with sparse and dense representations . Preprint, arXiv:2102.10073

    2021 arXiv

  9. [18]

    Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, Luke Zettlemoyer, and Scott Yih. 2024. https://arxiv.org/abs/2310.01352 Ra-dit: Retrieval-augmented dual instruction tuning . Preprint, ar...

    2024 arXiv

  10. [19]

    Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Wei Chen, and Xueqi Cheng. 2023. On the robustness of generative retrieval models: An out-of-distribution perspective. arXiv preprint arXiv:2306.12756

    2023 arXiv

  11. [20]

    Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2024. https://arxiv.org/abs/2407.06992 Robust neural information retrieval: An adversarial and out-of-distribution perspective . Preprint, arXiv:2407.06992

    2024 arXiv

  12. [21]

    Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi

    John X. Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. https://arxiv.org/abs/2005.05909 Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp . Preprint, arXiv:2005.05909

    2020 arXiv

  13. [22]

    Gustavo Penha, Arthur Câmara, and Claudia Hauff. 2022. https://arxiv.org/abs/2111.13057 Evaluating the robustness of retrieval pipelines with query variation generators . Preprint, arXiv:2111.13057

    2022 arXiv

  14. [23]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  15. [24]

    Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11:1316--1331

    2023

  16. [25]

    David Rau, Hervé Déjean, Nadezhda Chirkova, Thibault Formal, Shuai Wang, Vassilina Nikoulina, and Stéphane Clinchant. 2024. https://arxiv.org/abs/2407.01102 Bergen: A benchmarking library for retrieval-augmented generation . Preprint, arXiv:2407.01102

    2024 arXiv

  17. [26]

    Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, and 1 others. 1995. Okapi at trec-3. Nist Special Publication Sp, 109:109

    1995

  18. [27]

    implicit

    Xiaoyu Shen, Rexhina Blloshmi, Dawei Zhu, Jiahuan Pei, and Wei Zhang. 2024. Assessing “implicit” retrieval robustness of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8988--9003

    2024

  19. [28]

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023. https://arxiv.org/abs/2302.00093 Large language models can be easily distracted by irrelevant context . Preprint, arXiv:2302.00093

    2023 arXiv

  20. [29]

    Georgios Sidiropoulos and Evangelos Kanoulas. 2022. Analysing the robustness of dual encoders for dense retrieval against misspellings. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2132--2136

    2022

  21. [30]

    Jiuding Sun, Chantal Shaib, and Byron C. Wallace. 2023. https://arxiv.org/abs/2306.11270 Evaluating the zero-shot robustness of instruction-tuned language models . Preprint, arXiv:2306.11270

    2023 arXiv

  22. [31]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  23. [32]

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. https://arxiv.org/abs/2104.08663 Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models . Preprint, arXiv:2104.08663

    2021 arXiv

  24. [33]

    George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael Alvers, Dirk Weißenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, Yannis Almirantis, John Pavlopoulos, Nicolas Baskiotis, Patrick Gallinari, Thier...

    2015 doi

  25. [34]

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672

    2024 arXiv

  26. [35]

    Chong Xiang, Tong Wu, Zexuan Zhong, David Wagner, Danqi Chen, and Prateek Mittal. 2024. https://arxiv.org/abs/2405.15556 Certifiably robust rag against retrieval corruption . Preprint, arXiv:2405.15556

    2024

  27. [36]

    Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. https://arxiv.org/abs/2309.07597 C-pack: Packed resources for general chinese embeddings . Preprint, arXiv:2309.07597

    2024 arXiv

  28. [37]

    Guicai Xie, Ke Zhang, Lei Duan, Wei Zhang, and Zeqian Huang. 2024. https://aclanthology.org/2024.lrec-main.1470/ Typos correction training against misspellings from text-to-text transformers . In Proceedings of the 2024 Joint International Conference on Computational Linguisti...

    2024

  29. [38]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. https://doi.org/10.18653/v1/D18-1259 H otpot QA : A dataset for diverse, explainable multi-hop question answering . In Proceedings of the 2018 Conference...

    2018 doi

  30. [39]

    Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2024. https://arxiv.org/abs/2310.01558 Making retrieval-augmented language models robust to irrelevant context . Preprint, arXiv:2310.01558

    2024 arXiv

  31. [40]

    Yi Zhang, Yun Tang, Wenjie Ruan, Xiaowei Huang, Siddartha Khastgir, Paul Jennings, and Xingyu Zhao. 2025. Protip: Probabilistic robustness verification on text-to-image diffusion models against stochastic perturbation. In European Conference on Computer Vision, pages 455--472....

    2025

  32. [41]

    Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. 2024 a . https://doi.org/10.1145/3639372 Explainability for large language models: A survey . ACM Trans. Intell. Syst. Technol., 15(2)

    2024 doi

  33. [43]

    Yukun Zhao, Lingyong Yan, Weiwei Sun, Guoliang Xing, Shuaiqiang Wang, Chong Meng, Zhicong Cheng, Zhaochun Ren, and Dawei Yin. 2024 c . https://arxiv.org/abs/2403.14221 Improving the robustness of large language models via consistency alignment . Preprint, arXiv:2403.14221

    2024 arXiv

  34. [44]

    Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Zhenqiang Gong, and Xing Xie. 2024 a . https://arxiv.org/abs/2306.04528 Promptrobust: Towards evaluating the robustness of large language models on adversarial promp...

    2024 arXiv

  35. [45]

    Kaijie Zhu, Qinlin Zhao, Hao Chen, Jindong Wang, and Xing Xie. 2024 b . Promptbench: A unified library for evaluation of large language models. Journal of Machine Learning Research, 25(254):1--22

    2024

  36. [46]

    Shengyao Zhuang and Guido Zuccon. 2022. https://doi.org/10.1145/3477495.3531951 Characterbert and self-teaching for improving the robustness of dense retrievers on queries with typos . In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in...

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.