Pith. sign in

REVIEW 4 cited by

Evaluating the Retrieval Robustness of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2505.21870 v1 pith:FKZ2WVLW submitted 2025-05-28 cs.CL cs.AI

Evaluating the Retrieval Robustness of Large Language Models

Shuyang Cao , Karthik Radhakrishnan , David Rosenberg , Steven Lu , Pengxiang Cheng , Lu Wang , Shiyue Zhang This is my paper
classification cs.CL cs.AI
keywords robustnessllmsretrievalretrievedwhetherabilityalwaysbetter
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Retrieval-augmented generation (RAG) generally enhances large language models' (LLMs) ability to solve knowledge-intensive tasks. But RAG may also lead to performance degradation due to imperfect retrieval and the model's limited ability to leverage retrieved content. In this work, we evaluate the robustness of LLMs in practical RAG setups (henceforth retrieval robustness). We focus on three research questions: (1) whether RAG is always better than non-RAG; (2) whether more retrieved documents always lead to better performance; (3) and whether document orders impact results. To facilitate this study, we establish a benchmark of 1500 open-domain questions, each with retrieved documents from Wikipedia. We introduce three robustness metrics, each corresponds to one research question. Our comprehensive experiments, involving 11 LLMs and 3 prompting strategies, reveal that all of these LLMs exhibit surprisingly high retrieval robustness; nonetheless, different degrees of imperfect robustness hinders them from fully utilizing the benefits of RAG.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. The Powerless Noise: How Experimental Settings Shape the Reported Power of Noise

    cs.IR 2026-07 accept novelty 6.5

    The Power-of-Noise effect in RAG is an artifact of restrictive prompting and decoding, not a general benefit of random documents.

  2. The Powerless Noise: How Experimental Settings Shape the Reported Power of Noise

    cs.IR 2026-07 accept novelty 6.0

    The Power-of-Noise effect in RAG is reproducible only under the original constrained setup and disappears or weakens once instruction templates, longer outputs, and modern LLMs are used.

  3. Lost in the Evidence? Reproducing Document Position and Context Size Effects in RAG

    cs.IR 2026-05 unverdicted novelty 5.0

    Reproducibility study shows position and context size effects in RAG depend on topic sampling and retrieval quality, proposes calibration for stable trends, and releases code after finding discrepancies with prior ind...

  4. Retrieval Augmented Generation Framework for the Nepali Legal Domain Question Answering

    cs.CL 2026-04 conditional novelty 4.0

    A RAG pipeline using BM25 retrieval and GPT-o3 generation achieves 91% Precision@1 and 85% truthfulness for Nepali legal question answering on a curated 100-query benchmark.