REVIEW 2 major objections 1 minor 33 references
When Poison Fails After Retrieval: Revisiting Corpus Poisoning under Chunking and Reranking Pipelines
T0 review · 2 major / 1 minor · reviewed 2026-06-27 · grok-4.3
Pith's one-line read Existing corpus poisoning attacks on RAG systems lose effectiveness after reranking because chunking fragments adversarial signals and rerankers favor coherent local passages.
desk verdict The paper flags that chunking and reranking break many existing corpus poisoning attacks and offers CRCP as a joint-optimization fix, but the abstract leaves the actual gains and generalization unclear. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Chunk-aware and Rerank-Consistent Poisoning (CRCP), a framework that explicitly models chunking transformations during optimization to generate adversarial passages.
What would settle it
A test in which CRCP attack success rates drop to the level of prior methods when evaluated on a reranker or chunk size outside the paper's experimental set.
Extended reading notes
Core claim
Retrieval granularity mismatch causes existing attacks to fail after reranking; jointly optimizing retrieval relevance, reranker consistency, and chunk-boundary robustness produces locally self-contained adversarial passages that remain effective under varying chunking configurations and reranking strategies.
Load-bearing premise
Retrieval granularity mismatch is the primary cause of attack failure and the chosen benchmarks plus rerankers adequately represent production RAG pipelines.
Editorial extensions
If this is right
- Existing attacks become highly sensitive to chunk size and reranking strategies.
- CRCP produces substantially higher attack success rates across realistic multi-stage pipelines.
- Poisoning effectiveness must be measured after the full sequence of chunking, retrieval, and reranking rather than at retrieval alone.
- Security evaluations of RAG systems require testing consistency across retrieval stages.
Reading between the lines
- Attackers may need to simulate the target pipeline's chunking and reranking steps when crafting poisons.
- Defenses could prioritize reranker training that penalizes locally coherent but globally manipulated passages.
- The same granularity-mismatch issue could appear in other multi-stage retrieval systems beyond RAG.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript claims that existing corpus poisoning attacks on RAG systems substantially degrade after reranking despite high retrieval-stage relevance, due to retrieval granularity mismatch where document-level signals fragment during chunking and rerankers prefer locally coherent passages. It proposes CRCP (Chunk-aware and Rerank-Consistent Poisoning), a framework that jointly optimizes retrieval relevance, reranker consistency, and chunk-boundary robustness by explicitly modeling chunking transformations. Experiments on standard RAG benchmarks with multiple retrievers and rerankers are reported to show that prior methods are sensitive to chunk size and reranking while CRCP achieves substantially higher attack success rates and stronger robustness, highlighting a realism gap in RAG security evaluation.
Significance. If the results hold, the work identifies an important realism gap in current RAG security evaluations and reframes corpus poisoning as a multi-stage retrieval consistency problem rather than retrieval-only. The proposal of a chunk-aware optimization method and the use of multiple retrievers/rerankers in the evaluation are strengths that could guide more realistic attack research.
major comments (2)
- [Abstract, paragraph beginning 'We identify retrieval granularity mismatch as a key reason'] Abstract, paragraph beginning 'We identify retrieval granularity mismatch as a key reason': the claim that granularity mismatch is the primary cause of attack failure lacks direct supporting evidence such as ablations across reranker families or out-of-distribution chunkers. This is load-bearing for the central claim that CRCP's gains and the multi-stage consistency framing transfer beyond the specific open-source models and fixed chunk sizes tested.
- [Abstract (experimental claims)] Abstract (experimental claims): the abstract states that CRCP achieves substantially higher attack success rates and stronger robustness but supplies no metrics, dataset sizes, number of trials, statistical tests, or ablation details. Without these, it is impossible to assess whether the data support the claims of superiority over existing attacks, which is central to the contribution.
minor comments (1)
- The abstract would be strengthened by including at least one key quantitative result (e.g., ASR delta or robustness metric) to convey the magnitude of the reported improvements.
Simulated Author's Rebuttal
We thank the referee for their constructive feedback. We address the major comments point by point below.
read point-by-point responses
-
Referee: [Abstract, paragraph beginning 'We identify retrieval granularity mismatch as a key reason'] Abstract, paragraph beginning 'We identify retrieval granularity mismatch as a key reason': the claim that granularity mismatch is the primary cause of attack failure lacks direct supporting evidence such as ablations across reranker families or out-of-distribution chunkers. This is load-bearing for the central claim that CRCP's gains and the multi-stage consistency framing transfer beyond the specific open-source models and fixed chunk sizes tested.
Authors: The manuscript reports results across multiple retrievers and rerankers that demonstrate degradation after reranking and sensitivity to chunk size. We acknowledge that dedicated ablations on out-of-distribution chunkers would further strengthen the generality claim. We will add such experiments in the revision. revision: yes
-
Referee: [Abstract (experimental claims)] Abstract (experimental claims): the abstract states that CRCP achieves substantially higher attack success rates and stronger robustness but supplies no metrics, dataset sizes, number of trials, statistical tests, or ablation details. Without these, it is impossible to assess whether the data support the claims of superiority over existing attacks, which is central to the contribution.
Authors: Detailed metrics, dataset sizes, trial counts, and ablation results appear in Sections 4 and 5. We will revise the abstract to include key quantitative results (e.g., attack success rate deltas) to make the claims more self-contained. revision: yes
Circularity Check
No circularity: claims rest on experimental comparisons
full rationale
The paper presents an empirical study proposing CRCP based on observed degradation of prior attacks after reranking and chunking. No equations, fitted parameters, self-definitional constructs, or load-bearing self-citations appear in the abstract or described structure. Central claims (higher ASR and robustness for CRCP) are supported by reported benchmark results rather than reducing to inputs by construction. The granularity mismatch observation is framed as an experimental finding, not a self-referential derivation.
Assumptions & free parameters
assumptions (1)
- domain assumption Standard RAG benchmarks and common retriever/reranker implementations are representative of production pipelines.
invented entities (1)
-
CRCP (Chunk-aware and Rerank-Consistent Poisoning)
Cite this review
Pith. "Pith review of When Poison Fails After Retrieval: Revisiting Corpus Poisoning under Chunking and Reranking Pipelines." pith.science (2026). https://pith.science/paper/W2VEWW5S
@misc{pith2026260611265,
author = {Pith},
title = {Pith review of: When Poison Fails After Retrieval: Revisiting Corpus Poisoning under Chunking and Reranking Pipelines},
year = {2026},
howpublished = {\url{https://pith.science/paper/W2VEWW5S}},
note = {Machine review of arXiv:2606.11265}
}
read the original abstract
Retrieval-Augmented Generation (RAG) systems are vulnerable to corpus poisoning attacks that manipulate downstream model outputs through malicious knowledge injection. Existing studies mainly evaluate poisoning under simplified retrieval settings, overlooking practical RAG pipelines involving document chunking, dense retrieval, reranking, and grounded generation. In this paper, we revisit corpus poisoning under realistic multi-stage retrieval pipelines and show that many existing attacks substantially degrade after reranking despite achieving high retrieval-stage relevance. We identify retrieval granularity mismatch as a key reason for this failure: document-level adversarial signals are often fragmented during chunking, while rerankers favor locally coherent and answer-bearing passages rather than globally optimized semantic similarity. Based on this observation, we propose Chunk-aware and Rerank-Consistent Poisoning (CRCP), a poisoning framework that jointly optimizes retrieval relevance, reranker consistency, and chunk-boundary robustness. CRCP explicitly models chunking transformations during optimization to generate locally self-contained adversarial passages that remain effective under varying chunking configurations. Experiments on standard RAG benchmarks with multiple retrievers and rerankers show that existing poisoning methods are highly sensitive to chunk size and reranking strategies, whereas CRCP achieves substantially higher attack success rates and stronger robustness across realistic retrieval pipelines. Our findings highlight an important realism gap in current RAG security evaluation and suggest that poisoning in modern RAG systems should be studied as a multi-stage retrieval consistency problem rather than a retrieval-only problem.
Figures
Reference graph
Works this paper leans on
-
[1]
Retrieval-augmented generation for knowledge-intensive nlp tasks,
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K ¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt ¨aschel, et al., “Retrieval-augmented generation for knowledge-intensive nlp tasks,”NeurIPS, 2020
2020
-
[2]
Dense passage retrieval for open-domain question answering,
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih, “Dense passage retrieval for open-domain question answering,” inEMNLP, 2020
2020
-
[3]
Improving language models by retrieving from trillions of tokens,
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean- Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al., “Improving language models by retrieving from trillions of tokens,” inICML, 2022
2022
-
[4]
Wikichat: Stopping the hallucination of large language model chatbots by few-shot grounding on wikipedia,
Sina Semnani, Violet Yao, Heidi Zhang, and Monica Lam, “Wikichat: Stopping the hallucination of large language model chatbots by few-shot grounding on wikipedia,” inFindings of EMNLP, 2023
2023
-
[5]
Retrieval-based prompt selection for code-related few-shot learning,
Noor Nashid, Mifta Sintaha, and Ali Mesbah, “Retrieval-based prompt selection for code-related few-shot learning,” inICSE, 2023
2023
-
[6]
Docprompting: Generating code by retrieving the docs,
Shuyan Zhou, Uri Alon, Frank F. Xu, Zhengbao Jiang, and Graham Neubig, “Docprompting: Generating code by retrieving the docs,” in ICLR, 2023
2023
-
[7]
Phantom: General backdoor attacks on retrieval augmented language generation,
Harsh Chaudhari, Giorgio Severi, John Abascal, Anshuman Suri, Matthew Jagielski, Christopher A. Choquette-Choo, Milad Nasr, Cristina Nita-Rotaru, and Alina Oprea, “Phantom: General backdoor attacks on retrieval augmented language generation,”ACM Trans. AI Secur . Priv., 2026
2026
-
[8]
Typos that broke the rag’s back: Genetic attack on rag pipeline by simulating documents in the wild via low-level perturbations,
Sukmin Cho, Soyeong Jeong, Jeongyeon Seo, Taeho Hwang, and Jong C Park, “Typos that broke the rag’s back: Genetic attack on rag pipeline by simulating documents in the wild via low-level perturbations,” in Findings of EMNLP, 2024
2024
Show all 33 references
-
[9]
Hotflip: White-box adversarial examples for text classification,
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou, “Hotflip: White-box adversarial examples for text classification,” inACL, 2018
2018
-
[10]
Ctrlrag: Black- box document poisoning attacks for retrieval-augmented generation of large language models,
Runqi Sui, Xuejing Yuan, Di Tang, and Baojiang cui, “Ctrlrag: Black- box document poisoning attacks for retrieval-augmented generation of large language models,” inAAMAS, 2026
2026
-
[11]
Poisoning retrieval corpora by injecting adversarial passages,
Zexuan Zhong, Ziqing Huang, Alexander Wettig, and Danqi Chen, “Poisoning retrieval corpora by injecting adversarial passages,” in EMNLP, 2023
2023
-
[12]
A survey on rag with llms,
Muhammad Arslan, Hussam Ghanem, Saba Munawar, and Christophe Cruz, “A survey on rag with llms,”Procedia computer science, 2024
2024
-
[13]
A survey on rag meeting llms: Towards retrieval-augmented large language models,
Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li, “A survey on rag meeting llms: Towards retrieval-augmented large language models,” inSIGKDD, 2024
2024
-
[14]
Re2g: Retrieve, rerank, generate,
Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, Ankita Naik, Pengshan Cai, and Alfio Gliozzo, “Re2g: Retrieve, rerank, generate,” inNAACL, 2022
2022
-
[15]
Dynamicrag: Leveraging outputs of large language model as feedback for dynamic reranking in retrieval-augmented generation,
Jiashuo Sun, Xianrui Zhong, Sizhe Zhou, and Jiawei Han, “Dynamicrag: Leveraging outputs of large language model as feedback for dynamic reranking in retrieval-augmented generation,” inNeurIPS, 2025
2025
-
[16]
Prorank: Progressive context refinement for reliable retrieval-augmented generation,
Yihui Yang, Jianhui Ma, Xu An, Zheng Zhang, Mingfan Pan, Wuhong Wang, and Huiming Ding, “Prorank: Progressive context refinement for reliable retrieval-augmented generation,” inICASSP, 2026
2026
-
[17]
Maximizing rag efficiency: A comparative analysis of rag methods,
Tolga S ¸akar and Hakan Emekci, “Maximizing rag efficiency: A comparative analysis of rag methods,”Natural Language Processing, 2025
2025
-
[18]
Agentpoison: Red-teaming llm agents via poisoning memory or knowl- edge bases,
Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li, “Agentpoison: Red-teaming llm agents via poisoning memory or knowl- edge bases,”NeurIPS, 2024
2024
-
[19]
Poisonedrag: Knowledge corruption attacks to retrieval-augmented generation of large language models,
Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia, “Poisonedrag: Knowledge corruption attacks to retrieval-augmented generation of large language models,” inUSENIX Security, 2025
2025
-
[20]
Glue pizza and eat rocks- exploiting vulnerabilities in retrieval-augmented generative models,
Zhen Tan, Chengshuai Zhao, Raha Moraffah, Yifan Li, Song Wang, Jundong Li, Tianlong Chen, and Huan Liu, “Glue pizza and eat rocks- exploiting vulnerabilities in retrieval-augmented generative models,” in EMNLP, 2024
2024
-
[21]
Joint-gcg: Unified gradient- based poisoning attacks on retrieval-augmented generation systems,
Haowei Wang, Rupeng Zhang, Junjie Wang, Mingyang Li, Yuekai Huang, Dandan Wang, and Qing Wang, “Joint-gcg: Unified gradient- based poisoning attacks on retrieval-augmented generation systems,” in AAAI, 2026
2026
-
[22]
The RAG paradox: A black-box attack exploiting unintentional vulnerabilities in retrieval-augmented generation systems,
Chanwoo Choi, Jinsoo Kim, Sukmin Cho, Soyeong Jeong, and Buru Chang, “The RAG paradox: A black-box attack exploiting unintentional vulnerabilities in retrieval-augmented generation systems,” inFindings of EMNLP, 2025
2025
-
[23]
Cpa-rag: Covert poisoning attacks on retrieval-augmented generation in large language models,
Chunyang Li, Junwei Zhang, Anda Cheng, Zhuo Ma, Xinghua Li, and Jianfeng Ma, “Cpa-rag: Covert poisoning attacks on retrieval-augmented generation in large language models,”arXiv preprint arXiv:2505.19864, 2025
2025
-
[24]
RevPRAG: Revealing poisoning attacks in retrieval-augmented generation through LLM activation analysis,
Xue Tan, Hao Luan, Mingyu Luo, Xiaoyan Sun, Ping Chen, and Jun Dai, “RevPRAG: Revealing poisoning attacks in retrieval-augmented generation through LLM activation analysis,” inFindings of EMNLP, 2025
2025
-
[25]
Certifiably robust RAG against retrieval corruption,
Chong Xiang, Tong Wu, Zexuan Zhong, David Wagner, Danqi Chen, and Prateek Mittal, “Certifiably robust RAG against retrieval corruption,” in ICML 2024 Next Generation of AI Safety Workshop, 2024
2024
-
[26]
Astute rag: Overcoming imperfect retrieval augmentation and knowledge conflicts for large language models,
Fei Wang, Xingchen Wan, Ruoxi Sun, Jiefeng Chen, and Sercan O Arik, “Astute rag: Overcoming imperfect retrieval augmentation and knowledge conflicts for large language models,” inACL, 2025
2025
-
[27]
Instructrag: Instructing retrieval-augmented generation via self-synthesized rationales,
Zhepei Wei, Wei-Lin Chen, and Yu Meng, “Instructrag: Instructing retrieval-augmented generation via self-synthesized rationales,” inICLR, 2025
2025
-
[28]
Trustrag: enhancing robustness and trustworthiness in retrieval-augmented gen- eration,
Huichi Zhou, Kin-Hei Lee, Zhonghao Zhan, Yue Chen, Zhenhao Li, Zhaoyang Wang, Hamed Haddadi, and Emine Yilmaz, “Trustrag: enhancing robustness and trustworthiness in retrieval-augmented gen- eration,” inAAAI Workshop, 2026
2026
-
[29]
Secon-rag: A two-stage semantic filtering and conflict-free framework for trustworthy rag,
Xiaonan Si, Meilin Zhu, Simeng Qin, Lijia Yu, Lijun Zhang, Shuaitong Liu, Xinfeng Li, Ranjie Duan, Yang Liu, and Xiaojun Jia, “Secon-rag: A two-stage semantic filtering and conflict-free framework for trustworthy rag,”NeurIPS, 2025
2025
-
[30]
Ms marco: A human generated machine reading comprehension dataset,
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al., “Ms marco: A human generated machine reading comprehension dataset,”arXiv preprint arXiv:1611.09268, 2016
2016 arXiv
-
[31]
Natural questions: a benchmark for question answering research,
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al., “Natural questions: a benchmark for question answering research,”Transactions of the Association for Compu...
2019
-
[32]
Hotpotqa: A dataset for diverse, explainable multi-hop question answering,
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning, “Hotpotqa: A dataset for diverse, explainable multi-hop question answering,” in EMNLP, 2018
2018
-
[33]
Reconstructing context: Evaluating advanced chunking strategies for retrieval-augmented generation,
Carlo Merola and Jaspinder Singh, “Reconstructing context: Evaluating advanced chunking strategies for retrieval-augmented generation,” in International Workshop on Knowledge-Enhanced Information Retrieval. Springer, 2025
2025
Reviewed June 27, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.