Pith. sign in

REVIEW 2 major objections 1 minor 33 references

When Poison Fails After Retrieval: Revisiting Corpus Poisoning under Chunking and Reranking Pipelines

T0 review · 2 major / 1 minor · reviewed 2026-06-27 · grok-4.3

Pith's one-line read Existing corpus poisoning attacks on RAG systems lose effectiveness after reranking because chunking fragments adversarial signals and rerankers favor coherent local passages.

desk verdict The paper flags that chunking and reranking break many existing corpus poisoning attacks and offers CRCP as a joint-optimization fix, but the abstract leaves the actual gains and generalization unclear. read the letter →

arxiv 2606.11265 v1 pith:W2VEWW5S submitted 2026-06-09 cs.CR cs.AI

classification cs.CRcs.AI
keywords corpuspoisoningRAGsecurityretrieval-augmentedgenerationrerankingchunkingadversarialattacksmulti-stagepipelines
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper shows that many poisoning methods achieve high relevance at the initial retrieval stage but degrade sharply once chunking and reranking are added to the pipeline. It traces the drop to a mismatch where document-level attack signals break apart during chunking and rerankers select locally answer-bearing text rather than globally optimized similarity. To counter this, the authors introduce CRCP, which optimizes passages for retrieval relevance, reranker consistency, and robustness to different chunk boundaries at the same time. Experiments across standard benchmarks and multiple retriever-reranker combinations find that prior attacks are sensitive to chunk size and reranking choices while CRCP sustains higher attack success rates. The work concludes that corpus poisoning must be treated as a multi-stage consistency problem rather than a retrieval-only one.

What carries the argument

Chunk-aware and Rerank-Consistent Poisoning (CRCP), a framework that explicitly models chunking transformations during optimization to generate adversarial passages.

What would settle it

A test in which CRCP attack success rates drop to the level of prior methods when evaluated on a reranker or chunk size outside the paper's experimental set.

Watch

Extended reading notes

Core claim

Retrieval granularity mismatch causes existing attacks to fail after reranking; jointly optimizing retrieval relevance, reranker consistency, and chunk-boundary robustness produces locally self-contained adversarial passages that remain effective under varying chunking configurations and reranking strategies.

Load-bearing premise

Retrieval granularity mismatch is the primary cause of attack failure and the chosen benchmarks plus rerankers adequately represent production RAG pipelines.

Editorial extensions

If this is right

  • Existing attacks become highly sensitive to chunk size and reranking strategies.
  • CRCP produces substantially higher attack success rates across realistic multi-stage pipelines.
  • Poisoning effectiveness must be measured after the full sequence of chunking, retrieval, and reranking rather than at retrieval alone.
  • Security evaluations of RAG systems require testing consistency across retrieval stages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Attackers may need to simulate the target pipeline's chunking and reranking steps when crafting poisons.
  • Defenses could prioritize reranker training that penalizes locally coherent but globally manipulated passages.
  • The same granularity-mismatch issue could appear in other multi-stage retrieval systems beyond RAG.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The manuscript claims that existing corpus poisoning attacks on RAG systems substantially degrade after reranking despite high retrieval-stage relevance, due to retrieval granularity mismatch where document-level signals fragment during chunking and rerankers prefer locally coherent passages. It proposes CRCP (Chunk-aware and Rerank-Consistent Poisoning), a framework that jointly optimizes retrieval relevance, reranker consistency, and chunk-boundary robustness by explicitly modeling chunking transformations. Experiments on standard RAG benchmarks with multiple retrievers and rerankers are reported to show that prior methods are sensitive to chunk size and reranking while CRCP achieves substantially higher attack success rates and stronger robustness, highlighting a realism gap in RAG security evaluation.

Significance. If the results hold, the work identifies an important realism gap in current RAG security evaluations and reframes corpus poisoning as a multi-stage retrieval consistency problem rather than retrieval-only. The proposal of a chunk-aware optimization method and the use of multiple retrievers/rerankers in the evaluation are strengths that could guide more realistic attack research.

major comments (2)
  1. [Abstract, paragraph beginning 'We identify retrieval granularity mismatch as a key reason'] Abstract, paragraph beginning 'We identify retrieval granularity mismatch as a key reason': the claim that granularity mismatch is the primary cause of attack failure lacks direct supporting evidence such as ablations across reranker families or out-of-distribution chunkers. This is load-bearing for the central claim that CRCP's gains and the multi-stage consistency framing transfer beyond the specific open-source models and fixed chunk sizes tested.
  2. [Abstract (experimental claims)] Abstract (experimental claims): the abstract states that CRCP achieves substantially higher attack success rates and stronger robustness but supplies no metrics, dataset sizes, number of trials, statistical tests, or ablation details. Without these, it is impossible to assess whether the data support the claims of superiority over existing attacks, which is central to the contribution.
minor comments (1)
  1. The abstract would be strengthened by including at least one key quantitative result (e.g., ASR delta or robustness metric) to convey the magnitude of the reported improvements.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback. We address the major comments point by point below.

read point-by-point responses
  1. Referee: [Abstract, paragraph beginning 'We identify retrieval granularity mismatch as a key reason'] Abstract, paragraph beginning 'We identify retrieval granularity mismatch as a key reason': the claim that granularity mismatch is the primary cause of attack failure lacks direct supporting evidence such as ablations across reranker families or out-of-distribution chunkers. This is load-bearing for the central claim that CRCP's gains and the multi-stage consistency framing transfer beyond the specific open-source models and fixed chunk sizes tested.

    Authors: The manuscript reports results across multiple retrievers and rerankers that demonstrate degradation after reranking and sensitivity to chunk size. We acknowledge that dedicated ablations on out-of-distribution chunkers would further strengthen the generality claim. We will add such experiments in the revision. revision: yes

  2. Referee: [Abstract (experimental claims)] Abstract (experimental claims): the abstract states that CRCP achieves substantially higher attack success rates and stronger robustness but supplies no metrics, dataset sizes, number of trials, statistical tests, or ablation details. Without these, it is impossible to assess whether the data support the claims of superiority over existing attacks, which is central to the contribution.

    Authors: Detailed metrics, dataset sizes, trial counts, and ablation results appear in Sections 4 and 5. We will revise the abstract to include key quantitative results (e.g., attack success rate deltas) to make the claims more self-contained. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: claims rest on experimental comparisons

full rationale

The paper presents an empirical study proposing CRCP based on observed degradation of prior attacks after reranking and chunking. No equations, fitted parameters, self-definitional constructs, or load-bearing self-citations appear in the abstract or described structure. Central claims (higher ASR and robustness for CRCP) are supported by reported benchmark results rather than reducing to inputs by construction. The granularity mismatch observation is framed as an experimental finding, not a self-referential derivation.

Assumptions & free parameters 0 free parameters · 1 assumptions · 1 invented entities

The central claim rests on the new CRCP framework and on the premise that standard RAG benchmarks capture realistic multi-stage behavior; no free parameters are mentioned.

assumptions (1)
  • domain assumption Standard RAG benchmarks and common retriever/reranker implementations are representative of production pipelines.
    The paper evaluates on 'standard RAG benchmarks with multiple retrievers and rerankers' without further justification.
invented entities (1)
  • CRCP (Chunk-aware and Rerank-Consistent Poisoning)
    purpose: Poisoning framework that jointly optimizes retrieval relevance, reranker consistency, and chunk-boundary robustness.
    Introduced in the abstract as the proposed solution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Poison Fails After Retrieval: Revisiting Corpus Poisoning under Chunking and Reranking Pipelines." pith.science (2026). https://pith.science/paper/W2VEWW5S

@misc{pith2026260611265,
  author       = {Pith},
  title        = {Pith review of: When Poison Fails After Retrieval: Revisiting Corpus Poisoning under Chunking and Reranking Pipelines},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W2VEWW5S}},
  note         = {Machine review of arXiv:2606.11265}
}
read the original abstract

Retrieval-Augmented Generation (RAG) systems are vulnerable to corpus poisoning attacks that manipulate downstream model outputs through malicious knowledge injection. Existing studies mainly evaluate poisoning under simplified retrieval settings, overlooking practical RAG pipelines involving document chunking, dense retrieval, reranking, and grounded generation. In this paper, we revisit corpus poisoning under realistic multi-stage retrieval pipelines and show that many existing attacks substantially degrade after reranking despite achieving high retrieval-stage relevance. We identify retrieval granularity mismatch as a key reason for this failure: document-level adversarial signals are often fragmented during chunking, while rerankers favor locally coherent and answer-bearing passages rather than globally optimized semantic similarity. Based on this observation, we propose Chunk-aware and Rerank-Consistent Poisoning (CRCP), a poisoning framework that jointly optimizes retrieval relevance, reranker consistency, and chunk-boundary robustness. CRCP explicitly models chunking transformations during optimization to generate locally self-contained adversarial passages that remain effective under varying chunking configurations. Experiments on standard RAG benchmarks with multiple retrievers and rerankers show that existing poisoning methods are highly sensitive to chunk size and reranking strategies, whereas CRCP achieves substantially higher attack success rates and stronger robustness across realistic retrieval pipelines. Our findings highlight an important realism gap in current RAG security evaluation and suggest that poisoning in modern RAG systems should be studied as a multi-stage retrieval consistency problem rather than a retrieval-only problem.

Figures

Figures reproduced from arXiv: 2606.11265 by the authors.

Figure 1
Figure 1. An overview of the CRCP Attack. and chunk-boundary robustness. Unlike prior methods that optimize global document-level similarity, our approach ex￾plicitly models chunking transformations during optimization and encourages locally self-contained adversarial passages that remain effective across varying chunking configurations. Extensive experiments on standard RAG benchmarks using multiple retrievers and rerankers … view at source ↗
Figure 2
Figure 2. t-SNE Visualization: Embedding Distribution Comparison across Poisoning Methods and Retrieval Granularities. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Distribution of Signal Retention Rate (SRR) across [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Failure mode analysis of corpus poisoning attacks [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 2 canonical work pages

  1. [1]

    Retrieval-augmented generation for knowledge-intensive nlp tasks,

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K ¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt ¨aschel, et al., “Retrieval-augmented generation for knowledge-intensive nlp tasks,”NeurIPS, 2020

  2. [2]

    Dense passage retrieval for open-domain question answering,

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih, “Dense passage retrieval for open-domain question answering,” inEMNLP, 2020

  3. [3]

    Improving language models by retrieving from trillions of tokens,

    Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean- Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al., “Improving language models by retrieving from trillions of tokens,” inICML, 2022

  4. [4]

    Wikichat: Stopping the hallucination of large language model chatbots by few-shot grounding on wikipedia,

    Sina Semnani, Violet Yao, Heidi Zhang, and Monica Lam, “Wikichat: Stopping the hallucination of large language model chatbots by few-shot grounding on wikipedia,” inFindings of EMNLP, 2023

  5. [5]

    Retrieval-based prompt selection for code-related few-shot learning,

    Noor Nashid, Mifta Sintaha, and Ali Mesbah, “Retrieval-based prompt selection for code-related few-shot learning,” inICSE, 2023

  6. [6]

    Docprompting: Generating code by retrieving the docs,

    Shuyan Zhou, Uri Alon, Frank F. Xu, Zhengbao Jiang, and Graham Neubig, “Docprompting: Generating code by retrieving the docs,” in ICLR, 2023

  7. [7]

    Phantom: General backdoor attacks on retrieval augmented language generation,

    Harsh Chaudhari, Giorgio Severi, John Abascal, Anshuman Suri, Matthew Jagielski, Christopher A. Choquette-Choo, Milad Nasr, Cristina Nita-Rotaru, and Alina Oprea, “Phantom: General backdoor attacks on retrieval augmented language generation,”ACM Trans. AI Secur . Priv., 2026

  8. [8]

    Typos that broke the rag’s back: Genetic attack on rag pipeline by simulating documents in the wild via low-level perturbations,

    Sukmin Cho, Soyeong Jeong, Jeongyeon Seo, Taeho Hwang, and Jong C Park, “Typos that broke the rag’s back: Genetic attack on rag pipeline by simulating documents in the wild via low-level perturbations,” in Findings of EMNLP, 2024

Show all 33 references
  1. [9]

    Hotflip: White-box adversarial examples for text classification,

    Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou, “Hotflip: White-box adversarial examples for text classification,” inACL, 2018

  2. [10]

    Ctrlrag: Black- box document poisoning attacks for retrieval-augmented generation of large language models,

    Runqi Sui, Xuejing Yuan, Di Tang, and Baojiang cui, “Ctrlrag: Black- box document poisoning attacks for retrieval-augmented generation of large language models,” inAAMAS, 2026

  3. [11]

    Poisoning retrieval corpora by injecting adversarial passages,

    Zexuan Zhong, Ziqing Huang, Alexander Wettig, and Danqi Chen, “Poisoning retrieval corpora by injecting adversarial passages,” in EMNLP, 2023

  4. [12]

    A survey on rag with llms,

    Muhammad Arslan, Hussam Ghanem, Saba Munawar, and Christophe Cruz, “A survey on rag with llms,”Procedia computer science, 2024

  5. [13]

    A survey on rag meeting llms: Towards retrieval-augmented large language models,

    Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li, “A survey on rag meeting llms: Towards retrieval-augmented large language models,” inSIGKDD, 2024

  6. [14]

    Re2g: Retrieve, rerank, generate,

    Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, Ankita Naik, Pengshan Cai, and Alfio Gliozzo, “Re2g: Retrieve, rerank, generate,” inNAACL, 2022

  7. [15]

    Dynamicrag: Leveraging outputs of large language model as feedback for dynamic reranking in retrieval-augmented generation,

    Jiashuo Sun, Xianrui Zhong, Sizhe Zhou, and Jiawei Han, “Dynamicrag: Leveraging outputs of large language model as feedback for dynamic reranking in retrieval-augmented generation,” inNeurIPS, 2025

  8. [16]

    Prorank: Progressive context refinement for reliable retrieval-augmented generation,

    Yihui Yang, Jianhui Ma, Xu An, Zheng Zhang, Mingfan Pan, Wuhong Wang, and Huiming Ding, “Prorank: Progressive context refinement for reliable retrieval-augmented generation,” inICASSP, 2026

  9. [17]

    Maximizing rag efficiency: A comparative analysis of rag methods,

    Tolga S ¸akar and Hakan Emekci, “Maximizing rag efficiency: A comparative analysis of rag methods,”Natural Language Processing, 2025

  10. [18]

    Agentpoison: Red-teaming llm agents via poisoning memory or knowl- edge bases,

    Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li, “Agentpoison: Red-teaming llm agents via poisoning memory or knowl- edge bases,”NeurIPS, 2024

  11. [19]

    Poisonedrag: Knowledge corruption attacks to retrieval-augmented generation of large language models,

    Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia, “Poisonedrag: Knowledge corruption attacks to retrieval-augmented generation of large language models,” inUSENIX Security, 2025

  12. [20]

    Glue pizza and eat rocks- exploiting vulnerabilities in retrieval-augmented generative models,

    Zhen Tan, Chengshuai Zhao, Raha Moraffah, Yifan Li, Song Wang, Jundong Li, Tianlong Chen, and Huan Liu, “Glue pizza and eat rocks- exploiting vulnerabilities in retrieval-augmented generative models,” in EMNLP, 2024

  13. [21]

    Joint-gcg: Unified gradient- based poisoning attacks on retrieval-augmented generation systems,

    Haowei Wang, Rupeng Zhang, Junjie Wang, Mingyang Li, Yuekai Huang, Dandan Wang, and Qing Wang, “Joint-gcg: Unified gradient- based poisoning attacks on retrieval-augmented generation systems,” in AAAI, 2026

  14. [22]

    The RAG paradox: A black-box attack exploiting unintentional vulnerabilities in retrieval-augmented generation systems,

    Chanwoo Choi, Jinsoo Kim, Sukmin Cho, Soyeong Jeong, and Buru Chang, “The RAG paradox: A black-box attack exploiting unintentional vulnerabilities in retrieval-augmented generation systems,” inFindings of EMNLP, 2025

  15. [23]

    Cpa-rag: Covert poisoning attacks on retrieval-augmented generation in large language models,

    Chunyang Li, Junwei Zhang, Anda Cheng, Zhuo Ma, Xinghua Li, and Jianfeng Ma, “Cpa-rag: Covert poisoning attacks on retrieval-augmented generation in large language models,”arXiv preprint arXiv:2505.19864, 2025

  16. [24]

    RevPRAG: Revealing poisoning attacks in retrieval-augmented generation through LLM activation analysis,

    Xue Tan, Hao Luan, Mingyu Luo, Xiaoyan Sun, Ping Chen, and Jun Dai, “RevPRAG: Revealing poisoning attacks in retrieval-augmented generation through LLM activation analysis,” inFindings of EMNLP, 2025

  17. [25]

    Certifiably robust RAG against retrieval corruption,

    Chong Xiang, Tong Wu, Zexuan Zhong, David Wagner, Danqi Chen, and Prateek Mittal, “Certifiably robust RAG against retrieval corruption,” in ICML 2024 Next Generation of AI Safety Workshop, 2024

  18. [26]

    Astute rag: Overcoming imperfect retrieval augmentation and knowledge conflicts for large language models,

    Fei Wang, Xingchen Wan, Ruoxi Sun, Jiefeng Chen, and Sercan O Arik, “Astute rag: Overcoming imperfect retrieval augmentation and knowledge conflicts for large language models,” inACL, 2025

  19. [27]

    Instructrag: Instructing retrieval-augmented generation via self-synthesized rationales,

    Zhepei Wei, Wei-Lin Chen, and Yu Meng, “Instructrag: Instructing retrieval-augmented generation via self-synthesized rationales,” inICLR, 2025

  20. [28]

    Trustrag: enhancing robustness and trustworthiness in retrieval-augmented gen- eration,

    Huichi Zhou, Kin-Hei Lee, Zhonghao Zhan, Yue Chen, Zhenhao Li, Zhaoyang Wang, Hamed Haddadi, and Emine Yilmaz, “Trustrag: enhancing robustness and trustworthiness in retrieval-augmented gen- eration,” inAAAI Workshop, 2026

  21. [29]

    Secon-rag: A two-stage semantic filtering and conflict-free framework for trustworthy rag,

    Xiaonan Si, Meilin Zhu, Simeng Qin, Lijia Yu, Lijun Zhang, Shuaitong Liu, Xinfeng Li, Ranjie Duan, Yang Liu, and Xiaojun Jia, “Secon-rag: A two-stage semantic filtering and conflict-free framework for trustworthy rag,”NeurIPS, 2025

  22. [30]

    Ms marco: A human generated machine reading comprehension dataset,

    Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al., “Ms marco: A human generated machine reading comprehension dataset,”arXiv preprint arXiv:1611.09268, 2016

  23. [31]

    Natural questions: a benchmark for question answering research,

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al., “Natural questions: a benchmark for question answering research,”Transactions of the Association for Compu...

  24. [32]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering,

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning, “Hotpotqa: A dataset for diverse, explainable multi-hop question answering,” in EMNLP, 2018

  25. [33]

    Reconstructing context: Evaluating advanced chunking strategies for retrieval-augmented generation,

    Carlo Merola and Jaspinder Singh, “Reconstructing context: Evaluating advanced chunking strategies for retrieval-augmented generation,” in International Workshop on Knowledge-Enhanced Information Retrieval. Springer, 2025

Pith tools

Reviewed June 27, 2026 · model on record in the stance chip above.