REVIEW 4 major objections 6 minor 2 cited by
Chain-of-Thought Poisoning Attacks against R1-based Retrieval-Augmented Generation Systems
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims that wrapping erroneous knowledge in the RAG system's own chain-of-thought template lets attackers steer answers on R1-based RAG systems, raising attack success to 61 percent.
desk verdict A plausible new CoT-mimicry poisoning attack, but the headline gain is entangled with a teacher-student confound and thin statistics. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the reasoning template: the recurring chain-of-thought format (opening phrase, step-by-step document review, a transitional 'wait' pivot, a summary of evidence, closing </think> tag) that a target R1-based RAG system prints before its answer. The attack has two stages, extracting the template from observed responses and using an auxiliary LLM to wrap erroneous knowledge in that template, and those stages are what distinguish it from knowledge-only poisoning, because they aim to make adversarial documents look like the model's own past reasoning. The template is what carries the argument: without it, the injected document is just another passage asserting a wrong fact, and baseline numbers show such passages are filtered out more easily by deep-reasoning models.
What would settle it
On the same 100 queries, strip the reasoning-template phrasing from each poisoned document while keeping the factual content unchanged, and measure ASR on Qwen-7B-R1-distilled: if it does not fall from 61 toward the 51 of PRAG, then template imitation is not the causal ingredient.
Extended reading notes
Core claim
The paper's central claim is that deep-reasoning RAG systems, although more robust than standard RAG against knowledge corruption, are vulnerable to documents that imitate the system's own reasoning format. The attack extracts a fixed template from observed answers (begin with step-by-step review of context, introduce a 'wait, there's additional information' pivot, then summarize with 'so, putting it all together' and a </think> tag), then prompts an auxiliary LLM to generate a plausible reasoning chain that reaches the attacker's target answer. Injected into the knowledge base, such a document must first be retrieved (ASR$_r$=89) and then steer the generator (ASR$_g$=68.5), giving overall ASR=61, versus 51 for PRAG. The paper argues that the model mistakes the adversarial reasoning process for its own training-aligned reasoning and therefore prefers the embedded erroneous knowledge.
Load-bearing premise
The attack works only if the victim RAG system displays its full chain of thought in its answers, because the attacker needs that visible reasoning trace to extract the template that makes poisoned documents persuasive.
Editorial extensions
If this is right
- Any R1-style RAG system that exposes its chain of thought in output has a reproducible attack surface: an attacker who can observe a handful of answers can reconstruct the template and poison the knowledge base.
- The attack targets the generator more than the retriever: it achieves only 89 percent retrieval success (below most baselines) but the highest generator success (68.5 percent), so the reasoning format is what shifts the model's judgment.
- Deep reasoning does not make RAG systems immune to poisoning; it changes the level at which poisoning acts, from factual assertion to reasoning-process imitation.
- Scaling the backbone LLM reduces attack success for all methods, but chain-of-thought poisoning keeps its advantage across 1.5B, 7B, and 32B models.
- Attack documents built this way remain natural to human annotators, with document fluency close to PRAG and higher reasoning credibility than prompt hijacking.
Reading between the lines
- A cheap, testable defense follows directly: hide or truncate the chain of thought in production answers; since the attack's template-extraction step depends on observing full reasoning traces, systems that return only final answers should lose most of the observed advantage.
- The attack likely transfers to any reasoning model whose chain-of-thought format is stable and exposed, not only R1-style models; the same template imitation could be tried against other reasoning LLMs wrapped in RAG, regardless of which auxiliary model generates the poisoned text.
- Because the generator success rate (68.5) is far above the retrieval success rate (89), the approach's bottleneck is retrieval; an adversary who pairs template-wrapped documents with retriever-specific optimization techniques could plausibly push overall ASR still higher.
- A direct behavioral test would be to ask the generator why it trusts a poisoned document: the paper's hypothesis predicts the model will describe the document's reasoning process as credible rather than as a merely retrieved source, a difference from knowledge-only poisoning.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Chain-of-Thought Poisoning, a knowledge-base poisoning attack on R1-based retrieval-augmented generation (RAG) systems. The attacker first extracts a reasoning-process template by observing the victim RAG system's chain-of-thought outputs (Section 4.2), then uses an auxiliary LLM (DeepSeek-R1) to generate adversarial documents that wrap erroneous knowledge in that template (Section 4.3), and injects these documents into the knowledge base. Experiments on 100 MS MARCO queries compare the method against four baselines (NA, NPA, PHA, PRAG) on Qwen2.5-7B and Qwen-7B-R1-distilled RAG systems, reporting ASR=61 for the proposed method versus ASR=51 for PRAG on the R1-based system (Table 2), along with a naturalness study (Table 3) and a model-scaling experiment (Figure 3). The central claim is that adversarial documents mimicking the model's own chain-of-thought template are more likely to be referenced, yielding a reasoning-level attack surface beyond simple knowledge corruption.
Significance. If the central claim is sustained, the paper identifies a genuinely new attack surface for reasoning-capable RAG systems: poisoning at the level of the model's chain-of-thought style rather than only the factual content. This is timely and practically relevant given the deployment of R1-style models. The paper also provides a clean decomposition of attack success into retriever and generator components (Eq. 2), and it compares against four baselines including the strongest prior poisoning method (PRAG), which is a useful experimental anchor. However, the significance is conditional on isolating the proposed mechanism from a teacher-student confound and on demonstrating that the headline gains are not noise, since the reported differences are small relative to the evaluation scale (100 queries).
major comments (4)
- [Section 5.4 / Section 4.3] The main mechanism is confounded with teacher-student distributional alignment. Adversarial documents are generated by DeepSeek-R1 (Section 5.4), which is also the teacher model used to create the target Qwen-7B-R1-distilled model [7]. The reasoning template extracted in Section 4.2 is inherited from that same teacher, so the experimental condition simultaneously varies (1) alignment with the target's observed reasoning template and (2) presentation in the teacher's native style. Section 4.3 even states that the auxiliary generator should ideally be the same base model as the target, but no experiment varies the generator while holding the template fixed. The paper's Section 4.1 hypothesis ('the model may mistake adversarial documents for its own historical reasoning') is therefore not empirically separated from 'the model prefers text from its teacher distribution.' Please add a control that generates adversarial documents with a non-teacher LLM (e.g., Qwen2.5-7B or another model) using the same extracted template, and/or uses a template extracted from a different model, to isolate the effect of template alignment.
- [Table 2 / Section 6.1] The headline result (ASR 61 vs. 51 for PRAG on Qwen-7B-R1-distilled) is reported as a single run on 100 selected queries with no error bars, confidence intervals, or significance tests. The difference of 10 percentage points corresponds to only 10 queries, so it may lie within sampling noise. The generator-side comparison is even thinner: ASR_g 68.5 vs. 51.5, which on the ASR_r=89 subset is about 61 versus 46 successes. Please report results over multiple runs or seeds, and provide bootstrap confidence intervals or a paired significance test (e.g., McNemar's test) for the comparison against PRAG. Without this, the main ASR advantage is not statistically grounded.
- [Table 3 / Section 6.2] The naturalness claim is not supported by the reported agreement metrics. For the proposed method, Fleiss's Kappa is 0.520 (moderate agreement) and the Pearson correlation for document naturalness (PCCs) is only 0.355, which is low. The evaluation uses only 10 successfully attacked queries per method, and no confidence intervals are given for the N_r and N_d means. The text in Section 6.2 concludes that the method exhibits 'good naturalness,' but the low PCCs for the proposed method weaken this claim, especially relative to PRAG's PCCs of 0.639 and PHA's 0.843. Please report per-annotator scores, agreement statistics with confidence intervals, and a larger or more carefully selected sample, or temper the naturalness conclusion accordingly.
- [Section 3.3 / Section 4.2] The attack's feasibility depends on the victim RAG system exposing its full chain of thought, including internal reasoning markers such as '<think>...</think>' (Table 1). However, production RAG systems often hide the reasoning trace and return only the final answer; in that case the template-extraction step in Section 4.2 cannot be performed. The paper does not discuss or test this deployment scenario. Please add an explicit analysis of when the attack remains feasible (e.g., if only the final answer is visible, or if the reasoning is summarized) and, if possible, an experiment that simulates a hidden-reasoning setting to bound the attack's applicability.
minor comments (6)
- [Section 4 heading] The heading contains a typo: 'chain-of-thoyght' should be 'chain-of-thought.'
- [Section 5.2 / Table 2] The phrase 'all values have been centile-divided' is unclear; if it means values are percentages, please state that explicitly and use consistent notation (e.g., 'ASR is reported as a percentage').
- [Section 3.2 / Eq. (1)] Equation (1) uses d_q in the union in the second argument of G but the text defines the perturbed document as d'_t; please align the notation. Also, the indicator condition 'G(...) = y*_q' is written as a set membership symbol in the equation; please clarify.
- [Table 1] The caption mentions underlining to indicate thought-process formatting, but the underlining is not visible in the rendered example. Please use a visible formatting cue (italics, color, or a distinct font) or remove the reference to underlining.
- [Figure 3] The scaling plot would benefit from error bars and explicit axis labels; as shown, it is unclear whether each point is a single run and whether the differences between Ours and PoisonedRAG are within noise.
- [Section 5.1] The paper states that 100 queries with definite answers were selected from MS MARCO but gives no details on the selection criteria or whether the queries are the same across all methods and systems. Please provide the query selection process or a reference to the exact subset used.
Circularity Check
No significant circularity: the extracted template is an attack input, not a fitted prediction, and the headline comparison is grounded by identical evaluation across baselines.
full rationale
The paper does not exhibit circularity in its derivation chain. The reasoning template is extracted from the victim system's observed outputs (Section 4.2) and then used to construct adversarial documents (Section 4.3). This is a standard black-box attack procedure: the extracted template is an input to the attack, not a fitted parameter that numerically forces the reported ASR. The headline comparison (Ours ASR=61 vs PRAG ASR=51 on Qwen-7B-R1-distilled in Table 2) is measured on the same experimental setup for all methods, and the baselines provide independent grounding. No load-bearing result is justified by a self-citation: the authors' prior work appears only in related-work discussions (references [15]-[23], [27]) and is not used to establish the attack's effectiveness. The only notable issue is an empirical confound, not a circular reduction: Section 5.4 selects DeepSeek-R1 as the auxiliary generator, and DeepSeek-R1 is also the teacher behind Qwen-7B-R1-distilled (reference [7]); Section 4.3 notes that the auxiliary generator should ideally be the same base model, but no control varies the generator while holding the template fixed. This weakens the causal attribution of the ASR gain to the extracted template, but the gain is not forced by construction, so it belongs under validity/correctness risk rather than circularity. Score 0.
Assumptions & free parameters
free parameters (3)
- Reasoning template T =
Hand-extracted phrases, e.g., 'let's think step by step', '</think>'
- Number of retrieved passages k =
5
- Adversarial document rounds per query =
5
assumptions (4)
- ad hoc to paper The observed chain-of-thought format in target responses aligns with the model's training signals, so documents in that format are preferentially trusted.
- domain assumption The attacker can observe the target RAG system's full responses, including the exposed reasoning trace.
- domain assumption The target retriever and generator are static and do not adapt to injected documents.
- domain assumption The auxiliary LLM (DeepSeek-R1) can generate documents that match the target's reasoning style.
Cite this review
Pith. "Pith review of Chain-of-Thought Poisoning Attacks against R1-based Retrieval-Augmented Generation Systems." pith.science (2026). https://pith.science/paper/7YRKTLOM
@misc{pith2026250516367,
author = {Pith},
title = {Pith review of: Chain-of-Thought Poisoning Attacks against R1-based Retrieval-Augmented Generation Systems},
year = {2026},
howpublished = {\url{https://pith.science/paper/7YRKTLOM}},
note = {Machine review of arXiv:2505.16367}
}
read the original abstract
Retrieval-augmented generation (RAG) systems can effectively mitigate the hallucination problem of large language models (LLMs),but they also possess inherent vulnerabilities. Identifying these weaknesses before the large-scale real-world deployment of RAG systems is of great importance, as it lays the foundation for building more secure and robust RAG systems in the future. Existing adversarial attack methods typically exploit knowledge base poisoning to probe the vulnerabilities of RAG systems, which can effectively deceive standard RAG models. However, with the rapid advancement of deep reasoning capabilities in modern LLMs, previous approaches that merely inject incorrect knowledge are inadequate when attacking RAG systems equipped with deep reasoning abilities. Inspired by the deep thinking capabilities of LLMs, this paper extracts reasoning process templates from R1-based RAG systems, uses these templates to wrap erroneous knowledge into adversarial documents, and injects them into the knowledge base to attack RAG systems. The key idea of our approach is that adversarial documents, by simulating the chain-of-thought patterns aligned with the model's training signals, may be misinterpreted by the model as authentic historical reasoning processes, thus increasing their likelihood of being referenced. Experiments conducted on the MS MARCO passage ranking dataset demonstrate the effectiveness of our proposed method.
Figures
Forward citations
Cited by 2 Pith papers
-
Large Language Models in Misinformation Ecosystems: Misuse, Defense, and Vulnerability
A role-layer survey unifies LLM misuse, LLM-based defense, and LLM-centric verification vulnerabilities across content, social, evidence, and workflow layers, then lists three open challenges.
-
A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models
A structured literature survey concluding that reasoning capabilities do not automatically make LLMs more trustworthy and can introduce new vulnerabilities in safety, robustness, and privacy.
Reference graph
Works this paper leans on
-
[7]
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)
arXiv 2025
-
[1]
[n. d.]. LangChain. https://www.langchain.com/
-
[2]
Sukmin Cho, Soyeong Jeong, Jeongyeon Seo, Taeho Hwang, and Jong C Park
-
[3]
Jingsheng Gao, Linxu Li, Weiyuan Li, Yuzhuo Fu, and Bin Dai. 2024. SmartRAG: Jointly Learn RAG-Related Tasks From the Environment Feedback.arXiv preprint arXiv:2410.18141 (2024)
arXiv 2024
-
[4]
Luyu Gao and Jamie Callan. 2022. Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 2843– 2853
work page 2022
-
[5]
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Ji- awei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997 [cs]
arXiv 2024
-
[6]
Xinyan Guan, Jiali Zeng, Fandong Meng, Chunlei Xin, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun, and Jie Zhou. 2025. DeepRAG: Thinking to Retrieval Step by Step for Large Language Models. arXiv preprint arXiv:2502.01142 (2025)
arXiv 2025
-
[8]
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International conference on machine learning. PMLR, 3929–3938
2020
Show all 34 references
-
[9]
Zhibo Hu, Chen Wang, Yanfeng Shu, Hye-Young Paik, and Liming Zhu. 2024. Prompt perturbation in retrieval-augmented generation based large language models. In SIGKDD. 1119–1130
2024
-
[10]
Gautier Izacard and Édouard Grave. 2021. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. In EACL. 874–880
2021
-
[11]
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with retrieval augmented language models.arXiv preprint arXiv:2208.03299 1, 2 (2022), 4
2022 arXiv
-
[12]
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing s...
2020
-
[13]
Jiawei Liu, Yangyang Kang, Di Tang, Kaisong Song, Changlong Sun, Xiaofeng Wang, Wei Lu, and Xiaozhong Liu. 2022. Order-disorder: Imitation adversarial attacks for black-box neural ranking models. In SIGSAC. 2025–2039
2022
-
[14]
Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2024. Formalizing and benchmarking prompt injection attacks and defenses. In 33rd USENIX Security Symposium (USENIX Security 24) . 1831–1847
2024
-
[15]
Yu-An Liu, Ruqing Zhang, Jiafeng Guo, and Xueqi Cheng. 2025. On the Robustness of Generative Information Retrieval Models. In ECIR
2025
-
[16]
Yu-An Liu, Ruqing Zhang, Jiafeng Guo, and Maarten de Rijke. 2024. Robust Information Retrieval. In SIGIR. 3009–3012. Chain-of-Thought Poisoning Attacks against R1-based Retrieval-Augmented Generation Systems Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
2024
-
[17]
Yu-An Liu, Ruqing Zhang, Jiafeng Guo, and Maarten de Rijke. 2025. Robust Information Retrieval. In WSDM
2025
-
[18]
Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Wei Chen, Yixing Fan, and Xueqi Cheng. 2023. Black-Box Adversarial Attacks against Dense Retrieval Models: A Multi-View Contrastive Learning Method. In CIKM. 1647–1656
2023
-
[19]
Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Wei Chen, Yixing Fan, and Xueqi Cheng. 2023. Topic-Oriented Adversarial Attacks against Black-Box Neural Ranking Models. In SIGIR. 1700–1709
2023
-
[20]
Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, and Xueqi Cheng. 2025. Attack-in-the-Chain: Bootstrapping Large Language Models for Attacks against Black-box Neural Ranking Models. In AAAI
2025
-
[21]
Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2024. Multi-granular adversarial attacks against black-box neural ranking models. In SIGIR. 1391–1400
2024
-
[22]
Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2024. Robust neural information retrieval: An adversarial and out-of- distribution perspective. arXiv preprint arXiv:2407.06992 (2024)
2024 arXiv
-
[23]
Yu-An Liu, Ruqing Zhang, Mingkun Zhang, Wei Chen, Maarten de Rijke, Jiafeng Guo, and Xueqi Cheng. 2024. Perturbation-Invariant Adversarial Training for Neural Ranking Models: Improving the Effectiveness-Robustness Trade-Off. In AAAI, Vol. 38
2024
-
[24]
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human-generated machine reading comprehension dataset. (2016)
2016
-
[25]
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. TACL 11 (2023), 1316–1331
2023
-
[26]
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does llm safety training fail? NIPS 36 (2023), 80079–80110
2023
-
[27]
Chen Wu, Ruqing Zhang, Jiafeng Guo, Maarten De Rijke, Yixing Fan, and Xueqi Cheng. 2023. Prada: Practical black-box adversarial attacks against neural ranking models. ACM Transactions on Information Systems 41, 4 (2023), 1–27
2023
-
[28]
Sirui Xia, Xintao Wang, Jiaqing Liang, Yifei Zhang, Weikang Zhou, Jiaji Deng, Fei Yu, and Yanghua Xiao. 2024. Ground Every Sentence: Improving Retrieval- Augmented LLMs with Interleaved Reference-Claim Generation. arXiv preprint arXiv:2407.01796 (2024)
2024 arXiv
-
[29]
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al . 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 (2024)
2024 arXiv
-
[30]
Peitian Zhang, Zheng Liu, Shitao Xiao, Zhicheng Dou, and Jian-Yun Nie. 2024. A multi-task embedder for retrieval augmented llms. In ACL. 3537–3553
2024
-
[31]
Yucheng Zhang, Qinfeng Li, Tianyu Du, Xuhong Zhang, Xinkui Zhao, Zhengwen Feng, and Jianwei Yin. 2024. HijackRAG: Hijacking Attacks against Retrieval- Augmented Large Language Models. arXiv preprint arXiv:2410.22832 (2024)
2024 arXiv
-
[32]
Shuyan Zhou, Uri Alon, Frank F Xu, Zhengbao Jiang, and Graham Neubig. [n. d.]. DocPrompting: Generating Code by Retrieving the Docs. In ICLR
-
[33]
Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. 2024. Poisonedrag: Knowledge corruption attacks to retrieval-augmented generation of large lan- guage models. arXiv preprint arXiv:2402.07867 (2024)
2024 arXiv
-
[2024]
arXiv preprint arXiv:2404.13948 (2024)
Typos that Broke the RAG’s Back: Genetic Attack on RAG Pipeline by Simulating Documents in the Wild via Low-level Perturbations. arXiv preprint arXiv:2404.13948 (2024)
2024 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.