{"id":"6143f3d2-0571-403e-adc9-06b9a6f437cb","arxiv_id":"2501.00879","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TrustRAG combines K-means filtering with LLM self-assessment to defend RAG systems against corpus poisoning attacks, cutting attack success rates while preserving accuracy.","lead":"TrustRAG filters poisoned documents out of retrieval-augmented generation systems before the model reads them, using clustering and the model's own knowledge to spot contradictions. It claims large reductions in attack success rates across multiple language models and attack types.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The robustness claim is load-bearing on the untested assumption that malicious documents cluster in embedding space; a non-clustering adaptive attack could break Stage 1, and Stage 2 alone is insufficient at high poison rates.","rationale":"The reader's verdict set the weakest assumption at Stage 1's cluster hypothesis; I agree. The paper's central contribution is a general defense against 'corpus poisoning attacks', but every attack tested (PIA, PoisonedRAG, AD, Jamming) is generated with a shared optimization loop that tends to produce similar embeddings. The adaptive-attack section (Section 5.4) varies prompts but stays within the PoisonedRAG generator, so it does not decouple the attacks from the cluster prior. The ablation evidence is strong: removing K-means raises ASR to 39% at 100% poison (Table 10), so Stage 2 cannot carry the defense alone. Therefore the cluster assumption is truly load-bearing. The concrete test I propose directly constructs a non-clustering attack; if it succeeds, the claim 'outperforming all existing defenses' only holds for clustered attacks, and the paper should state that limitation. If it fails, the concern is resolved and the conditional accept stands.","tokens_in":33462,"tokens_out":6057,"duration_ms":54386,"concrete_test":"Run an adaptive attack on NQ with Llama3.1-8B at poison rates 20-100% where malicious documents are generated by maximizing retrieval success (Equation 3) while explicitly maximizing pairwise embedding distance (e.g., optimize each document with different seeds and a diversity penalty, or use a minimax objective). Measure TrustRAG Stage 1&2 ACC/ASR. Since clustered PoisonedRAG yields ASR ≤2% at 100% poison, define failure if ASR exceeds 15% or ACC falls below 70% at 100% poison. Repeat 5 seeds; report mean±std. If ASR remains low, the cluster assumption is not load-bearing; if it jumps, the headline claim must be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"TrustRAG's headline result depends on Stage 1 (K-means with k=2) correctly separating a malicious cluster from clean retrieved documents. That separation relies on the empirical claim in Section 3.2 and Figure 3 that attacker-optimized malicious documents are tightly clustered in embedding space. If an adversary can generate diverse malicious documents that are individually optimized but mutually dissimilar, K-means fails. The paper's 'Diverse-Context Multiple Injection Attack' (Section 5.4, Table 12) still uses PoisonedRAG variants with varied prompts, so the cluster structure is only perturbed, not removed. The importance of Stage 1 is confirmed by the ablation 'w/o K-Means' (Table 10): at 100% poison on NQ with Llama3.1-8B, ACC/ASR degrade from 83/2 to 55/39, and at 80% to 55/41. Thus Stage 2 alone is not sufficient at high poison rates, so a non-clustering attack would likely breach the claimed ASR reduction. Additionally, the fixed thresholds (ROUGE-L 0.25, cosine similarity 0.85, Appendix B.2) are tuned to the tested attacks; no sensitivity analysis is given, so the defense's parameters could be exploited by an attacker who knows them.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TrustRAG, a two-stage, training-free defense for retrieval-augmented generation (RAG) against corpus poisoning attacks. Stage 1 (Clean Retrieval) applies K-means clustering with k=2 to document embeddings, together with a ROUGE-L-based preservation heuristic, to filter out clusters of malicious documents. Stage 2 (Conflict Resolution) prompts the LLM to generate internal knowledge, consolidate consistent external and internal evidence, and self-assess whether to trust external or internal knowledge before producing the final answer. The method is evaluated against PoisonedRAG, prompt injection (PIA), adversarial decoding (AD), and jamming attacks on NQ, HotpotQA, and MS-MARCO, with Llama3.1-8B, MistralNemo-12B, and GPT4o, plus additional experiments on retrievers, model scales, and real-world datasets RedditQA and RAMDocs. The authors report large ASR reductions and ACC improvements over Vanilla RAG and over existing defenses such as RobustRAG, InstructRAG, and AstuteRAG.","tokens_in":33721,"tokens_out":4080,"duration_ms":38568,"significance":"If the reported results hold, TrustRAG would be a practically appealing defense: it is plug-and-play, requires no training, works across open- and closed-source LLMs, and maintains high accuracy even at 100% poison rates. The paper is unusually thorough in its experimental coverage: multiple attack families, multiple datasets, multiple embeddings, multiple retrievers, a scaling-law study, and two real-world evaluation sets. The perplexity analysis in Section 5.3 is a useful, honest negative result. The central caveat is that the empirical claims rest on the assumption that attacker-optimized malicious documents form a tight, separable cluster in embedding space, and on fixed thresholds (ROUGE-L 0.25, cosine 0.85) that are fitted to the tested attack distributions. The paper's own ablation shows that Stage 2 alone is not sufficient at high poison rates, so the cluster assumption is load-bearing. No error bars or significance tests are reported, and the 'diverse-context' adaptive attack still uses PoisonedRAG variants rather than an attack that explicitly breaks clustering.","major_comments":[{"comment":"The 'Diverse-Context Multiple Injection Attack' does not actually break the cluster assumption. The diverse malicious documents are generated by varying prompts within the PoisonedRAG framework, so they remain mutually similar in embedding space; the real poisoned rate after Stage-1 filtering is still substantial (e.g., NQ at 100% poison: RPR 22.0 after filtering), but the attack is not optimized for mutual dissimilarity or for evading K-means. This matters because the ablation in Table 10 shows that removing K-means at 100% poison on NQ drops ACC from 83 to 55 and raises ASR from 2 to 39, so Stage 2 alone is not sufficient at high poison rates. Please add an adaptive attack that explicitly targets the cluster assumption (e.g., diversity-regularized malicious documents, or per-document optimized embeddings that are mutually far apart) and report Stage-1 and end-to-end ACC/ASR for it.","section":"§5.4, Tables 12–13"},{"comment":"The thresholds ROUGE-L=0.25 and cosine similarity=0.85 are fixed values that appear to be selected from the observed attack distributions in Figure 2, but the paper provides no sensitivity analysis. Since the defense is a filtering mechanism, an adversary who knows these thresholds can attempt to craft documents that fall on the clean side of both thresholds, or that form a second clean-looking cluster. Please report ACC/ASR across a grid of both thresholds, and discuss how the decision boundary interacts with the attacker's optimization objectives in Eqs. (3)–(5).","section":"Appendix B.2, Figure 2"},{"comment":"All reported numbers are point estimates without error bars, confidence intervals, or significance tests. The headline claims are large ('up to 80%' ASR reduction, 'up to 30%' ACC improvement, and 'outperforming all existing defenses'), and several comparisons to ASTUTE RAG are within a few percentage points, especially in the low-ASR regime where a small number of successful attacks changes the percentage materially. Please report variance across at least a few independent runs, or bootstrap confidence intervals over queries, for the main tables.","section":"§4.2, Tables 1–16"},{"comment":"The theoretical motivation for K-means clustering is not fully supported by the threat model. Equation (5) bounds the embedding distance between the final malicious document and the attacker's initial retrieval-optimized document, not the distance among different malicious documents. The claim that malicious documents 'inherently possess high similarity' is an empirical observation about PoisonedRAG-like generation, not a consequence of Eqs. (3)–(5). Since the threat model in Section 2.2 permits arbitrary injection of documents, nothing prevents an adversary from generating malicious documents that are individually retrieval-optimized but mutually dissimilar. Please either formalize conditions under which the malicious set forms a separable cluster, or explicitly narrow the threat model and state that the defense targets attacks with clustered malicious documents.","section":"§3.2, Eqs. (3)–(5)"}],"minor_comments":[{"comment":"There are typos in the figures: 'Correct Anweser' should be 'Correct Answer' and 'MALICOUS' should be 'MALICIOUS'.","section":"Figure 1; Figure 2"},{"comment":"The entry '49,0' uses a comma instead of a decimal point; please fix the formatting.","section":"Table 1 (MistralNemo-12B, NQ row)"},{"comment":"The mechanism called 'N-gram Preservation' actually uses ROUGE-L, which is based on the longest common subsequence rather than n-gram overlap; please rename the component or clarify the terminology.","section":"§3.2"},{"comment":"Appendices H and K are both titled 'Impact of Different Retrievers' and contain overlapping results (Tables 11 and 14). Please consolidate them to avoid duplication.","section":"Appendices H and K"},{"comment":"For RAMDocs, only ACC is reported and ASR is not defined or listed, while for RedditQA both ACC and ASR are given. Please clarify whether ASR is applicable to RAMDocs and report it if so.","section":"Appendix M, Table 16"},{"comment":"The sentence 'in this paper, all the experiments are based on this setting' is confusing because the paper also reports results with 'Original Malicious Document w/ question' in the same section and in Table 13. Please specify which experiments use the w/o-question setting.","section":"§5.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical study with unusually broad coverage, and the code release is a plus. The main risk is that the robustness claim is stronger than the threat model and the adaptive-attack evaluation can justify. The authors should be asked to add a threshold sensitivity analysis and at least one attack that explicitly targets the clustering assumption; without those, the central 'up to 80% ASR reduction' claim is not fully supported. The duplicated appendix sections and small typos suggest the final version was assembled quickly, but they are fixable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful defense paper, better executed than most in this space. The two-stage pipeline—K-means cluster filtering plus ROUGE-L preservation, then LLM self-assessment with internal knowledge consolidation—is new as a combination, and the experiments are extensive: three datasets, three LLMs, four attack families, multiple retrievers and embedding models, plus real-world RedditQA and RAMDocs evaluation. The ablation study is well done; each component earns its place, and the runtime overhead (about 2x vanilla RAG) is reasonable. The observation that PoisonedRAG-style malicious documents cluster in embedding space is empirically demonstrated and useful.\n\nWhere the soft spots are, in order of importance. First, the cluster assumption is load-bearing, not an accelerator. The w/o K-means ablation shows that at 80–100% poison rates, Stage 2 alone lets ASR climb to about 39–41% on NQ, versus 1–2% with the full pipeline. If an adversary crafts mutually diverse malicious documents that do not form a tight embedding cluster, the defense loses most of its teeth at high poison rates. The paper's “diverse-context” experiments vary prompts but stay inside the PoisonedRAG family; they do not include an adversary that explicitly optimizes against K-means or builds a non-clustering set of malicious documents. Second, the thresholds (ROUGE-L 0.25, cosine 0.85) are fixed and hand-picked, with no sensitivity analysis; an attacker who knows them could tune documents to sit just above them. Third, no error bars or significance tests anywhere. Given the consistent margins this is minor, but it matters for the closer 20%-poison comparisons.\n\nThe paper is honest about its text-only scope but does not flag the clustering assumption as a limitation, which is worth raising in review. Overall, this is a solid, reproducible contribution to RAG robustness. It deserves a serious referee: the experiments are extensive, the method is practical, and the central claim likely holds for the tested attack families. I would send it to review with three requests: a sensitivity analysis of the two thresholds, an adaptive attack that explicitly targets the K-means filter by producing diverse malicious documents, and repeated-seed runs with error bars. Good reading-group material precisely because the gap between “defeats current attacks” and “robust against adversarial distribution shift” is the central tension in this literature.","headline":"TrustRAG is a solid, well-tested defense for the attack families it considers, but the clustering assumption it leans on is load-bearing and untested against a truly adaptive adversary; worth reviewing, but with clear asks.","tokens_in":34262,"tokens_out":2636,"would_cite":true,"duration_ms":25476,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TrustRAG defends retrieval-augmented generation against corpus poisoning by clustering malicious documents in embedding space and then having the LLM arbitrate between internal and external knowledge.","keywords":["retrieval-augmented generation","corpus poisoning attacks","K-means clustering filtering","LLM self-assessment","conflict resolution","robustness defenses","prompt injection","embedding-space clustering"],"falsifier":"Generate a set of targeted poisoning documents for the same question while explicitly minimizing their pairwise embedding similarity (e.g., optimizing each document independently with different random starts or different encoder surrogates) while keeping them individually retrievable; if the K-means filter then fails to separate them and the end-to-end attack success rate rises sharply above the paper's reported ASR, the cluster assumption is the load-bearing part of the defense.","tokens_in":33282,"feed_emoji":"🛡️","tokens_out":8720,"duration_ms":71457,"temperature":0.7,"pith_summary":"This paper proposes TrustRAG, a defense for retrieval-augmented generation (RAG) against corpus poisoning, where an attacker injects documents that are optimized to be retrieved and to steer the LLM toward a wrong answer. The paper's core claim is that current poisoning attacks create a detectable artefact: their malicious documents cluster tightly together in the embedding space, so a K-means step can remove them before generation, while a ROUGE-L check protects single clean documents from being discarded. After this 'clean retrieval' stage, a 'conflict resolution' stage makes the LLM generate its own internal knowledge, consolidate it with the surviving external documents, drop contradictions, and self-assess which source to trust for the final answer. On three question-answering datasets, four attack types, and three victim LLMs, the paper reports that TrustRAG lowers attack success rates by up to 80% and improves or maintains answer accuracy (up to +30%) across all tested baselines. Because the method is training-free and prompt-based, the authors present it as a plug-and-play module for open and closed LLMs alike.","feed_headline":"Two-stage filter cuts RAG poisoning attacks by up to 80%","feed_subtitle":"TrustRAG clusters malicious documents in embedding space, then has the LLM verify answers against its own knowledge.","key_machinery":"The machinery is the two-stage filter. Stage 1 is a K-means clustering with $k=2$ over the embeddings of the documents returned by the retriever; it removes the group whose members are most similar to each other, relying on the paper's empirical finding (Figure 3) that multiple malicious documents form a tight cluster. The ROUGE-L score (with a threshold of 0.25) acts as the n-gram preservation mechanism that keeps Stage 1 from discarding clean documents in single-injection cases. Stage 2 is a prompt-based self-assessment pipeline: the LLM first generates its own answer from memory at temperature 0, then consolidates internal and external documents, filters contradictions, and finally decides which source is more reliable. This machinery is what makes the defense training-free, retriever-agnostic, and applicable to closed-source models.","core_discovery":"Corpus poisoning attacks against RAG systems inject documents that are optimized to be retrieved for a target question and to steer the model toward a wrong answer. The key empirical observation in this paper is that these machine-generated malicious documents end up unusually similar to one another in the embedding space used by the retriever, even when they are paraphrased. TrustRAG's first stage runs K-means with $k=2$ on the retrieved documents and removes the tight cluster, while a ROUGE-L threshold prevents clean documents from being deleted when only one malicious document is present. After filtering, the second stage prompts the LLM to generate its own internal knowledge at temperature 0, consolidate the surviving external documents with that knowledge, discard documents that conflict, and self-assess whether the internal or external source should determine the final answer. The paper reports that on NQ, HotpotQA, and MS-MARCO, across three victim models and four attack types, this two-stage pipeline lowers attack success rates by up to 80% (often under 5% in high-poison regimes) and lifts accuracy by up to 30%, outperforming the RobustRAG, InstructRAG, and AstuteRAG baselines.","pith_inferences":["If the cluster property is the true load-bearing assumption, an adversary who forces malicious documents to spread out in embedding space (for instance by per-document optimization with different random seeds) would collapse the defense to Stage 2; the paper's own data suggest Stage 2 alone is weaker, with ASR rising to 28% on original poisoned documents at 100% poison rate.","A paraphrase-based attack that keeps malicious documents semantically similar but lexically distinct could evade the ROUGE-L preservation check, since that check assumes clean and malicious pairs have low word overlap.","The self-assessment stage importing the LLM's internal knowledge makes defense quality scale with model capability; for long-tail or novel queries where parametric knowledge is missing, the internal-knowledge veto could be silent or wrong.","A concrete extension is to evaluate TrustRAG against a genuinely diverse multi-injection attack whose documents are individually optimized to be retrievable but jointly dissimilar in embedding space, and to measure both the filter's F1 and the end-to-end ASR."],"forward_implications":["Under the tested attacks, the defense keeps attack success rates near or below 5% even when all retrieved documents are malicious, whereas vanilla RAG exceeds 90% ASR in the same settings.","Because the method is prompt-based and retriever-agnostic, it can be layered onto existing RAG pipelines without retraining, and the paper demonstrates this with three retrievers and three embedding models.","The internal-knowledge stage lets the model overrule poisoned external evidence, which also appears to reduce ordinary retrieval noise: clean-setting accuracy is not degraded and sometimes improves.","The scaling experiment shows that larger LLMs benefit more from the second stage, suggesting the defense's upper bound tracks the model's parametric knowledge rather than the filter's.","The framework sustains high accuracy and low attack success rates under diverse-context variations of PoisonedRAG, indicating that even when the first stage leaks some malicious documents, the second stage can still neutralize them."],"supporting_citations":[{"why":"Defines the PoisonedRAG corpus poisoning attack that motivates the threat model and supplies the primary malicious documents whose clustering behavior the paper exploits.","marker":"Zou et al. (2024)"},{"why":"Formalizes the query-specific adversarial prompt attacks and the retriever optimization objective that the paper's attacker model is built on.","marker":"Tan et al. (2024)"},{"why":"Provides the Adversarial Decoding attack used to generate readable malicious documents in the paper's single-injection evaluation.","marker":"Zhang et al. (2025)"},{"why":"RobustRAG baseline that aggregates isolated responses; the paper argues it collapses when malicious documents outnumber clean ones.","marker":"Xiang et al. (2024)"},{"why":"AstuteRAG baseline and source of the observation that roughly 70% of retrieved documents do not directly contain true answers, motivating the internal-knowledge stage.","marker":"Wang et al. (2024)"},{"why":"Supplies the ROUGE-L score used in the n-gram preservation mechanism that keeps Stage 1 from over-filtering clean documents.","marker":"Lin (2004)"},{"why":"Defines the Jamming denial-of-service attack tested as a single injection and the perplexity-based detector whose effectiveness the paper empirically contradicts.","marker":"Shafran et al. (2024)"},{"why":"Supports the claim that LLM internal knowledge can supplement or rebut retrieved external information, the basis of Stage 2's design.","marker":"Sun et al. (2022)"}],"fun_headline_variants":["Cluster filtering stops 80% of RAG poisoning attacks","TrustRAG: detect malicious clusters, then LLM self-check","Two-stage defense: cluster poison docs, verify with internal knowledge","80% cut in RAG attacks: cluster them out, then self-assess","RAG poisoning? TrustRAG clusters the bad docs and checks internally"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The defense presumes that malicious documents produced by an attack will crowd together in embedding space; if an adversary generates documents that are varied enough to avoid forming a cluster, the K-means stage cannot separate them, and the second stage must do all the work.","fun_headline_variants_meta":{"raw":{"variants":["Cluster filtering stops 80% of RAG poisoning attacks","TrustRAG: detect malicious clusters, then LLM self-check","Two-stage defense: cluster poison docs, verify with internal knowledge","80% cut in RAG attacks: cluster them out, then self-assess","RAG poisoning? TrustRAG clusters the bad docs and checks internally"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1490,"prompt_tokens":943,"completion_tokens":547,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":469}},"tokens_in":559,"tokens_out":547,"duration_ms":5602,"temperature":1.0,"reasoning_tokens":469,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:39:22.182829+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate a set of targeted poisoning documents for the same question while explicitly minimizing their pairwise embedding similarity (e.g., optimizing each document independently with different random starts or different encoder surrogates) while keeping them individually retrievable; if the K-means filter then fails to separate them and the end-to-end attack success rate rises sharply above the paper's reported ASR, the cluster assumption is the load-bearing part of the defense.","supporting_citations":[],"review_version":1}