{"id":"2f39a36f-d0e2-4d3d-b338-ad4467b2e940","arxiv_id":"2508.18652","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A universal knowledge-corruption attack uses as few as 100 crafted texts to hijack responses to thousands of diverse user queries in retrieval-augmented generation.","lead":"This paper presents UniC-RAG, an attack that poisons a RAG system's knowledge database with a small number of crafted texts to redirect or block answers to thousands of unrelated user queries. A generalist should care because if it works, AI assistants that ground answers in retrieved documents become cheaply and widely exploitable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's headline 'over 90% ASRs' is not supported by Table 2: only HotpotQA exceeds 90% ASR; NQ, MS-MARCO, and Wikipedia range from 68% to 87%, and the aggregate ASR is about 81%.","rationale":"The most load-bearing condition for the central claim is not a hidden threat-model assumption: the paper explicitly assumes white-box retriever access, and the reader already flags that. The stronger problem is that the advertised quantitative outcome conflicts with the paper's own experimental table. This is an internal inconsistency, not a matter of consensus. If the abstract's numbers are corrected, the contribution narrows but does not disappear: the method still achieves high attack rates on HotpotQA and non-trivial rates elsewhere. A cross-retriever transfer test would be valuable, but it would test a stated limitation rather than the headline claim. The abstract's 'over 90%' assertion is the single most load-bearing sentence because it is what a reader will carry away, and it is falsified by Table 2. Therefore the reader's CONDITIONAL verdict stands, with the requested revisions focusing on accurate, dataset-specific reporting.","tokens_in":24412,"tokens_out":8456,"duration_ms":96874,"concrete_test":"Reproduce Table 2's final UniC-RAG row from the reported numbers or released code, and compute the aggregate mean ASR across the 12 dataset/top-k cells. Optionally rerun with m=2000, n=100 on NQ, HotpotQA, MS-MARCO, and Wikipedia and record exact ASR values. If the aggregate mean is below 90% or only HotpotQA exceeds 90%, the abstract's universal 'over 90% ASRs' claim is unsupported and should be revised to per-dataset ranges.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim is the abstract's statement that UniC-RAG 'could achieve over 90% RSRs and ASRs by injecting 100 adversarial texts ... to simultaneously attack 500-2,000 queries.' This is the claim the evaluation must support. Table 2's full method row ('Similarity Based Clustering') shows ASR >90% only on HotpotQA (90.8/91.4/92.2 for top-5/10/20). On NQ the ASR is 82.2-87.4, on MS-MARCO it is 73.2-78.0, and on the Wikipedia-scale database it is 68.2-77.0. Averaging the 12 ASR cells gives about 81.2%, not over 90%. The RSR average is about 93.2%, but the abstract states RSRs and ASRs jointly. Additionally, the Figure 3 scalability results for m up to 2,000 are shown only as unlabeled curves, and the text says RSR decreases monotonically with m and ASR declines after a peak, so the 2,000-query regime is not established at >90%. Thus the strongest advertised result appears to be a cherry-picked best case (HotpotQA) rather than a general property of the method. The underlying optimization may still be a valid contribution, but the central claim as stated is internally inconsistent with the paper's own tables.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes UniC-RAG, a universal knowledge-corruption attack against retrieval-augmented generation (RAG). Given a set of m target queries, the method partitions them into n clusters using a balanced similarity-based clustering algorithm, then for each cluster optimizes an adversarial text consisting of a retrieval prefix (optimized with HotFlip) and a fixed prompt-injection suffix. The claimed contribution is that 100 injected texts can simultaneously attack 500–2,000 diverse queries in databases containing millions of texts, with over 90% retrieval success rate (RSR) and attack success rate (ASR). The evaluation covers four datasets (NQ, HotpotQA, MS-MARCO, and a 47.8M-text Wikipedia dump), four retrievers, seven LLMs, several baselines, and three defenses.","tokens_in":126,"tokens_out":4064,"duration_ms":68492,"significance":"If the headline claim were fully substantiated, the paper would describe a practically important attack: a small number of injected texts could redirect or block responses for a broad set of user queries in RAG systems, with implications for phishing, denial-of-service, and harmful-command attacks. The work has several genuine strengths: it is the first to formulate the universal multi-query attack as an optimization problem with explicit clustering; it provides a systematic empirical study across datasets, retrievers, and LLMs; it evaluates three defenses; and it compares against multiple baselines. The balanced clustering and greedy initialization ideas are sensible and appear to improve over simple baselines. However, the central quantitative claim in the abstract is not supported by the paper’s own tables, and several evaluation choices weaken the evidence. The method may still be a valid contribution, but the advertised result needs substantial revision or additional experiments.","major_comments":[{"comment":"The abstract claims “over 90% RSRs and ASRs by injecting 100 adversarial texts… to simultaneously attack 500–2,000 queries.” Table 2 does not support this joint claim. Only HotpotQA has ASR ≥ 90.8%; NQ ranges from 82.2% to 87.4%, MS-MARCO from 73.2% to 78.0%, and Wikipedia from 68.2% to 77.0%. Averaging the 12 ASR cells gives approximately 81%, not over 90%. The RSR average is about 93%, but the abstract states RSRs and ASRs jointly. Moreover, §5.3.2 and Figure 3 report that RSR decreases monotonically with m and ASR declines after a peak; no experiment shows >90% at m=2,000. The headline result should be revised to reflect the actual data, or new experiments must be provided that support the stated range.","section":"Abstract and §5.2, Table 2"},{"comment":"No error bars, confidence intervals, or repeated-seed results are reported anywhere in the paper. RSR and ASR are proportions computed from a single sample of 500 queries, and many comparisons differ by only 1–2 percentage points. Without repeated runs, it is unclear whether differences between clustering methods, baselines, or hyperparameter settings are statistically meaningful. The paper should report variance over at least several independent runs. In addition, the axes in Figure 3 lack numeric tick labels and the curves are not annotated, so the reader cannot verify the quantitative behavior described in the text.","section":"§5.3.2, Figure 3"},{"comment":"The text says UniC-RAG “consistently achieves high RSRs and ASRs” across retrievers, but Table 7 shows DPR-Single and DPR-Multi with top-5 RSR/ASR of 61.4/59.6 and 64.2/62.0, respectively, and top-20 ASR of 73.0 and 77.6. These are far below the HotpotQA values with Contriever (RSR/ASR 99.6/92.2 at top-20). Given that the threat model assumes white-box access to the exact retriever, the retriever-dependence of the attack is directly relevant to the claimed universality. The paper should either qualify the claim or investigate why DPR variants are substantially less vulnerable.","section":"§5.3.1, Table 7"},{"comment":"The threat model states that “the adversarial texts should also be able to transfer to queries beyond those in Q” and discusses a “more challenging scenario where the attacker does not know the user query set Q” and performs a transfer attack to an unseen query set. No experiment in Section 5 evaluates transfer to a disjoint or unseen query set. The paraphrasing defense experiment uses GPT-4o-mini to paraphrase queries in the same Q, not a held-out set with different topics. The transferability claim is a stated goal and should be evaluated, or removed from the threat model.","section":"§3.1 vs. §5"}],"minor_comments":[{"comment":"The notation “E(Pr_i ⊕ Pg_i)” is ambiguous: E is an encoder that takes text as input, so the concatenation should be described as a token sequence, not a vector operation. Define ⊕ explicitly.","section":"§4.3, Eq. (4)"},{"comment":"The abstract claims “over 90% RSRs and ASRs” while §5.2 itself reports an average ASR of 81.2%. These statements are inconsistent and should be reconciled.","section":"Abstract vs. §5.2"},{"comment":"The x-axes show hyperparameter values but the y-axis labels and axis ticks are missing; the reader cannot read off the exact RSR/ASR values for m=2000, n=200, etc. Add numeric labels or a table.","section":"Figure 3"},{"comment":"The Prompt Injection baseline is described as achieving RSR=0 and ASR=0. This is expected because the baseline makes no attempt to optimize retrieval. State explicitly that this baseline is included only as a sanity check on the need for retrieval optimization, not as a comparable attack.","section":"§5.2, Table 2"},{"comment":"In Table 3, PoisonedRAG and Jamming rows show RSR and ASR nearly identical. This suggests ASR is computed only over the queries for which the adversarial text was retrieved, while for UniC-RAG the two metrics differ. Clarify the denominator used for ASR in each table so comparisons are apples-to-apples.","section":"§5.2, Table 3"},{"comment":"The RAG system prompt in Appendix A says “Answer the query given the information in those contexts.” It would help to state whether the LLM is instructed to ignore context that is irrelevant or conflicting, since that affects the ASR for the malicious-link objective.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a security venue and the attack idea is novel enough to merit consideration. The main issue is that the paper's strongest advertised result—over 90% ASR across 500–2,000 queries—is not supported by its own tables. I would not reject the paper outright, but the authors must either substantially soften the claim or provide new experiments (e.g., per-dataset and per-m results, with repeated runs) that directly support it. I would also ask the authors to clarify the evaluation denominator for ASR in baseline comparisons and to run a transfer experiment if they keep the transferability claim in the threat model."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"X, here's my read.\n\nThe genuinely new thing is the threat model: one small set of adversarial texts that can simultaneously hit a large, diverse set of queries. Prior work targets single queries, similar-topic queries, or trigger-based backdoors. That distinction is real, and it's worth taking seriously. The balanced similarity-based clustering to make the optimization tractable, and the greedy initialization from the previous prefix, are both sensible engineering choices. The experiments span four datasets, several retrievers, and seven LLMs, which gives the empirical core some breadth.\n\nBut the abstract's central claim is not supported by the paper's own data. It says UniC-RAG achieves over 90% RSR and ASR by injecting 100 texts to attack 500-2,000 queries. In Table 2, ASR over 90% appears only on HotpotQA. On NQ it's 82-87%, on MS-MARCO 73-78%, and on Wikipedia 68-77%. The four-dataset average ASR is about 81%, as the text itself says in Section 5.2. So the 'over 90%' claim is a cherry-picked best case. That's not a fatal flaw to the method, but it is a serious reporting problem, and the abstract needs to be rewritten to match the aggregate results.\n\nOther soft spots: no error bars or repeated seeds, so we don't know how stable the RSR/ASR numbers are. The threat model promises a transfer scenario where the attacker uses a proxy query set Q' and attacks unseen Q, but I couldn't find any evaluation of that. Since RSR is measured on the same retriever used for optimization, transfer is exactly the open question. Baseline comparisons are not fully apples-to-apples because PoisonedRAG and Jamming are single-query attacks and get only 100 texts in the main table, though the paper does give them 500 texts in Table 3 and UniC-RAG still holds up. No code or data is released, which makes it harder to check the optimization details.\n\nThe authors are honest about the white-box retriever assumption and the retrieval/manipulation trade-off in Section 7, which I credit. But as it stands, the paper is a solid idea with over-claimed headline results.\n\nI'd send it to peer review, because the core contribution deserves referee time and the overclaiming is correctable. I'd ask the authors to fix the abstract, add seeds/error bars, and evaluate the transfer setting.","headline":"The universal-attack setup is worth your time, but the paper's headline number—'over 90%'—is contradicted by its own Table 2; average ASR is about 81%.","tokens_in":25300,"tokens_out":2975,"would_cite":true,"duration_ms":30163,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a RAG system's knowledge base can be corrupted with just 100 injected texts so that over 90% of a diverse set of 500–2,000 user queries are redirected to attacker-chosen outputs.","keywords":["retrieval-augmented generation","knowledge corruption attack","universal poisoning","prompt injection","dense retrieval","adversarial text optimization","denial-of-service","clustering"],"falsifier":"Take the 100 crafted texts from this paper, plug them into a RAG system whose retriever is a different, closed-source embedding model (or a retriever fine-tuned on a different domain), and measure retrieval success; separately measure attack success on an LLM explicitly trained to ignore instructions that appear in retrieved documents. If retrieval success falls to the baseline retrieval rate, or attack success collapses while retrieval success stays high, the universality claim is bounded by retriever access and LLM compliance.","tokens_in":1698,"feed_emoji":"🎯","tokens_out":1746,"duration_ms":77270,"temperature":0.7,"pith_summary":"This paper argues that retrieval-augmented generation (RAG) systems—applications that answer a user's question by first retrieving a few relevant texts from a large database and then feeding them to an LLM—are vulnerable to a new kind of \"universal\" knowledge corruption. The authors claim that an attacker who can insert a small number of crafted texts into the knowledge database can simultaneously compromise hundreds or thousands of unrelated user queries, not just one query or one topic. Concretely, UniC-RAG is said to reach over 90% retrieval success and attack success by injecting 100 texts into databases with millions of entries to target 500–2,000 diverse questions. The method's core move is to group the target queries into balanced, semantically similar clusters and then optimize one adversarial text per cluster so that the group's queries all retrieve it; a fixed prompt-injection suffix attached to every text then steers the LLM toward the attacker's chosen response. If the results hold, any widely deployed RAG service built on an openly available retriever is cheap to attack at scale, and the defenses tested—paraphrasing, larger context windows, and robust-RAG pipelines—do not stop it.","feed_headline":"100 poisoned texts redirect 2,000 AI search queries at once","feed_subtitle":"New attack groups similar questions, optimizes one malicious text per group, and beats current defenses—so knowledge-base poisoning scales.","key_machinery":"The central object is the two-part adversarial text P = P_r ⊕ P_g: a retrieval prefix that makes the text surface in the top-k results for many queries, and a fixed universal prompt-injection suffix that, once retrieved, makes the LLM obey the attacker's objective. The mechanism that makes \"one text, many queries\" work is balanced similarity-based clustering: queries are embedded, partitioned into nearly equal-size groups of semantically similar questions, and one adversarial text is optimized per group. The optimization maximizes the average embedding similarity between the adversarial text and the group's queries—so retrieval is achieved by being close to the cluster's centroid in embeddin","core_discovery":"On its own terms, UniC-RAG establishes a scaling result for poisoning RAG systems: with a white-box retriever, a universal attack can be constructed whose cost grows with the number of clusters, not the number of queries. The paper formalizes the attacker's problem as maximizing, over a set of injected texts Γ, the fraction of queries in Q whose LLM output satisfies a fixed objective O. It solves this by partitioning Q with a balanced similarity-based clustering algorithm that keeps cluster sizes nearly equal while maximizing intra-cluster semantic similarity, and then, for each cluster, performing token-level gradient optimization of a retrieval prefix P_r that maximizes average embedding s","pith_inferences":["If the same semantic-clustering trick transfers to black-box settings, a realistic attacker could pick any public target topic, embed a sample of queries with a surrogate retriever, and still hit a meaningful fraction; the paper only establishes transfer across paraphrases, not across retriever architectures.","The clustering step is reusable beyond attacks: any embedding-space content placement problem—like inserting one document that should be surfaced for many queries—can be recast as balanced similarity clustering, so the technique may have legitimate uses in search and recommendations.","The reported trade-off, in which longer retrieval prefixes raise retrieval success but can dilute the injection suffix and lower attack success, predicts that content filters that truncate or re-rank long documents could partially blunt the attack without dedicated defenses.","Since adversarial texts are optimized against public open-source retrievers, defenders could monitor for near-duplicate embedding patterns across injected texts; the attack's reliance on a shared universal suffix is a detectable signature."],"forward_implications":["Injecting 100 texts can redirect or block responses for 500–2,000 diverse queries in a database of millions of texts, so the attack budget scales with the number of clusters, not with the number of victims.","Because the same suffix serves all clusters, the attacker can switch objectives—malicious link, harmful command, denial-of-service—without redoing the retrieval optimization.","Paraphrasing user queries does not neutralize the attack, since paraphrases keep semantic meaning and the adversarial text is optimized in embedding space.","Expanding the context window makes the attack more likely to succeed, contrary to its effect on single-query poisoning, because more retrieved slots mean more chances for an adversarial text to appear.","Robust-RAG defenses that use instruction-tuned rationalization reduce attack success but still leave it high (for example, 70.4% for a denial-of-service objective), so current defenses are insufficient."],"supporting_citations":[{"why":"Supplies the two-part adversarial-text decomposition (retrieval prefix plus content) and the single-query poisoning baseline and context-window-expansion defense that UniC-RAG is compared against.","marker":"[8]"},{"why":"Supplies the denial-of-service prompts used as the injection suffix and the LLM-based evaluation method for measuring whether a query was refused.","marker":"[9]"},{"why":"Supplies the gradient-based token-level text optimization used to maximize retrieval similarity for each cluster.","marker":"[33]"},{"why":"Supplies the prompt-injection technique used as the default method for crafting the universal suffix P_g.","marker":"[37]"},{"why":"Supplies the corpus-poisoning optimization baseline and the top-k retrieval-success metric; the paper's extended baseline is built on it.","marker":"[45]"},{"why":"Supplies the paraphrasing defense that UniC-RAG is shown to remain effective against.","marker":"[46]"},{"why":"Supplies the robust-RAG defense that generates rationales, evaluated against UniC-RAG's denial-of-service objective.","marker":"[47]"}],"fun_headline_variants":["Universal poisoning: 100 texts hijack 2,000 RAG queries","Cluster-based attack scales: poison 2,000 queries with 100 texts","Universal RAG attack: 100 injected texts beat 2,000 queries","Scaling poisoning: cluster attacks hit diverse queries via 100 texts","100 texts poison 2,000 queries: universal RAG attack scales"],"cache_read_input_tokens":27008,"weakest_assumption_plain":"The load-bearing premise is that the attacker can run the exact retriever model in white-box form to optimize the prefixes, and that the target LLM will follow the prompt-injection suffix once it appears in retrieved context.","fun_headline_variants_meta":{"raw":{"variants":["Universal poisoning: 100 texts hijack 2,000 RAG queries","Cluster-based attack scales: poison 2,000 queries with 100 texts","Universal RAG attack: 100 injected texts beat 2,000 queries","Scaling poisoning: cluster attacks hit diverse queries via 100 texts","100 texts poison 2,000 queries: universal RAG attack scales"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000465,"raw_usage":{"total_tokens":2187,"prompt_tokens":804,"completion_tokens":1383,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":1299}},"tokens_in":548,"tokens_out":1383,"duration_ms":11177,"temperature":1.0,"reasoning_tokens":1299,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:19:07.472516+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 100 crafted texts from this paper, plug them into a RAG system whose retriever is a different, closed-source embedding model (or a retriever fine-tuned on a different domain), and measure retrieval success; separately measure attack success on an LLM explicitly trained to ignore instructions that appear in retrieved documents. If retrieval success falls to the baseline retrieval rate, or attack success collapses while retrieval success stays high, the universality claim is bounded by retriever access and LLM compliance.","supporting_citations":[{"cited_title":"Poisonedrag: Knowledge poisoning attacks to retrieval-augmented generation of large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the two-part adversarial-text decomposition (retrieval prefix plus content) and the single-query poisoning baseline and context-window-expansion defense that UniC-RAG is compared against."},{"cited_title":"Machine against the rag: Jamming retrieval-augmented generation with blocker documents,","cited_arxiv_id":null,"evidence_quote":"Supplies the denial-of-service prompts used as the injection suffix and the LLM-based evaluation method for measuring whether a query was refused."},{"cited_title":"Hotflip: White-box adversarial examples for text classification,","cited_arxiv_id":null,"evidence_quote":"Supplies the gradient-based token-level text optimization used to maximize retrieval similarity for each cluster."},{"cited_title":"Formalizing and benchmarking prompt injection attacks and defenses,","cited_arxiv_id":null,"evidence_quote":"Supplies the prompt-injection technique used as the default method for crafting the universal suffix P_g."},{"cited_title":"Poisoning retrieval corpora by injecting adversarial passages,","cited_arxiv_id":null,"evidence_quote":"Supplies the corpus-poisoning optimization baseline and the top-k retrieval-success metric; the paper's extended baseline is built on it."},{"cited_title":"Baseline defenses for adversarial attacks against aligned language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the paraphrasing defense that UniC-RAG is shown to remain effective against."},{"cited_title":"InstructRAG: Instructing retrieval-augmented generation via self-synthesized rationales,","cited_arxiv_id":null,"evidence_quote":"Supplies the robust-RAG defense that generates rationales, evaluated against UniC-RAG's denial-of-service objective."}],"review_version":1}