{"id":"0ae1e52c-3f6b-4e9d-9177-c179eaf5d3fc","arxiv_id":"2608.02678","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A single poisoned document that explicitly names and refutes the correct answer can steer RAG systems toward attacker-chosen wrong answers, with effects that vary by target model.","lead":"This paper introduces DenialRAG, a corpus-poisoning attack on retrieval-augmented question answering that deliberately names the correct answer inside the poisoned document, denies it, and substitutes a wrong answer. On smaller models such as Mistral-7B, this 'embedded denial' outperformed four published attacks, and it retained partial effectiveness under all five tested defenses.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Strict substring match counts refutations as attack successes, inflating DenialRAG ASR and confounding the denial-ablation's comparative claims.","rationale":"The paper has a clear threat model, a systematic evaluation across models and defenses, and a sensible ablation design. However, the outcome variable is the linchpin: every ASR number, every comparison, and the ablation's 'most influential component' conclusion rely on the substring match. The reader's weakest assumption correctly identifies this. A strict re-annotation is a single, feasible check that would resolve whether the headline numbers are real. This does not reject the underlying concept; it suggests the reported magnitudes may be overestimated. If the stricter metric confirms similar ASR, the paper is substantially stronger; if not, the comparative claims need revision.","tokens_in":16842,"tokens_out":5573,"duration_ms":62396,"concrete_test":"Re-annotate the no-defense Mistral-7B responses on NQ (N=100) using a strict success definition: the final answer selects Y and does not negate it (e.g., does not contain 'not 24' or '24 is incorrect'). Count how many of the original 89 'successes' are actually refutations. Compute ASR under the strict definition. If the strict ASR is below 79% or differs by more than 10 points from the reported 89%, the substring metric materially inflates the results.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's success criterion (§3.4) counts a response as successful if it merely contains the attacker-chosen answer Y via strict substring matching. Because DenialRAG's poisoned passage explicitly names and denies X, generated responses often quote the passage: 'The answer is not 24, it is 23' contains '24' and would be scored as a success for Y=24 even though the model explicitly rejected Y. The flaw is disproportionately damaging to DenialRAG: the denial component makes Y more likely to be mentioned in a refutation, so the ablation's 17.1 pp drop when removing the denial may reflect reduced mention frequency, not reduced adoption of Y. All comparative tables (Tables 1–3) and the model-regime analysis are based on this metric, so the central claim that DenialRAG achieves 89/94/86% ASR on Mistral-7B is unsupported until this is addressed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DenialRAG, a single-document RAG corpus-poisoning attack in which one short (≤100-word) passage explicitly names the correct answer X, denies it, and asserts an attacker-chosen wrong answer Y, with the goal of steering the generator to output Y. The attack is evaluated on three BEIR QA datasets (NQ, HotpotQA, MS-MARCO), eight target LLMs, four published single-document attack baselines, and five inference-time defenses. The central empirical claims are that DenialRAG achieves the highest attack success rate on all three Mistral-7B datasets (89/94/86% ASR), remains competitive on several other models, and that the embedded denial is the most influential component, based on a component ablation showing a 17.1-percentage-point mean ASR drop when denial is removed. The paper also reports a stability analysis across cost-tier and frontier models and retrieval-depth/position sensitivity probes.","tokens_in":17047,"tokens_out":4586,"duration_ms":53977,"significance":"If the qualitative results survive re-measurement, the paper makes a useful contribution to the RAG-poisoning literature: it identifies a plausible attack mechanism—explicitly naming and refuting the correct answer inside the poisoned passage—that interacts non-trivially with model family, model tier, and defense strategy. The systematic sweep over models, baselines, and defenses is valuable, and the attack definition is clear and independent of the measured outcomes, so there is no circularity. However, the central quantitative claims rest on a success criterion that likely over-counts DenialRAG successes, so the currently reported magnitudes and comparative rankings are not established. The paper's strengths include a well-specified threat model, a large and clearly presented evaluation matrix, and a reproducible configuration (Contriever retriever, seed 12, T=0.1) that should allow the authors to re-score their traces.","major_comments":[{"comment":"The success criterion counts a response as successful if it contains the attacker-chosen answer Y via strict substring matching. This also counts responses that explicitly reject Y. For example, with X=23 and Y=24, a response such as \"The answer is not 24; it is 23\" contains the substring '24' and would be scored as a successful attack. Because DenialRAG's defining component is a passage that names Y and denies X, generated responses are likely to mention both values even when the model ultimately adopts X. Consequently, the headline ASRs (89/94/86% on Mistral-7B) and the comparative conclusions in Tables 1–3 do not establish that the model was steered to Y. The authors should re-score the generated responses with a negation-aware metric (e.g., excluding 'not Y' patterns or extracting the final answer), and separately report Y-mention and Y-adoption rates.","section":"§3.4, §5.1"},{"comment":"The component ablation's main claim—that removing the denial causes the largest ASR drop (17.1 percentage points on average)—is confounded by the same substring metric. The denial sentence explicitly contains both X and Y and invites the model to mention Y in the course of refuting it. Removing the denial can therefore reduce Y-mention frequency without reducing genuine adoption of Y. The claim that the embedded denial is 'the most influential tested component' is unsupported until responses are classified by semantic adoption rather than by substring containment. This is particularly important because the ablation is used to argue that DenialRAG's mechanism is specifically the denial, not merely assertion of Y.","section":"§5.4, Table 3"},{"comment":"All ASR values are point estimates over N=100 queries with a single seed (seed 12) and temperature 0.1, and no confidence intervals or significance tests are reported. Many comparisons in the paper are small in magnitude, e.g., Table 1 NQ/GPT-4o-mini shows PIA-direct at 86% versus DenialRAG at 84%, and several 'highest' or 'competitive' claims rest on differences of a few points. The authors should report binomial confidence intervals and, where comparing attacks or defenses, use paired tests across the same 100 queries, or run multiple seeds. This is needed to support the paper's model-dependence claims beyond individual point estimates.","section":"§5.1, Tables 1–2"},{"comment":"The main evaluation reports generator-side ASR but does not report how often the poisoned document is actually retrieved into the top-k context. A single-document poisoning attack is only meaningful end-to-end if the poisoned passage is retrieved with sufficiently high probability. Tables 4 and 5 explicitly condition on the poisoned passage being included in the retrieved set, so they do not address this. The authors should report the retrieval rate (fraction of queries for which the poisoned passage appears in the top-5) per dataset and model, since a high generator-side ASR combined with low retrieval would materially change the practical threat assessment.","section":"§5.1, Tables 4–5"}],"minor_comments":[{"comment":"The header contains a typo: 'RFFECTIVENESS' should be 'EFFECTIVENESS'.","section":"Table 1"},{"comment":"Some cells appear without clear delimiters (e.g., '7789' in the NQ/Mistral-7B row). Use explicit spacing or cell boundaries in the camera-ready version.","section":"Table 1"},{"comment":"The figure caption refers to four 'colour-coded' parts, but the text is not visibly color-coded in the submitted version. Label the four components directly in the figure or remove the color reference.","section":"Figure 2"},{"comment":"The reproducibility statement says code and traces will be released after acceptance. For a security paper whose main evidence is empirical, a reviewable artifact or at least the exact prompt templates and a sample of scored traces should be made available to reviewers.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim currently hinges on a metric artifact, but the issue is addressable by re-scoring the existing traces with a negation-aware or answer-extraction metric. If the re-scored results still show DenialRAG dominating on Mistral-7B, the paper could be a solid empirical contribution. If the effect disappears, the contribution reduces to a cautionary negative result. I see no circularity or fabrication; the attack definition is clean. I would encourage the editor to give the authors the opportunity to re-analyze, but the current version is not acceptable as is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is genuinely new: instead of avoiding the correct answer X, DenialRAG names it, denies it, and resolves the conflict inside the poisoned passage. That is a real departure from PoisonedRAG, AuthChain, and CorruptRAG, and the systematic sweep across eight models, three datasets, and five defenses is more thorough than most attack papers. The ablation is also well designed and shows, on its own terms, that the denial component matters.\n\nThe problem is the success metric. The paper counts a response as successful if it merely contains the attacker-chosen answer Y via strict substring matching. Because DenialRAG explicitly names both X and Y, a model that rejects the attack can easily produce \"the answer is not 24, it is 23\"—which contains 24 and is scored as a success. Baselines that avoid X are less likely to elicit this kind of refutation mention, so the bias hits DenialRAG hardest. That means the headline numbers (89/94/86 on Mistral-7B) and the 17.1-point ablation drop could partly reflect mention frequency rather than genuine adoption of Y. The reader's stress-test note is correct, and the paper does not address it.\n\nOther soft spots are more minor but worth listing. N=100 per cell with no confidence intervals or significance tests makes 1–2 percentage-point differences uninterpretable. The paper says it keeps only queries where the clean RAG system answers correctly but does not report how many were excluded, which matters for comparability. The attack-generation LLM is unnamed, and code/data are promised only after acceptance.\n\nNone of this kills the underlying idea. The mechanism is plausible and the framing is coherent. But the empirical support, as it stands, is not solid enough to accept the specific claims. I would want a re-run with a stricter criterion—for example, exact final-answer matching, or at least counting cases where the response explicitly rejects Y as failures.\n\nWho is this for? Researchers working on RAG security and on evaluation methodology for poisoning attacks. It is a useful trigger for discussion of metric design, but I would not cite the numbers yet. Send it to peer review—it deserves a serious referee—but expect a major revision focused on the success criterion and uncertainty reporting.","headline":"Novel attack idea with a real evaluation gap: the substring-match metric likely inflates DenialRAG's reported ASR, and the numbers need re-analysis before the comparative claims can be trusted.","tokens_in":17527,"tokens_out":2325,"would_cite":false,"duration_ms":29909,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DenialRAG: a single poisoned passage that names the correct answer and denies it steers RAG systems to wrong answers more reliably than merely asserting a false one.","keywords":["DenialRAG","RAG poisoning","corpus poisoning","single-document attack","retrieval-augmented generation","knowledge conflict","embedded denial","attack success rate"],"falsifier":"Re-run the Mistral-7B NQ, HotpotQA, and MS-MARCO evaluations with a stronger scoring rule that counts a success only when Y appears as the asserted answer (for example, not immediately preceded by 'not', 'incorrectly', or 'actually wrong'), and compare the resulting ASRs with the reported 89%, 94%, and 86%. A large drop would falsify the claim that the embedded denial makes the model adopt Y; instead it would mean the model often mentions Y while rejecting it.","tokens_in":16730,"feed_emoji":"⚠️","tokens_out":6360,"duration_ms":51799,"temperature":0.7,"pith_summary":"This paper proposes DenialRAG, a corpus-poisoning attack on retrieval-augmented generation (RAG) systems that needs only a single inserted document. The core idea inverts the usual assumption that a poisoned passage should avoid mentioning the correct answer: DenialRAG explicitly names the correct answer X, explains why it is wrong, and asserts an attacker-chosen answer Y inside the same short passage. The paper reports that this approach outperforms four published single-document attacks on all three datasets when the generator is Mistral-7B, with attack success rates of 89%, 94%, and 86%, and remains competitive on several other models. It argues that the embedded denial, not the bare assertion of Y, does the work, since removing it costs 17.1 percentage points of mean success. The broader claim is that RAG poisoning risk cannot be captured by a single attack family: effectiveness depends on the interaction between the poisoning mechanism and the target model.","feed_headline":"One denied passage hijacks RAG answers 94% of the time","feed_subtitle":"A poison document that names the true answer, calls it wrong, and asserts a false one beats all baselines on Mistral-7B","key_machinery":"The embedded-denial passage: a four-part, at-most-100-word poisoned document that (1) opens by asserting the attacker's answer Y as definitive, (2) incorporates salient entities from the question for retrieval alignment, (3) explicitly names the correct answer X and gives a reason it should be rejected, and (4) closes by reaffirming Y with an authority-style detail. Its work is to make the X-versus-Y conflict self-contained: the model never has to weigh X against Y, because the passage has already settled the question in favor of Y before generation begins. The ablation identifies part (3) as the load-bearing component.","core_discovery":"DenialRAG is a single-document, black-box attack on RAG question answering. From a target question, the correct answer X, and a wrong answer Y, two LLM calls produce a ≤100-word passage that opens by asserting Y, weaves in the question's salient terms, explicitly denies X with a reason earlier sources were wrong, and closes with an institutional citation. The mechanism is the embedded denial: the passage resolves the X-versus-Y conflict locally, so the generator sees the correct answer already acknowledged, discounted, and replaced. The paper's central claim is that this denial is the most influential component, removing it causes the largest drop (17.1-point mean), and that the attack reach","pith_inferences":["A natural next test, not run in the paper, is whether responses like \"the answer is not 24, it is 23\" are being counted as a success; the strict substring match would count them, so the true adoption rate of Y could be lower than the reported 89–94%.","The same denial structure should transfer to other fact-asserting RAG tasks such as entity linking or timeline questions, because the mechanism acts on the internal X-versus-Y conflict rather than on any dataset-specific surface form.","The component ablation predicts that defenses which ask the model to cross-check against parametric knowledge will interact with denial position: since moving the denial later already weakens the attack, a skeptical prompt is likely to land harder on late-denial passages.","The stability result suggests an attack zoo: future RAG risk assessment should include at least one denial-style, one assertion-style, and one update-style poisoning document, since their frontier-model transfer curves differ."],"forward_implications":["A single adversarial document can override the model's correct answer even when it openly contradicts it, on models as strong as Mistral-7B.","Explicit denial of the correct answer is the main driver of this attack; bare assertion of Y is weaker by a 17.1-point mean.","No tested inference-time defense is uniformly effective: each leaves residual attack success in at least one setting.","Attack ranking is model-dependent; for example, PIA-direct dominates on GPT-5.2 while CorruptRAG-AK dominates on GPT-5.5.","Retrieval depth and position within the context do not materially change DenialRAG's success on the two tested models."],"supporting_citations":[{"why":"PoisonedRAG-N1: the direct-assertion baseline DenialRAG must beat, and the source of the evaluation protocol (datasets, retriever, substring-match ASR).","marker":"[5]"},{"why":"AuthChain: the citation-fabrication baseline whose single-document passage framework DenialRAG extends with a denial component.","marker":"[11]"},{"why":"CorruptRAG-AK: the freshness-update framing baseline that dominates in some frontier-model cells.","marker":"[12]"},{"why":"PIA-direct: the assertive injection baseline against which DenialRAG compares.","marker":"[6]"},{"why":"Contriever: the dense retriever used in all experiments, standard in the baselines.","marker":"[13]"},{"why":"Supplies the motivating behavior: when context conflicts with parametric knowledge, instruction-tuned LLMs often follow the context, so an embedded denial can override the correct answer.","marker":"[23]"},{"why":"Supports the stability explanation that frontier models cross-check retrieved claims against parametric knowledge more aggressively.","marker":"[26]"},{"why":"BEIR: source of the three open-domain QA datasets (NQ, HotpotQA, MS-MARCO) used for evaluation.","marker":"[28]"}],"fun_headline_variants":["DenialRAG: deny the true answer in one passage, hijack RAG","One denied passage beats all baselines on Mistral-7B","Poison RAG with a passage that names then denies the truth","Explicit denial of correct answer in single doc flips RAG","DenialRAG: embedded denial is top driver of RAG poisoning"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The results assume the success metric truly captures the model adopting the attacker's answer: any response that contains the answer string Y, even \"the answer is not 24, it is 23\", counts as a successful attack, so if most matches are in fact rejections, the reported ASRs and every comparison built on them collapse.","fun_headline_variants_meta":{"raw":{"variants":["DenialRAG: deny the true answer in one passage, hijack RAG","One denied passage beats all baselines on Mistral-7B","Poison RAG with a passage that names then denies the truth","Explicit denial of correct answer in single doc flips RAG","DenialRAG: embedded denial is top driver of RAG poisoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000235,"raw_usage":{"total_tokens":1382,"prompt_tokens":838,"completion_tokens":544,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":448}},"tokens_in":582,"tokens_out":544,"duration_ms":5545,"temperature":1.0,"reasoning_tokens":448,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T00:04:58.570366+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Mistral-7B NQ, HotpotQA, and MS-MARCO evaluations with a stronger scoring rule that counts a success only when Y appears as the asserted answer (for example, not immediately preceded by 'not', 'incorrectly', or 'actually wrong'), and compare the resulting ASRs with the reported 89%, 94%, and 86%. A large drop would falsify the claim that the embedded denial makes the model adopt Y; instead it would mean the model often mentions Y while rejecting it.","supporting_citations":[{"cited_title":"{PoisonedRAG}: Knowledge corruption attacks to {Retrieval-Augmented} generation of large language models,","cited_arxiv_id":null,"evidence_quote":"PoisonedRAG-N1: the direct-assertion baseline DenialRAG must beat, and the source of the evaluation protocol (datasets, retriever, substring-match ASR)."},{"cited_title":"One shot dominance: Knowledge poisoning attack on retrieval-augmented generation systems","cited_arxiv_id":null,"evidence_quote":"AuthChain: the citation-fabrication baseline whose single-document passage framework DenialRAG extends with a denial component."},{"cited_title":"Practical poisoning attacks against retrieval-augmented generation,","cited_arxiv_id":null,"evidence_quote":"CorruptRAG-AK: the freshness-update framing baseline that dominates in some frontier-model cells."},{"cited_title":"Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection,","cited_arxiv_id":null,"evidence_quote":"PIA-direct: the assertive injection baseline against which DenialRAG compares."},{"cited_title":"Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts,","cited_arxiv_id":null,"evidence_quote":"Supplies the motivating behavior: when context conflicts with parametric knowledge, instruction-tuned LLMs often follow the context, so an embedded denial can override the correct answer."},{"cited_title":"What is seen cannot be unseen: The disruptive effect of knowledge conflict on large language models,","cited_arxiv_id":null,"evidence_quote":"Supports the stability explanation that frontier models cross-check retrieved claims against parametric knowledge more aggressively."},{"cited_title":"Beir: A heterogenous benchmark for zero-shot evaluation of informa- tion retrieval models,","cited_arxiv_id":null,"evidence_quote":"BEIR: source of the three open-domain QA datasets (NQ, HotpotQA, MS-MARCO) used for evaluation."}],"review_version":1}