{"id":"c3f9d136-3bb7-4e20-8ac6-f491e7d84513","arxiv_id":"2608.08100","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"RAG-IDS protects a retrieval-augmented intrusion detection pipeline with a label-embedding consistency check, but the protected system only beats the unprotected one at 1% poisoning and underperforms it at 10% and above.","lead":"This paper presents RAG-IDS, a retrieval-augmented AI pipeline for network intrusion detection, and a defense layer meant to resist poisoned knowledge-base entries and prompt-injection payloads. The defense preserves clean-level accuracy only when the knowledge base is barely poisoned, and at higher poison rates it performs slightly worse than the unprotected pipeline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The defense's central recovery claim fails the same-rate test: at 10–30% poisoning, defended macro-F1 is below undefended, and the LECC component credited with robustness is the one that degrades performance.","rationale":"The paper is unusually transparent: it reports absolute F1 and FPR alongside R, documents the calibration assumption, and includes a full ablation. I am not questioning the honesty of the experiments. The load-bearing problem is that the chosen metric R is normalized by clean undefended performance, so it cannot answer whether the defense helps under the attack it is designed to counter. The paper's own Table V shows defended F1 falling below undefended F1 at 10%, 20%, and 30% poison rates, and Table VIII shows that the LECC component credited with robustness is the one that degrades both absolute F1 and R at the two ablated high rates. This is an internal-consistency failure of the central claim, not merely a disagreement with an external baseline. The reader's rejection is therefore supported. My emphasis differs slightly from the reader's stated weakest_assumption: the clean-calibration and adaptive-attack gaps are important but secondary, because even granting clean calibration and the PoisonedRAG attack family, the defense does not outperform undefended retrieval at meaningful poison rates. A revision could salvage the paper as a scoped low-rate defense study or as a negative result for LECC-style demotion in this embedding space, but the headline claim as written should be rejected.","tokens_in":19474,"tokens_out":7921,"duration_ms":86455,"concrete_test":"Re-run CEXP04 paired by seed and compute Δ=F1_def−F1_undef and R_same=F1_def/F1_undef at every p∈{0.01,0.05,0.10,0.20,0.30} with the same three-seed statistics; also compute ΔR=R(D2-only)−R(Undefended) from the Table VIII data at p=0.10 and p=0.30. If the confidence intervals at p≥0.10 exclude positive Δ, or if R_same<1, then the paper's recovery claim must be revised to a low-rate-only benefit, and the sentence claiming LECC is the primary contributor to robustness should be removed or reworded to say it shapes the defended path without improving attacked performance.","verdict_should_be":"REJECT","load_bearing_attack":"The strongest_claim is that R=F1_def/F1_clean,undef measures recovery under poisoning, but the operative question for a defense is whether defended F1 beats undefended F1 at the same poison rate. Table V shows it does not: at 10%, defended F1 is 0.2157±0.0049 vs undefended 0.2248±0.0027; at 20%, 0.1863±0.0149 vs 0.2005±0.0130; at 30%, 0.1540±0.0064 vs 0.1671±0.0029. Only at 1% is defended F1 slightly above undefended, and 5% is parity. More damaging, the component described as 'the primary contributor to robustness' (LECC/D2) is the one that lowers performance. In the CEXP05 ablation of Table VIII, D2-only has lower F1 than undefended at both ablated rates (10%: 0.210 vs 0.222; 30%: 0.153 vs 0.170) and lower R as well (0.794 vs 0.838 at 10%; 0.579 vs 0.643 at 30%). Thus the claimed recovery is not demonstrated against the attack the defense is designed to counter; at the high-end rates the paper stresses, the active defense component is harmful. The paper discloses these absolute comparisons, but centering R against clean undefended performance hides the fact that the defense does not beat doing nothing under the attack.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RAG-IDS, a three-tier retrieval-augmented intrusion detection framework with a defense module (soft trust filter D1, label-embedding consistency check LECC/D2, and prompt sanitizer D3) intended to preserve classification quality under knowledge poisoning and prompt injection. Using CIC-UNSW-NB15, the authors report recovery ratios R = F1_defended / F1_clean_undefended ranging from R=1.002 at 1% poisoning to R=0.573 at 30%, negligible clean overhead, and low label-flip success in the multi-document prompt-injection setting. Ablations identify LECC as the component that shapes the defended pipeline.","tokens_in":19760,"tokens_out":4634,"duration_ms":46708,"significance":"The paper is commendably transparent: it reports absolute F1 values alongside recovery ratios, includes per-seed results, an ablation study, latency measurements, and an explicit discussion of limitations, calibration assumptions, and out-of-scope adaptive attacks. The experimental setup is described in unusual detail for a preprint, and the authors are honest that clean RAG-IDS accuracy is far below tree-based baselines. However, the significance of the work depends entirely on the claim that the defense recovers classification quality under retrieval poisoning, and that claim is not supported by the reported data. A defense that performs worse than the undefended pipeline at the same poison rate does not demonstrate retrieval-boundary defense. Because the central contribution fails on its own terms, the paper cannot be accepted in its current form.","major_comments":[{"comment":"The recovery metric R is defined against clean undefended F1, but the appropriate baseline for a defense under attack is the undefended pipeline at the same poison rate. Table V shows that at 10%, 20%, and 30% poisoning, defended macro-F1 is lower than undefended macro-F1 (0.2157 vs 0.2248; 0.1863 vs 0.2005; 0.1540 vs 0.1671), and defended FPR is higher at each of these rates. The paper acknowledges this in the text ('Absolute defended F1 can fall slightly below undefended F1 at the same rate') but the consequence is not minor: at 10% and 30% the differences are substantial relative to the standard deviations, and the direction is consistently negative. Thus the defense does not reduce the impact of poisoning compared to doing nothing; it increases it. The headline claim of 'recovery' is therefore not demonstrated.","section":"§VI-D, Table V"},{"comment":"The ablation directly contradicts the claim that LECC is the primary contributor to robustness. At p=0.10, D2-only yields F1=0.210 versus undefended F1=0.222; at p=0.30, D2-only yields F1=0.153 versus undefended F1=0.170. The R values for D2-only (0.794 and 0.579) are also below the undefended R values (0.838 and 0.643). The authors state that the ablation 'explains who shapes the defended pipeline, not that demotion always raises absolute F1 under attack,' but this evades the real issue: the component credited with robustness is the component that degrades performance. On the evidence in Table VIII, LECC is the primary contributor to the defense's deficit, not to its robustness.","section":"§VI-E, Table VIII"},{"comment":"The LECC design is explicitly tailored to PoisonedRAG-style relabeling, and the evaluation includes only that exact attack family. The paper acknowledges in Section VII that adaptive embedding attacks are out of scope and that centroid calibration on a clean KB contradicts a fully writable threat model. A targeted defense against a specific attack is a legitimate research goal, but the conclusion that RAG-IDS is 'attack-resilient' goes beyond the evidence. More importantly, the circularity is secondary: even against the attack the defense is designed for, the defense fails to beat the undefended baseline at the same poison rate. The evaluation therefore does not establish that LECC provides any protective value.","section":"§IV-A.2 and §VII"},{"comment":"The low-volume poisoning probe (CEXP08) shows that injecting one or two PoisonedRAG-style documents leaves both undefended and defended F1 unchanged, so the defense provides no measurable benefit in the operationally realistic low-count regime. The defense only engages at percent-scale write shares, and in exactly that regime it lowers F1 relative to undefended retrieval. There is consequently no demonstrated poisoning scenario in which the defense improves classification quality. The paper's own data support the opposite conclusion: the defended pipeline is either neutral or worse than the undefended pipeline under the tested attacks.","section":"§VI-D, CEXP08, and §VII"}],"minor_comments":[{"comment":"The term 'recovery' is misleading when R is defined as F1_defended / F1_clean_undefended and every reported R below 1 indicates a loss relative to clean. At 1% poisoning R=1.002 is parity, not recovery; the abstract's phrase 'recovery relative to clean undefended performance' would be more accurately rendered as 'retained fraction of clean undefended F1' or 'robustness ratio.'","section":"Abstract and §VI-D"},{"comment":"The tables are transparent about the difference between pooled and macro-averaged FPR, but the reader cannot numerically compare FPR across rows. Since the FPR gap is used to motivate hybrid deployment, reporting at least one row with a single pooling convention would strengthen the presentation.","section":"Table II and Table III"},{"comment":"Two clean macro-F1 values are reported for RAG-IDS: 0.1237 under the CEXP02 200-sample protocol and 0.2697 under the CEXP04 N=499 protocol. The text explains that they are not directly comparable, but the existence of two clean baselines will confuse readers; a single naming convention (e.g., 'CEXP02 clean characterization' vs. 'CEXP04 clean baseline') would help.","section":"§VI-C and §VI-D"}],"recommendation":"reject","confidential_remarks":"The paper is honest and unusually detailed, but the central claim is refuted by the paper's own tables: the defended pipeline does not beat the undefended pipeline at the same poison rate, and the component identified as the main contributor to robustness is the one that lowers F1. This is not a presentation issue; it is a failure of the core contribution. The prompt-injection results are also weak, since multi-document robustness is mostly a dilution effect and the defense is a null operation in the single-document setting. I see no easy fix within the manuscript's scope; a revised defense or a substantially different evaluation would be needed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the LECC design and the IDS-specific evaluation are genuinely new, but the paper's headline recovery claim does not survive the same-rate comparison that the authors themselves tabulate.\n\nWhat is actually good. LECC is a concrete, targeted defense against relabeled-document attacks, and evaluating RAG poisoning defenses on an IDS workload with extreme class imbalance is a real gap. The paper is also unusually honest: it reports absolute F1 and FPR values, clearly separates the CEXP02 characterization from the CEXP04 adversarial protocol, includes a negative external baseline, and explicitly lists the calibration and attack-model limitations. That transparency is not common.\n\nThe soft spot is load-bearing. Recovery R is defined relative to clean undefended F1, so R=0.57 at 30% poison reads as if the defense retains over half of clean performance. But in Table V, defended F1 at 30% is 0.154 vs 0.167 undefended at the same poison rate; at 10% and 20% defended is also below undefended. So at the rates the paper stresses, the active defense is worse than doing nothing under the same attack. The ablation confirms the trouble: D2/LECC, credited as the primary robustness contributor, is precisely the component that drags F1 below undefended at 10% and 30%. The metric choice is not a minor presentational issue; it hides the fact that the defense does not beat the undefended baseline under attack.\n\nThe other limits—evaluation only against the PoisonedRAG relabeling family that LECC was built to detect, and defense calibration on a clean KB—are stated candidly in Section VII, but they are not minor. They are part of the same pattern: the strong claim is scoped so narrowly that it does not demonstrate retrieval-boundary robustness in a meaningful sense.\n\nNone of this makes the paper worthless. The LECC mechanism may be salvageable with better thresholding or in combination with other signals, and the IDS-specific evaluation is a useful starting point for follow-up work. But as submitted, the central claim contradicts the paper's own numbers.\n\nRecommendation: This deserves referee time, not a desk reject, but I would expect a major revision. The authors should either rework the defense so that defended F1 beats undefended at high poison rates, or reframe the paper as a tradeoff study that honestly reports where the defense helps and where it hurts. If they keep the recovery metric, they need to show the same-rate comparison in the main text and adjust the conclusion.","headline":"LECC is a real idea and the IDS evaluation is new, but the defense's headline recovery claim collapses under the same-rate comparison the paper itself tabulates.","tokens_in":20419,"tokens_out":3256,"would_cite":false,"duration_ms":30509,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RAG-IDS's retrieval-boundary defense restores clean detection quality under knowledge poisoning, from $R=1.002$ at 1% poison to $R=0.573$ at 30%, while multi-document context limits prompt-injection label flips to 0.6–2.4%.","keywords":["adversarial defense","intrusion detection system","knowledge poisoning","large language models","prompt injection","retrieval-augmented generation","label-embedding consistency check","retrieval poisoning"],"falsifier":"Run the defense against a 30% poison batch whose injected documents are embedded near the benign-class centroid while labeled benign, and compare defended macro-F1 with the reported $R=0.573$; if recovery collapses toward the undefended values, the LECC centroid check is the failure point, not the cure.","tokens_in":19221,"feed_emoji":"🛡️","tokens_out":10779,"duration_ms":97500,"temperature":0.7,"pith_summary":"Retrieval-augmented generation lets an LLM classify network flows by retrieving similar labeled history, but the vector knowledge base is a writable surface: an attacker can inject attack flows relabeled as benign, or hide instruction overrides inside retrieved documents. The paper claims that a defense placed at the retrieval boundary—soft trust scoring, label-embedding consistency checking (LECC), and prompt sanitization—can recover most of the clean classification quality under such attacks. On a 10-class network-traffic benchmark, the recovery ratio relative to clean undefended macro-F1 stays near 1.0 at 1% knowledge-base poisoning and falls to 0.573 at 30%, with negligible clean overhead; multi-document context keeps prompt-injection label flips between 0.6% and 2.4% instead of the 35–55% seen with single-document retrieval. If true, a RAG-based detector can serve as an explainable, attack-resilient layer in a hybrid IDS rather than a drop-in replacement for classical classifiers.","feed_headline":"At 30% knowledge poisoning, defended IDS keeps 57% of clean quality","feed_subtitle":"Label-embedding checks stop relabeled attack flows, and multi-document context keeps prompt-injection flips under 3%.","key_machinery":"The load-bearing object is the label-embedding consistency check (LECC). For a retrieved document with embedding $\\mathbf{e}_{d_i}$ and stated label $\\ell$, LECC computes distances $\\delta_c=\\|\\mathbf{e}_{d_i}-\\boldsymbol{\\mu}_c\\|$ to all class centroids and the distance to the stated label's centroid $\\delta_\\ell$; if the nearest centroid is not the stated label, it applies penalty $s_2=(\\delta_\\ell-\\delta_{\\min})/(\\delta_\\ell+\\varepsilon)$, otherwise it falls back to a per-class 95th-percentile outlier threshold calibrated on clean data. This penalty enters a composite score $\\hat{s}(d_i)=s_{\\mathrm{ret}}(d_i)-\\lambda(s_1+s_2+s_3)$ with $\\lambda=0.3$, where $s_1$ is the soft trust-score deficit below cosine threshold $\\theta=0.40$ and $s_3$ is a prompt-injection score from regex and exemplar similarity; documents are reranked by $\\hat{s}$ and demoted rather than removed. LECC is the component that makes the defense work against the relabeling attack, because that attack preserves original embeddings.","core_discovery":"The paper's central discovery is that one consistency check carries the defense: when an attack flow is relabeled as benign, its embedding stays close to its true attack-class centroid and far from the stated benign centroid, so comparing each retrieved document's embedding against all class centroids exposes the relabeling. The full defense combines this label-embedding consistency check with a soft trust-score penalty for documents below a similarity threshold and a prompt-sanitizer score, reranking by the composite score instead of deleting documents. Under retrieval poisoning with the relabeling attack, defended macro-F1 matches clean undefended performance at 1% poison ($R=1.002\\pm0.006$) and retains $R=0.573\\pm0.057$ at 30%, with clean overhead within noise; the ablation shows LECC alone reproduces the defended path. Under prompt injection, five-document context limits label-flip success to 0.6–2.4%, while single-document retrieval fails at 35–55% and demotion becomes a null operation. The paper frames these as defense-study claims: absolute clean macro-F1 remains low (about 0.27 on the adversarial query set), so the contribution is recovery and explainability, not replacing high-throughput ML classifiers.","pith_inferences":["We infer that an adversary who can poison the calibration set or shift poison embeddings toward the benign-class centroid will bypass LECC, since every defense term reduces to distance checks against centroids the adversary can estimate.","We infer that the reported recovery ratio would shrink if clean retrieval improved: the recovery ratio divides defended macro-F1 by clean undefended macro-F1, so a stronger clean baseline makes the same absolute defended quality look worse.","We infer that the defense likely transfers to other traffic benchmarks by re-fitting centroids and thresholds, but transfer is untested because the paper only measures one dataset."],"forward_implications":["Because LECC alone reproduces the full defense under poisoning, future RAG-IDS designs should treat embedding-label consistency as the primary control point rather than adding more filters.","Because hard cosine filtering underperforms soft demotion at every poison rate, defenders should demote suspicious documents instead of deleting them, preserving diverse context.","Because multi-document context keeps prompt-injection label-flip rates at 0.6–2.4% while single-document retrieval reaches 35–55%, retrieval depth should be treated as a security parameter.","Because the defense adds about 5 ms per query versus roughly 1.9 s of LLM generation, securing the retrieval boundary costs almost nothing in latency."],"supporting_citations":[{"why":"Defines the relabeling attack (attack flows stored under a benign label with embeddings intact) that the LECC centroid check is designed to catch.","marker":"[1]"},{"why":"Establishes the indirect prompt-injection threat model used for attack surface A2.","marker":"[3]"},{"why":"Supplies the re-extracted network-traffic benchmark on which every experiment is run.","marker":"[8]"},{"why":"Defines retrieval-augmented generation, the pipeline architecture whose retrieval layer is being secured.","marker":"[9]"},{"why":"Produces the text embeddings used for retrieval, centroid computation, and label-embedding distance checks.","marker":"[11]"},{"why":"Generates the class-balanced oversampled training samples from which knowledge-base documents are drawn.","marker":"[22]"},{"why":"Motivates the absolute-count 1-2 document poisoning probe that complements the rate sweep.","marker":"[35]"},{"why":"Provides the hard-cosine filtering baseline against which soft demotion with LECC is compared.","marker":"[36]"}],"fun_headline_variants":["RAG-IDS recovers full quality at 1% poison, 57% at 30%","Label-embedding check stops relabeled attacks in RAG IDS","Multi-doc RAG cuts prompt-injection flips to under 3%","Embedding centroid check exposes poisoned RAG retrievals","RAG-IDS: 100% recovery at 1% poison, 57% at 30%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The defense's centroid positions and per-class outlier thresholds are calibrated on a clean knowledge base before any poison is injected, so an adversary who corrupts the calibration set or writes poison embeddings near the benign-class centroid can bypass every check; the paper explicitly leaves such adaptive embedding attacks out of scope.","fun_headline_variants_meta":{"raw":{"variants":["RAG-IDS recovers full quality at 1% poison, 57% at 30%","Label-embedding check stops relabeled attacks in RAG IDS","Multi-doc RAG cuts prompt-injection flips to under 3%","Embedding centroid check exposes poisoned RAG retrievals","RAG-IDS: 100% recovery at 1% poison, 57% at 30%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000834,"raw_usage":{"total_tokens":3679,"prompt_tokens":1025,"completion_tokens":2654,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":2544}},"tokens_in":641,"tokens_out":2654,"duration_ms":42452,"temperature":1.0,"reasoning_tokens":2544,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:25:25.137248+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the defense against a 30% poison batch whose injected documents are embedded near the benign-class centroid while labeled benign, and compare defended macro-F1 with the reported $R=0.573$; if recovery collapses toward the undefended values, the LECC centroid check is the failure point, not the cure.","supporting_citations":[{"cited_title":"PoisonedRAG: Knowledge Cor- ruption Attacks to Retrieval-Augmented Generation,","cited_arxiv_id":null,"evidence_quote":"Defines the relabeling attack (attack flows stored under a benign label with embeddings intact) that the LECC centroid check is designed to catch."},{"cited_title":"Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injec- tion,","cited_arxiv_id":null,"evidence_quote":"Establishes the indirect prompt-injection threat model used for attack surface A2."},{"cited_title":"Poisoning and Evasion: Deep Learning-Based NIDS under Adversarial Attacks,","cited_arxiv_id":null,"evidence_quote":"Supplies the re-extracted network-traffic benchmark on which every experiment is run."},{"cited_title":"Retrieval-Augmented Generation for Knowledge- Intensive NLP Tasks,","cited_arxiv_id":null,"evidence_quote":"Defines retrieval-augmented generation, the pipeline architecture whose retrieval layer is being secured."},{"cited_title":"SMOTE: Synthetic Minority Over-sampling Technique,","cited_arxiv_id":null,"evidence_quote":"Generates the class-balanced oversampled training samples from which knowledge-base documents are drawn."}],"review_version":1}