{"id":"a721597a-2a10-4662-a03d-c3cfa01c612e","arxiv_id":"2605.20123","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"BiRD defends RAG systems against poisoning attacks by using a dual-signal framework that combines forward ranking for semantic relevance with backward ranking for context consistency, reducing attack success rates by up to 54% and improving accuracy by up to 56%.","lead":"BiRD is a defense for retrieval-augmented generation that detects poisoned documents by checking alignment between forward and backward rankings. A smart generalist might read it because RAG systems are widely used in AI tools and this offers a low-cost way to improve their resistance to manipulation.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"The bidirectional ranking alignment pattern may be an artifact of the specific PoisonedRAG attack and retrieval setups rather than a general discriminator.","rationale":"The reader's weakest assumption correctly isolates the generalizability of the alignment pattern as the key unverified premise. My concern is a direct technical elaboration of that assumption: without evidence that the pattern survives retriever or attack variation, the headline performance numbers rest on an untested inductive step. This matches the reader's low-confidence UNVERDICTED stance and justifies moving to CONDITIONAL once the concrete cross-retriever check is performed.","tokens_in":1715,"tokens_out":407,"duration_ms":28980,"concrete_test":"Re-run the PoisonedRAG evaluation on the same 3 datasets but replace the original retriever with a different one (e.g., switch from BM25 to a dense retriever like Contriever) while keeping the same LLM and attack parameters; if the ASR reduction drops below 30% or task accuracy gain disappears, the pattern is not retrieval-agnostic.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that poisoned documents reliably exhibit stronger alignment between their backward rankings and the query's forward ranking, and that this signal is robust enough to yield up to 54% ASR reduction while boosting accuracy. This hinges on the pattern being intrinsic to poisoning rather than tied to how PoisonedRAG constructs documents or how the chosen retrievers compute scores. If the alignment is measured only on top-k results or is sensitive to embedding model choice, the dual-signal framework could degrade when the attack is adapted or when a different retriever (e.g., one using dense vs. sparse retrieval) is substituted. The abstract reports results across 3 datasets/3 retrievers/3 LLMs but provides no indication of statistical significance tests, variance across runs, or ablation removing the backward-ranking component, leaving open whether the reported gains are driven by the claimed mechanism or by incidental filtering effects.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims to have identified a key discriminative pattern in RAG systems: poisoned documents exhibit significantly stronger alignment between their backward rankings and the query's forward ranking. Building on this observation, it proposes BiRD, a bidirectional ranking defense that uses a dual-signal framework—forward ranking to assess semantic content relevance and backward ranking to quantify ranking context consistency. The approach is evaluated across 3 datasets, 3 retrievers, and 3 LLMs under 2 attack scenarios, reporting up to 54% reduction in PoisonedRAG attack success rate, up to 56% improvement in task accuracy, and average added latency under 1 second.","tokens_in":1914,"tokens_out":607,"duration_ms":52537,"significance":"If the bidirectional ranking alignment pattern is shown to be a robust, generalizable property of poisoning rather than tied to specific attack constructions or retriever choices, BiRD would represent a meaningful advance by resolving the efficiency-robustness trade-off in prior semantic or voting-based defenses. The multi-configuration evaluation across datasets/retrievers/LLMs is a strength that supports broader applicability claims.","major_comments":[{"comment":"Abstract: The central performance claims (up to 54% ASR reduction and 56% accuracy improvement) are stated as maxima without identifying the exact dataset/retriever/LLM configuration, reporting variance across runs, or including statistical significance tests; this directly affects whether the data support the claimed effectiveness of the dual-signal mechanism.","section":"Abstract"},{"comment":"Method (dual-signal framework): The defense rests on the assumption that stronger backward-forward ranking alignment is intrinsic to poisoned documents and a reliable discriminator; without an ablation that isolates or removes the backward-ranking signal, it remains unclear whether reported gains derive from the proposed mechanism or from incidental top-k filtering effects.","section":"Method"},{"comment":"Experiments: Results are shown across 3 retrievers, but the evaluation does not test adapted variants of PoisonedRAG or alternative embedding models (dense vs. sparse); this leaves open whether the alignment pattern persists when the attack or retrieval setup changes, which is load-bearing for the generalizability of the defense.","section":"Experiments"}],"minor_comments":[{"comment":"Notation for forward and backward rankings could be introduced with a small illustrative example early in the method section to improve readability.","section":"Method"},{"comment":"The abstract mentions '2 attack scenarios' but does not name them; adding the names (e.g., PoisonedRAG and the second scenario) would aid quick assessment.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to fit the journal's scope in security for generative systems. The citation list should be checked to ensure prior ranking-based RAG analyses are adequately referenced."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback and the recommendation for major revision. We address each major comment point by point below, indicating where we agree and the revisions we will make to strengthen the manuscript.","responses":[{"response":"We agree that the abstract would benefit from greater specificity. In the revised version, we will update the abstract to identify the exact dataset/retriever/LLM configuration that achieves the reported maxima. We will also ensure the experimental results section reports variance across runs and includes statistical significance tests to more rigorously support the effectiveness of the dual-signal mechanism.","revision_made":"yes","referee_comment":"[Abstract] Abstract: The central performance claims (up to 54% ASR reduction and 56% accuracy improvement) are stated as maxima without identifying the exact dataset/retriever/LLM configuration, reporting variance across runs, or including statistical significance tests; this directly affects whether the data support the claimed effectiveness of the dual-signal mechanism."},{"response":"This is a fair and important point. To clarify that the gains stem from the bidirectional mechanism rather than incidental top-k effects, we will add an ablation study in the revised manuscript. The ablation will compare the full BiRD dual-signal framework against a forward-ranking-only variant, thereby isolating the contribution of the backward-ranking signal.","revision_made":"yes","referee_comment":"[Method] Method (dual-signal framework): The defense rests on the assumption that stronger backward-forward ranking alignment is intrinsic to poisoned documents and a reliable discriminator; without an ablation that isolates or removes the backward-ranking signal, it remains unclear whether reported gains derive from the proposed mechanism or from incidental top-k filtering effects."},{"response":"We acknowledge the value of broader testing for generalizability. While our evaluation already spans three retrievers under two attack scenarios, we agree that adapted PoisonedRAG variants and explicit dense-versus-sparse comparisons would provide stronger evidence. We will revise the discussion section to explicitly address this limitation and outline it as future work; however, we cannot perform these additional experiments within the current revision timeline.","revision_made":"partial","referee_comment":"[Experiments] Experiments: Results are shown across 3 retrievers, but the evaluation does not test adapted variants of PoisonedRAG or alternative embedding models (dense vs. sparse); this leaves open whether the alignment pattern persists when the attack or retrieval setup changes, which is load-bearing for the generalizability of the defense."}],"tokens_in":1466,"tokens_out":537,"duration_ms":38829,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The key takeaway is that this paper identifies a bidirectional ranking alignment pattern that distinguishes poisoned documents from benign ones in RAG systems and builds a simple dual-signal defense around it. Forward ranking handles semantic relevance while backward ranking checks consistency in the ranking context, which directly targets a gap in prior semantic-only or voting defenses. The approach is straightforward and avoids heavy computation, which fits real deployment needs. They test across three datasets, three retrievers, and three LLMs under two attack scenarios, and the numbers show up to 54% lower attack success on PoisonedRAG plus up to 56% better task accuracy with under one second added latency on average. That combination of robustness and speed is the practical strength here. The evaluation coverage gives a reasonable picture of behavior under the conditions they chose. Where it could be softer is on how general the alignment pattern really is. It might tie more closely to the way PoisonedRAG constructs poisoned documents or to the specific retrievers and scoring methods used in the tests. If the signal weakens with adapted attacks or different embedding models, the reported gains could shrink. The summary does not include variance numbers, statistical tests, or ablations that isolate the backward-ranking component, so it is hard to tell exactly how much that signal drives the results versus incidental filtering. This paper is for researchers and engineers working on security for deployed RAG systems. Anyone looking at lightweight ways to harden retrieval against poisoning would find the core idea and efficiency results useful. It engages honestly with the limitations of existing methods and presents a new angle worth checking. I would send this to peer review.","headline":"BiRD spots a bidirectional ranking alignment pattern in poisoned RAG documents and turns it into a low-overhead dual-signal defense that reports solid gains on tested attacks.","tokens_in":2398,"tokens_out":398,"would_cite":false,"duration_ms":37469,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[{"relation":"unclear","rs_module":"IndisputableMonolith/Foundation/AlexanderDuality.lean","rs_theorem":"alexander_duality_circle_linking","paper_passage":"poisoned documents exhibit significantly stronger alignment between their backward rankings and the query's forward ranking"},{"relation":"unclear","rs_module":"IndisputableMonolith/Cost/FunctionalEquation.lean","rs_theorem":"washburn_uniqueness_aczel","paper_passage":"S(dq_i) = r_i_cr / (1 - r_i_cc) with Spearman rank correlation"}],"headline":"BiRD bidirectional ranking defense is applied RAG security work with no overlap to RS forcing chain or J-cost structures","alignment":"orthogonal","rationale":"The paper's core machinery (forward/backward ranking via Spearman ρ, composite score S = r_cr/(1-r_cc), clustering of poisoned embeddings) addresses adversarial retrieval in NLP systems. RS derives J(x), φ, 8-tick periodicity, D=3 via Alexander duality, and constants from a single distinction with zero adjustable parameters. No shared primitives, cost functions, or topological forcing appear; the domains (cs.CR vs. foundational physics/logic) are disjoint.","tokens_in":57425,"confidence":"high","tokens_out":298,"duration_ms":11735,"cache_read_input_tokens":38528,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"BiRD defends RAG by spotting poisoned documents through unusually strong alignment between their backward rankings and the query's forward ranking.","keywords":["retrieval augmented generation","adversarial defense","poisoning attacks","bidirectional ranking","RAG security","ranking structures","adversarial robustness"],"falsifier":"A test on a fresh dataset or attack method where poisoned documents no longer display the claimed stronger backward-forward alignment would falsify the central pattern.","tokens_in":2620,"feed_emoji":"🛡️","tokens_out":485,"duration_ms":21462,"temperature":0.7,"pith_summary":"The paper sets out to prove that poisoned documents in retrieval-augmented generation systems display a consistent ranking pattern that benign documents lack, allowing a lightweight defense to filter them without heavy semantic analysis. This matters because existing defenses either demand high computation or lose effectiveness against strong poisoning attacks, restricting reliable use of RAG. By combining forward ranking for content relevance with backward ranking for context consistency, the approach aims to cut attack success while raising overall accuracy and keeping added delay under one second. If the pattern holds, RAG deployments could become safer in practice without trading performance for security.","feed_headline":"Bidirectional ranking cuts RAG poisoning attacks by 54%","feed_subtitle":"The defense checks alignment between backward rankings and query forward rankings to filter poisoned documents with under one second added延迟","key_machinery":"The bidirectional ranking defense mechanism that pairs forward ranking for semantic content relevance with backward ranking for ranking context consistency.","core_discovery":"The central claim is that poisoned documents exhibit significantly stronger alignment between their backward rankings and the query's forward ranking than benign ones do. The authors build BiRD on a dual-signal framework that uses forward ranking to judge semantic relevance and backward ranking to measure ranking context consistency, directly addressing the prior focus on content alone. Experiments across three datasets, three retrievers, three LLMs, and two attack scenarios show this reduces PoisonedRAG attack success by up to 54 percent while lifting task accuracy by up to 56 percent with under one second of extra latency on average.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Backward rank alignment flags RAG poisoned documents","BiRD defends RAG using bidirectional ranking consistency","Strong backward-forward rank match identifies RAG poisons","BiRD filters poisoned docs with dual ranking signals"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Poisoned documents will reliably show stronger alignment between their backward rankings and the query's forward ranking across varied datasets, retrievers, models, and attacks.","fun_headline_variants_meta":{"raw":{"variants":["Backward rank alignment flags RAG poisoned documents","BiRD defends RAG using bidirectional ranking consistency","Strong backward-forward rank match identifies RAG poisons","BiRD filters poisoned docs with dual ranking signals"]},"model":"grok-4.3","cost_usd":0.008469,"raw_usage":{"total_tokens":3763,"prompt_tokens":697,"num_sources_used":0,"completion_tokens":49,"cost_in_usd_ticks":84690500,"prompt_tokens_details":{"text_tokens":697,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3017,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":697,"tokens_out":49,"duration_ms":31151,"temperature":1.0,"reasoning_tokens":3017,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-20T03:41:25.246468+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A test on a fresh dataset or attack method where poisoned documents no longer display the claimed stronger backward-forward alignment would falsify the central pattern.","supporting_citations":[],"review_version":1}