{"id":"318d7344-1466-4de0-91c5-5acfba05fbc7","arxiv_id":"2606.26793","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"MIRROR is a novelty-constrained memory-guided MCTS red-teaming framework that reports higher attack success rates and lower variance than specialized baselines across four surfaces on agentic RAG, with release of ART-SafeBench.","lead":"MIRROR applies memory-guided Monte Carlo tree search with an explicit novelty gate to generate attacks across text, image, direct-query, and orchestrator surfaces on multimodal agentic RAG systems. A smart generalist might read it to see how red-teaming can move from recycled templates to more systematic, cross-surface coverage for AI security evaluation.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Novelty Gate's claimed separation of retrieval guidance from prompt copying lacks ablation isolating its contribution to ASR gains and variance reduction.","rationale":"The reader's weakest_assumption directly identifies the untested premise required for the memory-guided benefit. Because the full text was referenced but the provided abstract supplies the only quantitative claims, the same assumption remains the single most load-bearing point; an ablation would convert the UNVERDICTED status to CONDITIONAL rather than ACCEPT or REJECT.","tokens_in":1783,"tokens_out":340,"duration_ms":20881,"concrete_test":"Re-run the four-surface evaluation with an ablated variant that removes the Novelty Gate (or replaces it with a random reject); if ASR on image poisoning falls below 70% or cross-surface CV rises above 0.6 while query cost increases, the gate is load-bearing; otherwise the performance edge is not explained by the novelty constraint.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that the deterministic Novelty Gate (rejecting normalized matches to the retrieval set) lets memory inform MCTS priors without enabling copying or harming effectiveness. This is asserted in the abstract and is the explicit justification for the memory-guided component over baselines. No section quantifies how often the gate triggers, whether disabling it increases duplication rates above the reported 73-84% baseline, or whether ASR on image poisoning (76%) and orchestrator attacks (97%) degrades without it. If the gate is either too permissive or too restrictive in practice, the cross-surface stability (CV 0.47) and query-cost advantage cannot be attributed to the stated mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript presents MIRROR, a framework for red-teaming multimodal agentic RAG systems using memory-guided Monte Carlo Tree Search (MCTS) conditioned on retrieved context under a deterministic Novelty Gate that rejects candidates matching the retrieval set. It reports improved attack success rates (ASR) across four attack surfaces—76% on image poisoning (vs. 52% baselines), 97% on orchestrator attacks at half the query cost—and the lowest cross-surface variance (CV=0.47), while releasing the ART-SafeBench dataset with over 41k records.","tokens_in":1926,"tokens_out":359,"duration_ms":27044,"significance":"If the results hold after verification, the work provides a unified cross-surface red-teaming method that addresses the limitations of surface-specific approaches and highlights the potential of novelty-constrained memory guidance in MCTS for security assessments. The benchmark release adds value for reproducibility in the field.","major_comments":[{"comment":"The performance advantages and variance reduction are attributed to the Novelty Gate, but no ablation is reported isolating its effect on ASR, duplication rates, or query efficiency. Without quantifying how often the gate triggers or comparing to a version without it, the claim that it allows retrieval to inform priors without enabling copying cannot be substantiated.","section":"Abstract (Novelty Gate)"},{"comment":"Concrete ASR numbers, query costs, and CV=0.47 are given without error bars, statistical significance tests, or full details on how baselines were implemented and datasets constructed for each surface, which are necessary to support the cross-surface stability claim.","section":"Results (ASR and variance claims)"}],"minor_comments":[{"comment":"The abstract mentions 'normalized comparison' for the Novelty Gate but does not specify the normalization method or similarity metric used.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments. We address each major point below and commit to revisions that will strengthen the empirical support for our claims without altering the core contributions.","responses":[{"response":"We agree that an explicit ablation is required to substantiate the Novelty Gate's contribution. In the revised manuscript we will add a dedicated ablation section comparing full MIRROR against an otherwise identical variant with the gate disabled. We will report ASR, exact duplication rates, query counts, and the empirical trigger frequency of the gate across the four surfaces. This will directly quantify how the deterministic rejection step prevents copying while still allowing retrieval-conditioned priors.","revision_made":"yes","referee_comment":"[Abstract (Novelty Gate)] The performance advantages and variance reduction are attributed to the Novelty Gate, but no ablation is reported isolating its effect on ASR, duplication rates, or query efficiency. Without quantifying how often the gate triggers or comparing to a version without it, the claim that it allows retrieval to inform priors without enabling copying cannot be substantiated."},{"response":"We accept that error bars, significance testing, and expanded methodological detail are necessary. The revised version will include standard-error bars on all ASR and cost figures, paired statistical tests (e.g., McNemar or Wilcoxon) comparing MIRROR to each baseline per surface, and an expanded appendix providing the precise baseline implementations, prompt templates, and dataset-construction procedures used for every attack surface. These additions will allow readers to evaluate the reported cross-surface stability (CV=0.47) with appropriate statistical context.","revision_made":"yes","referee_comment":"[Results (ASR and variance claims)] Concrete ASR numbers, query costs, and CV=0.47 are given without error bars, statistical significance tests, or full details on how baselines were implemented and datasets constructed for each surface, which are necessary to support the cross-surface stability claim."}],"tokens_in":1384,"tokens_out":417,"duration_ms":22185,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"MIRROR combines memory-guided Monte Carlo tree search with a deterministic novelty gate that rejects normalized matches to the retrieval set. The goal is to let retrieved context shape attack search without falling back into the 73-84% template duplication seen in prior work.\n\nThe concrete advance is the unified framework that targets four surfaces—text poisoning, image injection, direct queries, and orchestrator manipulation—on a multimodal agentic RAG target. It also ships ART-SafeBench with 41k+ records and runtime adapters. That data release is the part most likely to stick around.\n\nThe reported results show 76% ASR on image poisoning against a 52% baseline, 97% on orchestrator attacks at roughly half the query cost, and the lowest cross-surface variance (CV 0.47). Specialized baselines do drop off sharply on some surfaces, as claimed.\n\nThe soft spot is the novelty gate. The abstract presents it as the mechanism that lets memory guide search without copying, yet no numbers appear on rejection frequency, no ablation compares runs with the gate disabled, and no check shows whether ASR or variance numbers move when the gate is removed. Without those checks the performance edge cannot be cleanly attributed to the stated design choice.\n\nThis work is aimed at researchers building or evaluating security tools for retrieval-augmented agents. Anyone who needs a multi-surface benchmark or is extending MCTS-style red-teaming will find usable pieces. It is worth sending to referees because the problem area is active, the data contribution is real, and the central technique is distinct from prior surface-specific methods, even though the gate's contribution needs tighter evidence.","headline":"MIRROR gives a new cross-surface red-teaming method plus a sizable benchmark release, but the novelty gate's role in the gains is not yet isolated.","tokens_in":2469,"tokens_out":411,"would_cite":false,"duration_ms":22804,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"MIRROR's novelty-constrained MCTS unifies red-teaming across four attack surfaces on multimodal agentic RAG systems.","keywords":["red-teaming","agentic RAG","Monte Carlo tree search","novelty constraint","multimodal attacks","prompt injection","attack success rate"],"falsifier":"A test where candidates rejected by the novelty gate still achieve high attack success would falsify the benefit of the constraint.","tokens_in":2682,"feed_emoji":"🛡️","tokens_out":620,"duration_ms":34992,"temperature":0.7,"pith_summary":"The paper introduces MIRROR as a single framework for attacking agentic RAG systems on text poisoning, image injection, direct queries, and orchestrator manipulation. It combines memory-guided Monte Carlo tree search with an explicit novelty gate that prevents copying retrieved prompts. Results show improved attack success rates, such as 76% on image poisoning versus 52% for baselines, 97% on orchestrator attacks at lower cost, and more consistent performance across surfaces. This matters because specialized methods fail when switched between surfaces, while a unified approach could better expose vulnerabilities in complex AI agents. The work also releases a benchmark dataset for further testing.","feed_headline":"Novelty gate in MCTS raises RAG attack success to 97 percent","feed_subtitle":"Unified framework outperforms surface-specific baselines on four attack types while cutting query costs in half.","key_machinery":"The deterministic Novelty Gate, which rejects any candidate matching the retrieval set under normalized comparison, allowing retrieval to inform search without prompt copying in the memory-guided MCTS.","core_discovery":"MIRROR performs memory-guided Monte Carlo tree search conditioned on retrieved context under a deterministic Novelty Gate that rejects matching candidates, enabling cross-surface red-teaming that attains 76% ASR on image poisoning, 97% on orchestrator attacks at half query cost, and lowest variance compared to surface-specific baselines.","pith_inferences":["Similar novelty constraints could apply to other search-based attack methods beyond MCTS.","If the gate works, it suggests retrieval-augmented systems need defenses against memory-informed attacks.","Extending to more surfaces or non-multimodal RAG might reveal additional weaknesses.","Lower cost at high ASR implies scalable red-teaming for larger deployments."],"forward_implications":["Unified red-teaming becomes feasible without surface-specific tuning.","Attack success rates improve on image poisoning and orchestrator attacks while reducing query costs.","Cross-surface variance decreases, making evaluations more reliable.","Specialized baselines like suffix optimization fail on some surfaces where MIRROR succeeds.","The released ART-SafeBench enables standardized testing across 41k+ records."],"fun_headline_variants":["MIRROR novelty-constrained MCTS red-teams agentic RAG at 97 percent ASR","Novelty gate in memory-guided MCTS for cross-surface RAG attack surfaces","97 percent ASR on orchestrator attacks using novelty-constrained MCTS","Lowest cross-surface variance with memory-guided MCTS novelty constraint"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The novelty gate prevents prompt copying while still letting retrieved context guide the search effectively.","fun_headline_variants_meta":{"raw":{"variants":["MIRROR novelty-constrained MCTS red-teams agentic RAG at 97 percent ASR","Novelty gate in memory-guided MCTS for cross-surface RAG attack surfaces","97 percent ASR on orchestrator attacks using novelty-constrained MCTS","Lowest cross-surface variance with memory-guided MCTS novelty constraint"]},"model":"grok-4.3","cost_usd":0.006087,"raw_usage":{"total_tokens":2877,"prompt_tokens":669,"num_sources_used":0,"completion_tokens":80,"cost_in_usd_ticks":60874500,"prompt_tokens_details":{"text_tokens":669,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2128,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":669,"tokens_out":80,"duration_ms":23444,"temperature":1.0,"reasoning_tokens":2128,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T04:29:47.295511+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A test where candidates rejected by the novelty gate still achieve high attack success would falsify the benefit of the constraint.","supporting_citations":[],"review_version":1}