{"id":"800ce9ea-f517-480a-a44f-b20ee26a0ca4","arxiv_id":"2607.21951","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A PAIR-style attacker editing a single retrieved webpage moved a chosen business to rank 1 in 50 percent of 124 trials across two Claude models, and most winning edits succeeded again in fresh sessions.","lead":"This paper shows that a chatbot that answers recommendation questions by reading webpages can be steered into ranking a chosen business first, by editing text on just one webpage it reads. The authors tried 23 editing tactics across two commercial AI models and found that over half of the trials moved the target business to the top of the list.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on untested replay fidelity: live baseline ranks were never compared with pristine replay ranks, so the 50% ASR may measure the search_result harness, not web-RAG.","rationale":"The paper is methodologically honest and internally consistent: it includes pristine controls, a deterministic parser audit, fresh-session validation, and an explicit limitations section that all support the replay-scoped claim. I found no internal inconsistency that would overturn the reported measurements. The soft spot is not the replay experiment itself but the inference from it to live web-RAG recommenders. The title, abstract, and threat model describe a webpage-editing adversary acting on live retrieved pages, while the measured attack operates on web_fetch text replayed through search_result blocks. The paper acknowledges this gap, which is why the concern is about external validity rather than internal validity. However, because the headline quantities (ASR, fresh-session persistence, family ordering) are measured only in replay, the unverified assumption that replay preserves ranking behavior is load-bearing for the central claim. The paper already has the data needed to test this directly in the pristine control, so the proposed check is inexpensive and conclusive: if modal replay ranks match live baseline ranks across all eight contexts, the concern is resolved; if not, the aggregate success rates must be re-attributed to the replay harness. Given that the verdict is already CONDITIONAL and the limitation is disclosed, no change in verdict is warranted, but the condition should be made explicit and discharged with the live-vs-replay comparison.","tokens_in":18867,"tokens_out":8221,"duration_ms":74835,"concrete_test":"For each of the eight query-model contexts, recover the target's rank in the stored live-search baseline y0 (Section 4 Stage 1 / Algorithm 1) and compare it with the per-session target ranks from the 20 pristine replays already collected in Section 9.2. Report the full rank distribution, including unranked cases. If any context's modal replay rank differs from its live baseline rank, the replay harness changes ranking behavior on unmodified content, so the reported ASR and 0.805 reproduction rate cannot be attributed to a live web-RAG pipeline. A single mismatch is decisive because the unmodified source content is supposed to be identical; matching ranks in all eight contexts would remove the most direct objection. If the original y0 answers were not retained, rerun the live search for the same queries and compare those rankings with pristine replays of the same source sets.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's empirical results all come from the custom-RAG replay platform. Section 4 Stage 2 stores web_fetch text as the pristine source because the server-side representation used during live search is not available as an editable and replayable artefact, and Stage 4 re-injects it as top-level search_result blocks rather than the production web_search_tool_result path. Section 10 explicitly states that fixed-context replay is not behaviourally equivalent to a production live-search pipeline. The unmodified-source control in Section 9.2 checks only that replay is stable across 20 fresh sessions and that no spontaneous rank-1 drift occurs; it never compares replay rankings with the live baseline answer y0 used to select targets. If the search_result input format causes the model to weight supplied pages differently than the live web-search path does, then the 62/124 rank-1 outcomes, the 0.805 fresh-session rate, and the family comparisons describe the harness, not a real web-RAG recommender. This risk is concrete: related work [31] shows attacks that succeed when content is supplied directly can fail after retrieval and reranking. Because every headline quantity is measured only in replay, replay fidelity is the load-bearing premise; the paper's own limitation statement identifies but does not resolve it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SIREN, an automated attacker–judge method for competitive rank manipulation in web-augmented LLM recommenders. SIREN captures live web pages via Anthropic's server-side web_search and web_fetch tools, applies offline edits to a single captured page using a taxonomy of 23 content-poisoning techniques across six families, and replays the fixed source set to the target model as search_result blocks. It is evaluated on two Claude models (Haiku 4.5 and Sonnet 5) across four anonymized recommendation queries in eight query–model contexts, totaling 124 technique trials. The authors report a rank-1 attack success rate of 50.0% (62/124), a mean fresh-session reproduction rate of 0.805 for successful payloads, and descriptive findings that declarative ranking claims and seeded lists outperform directive-form injections. The paper includes pristine controls, parser-audited rank extraction, and explicit limitations regarding the replay setting.","tokens_in":19064,"tokens_out":9681,"duration_ms":83456,"significance":"The manuscript's strengths are its controlled fixed-context platform, the systematic taxonomy, and an unusually candid set of sensitivity analyses. The deterministic parser audit (62/63 judge decisions confirmed; a stricter explicit-enumeration rule lowers the ASR from 50.0% to 46.8%), the pristine replay control, the fresh-session validation (37/62 payloads always rank 1, two never), the disclosure of the reduced-sweep selection issue, and the explicit statement in Section 10 that the replay is not behaviorally equivalent to a production pipeline all indicate careful empirical practice. If the replay platform faithfully approximates the live web-search condition, the paper would be a significant advance: it would be the first systematic comparison of content-edit types for rank manipulation under a fixed source context, and the declaration-versus-directive finding is a plausible and actionable security insight. The cross-model transfer asymmetry (Section 9.7) is also a valuable observation.","major_comments":[{"comment":"The load-bearing premise of the headline results—that the search_result replay preserves the ranking behavior of the production web-RAG pipeline—is never tested. The paper stores web_fetch text as the pristine source because the server-side representation is unavailable, and re-injects it as top-level search_result blocks rather than through the production web_search_tool_result path. The pristine control in Section 9.2 replays unmodified sources 20 times per context and only shows the absence of spontaneous rank-1 drift; it does not compare the replay ranking to the stored live baseline answer y0 used for target selection. Without such a fidelity check, the 62/124 rank-1 outcomes and the 0.805 fresh-session rate may characterize the search_result harness rather than a live web-RAG recommender. This concern is concrete: the paper itself cites [31] showing that attacks effective when content is supplied directly can fail after retrieval and reranking, and Section 10 admits that fixed-context replay is not behaviorally equivalent to a production live-search pipeline. Because every headline quantity is measured only in replay, I request that the authors either (a) validate replay fidelity by comparing pristine-replay rankings against the stored live baseline y0 for each of the eight contexts, reporting per-context agreement, or (b) explicitly rescope the abstract, title, and conclusion claims to the search_result interface so that 'web-RAG recommenders' is not the unqualified object of study.","section":"Section 4 (Stages 2 and 4) and Section 10 (Limitations)"},{"comment":"The attacker and judge are always claude-sonnet-5, so in the Sonnet-target runs (R3, R4, R7, R8) the attacker, judge, and target are the same model, whereas in the Haiku-target runs (R1, R2, R5, R6) the attacker and judge are a different model. This confounds the target-model comparison in Tables 6 and 7: any apparent difference in attack success between Haiku and Sonnet could be due to the degree of attacker–target weight sharing rather than to target-model robustness to the content edits themselves. The paper notes that source discovery and sampling differ across targets and that the comparison is descriptive, but it never acknowledges this particular confound, which is directly relevant to the conclusion in Section 11 that 'no uniform ordering between the target models' is supported. I recommend either running at least one Sonnet-target condition with a non-Sonnet attacker/judge to quantify the effect, or explicitly adding this as a limitation in the interpretation of the target-model comparisons.","section":"Section 8 (Models) and Section 9.6 (Success vs. Target Model)"}],"minor_comments":[{"comment":"The stored baseline answer y0 is used in Algorithm 1 and Section 9.2, but Section 4 Stage 1 only says that the returned URLs are stored as source objects; please clarify explicitly how and when the live baseline answer y0 is captured for each context.","section":"Section 4 (Stage 1)"},{"comment":"The 124 trials are nested in eight contexts with per-context ASR ranging from 0.261 to 1.000, and the paper correctly warns about non-independence in Section 9.1 and Section 10. However, the abstract presents only the pooled 62/124 figure; please add a context-level range or confidence interval to the abstract, or at least state explicitly that the pooled rate is descriptive of these eight contexts only.","section":"Section 9.1 and Appendix A"},{"comment":"The pristine-control experiment uses 20 fresh sessions per context, but the number of Haiku temperature-0.0 generations that produced distinct answer texts is reported for only one context; please report for all contexts whether the target rank remained stable, even when answer text varied.","section":"Section 9.2"},{"comment":"The sentence 'The results show a descriptiveclaim-beats-directivepattern' is missing spaces and should read 'a descriptive claim-beats-directive pattern.' There are several similar spacing typos in the text (e.g., 'rank4and' in Section 9.5, '16distinct' in Section 9.2, 'descriptiveclaim-beats-directive' in Section 9.5, and 'All620.80537' in Table 4).","section":"Section 9.5"},{"comment":"The description of how overlapping edits are handled (first-listed edit wins, reverse-offset application) is clear, but please also state how edits that target the same element boundary are disambiguated, and confirm that the splice operation never corrupts the original source when an attacker submits a malformed offset.","section":"Section 5 (Element-indexed editing)"},{"comment":"The paper notes that some semantic eligibility rules are enforced through the attacker prompt rather than the harness. Please provide at least one example of such a rule and explain why it was not mechanically enforced, as this affects the reproducibility of the technique taxonomy.","section":"Section 10 (Constraint enforcement)"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of cs.IR and the empirical work is generally careful, with honest sensitivity checks. The main concern is the unvalidated replay fidelity; if the authors can provide a live-vs-replay baseline comparison or rescope the claims, the paper could become a solid contribution. The attacker/judge model confound should also be addressed. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is the most careful controlled study I have seen of content edits on LLM ranking under a fixed supplied source set. The headline numbers are real for the replay harness. Whether they transfer to a production web-RAG pipeline is not established, and the authors mostly say so.\n\nWhat it does well: it adapts PAIR to iterative webpage editing, defines 23 techniques grouped into six families, and keeps the surrounding source set and order fixed so rank changes can be attributed to the edited content. The controls are honest: pristine replay over 20 fresh sessions, parser-versus-judge disagreement, a stricter explicit-enumeration rule, and fresh-session validation of successful payloads. The 0.805 fresh-session persistence is a real result for the harness, as is the descriptive claim-vs-directive pattern. The limitations section is unusually upfront.\n\nThe soft spot is the one that matters. Source capture uses web_fetch text because the server-side representation used during live search is not available, and replay re-injects sources as top-level search_result blocks rather than through the live web_search_tool_result path. The unmodified-source control checks that replay is stable; it never compares replay ranks with the live baseline answer y0. Without that comparison we do not know if the harness weights supplied pages the way the production path does. Related work [31] already shows that attacks which succeed when content is supplied directly can fail after retrieval and reranking. So the 50.0% ASR and 0.805 fresh-session rate characterise the replay platform unless the equivalence is shown. The paper explicitly bounds its claim to fixed-context replay, but the abstract's 'live webpages' framing invites a wider reading, and the missing baseline check is the gap a referee should push on.\n\nSmaller issues, in proportion: 124 trials are nested in eight contexts and aggregate rates have no cluster-adjusted intervals; one reduced-sweep technique was retained below the parser-confirmed threshold (disclosed); and no code, data, or prompts are released. None of these is disqualifying, but they keep confidence moderate.\n\nWho it is for: people working on generative engine optimisation, indirect prompt injection, and LLM ranking integrity will get real value from the taxonomy and the controlled methodology. It deserves serious peer review. I would send it out with a request for the live-versus-replay baseline comparison and for released artifacts or a clear reason no artifacts can be shared.","headline":"The 62/124 rank-1 rate is a real result about a fixed-context replay harness; the live web-RAG transfer question is explicitly left open, and the missing live-baseline comparison is the paper's true soft spot.","tokens_in":19611,"tokens_out":3460,"would_cite":true,"duration_ms":31565,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Content-only edits to a single retrieved webpage can move a chosen real entity to the top of an LLM-generated ranked recommendation: in 62 of 124 technique trials across eight query–model contexts, SIREN reached rank 1, and most…","keywords":["large language models","retrieval-augmented generation","indirect prompt injection","generative engine optimisation","recommendation manipulation","red-teaming","rank manipulation","web content poisoning"],"falsifier":"Run SIREN's successful payloads through a production web-RAG pipeline with live retrieval, reranking, and the target's real search path, and count rank-1 outcomes: if the rate drops far below 0.805 (or below the 50 percent trial success), the replay harness—not the model's live ranking—carried the effect. A cheaper check: compare the same edit injected as a search_result block against the same edit delivered through the actual web_search_tool_result path the production system uses.","tokens_in":18659,"feed_emoji":"🎯","tokens_out":5003,"duration_ms":38211,"temperature":0.7,"pith_summary":"The paper aims to show that a web-enabled LLM's ranked recommendations are manipulable by editing one retrieved webpage, without touching the query, the retrieval process, or the model. It introduces SIREN, an automated attacker–judge loop adapted from jailbreaking, which iteratively applies one of 23 content-poisoning techniques to a captured page and replays the fixed source set to the target model. Across two production Claude models, rank 1 was reached in 62 of 124 trials and reproduced in fresh sessions at a mean rate of 0.805. If true, this means any party controlling a page that a web-RAG assistant fetches—a business, a listicle writer, a user-contributed page—can influence the shortlist consumers see, and defenses must look beyond sanitizing hidden markup toward provenance and cross-source corroboration.","feed_headline":"One edited webpage moved an LLM's top pick in 62 of 124 trials","feed_subtitle":"SIREN's content-poisoning methods kept rank 1 in most fresh-session replays—and declarative claims beat directives.","key_machinery":"The load-bearing device is the custom-RAG replay platform: live URLs are fetched once via Anthropic's server-side web_fetch, one source is parsed into an element index (content blocks, links, images), offline edits are applied as character-level splices to a pristine copy, and the unchanged source set is re-supplied to the target as search_result blocks in the same order. This keeps retrieval and source composition fixed, so any change in the model's ranking is attributable to the edited content. On top of it runs the PAIR-adapted loop: attacker proposes edits, target answers, judge extracts rank, and the result feeds the next proposal until rank 1 or budget exhaustion.","core_discovery":"SIREN demonstrates that competitive rank manipulation of real entities in web-RAG recommenders can be achieved through content-level edits to one already-retrieved source, under a fixed supplied context. The method captures live pages with Anthropic's web tools, parses the editable page into addressable elements, and has an attacker model propose edits restricted to one of 23 techniques while a judge extracts the target entity's rank and feeds it back, until rank 1 or the 20-iteration budget. Empirically, the loop reached rank 1 in 62 of 124 trials; successful payloads replayed in fresh sessions retained rank 1 with mean rate 0.805. The paper also finds that declarative ranking claims and seeded lists are generally more effective than directive-form injections, and that cross-model transfer is asymmetric.","pith_inferences":["If the replay proxy overestimates real-world transfer (as earlier work suggests attacks can be filtered by retrieval and reranking), the 0.805 persistence rate may be an upper bound for live pipelines; a direct live-search replication would settle this.","The asymmetry in cross-model transfer (Haiku-origin payloads often fail on Sonnet, Sonnet-origin often succeed on Haiku) hints that the same webpage edit can shift a recommendation system's ranking in opposite directions across models—an implication for any single 'ranking SEO' practice.","The mechanism generalizes beyond tourism rankings: any list-producing web-RAG query (product comparisons, hiring shortlists, answer lists) exposes the same single-page attack surface, and the 23-technique taxonomy provides a ready audit kit.","The finding that a model attributes an inserted claim as the business 'claiming to be' at rank 2 but states it directly at rank 1 suggests the model's trust in the entity is itself being manipulated—a testable hypothesis about model credulity under rank position."],"forward_implications":["A page owner who can legitimately control a retrieved page has a practical way to influence top-of-list LLM recommendations, since declarative content (ranked lists, FAQ claims, metadata) often works without being flagged as an injection.","Defenses limited to stripping hidden markup or detecting directive instructions will miss the most effective payload classes; provenance and cross-source corroboration are the natural next step.","Success rates and persistence vary by target model and query context, so model-agnostic defenses cannot be assumed; evaluation must be per-context.","Because replay keeps sources fixed, the measured effects isolate content influence; extending to live retrieval pipelines is required before claiming real-world attack rates of 50 percent.","The iterative loop matters: most successes took multiple attempts, so single-shot content edits are a weaker description of the threat than the adaptive attacker."],"supporting_citations":[{"why":"Supplies the PAIR attacker–judge loop that SIREN adapts from jailbreak search to webpage editing.","marker":"[2]"},{"why":"Defines the GEO-Bench rank-shift metrics (NRG, Success@k) that the paper adopts for comparability.","marker":"[17]"},{"why":"Establishes the web-content-pollution setting of locally rewriting retrieved pages, which SIREN extends to promoting a real entity among real competitors.","marker":"[13]"},{"why":"Frames generative engine optimization, the organic-content counterpart whose adversarial version SIREN studies.","marker":"[1]"},{"why":"Reports that attacks effective when content is supplied directly may fail to reach the generator after retrieval and reranking, the boundary condition for SIREN's replay platform.","marker":"[31]"},{"why":"Provides the indirect prompt injection mechanism that motivates the taxonomy's injection families.","marker":"[5]"}],"fun_headline_variants":["SIREN: one edited webpage flips LLM picks to #1 in 62/124 tries","Edited source alone moves LLM top pick 62 times in 124 tests","LLM recommenders spoofed by single-page edits: 62 rank-1 wins","SIREN payloads keep rank 1 in 80% of fresh LLM sessions","Content edits beat retrieval in LLM ranking attacks: SIREN hits #1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The text that Anthropic's web_fetch tool returns is treated as the faithful 'pristine' page the model actually sees during a live search, and replaying captured sources as search_result blocks is assumed to preserve the model's ranking behavior; the paper notes the server-side search representation is unavailable and that replay is not behaviorally equivalent to a production live-search pipeline.","fun_headline_variants_meta":{"raw":{"variants":["SIREN: one edited webpage flips LLM picks to #1 in 62/124 tries","Edited source alone moves LLM top pick 62 times in 124 tests","LLM recommenders spoofed by single-page edits: 62 rank-1 wins","SIREN payloads keep rank 1 in 80% of fresh LLM sessions","Content edits beat retrieval in LLM ranking attacks: SIREN hits #1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000408,"raw_usage":{"total_tokens":2169,"prompt_tokens":1048,"completion_tokens":1121,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":664,"completion_tokens_details":{"reasoning_tokens":1021}},"tokens_in":664,"tokens_out":1121,"duration_ms":9590,"temperature":1.0,"reasoning_tokens":1021,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:28:33.150094+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SIREN's successful payloads through a production web-RAG pipeline with live retrieval, reranking, and the target's real search path, and count rank-1 outcomes: if the rate drops far below 0.805 (or below the 50 percent trial success), the replay harness—not the model's live ranking—carried the effect. A cheaper check: compare the same edit injected as a search_result block against the same edit delivered through the actual web_search_tool_result path the production system uses.","supporting_citations":[{"cited_title":"GEO-Bench: Benchmarking Ranking Manipulation in Generative Engine Optimization","cited_arxiv_id":"2605.29107","evidence_quote":"Defines the GEO-Bench rank-shift metrics (NRG, Success@k) that the paper adopts for comparability."},{"cited_title":"One Polluted Page Is Enough: Evaluating Web Content Pollution in Generative Recommenders","cited_arxiv_id":"2606.13610","evidence_quote":"Establishes the web-content-pollution setting of locally rewriting retrieved pages, which SIREN extends to promoting a real entity among real competitors."},{"cited_title":"Can It Reach the Generator? Investigating the Survival of Prompt-Injection Attacks in Realistic RAG Settings","cited_arxiv_id":"2605.28017","evidence_quote":"Reports that attacks effective when content is supplied directly may fail to reach the generator after retrieval and reranking, the boundary condition for SIREN's replay platform."}],"review_version":2}