Pith. sign in

REVIEW 3 major objections 4 minor 51 references

Replacing one web agent with a team of specialized web agents changes which attacks succeed, even when the task, tools, and browser stay the same.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 01:00 UTC pith:PVO2BSCY

load-bearing objection A genuinely useful SAS-vs-MAS testbed and two solid attack case studies, but the marquee Telephone Loop result rests on a framework error signal with no no-adversary control—easy to fix, but load-bearing as written. the 3 major comments →

arxiv 2608.00202 v1 pith:PVO2BSCY submitted 2026-07-31 cs.CR

From Monoliths to Swarms: A Study of Attack Surface Evolution in the Transition to Multi-Agent Web Systems

Yashaswi Malla , Sandra Siby This is my paper
classification cs.CR
keywords multi-agent systemsweb agentsLLM securityattack surfaceprompt injectiondelegation loopdenial of serviceattack taxonomy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that the shift from a single web agent to a multi-agent crew is not security-neutral: with the same user task, the same tool surface, and the same browser substrate, splitting work across specialized sub-agents creates structural vulnerabilities that a single agent does not have. It proposes a taxonomy of seven threat vectors and five attack classes for web-based multi-agent systems, then builds a controlled testbed that swaps only the architecture. Three adversarial scenarios are evaluated; the headline finding is a new attack that makes agents delegate in a circle, crashing the run, which succeeds about 80% of the time on three of four frontier models and cannot even be attempted against a single agent. The results imply that defenses such as prompt hardening and reasoning mode do not transfer reliably across models or architectures.

Core claim

The central claim is that architecture alone changes attack outcomes: a fully external, web-only adversary who controls only page content can compromise a multi-agent crew in ways that are inert against a single-agent system. The Telephone Loop attack first harvests an agent's role from one attacker page, then uses that metadata on a second page to force two agents to delegate the same task back and forth; because delegations are tool calls in the framework, the unresolved cycle produces an orphaned-tool-call error and terminates the run. The attack averaged 80% success across three frontier models at baseline, while the fourth model detected the manipulation in 92% of trials. For the other

What carries the argument

The key machinery is the delegation protocol of the multi-agent framework: an agent can hand a task to a peer as a tool call, and the call completes only when the peer responds. The Telephone Loop weaponizes this by chaining two attacker-controlled pages—one that captures the visiting agent's role identity and one that instructs the second agent to delegate resolution of a fabricated survey back to the first—so the same delegation is never resolved, producing a circular-delegation error that crashes the run. The testbed holds the user task, LLM backend, tool set, browser substrate, and adversarial workload fixed and varies only single-agent versus multi-agent configuration.

Load-bearing premise

The load-bearing premise is that the framework's orphaned-tool-call error is a reliable, attack-specific sign of delegation-loop compromise; if that error occurs spontaneously under normal operation, the reported success rates for Telephone Loop are inflated.

What would settle it

Run the same multi-agent task with the adversarial payload removed—without the role-capture and hand-back instructions—and count how often the framework still reports an orphaned-tool-call error. If the baseline error rate is comparable to the roughly 80% attack success rate, the attack's measured success is an artifact of framework instability.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Multi-agent web systems have a different and, for the same task, often larger attack surface than single-agent systems, so architecture decisions carry security consequences beyond performance.
  • A purely web-side adversary with no access to prompts, code, or tools can still cause a denial of service by inducing delegation cycles.
  • Prompt hardening is not a portable defense: it collapses one model's attack success from 100% to 8% while barely affecting others; reasoning mode does not reliably help and can even make attacks more successful.
  • Models differ sharply in security awareness: some resist silently without warning the user, which matters for real deployments where a user must know an attack was attempted.
  • The proposed taxonomy gives a structured way to map known and future web-based multi-agent attacks to underlying threat vectors.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reported success signal for Telephone Loop is the framework's own orphaned-tool-call error; without a control run that omits the adversarial payload, the headline success rate could include natural framework instability.
  • If the architectural component is what really drives the vulnerability, similar delegation cycles could likely be triggered in other role-based multi-agent frameworks, not just the one tested, suggesting the attack generalizes beyond a single stack.
  • The model that resists does so mainly by flagging a conspicuous 'unusual' pattern; the paper itself notes that stealthier variants avoiding that dark pattern could still compromise it, implying the measured robustness may be brittle.
  • The silent-resistance behavior of other models implies that even when an attack fails, users may never learn it happened; security monitoring for multi-agent systems may need to instrument delegation logs rather than rely on agent warnings.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies how the transition from single-agent (SAS) to multi-agent (MAS) LLM web agents changes the security attack surface. It contributes a taxonomy of MAS-specific threat vectors and attack patterns, and a testbed, WEBMASLAB, that keeps user task, tool surface, browser substrate, and web environment fixed while varying only architecture. Three attack scenarios are evaluated: Vault Mirage (credential phishing), Header Heist (session-token exfiltration), and the new Telephone Loop (delegation-cycle denial of service). Experiments sweep four frontier models (Claude Sonnet 4.5/4.6, GPT-5.2/5.4) and three conditions (baseline, prompt-hardened, reasoning-enabled). The main reported findings are that attack resistance is model-dependent, that MAS can destroy the resistance of a capable SAS (e.g., GPT-5.4 in Vault Mirage), and that the Telephone Loop attack compromises MAS at high baseline rates for three of four models, with prompt hardening effective for one model only.

Significance. If the claims hold, this is a useful and timely contribution. The controlled architecture-only design is a genuine strength: no prior work, as cited by the authors, directly compares SAS and MAS on identical web-agent tasks with the same tool surface and browser substrate. The threat model is also comparatively clean — a fully external, web-only adversary with no inside-agent access. The taxonomy is grounded in prior work and clearly maps attack patterns to threat vectors. The paper additionally ships a testbed and notes that it is released, which improves reproducibility. The main significance hinges on the Telephone Loop result: it is presented as a new MAS-specific attack that is inert against SAS and achieves an average 80% baseline ASR on three frontier models. That result currently rests on an insecure success criterion and a small number of trials, so the significance is conditional on fixing those issues.

major comments (3)
  1. [§5.1.3, §6.3, Table 4] The Telephone Loop success criterion is defined exclusively by CrewAI reporting an orphaned-tool-call error (§5.1.3: 'Our evaluation harness logs a trial as a successful attack if the underlying agent framework, CrewAI, reports an orphaned tool-call error'). No control trials are reported in which the same task and the same pages are run without the adversarial role-harvesting and cross-role hand-back payload. This is load-bearing because §7.1 documents that MAS agents fail for non-adversarial reasons in the otherwise similar Header Heist scenario — 22/26 GPT-MAS failures came from an agent erroneously concluding it lacked a tool that was in fact available. If background CrewAI instability can produce the same error signal, the reported 80% baseline ASR could substantially overstate the attack's causal effect. The paper should add no-adversary control trials, report the false-positive ra
  2. [§6.1–§6.3, Tables 2–4] All ASR figures are based on 12 trials per model×architecture×condition cell, but no confidence intervals or significance tests are reported. The headline percentages are therefore less precise than they appear. For example, if GPT-5.4's Telephone Loop baseline ASR of 58% corresponds to 7/12 trials, the 95% binomial CI is roughly [28%, 85%]; GPT-5.4's Vault Mirage MAS ASR of 69% (8/12) has a similarly wide interval. Statements such as 'averaging 80% across them at baseline' or 'collapses one model's ASR from 100% to 8%' would be more defensible with at least binomial CIs, and the central Telephone Loop cells would benefit from additional trials. Without this, the exact numeric claims outrun the measurement.
  3. [§5.1.3, §6.3, Abstract] The paper states that the Telephone Loop attack is 'inert against SAS' and lists this as part of the SAS-vs-MAS evidence. But by the paper's own definition (§5.1.3), the attack's prerequisites do not exist in SAS, so it is impossible rather than empirically inert. The SAS-vs-MAS architecture comparison, which is the paper's stated RQ3, is therefore not directly measured for this scenario. The architecture-specific claim is supported only if the control condition in the first major comment is added and the framework-error signal is shown to be attack-induced. The abstract and introduction should be reworded to distinguish 'not applicable to SAS' from 'empirically failed against SAS.'
minor comments (4)
  1. [Abstract / §6.3] The abstract says Claude Sonnet 4.6 'resists the attack with a 92% detection rate,' but §6.3 reports that in every trial this model still visited the first attacker site and exfiltrated its CrewAI role metadata. The 92% detection rate reflects refusal to proceed to the second stage, not prevention of metadata leakage. The authors acknowledge this 'false sense of security' in the text, but the abstract should not present 92% detection as equivalent to resisting the full attack.
  2. [§6.3, Table 4 / Figure 10] Table 4 reports per-model ASR pooled across all conditions, while the abstract's 80% average refers only to baseline. The paper should explicitly state which trials are included in the abstract claim and, ideally, also report per-condition ASR with the same level of detail as Figure 10.
  3. [§5.3] The LLM-as-a-judge for security awareness uses GPT-5.2, which is also one of the evaluated models. The authors state they manually validated the labels, which is good, but the paper would be stronger if it reported inter-annotator agreement or at least the number of judge outputs that were manually corrected.
  4. [§7.2] The limitations section is candid about the closed web environment, fixed mesh topology, and lack of open-source models. It would also be useful to state that the 'assume injection has already happened' step in Telephone Loop (§5.1.3) is an assumption about how the malicious link reaches the agent, and that the paper does not evaluate an end-to-end delivery mechanism such as a spam email or malicious forum post.

Circularity Check

0 steps flagged

No significant circularity: the paper's central claims are measured empirical outcomes, not derived quantities, and no load-bearing self-citation chain is present.

full rationale

The paper's central claims are empirical outcomes, not derived quantities. ASR, exposure, and detection are measured per trial from ground-truth side effects such as captured credentials in attacker logs, uploaded session payloads, or a framework-reported orphaned tool-call error. There is no fitted parameter later presented as a prediction, and the SAS/MAS comparison holds task, tools, browser substrate, and backbone model fixed, so the architecture contrast is implemented rather than assumed. The taxonomy in Section 3 is constructed from external prior work and each attack pattern is mapped to published references; no uniqueness theorem or ansatz is imported from the authors' own prior publications. The only self-referential element is the use of GPT-5.2 as an LLM-as-a-judge for security-awareness labels in Section 5.3, but the paper states that the judge output is manually validated ('We also validate the output of the LLM-as-a-judge manually to ensure the labels have been assigned correctly'), and this label is not used in any derivation of the headline ASR. The Telephone Loop success criterion in Section 5.1.3 uses CrewAI's orphaned tool-call error as a proxy for circular delegation; the absence of control trials without the adversarial payload is a potential threat to measurement validity rather than a circularity in which the result is equivalent to its inputs by construction. The paper itself documents a related non-adversarial MAS failure mode in Section 7.1, which is a legitimate empirical limitation but not a circular step. Because the load-bearing claims are directly measured against ground-truth artifacts and no fitted quantity is relabeled as a prediction, no circular step can be exhibited. Score 0.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The paper reports measured aggregate rates; no numbers are fitted to data or tuned to make attacks succeed. The main axioms are evaluation-proxy assumptions, the representativeness of one MAS topology, the validity of the LLM judge, the testbed's threat-model design, and the completeness of the taxonomy.

axioms (5)
  • ad hoc to paper CrewAI orphaned-tool-call error is a valid proxy for attacker-induced delegation loop
    This is the sole success label for Telephone Loop (§5.1.3, §6.3); no control condition measures the baseline error rate of the framework.
  • domain assumption The fixed sequential mesh topology and CrewAI orchestration represent web-based MAS
    The paper uses one topology and acknowledges in §7.2 that hierarchical/linear/debate topologies may differ.
  • domain assumption GPT-5.2 as LLM-as-a-judge yields reliable security-awareness labels
    Used in §5.3 to score detection; authors mention manual validation but report no agreement metric.
  • domain assumption BrowserProfileTool can access and decrypt saved Chrome passwords on demand
    Required for Vault Mirage; a normal browser would not expose decrypted passwords to an LLM via a tool.
  • domain assumption Literature-derived taxonomy of seven threat vectors and thirteen patterns is complete
    §3.1 describes aggregation/mapping, but no formal completeness argument is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 20482 in / 13238 out tokens · 127894 ms · 2026-08-04T01:00:08.420139+00:00 · methodology

0 comments
read the original abstract

Large Language Model (LLM)-based web agents are increasingly evolving from single-agent systems (SAS) to multi-agent systems (MAS). While MAS can lead to improved task performance by decomposing complex tasks across specialized sub-agents, such role decomposition introduces new structural attack surfaces that are absent in SAS. This expanded attack surface remains poorly understood and inadequately categorized. To address this, we propose a taxonomy to categorize attack vectors specific to web-based MAS, accounting for vulnerabilities introduced or amplified by the involvement of multiple agents. We further present a test-bed WebMASLab to analyze web agent security against a fully external, web-only adversary. To isolate the effect of architecture, we keep the user task, tool surface, and browser substrate fixed, and compare single- and multi-agent setups. We evaluate three adversarial scenarios, across three conditions (baseline, prompt-hardened, and reasoning-enabled), including a novel MAS-specific Telephone Loop attack that exploits cross-agent delegation to create cyclical task loops. The attack is inert against SAS but compromises MAS when powered by three of the four frontier models evaluated (Claude Sonnet 4.5, GPT-5.2, GPT-5.4), averaging 80% across them at baseline. Only the fourth model, Claude Sonnet 4.6, resists the attack with a 92% detection rate. For the rest, the detection is 0% at baseline, reaching 33% with prompt-hardening for one model. We also show that obvious defenses do not generalize; prompt-hardening collapses one model's ASR from 100% to 8% while providing only modest reduction to the others. Our findings demonstrate that the transition from single- to multi-agent web systems changes the security landscape. Role specialization may not only lead to performance optimization but also introduce new architectural risks that require further study and defenses.

Figures

Figures reproduced from arXiv: 2608.00202 by Sandra Siby, Yashaswi Malla.

Figure 1
Figure 1. Figure 1: MAS architecture in WEBMASLAB . The Executive Layer. This layer manages initial routing and sensitive credential handling. It comprises two agents: (i) a Dispatcher agent, that parses and classifies incoming user requests into one of the discrete domain categories, including Shopping, Social Media, Forms, General, and File, and (ii) a Credential Manager agent, that is the only entity with access to sensiti… view at source ↗
Figure 2
Figure 2. Figure 2: Attack A: Vault Mirage Attack. Red boxes are attacker con [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Attack B: Header Heist Attack. Red boxes are attacker controlled [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Attack C: Telephone Loop Attack. Red boxes are attacker [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Vault Mirage Attack: Outcome composition by model and ar [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Vault Mirage Attack: ASR by experimental condition for MAS [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Header Heist Attack: Attack success rate by condition and model [PITH_FULL_IMAGE:figures/full_fig_p011_8.png] view at source ↗
Figure 7
Figure 7. Figure 7: Header Heist Attack: Per-run outcome composition by model [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Header Heist Attack: Detection (bottom) and exposure (top) rates. [PITH_FULL_IMAGE:figures/full_fig_p012_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Telephone Loop Attack: Outcome composition by model and [PITH_FULL_IMAGE:figures/full_fig_p012_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Base Backstory Prompt for File Handler Specialist [PITH_FULL_IMAGE:figures/full_fig_p015_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Hardened Backstory Prompt for Credential Manager [PITH_FULL_IMAGE:figures/full_fig_p015_12.png] view at source ↗
Figure 15
Figure 15. Figure 15: Prompt used for LLM-as-a-judge to classify agent final re [PITH_FULL_IMAGE:figures/full_fig_p016_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Attacker payload for Vault Mirage B.2. Header Heist Adversarial Postmill Post [PITH_FULL_IMAGE:figures/full_fig_p016_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Attacker payload for Header Heist. We also add upvotes and [PITH_FULL_IMAGE:figures/full_fig_p016_17.png] view at source ↗
Figure 20
Figure 20. Figure 20: Header Heist Attack: Detection (bottom) and exposure (top) [PITH_FULL_IMAGE:figures/full_fig_p017_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Header Heist Attack: Further breakdown of the attack failure [PITH_FULL_IMAGE:figures/full_fig_p017_21.png] view at source ↗
Figure 19
Figure 19. Figure 19: Vault Mirage Attack: Detection rate and exposure rate by model [PITH_FULL_IMAGE:figures/full_fig_p017_19.png] view at source ↗
Figure 22
Figure 22. Figure 22: Telephone Loop Attack: Detection rate (top) and exposure rate [PITH_FULL_IMAGE:figures/full_fig_p017_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Telephone Loop Attack: API crash error after orphaned tool call [PITH_FULL_IMAGE:figures/full_fig_p018_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Header Heist Attack: E-commerce Specialist erroneously con [PITH_FULL_IMAGE:figures/full_fig_p018_24.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 14 linked inside Pith

  1. [1]

    Openhands: An open platform for ai software developers as generalist agents,

    X. Wang, B. Li, Y . Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y . Song, B. Li, J. Singhet al., “Openhands: An open platform for ai software developers as generalist agents,” inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 65 882–65 919

    2025

  2. [2]

    Openwebagent: An open toolkit to enable web agents on large language models,

    I. L. Iong, X. Liu, Y . Chen, H. Lai, S. Yao, P. Shen, H. Yu, Y . Dong, and J. Tang, “Openwebagent: An open toolkit to enable web agents on large language models,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), 2024, pp. 72–81

    2024

  3. [3]

    Browser-use: Make websites accessible for ai agents,

    M. Gretter and I. Møller, “Browser-use: Make websites accessible for ai agents,” https://github.com/browser-use/browser-use, 2024, ac- cessed: 2026-06-11

    2024

  4. [4]

    Perplexity AI, “Comet,” https://www.perplexity.ai/comet, 2026, ac- cessed: 2026-06-11

    2026

  5. [5]

    Multi-agent collaboration: Harnessing the power of intelligent llm agents,

    Y . Talebirad and A. Nadiri, “Multi-agent collaboration: Harnessing the power of intelligent llm agents,”arXiv preprint arXiv:2306.03314, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Magentic-one: A generalist multi-agent system for solving complex tasks,

    A. Fourney, G. Bansal, H. Mozannar, C. Tan, E. Salinas, F. Niedtner, G. Proebsting, G. Bassman, J. Gerrits, J. Alberet al., “Magentic-one: A generalist multi-agent system for solving complex tasks,”arXiv preprint arXiv:2411.04468, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Autogen: Enabling next-gen llm applications via multi-agent conversations,

    Q. Wu, G. Bansal, J. Zhang, Y . Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liuet al., “Autogen: Enabling next-gen llm applications via multi-agent conversations,” inFirst conference on language modeling, 2024

    2024

  8. [8]

    Webarena: A realistic web environ- ment for building autonomous agents,

    S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y . Bisk, D. Friedet al., “Webarena: A realistic web environ- ment for building autonomous agents,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 15 585–15 606

    2024

  9. [9]

    Workarena: How capable are web agents at solving common knowl- edge work tasks?

    A. Drouin, M. Gasse, M. Caccia, I. H. Laradji, M. Del Verme, T. Marty, L. Boisvert, M. Thakkar, Q. Cappart, D. Vazquezet al., “Workarena: How capable are web agents at solving common knowl- edge work tasks?”arXiv preprint arXiv:2403.07718, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    Safearena: Evaluating the safety of autonomous web agents,

    A. D. Tur, N. Meade, X. H. L `u, A. Zambrano, A. Patel, E. Durmus, S. Gella, K. Sta´nczak, and S. Reddy, “Safearena: Evaluating the safety of autonomous web agents,”arXiv preprint arXiv:2503.04957, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Agentdam: Privacy leakage evaluation for autonomous web agents,

    A. Zharmagambetov, C. Guo, I. Evtimov, M. Pavlova, R. Salakhut- dinov, and K. Chaudhuri, “Agentdam: Privacy leakage evaluation for autonomous web agents,”Advances in Neural Information Processing Systems, vol. 38, 2026

    2026

  12. [12]

    Tamas: Benchmarking adversarial risks in multi-agent llm systems,

    I. Kavathekar, H. Jain, A. Rathod, P. Kumaraguru, and T. Ganu, “Tamas: Benchmarking adversarial risks in multi-agent llm systems,” arXiv preprint arXiv:2511.05269, 2025

    arXiv 2025

  13. [13]

    Trism for agentic ai: A review of trust, risk, and security management in llm- based agentic multi-agent systems,

    S. Raza, R. Sapkota, M. Karkee, and C. Emmanouilidis, “Trism for agentic ai: A review of trust, risk, and security management in llm- based agentic multi-agent systems,”arXiv preprint arXiv:2506.04133, 2025

    arXiv 2025

  14. [14]

    Seven security challenges that must be solved in cross- domain multi-agent llm systems,

    R. Ko, J. Jeong, S. Zheng, C. Xiao, T.-W. Kim, M. Onizuka, and W.-Y . Shin, “Seven security challenges that must be solved in cross- domain multi-agent llm systems,”arXiv preprint arXiv:2505.23847, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    Prompt injection attack against llm-integrated applications,

    Y . Liu, G. Deng, Y . Li, K. Wang, Z. Wang, X. Wang, T. Zhang, Y . Liu, H. Wang, Y . Zheng, L. Y . Zhang, and Y . Liu, “Prompt injection attack against llm-integrated applications,” 2025. [Online]. Available: https://arxiv.org/abs/2306.05499

    Pith/arXiv arXiv 2025

  16. [16]

    Collaborative shad- ows: Distributed backdoor attacks in llm-based multi-agent systems,

    P. Zhu, L. Li, Y . Lyu, L. Sun, S. Su, and J. Shao, “Collaborative shad- ows: Distributed backdoor attacks in llm-based multi-agent systems,” arXiv preprint arXiv:2510.11246, 2025

    arXiv 2025

  17. [17]

    Gpt-4v (ision) is a generalist web agent, if grounded,

    B. Zheng, B. Gou, J. Kil, H. Sun, and Y . Su, “Gpt-4v (ision) is a generalist web agent, if grounded,”arXiv preprint arXiv:2401.01614, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks,

    J. Y . Koh, R. Lo, L. Jang, V . Duvvur, M. Lim, P.-Y . Huang, G. Neubig, S. Zhou, R. Salakhutdinov, and D. Fried, “Visualwebarena: Evaluating multimodal agents on realistic visual web tasks,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 881–905

    2024

  19. [19]

    Assistantbench: Can web agents solve realistic and time- consuming tasks?

    O. Yoran, S. J. Amouyal, C. Malaviya, B. Bogin, O. Press, and J. Berant, “Assistantbench: Can web agents solve realistic and time- consuming tasks?” inProceedings of the 2024 Conference on Empir- ical Methods in Natural Language Processing, 2024, pp. 8938–8968

    2024

  20. [20]

    The browsergym ecosystem for web agent research,

    D. Chezelles, T. Le Sellier, S. O. Shayegan, L. K. Jang, X. H. L`u, O. Yoran, D. Kong, F. F. Xu, S. Reddy, Q. Cappartet al., “The browsergym ecosystem for web agent research,”arXiv preprint arXiv:2412.05467, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    St-webagentbench: A benchmark for evaluating safety and trustwor- thiness in web agents,

    I. Levy, B. Wiesel, S. Marreed, A. Oved, A. Yaeli, and S. Shlomov, “St-webagentbench: A benchmark for evaluating safety and trustwor- thiness in web agents,”arXiv preprint arXiv:2410.06703, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Agentwebbench: Bench- marking multi-agent coordination in agentic web,

    S. Zhong, K. Shen, and C. Xiong, “Agentwebbench: Bench- marking multi-agent coordination in agentic web,”arXiv preprint arXiv:2604.10938, 2026

    Pith/arXiv arXiv 2026

  23. [23]

    Formalizing and benchmarking prompt injection attacks and defenses,

    Y . Liu, Y . Jia, R. Geng, J. Jia, and N. Z. Gong, “Formalizing and benchmarking prompt injection attacks and defenses,” in33rd USENIX Security Symposium (USENIX Security 24). Philadelphia, PA: USENIX Association, Aug. 2024, pp. 1831–1847. [Online]. Available: https://www.usenix.org/conference/ usenixsecurity24/presentation/liu-yupei

    2024

  24. [24]

    Not what you’ve signed up for: Compromising real- world llm-integrated applications with indirect prompt injection,

    K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real- world llm-integrated applications with indirect prompt injection,” in Proceedings of the 16th ACM workshop on artificial intelligence and security, 2023, pp. 79–90

    2023

  25. [25]

    Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for llm agents,

    E. Debenedetti, J. Zhang, M. Balunovic, L. Beurer-Kellner, M. Fis- cher, and F. Tram`er, “Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for llm agents,”Advances in Neural Information Processing Systems, vol. 37, pp. 82 895–82 920, 2024

    2024

  26. [26]

    Agent security bench (asb): Formalizing and benchmark- ing attacks and defenses in llm-based agents,

    H. Zhang, J. Huang, K. Mei, Y . Yao, Z. Wang, C. Zhan, H. Wang, and Y . Zhang, “Agent security bench (asb): Formalizing and benchmark- ing attacks and defenses in llm-based agents,” inInternational Con- ference on Learning Representations, vol. 2025, 2025, pp. 35 331– 35 366

    2025

  27. [27]

    Wainjectbench: Benchmarking prompt injection detections for web agents,

    Y . Liu, R. Xu, X. Wang, Y . Jia, and N. Z. Gong, “Wainjectbench: Benchmarking prompt injection detections for web agents,”arXiv preprint arXiv:2510.01354, 2025

    arXiv 2025

  28. [28]

    Web fraud attacks against llm-driven multi-agent systems,

    D. Kong, H. Peng, Y . Zhang, L. Zhao, Z. Xu, S. Lin, C. Lin, and M. Han, “Web fraud attacks against llm-driven multi-agent systems,” arXiv preprint arXiv:2509.01211, 2025

    arXiv 2025

  29. [29]

    Manipulating llm web agents with indirect prompt injection attack via html accessibility tree,

    S. Johnson, V . Pham, and T. Le, “Manipulating llm web agents with indirect prompt injection attack via html accessibility tree,”arXiv preprint arXiv:2507.14799, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    Muzzle: Adaptive agentic red- teaming of web agents against indirect prompt injection attacks,

    G. Syros, E. Rose, B. Grinstead, C. Kerschbaumer, W. Robertson, C. Nita-Rotaru, and A. Oprea, “Muzzle: Adaptive agentic red- teaming of web agents against indirect prompt injection attacks,”

  31. [31]

    Waaa! web adversaries against agentic browsers,

    S. Datta, A. Nahapetyan, W. Enck, and A. Kapravelos, “Waaa! web adversaries against agentic browsers,”arXiv preprint arXiv:2605.05509, 2026

    Pith/arXiv arXiv 2026

  32. [32]

    Agentic browsers and the same- origin policy,

    F. Roesner and D. Kohlbrenner, “Agentic browsers and the same- origin policy,” 2026

    2026

  33. [33]

    A survey on llm- based multi-agent systems: workflow, infrastructure, and challenges,

    X. Li, S. Wang, S. Zeng, Y . Wu, and Y . Yang, “A survey on llm- based multi-agent systems: workflow, infrastructure, and challenges,” Vicinagearth, vol. 1, no. 1, p. 9, 2024

    2024

  34. [34]

    Converse: Benchmarking contextual safety in agent-to-agent conversations,

    A. Gomaa, A. Salem, and S. Abdelnabi, “Converse: Benchmarking contextual safety in agent-to-agent conversations,” inFindings of the Association for Computational Linguistics: EACL 2026, 2026, pp. 3246–3268

    2026

  35. [35]

    Magpie: a benchmark for multi-agent contextual privacy evaluation,

    G. Juneja, J. N. S. Pasupulati, A. Albalak, W. Hua, and W. Y . Wang, “Magpie: a benchmark for multi-agent contextual privacy evaluation,” arXiv preprint arXiv:2510.15186, 2025

    arXiv 2025

  36. [36]

    Terrarium: Revisiting the blackboard for multi-agent safety, privacy, and security studies,

    M. Nakamura, A. Kumar, S. Mahmud, S. Abdelnabi, S. Zilber- stein, and E. Bagdasarian, “Terrarium: Revisiting the blackboard for multi-agent safety, privacy, and security studies,”arXiv preprint arXiv:2510.14312, 2025

    arXiv 2025

  37. [37]

    Prompt infection: Llm-to-llm prompt injection within multi-agent systems,

    D. Lee, M. Tiwari, and B. Miranda, “Prompt infection: Llm-to-llm prompt injection within multi-agent systems,” inEuropean Sympo- sium on Research in Computer Security. Springer, 2025, pp. 511– 520

    2025

  38. [38]

    Agents under siege: Breaking pragmatic multi-agent llm systems with optimized prompt attacks,

    R. Shahroz, Z. Tan, S. Yun, C. Fleming, and T. Chen, “Agents under siege: Breaking pragmatic multi-agent llm systems with optimized prompt attacks,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, pp. 9661–9674

    2025

  39. [39]

    Breaking and fixing defenses against control-flow hijacking in multi-agent systems,

    R. Jha, H. Triedman, J. Wagle, and V . Shmatikov, “Breaking and fixing defenses against control-flow hijacking in multi-agent systems,” arXiv preprint arXiv:2510.17276, 2025

    arXiv 2025

  40. [40]

    The trust paradox in llm-based multi-agent systems: When collaboration becomes a security vulnerability,

    Z. Xu, M. Qi, S. Wu, L. Zhang, Q. Wei, H. He, and N. Li, “The trust paradox in llm-based multi-agent systems: When collaboration becomes a security vulnerability,”IEEE Transactions on Computa- tional Social Systems, 2026

    2026

  41. [41]

    Multi-agent systems execute arbitrary malicious code,

    H. Triedman, R. Jha, and V . Shmatikov, “Multi-agent systems execute arbitrary malicious code,”arXiv preprint arXiv:2503.12188, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    Privacy practices of browser agents,

    A. Ukani, H. Haddadi, A. S. Shamsabadi, and P. Snyder, “Privacy practices of browser agents,”arXiv preprint arXiv:2512.07725, 2025

    arXiv 2025

  43. [43]

    Why are web ai agents more vulnerable than standalone llms? a security analysis,

    J. Y . F. Chiang, S. Lee, J.-B. Huang, F. Huang, and Y . Chen, “Why are web ai agents more vulnerable than standalone llms? a security analysis,”arXiv preprint arXiv:2502.20383, 2025

    arXiv 2025

  44. [44]

    From the wild web to the zoo: Benchmarking web agents with a realistic simulator

    B. Grinstead, M. Meireles, C. Kerschbaumer, and C. Allen, “From the wild web to the zoo: Benchmarking web agents with a realistic simulator.”

  45. [45]

    CrewAI: Cutting-edge framework for orchestrating role-playing, autonomous ai agents,

    J. Moura, “CrewAI: Cutting-edge framework for orchestrating role-playing, autonomous ai agents,” https://github.com/crewAIInc/ crewAI, 2024, accessed: 2026-06-11

    2024

  46. [46]

    Introducing Claude Sonnet 4.6,

    Anthropic, “Introducing Claude Sonnet 4.6,” https://www.anthropic. com/news/claude-sonnet-4-6, February 2026, accessed: 2026-06-11

    2026

  47. [47]

    Introducing Claude Sonnet 4.5,

    ——, “Introducing Claude Sonnet 4.5,” https://www.anthropic.com/ news/claude-sonnet-4-5, September 2025, accessed: 2026-06-11

    2025

  48. [48]

    Introducing GPT-5.2,

    OpenAI, “Introducing GPT-5.2,” https://openai.com/index/ introducing-gpt-5-2/, December 2025, accessed: 2026-06-11

    2025

  49. [49]

    Introducing GPT-5.4,

    ——, “Introducing GPT-5.4,” https://openai.com/index/ introducing-gpt-5-4/, March 2026, accessed: 2026-06-11

    2026

  50. [50]

    Trading inference-time compute for adversarial robustness,

    W. Zaremba, E. Nitishinskaya, B. Barak, S. Lin, S. Toyer, Y . Yu, R. Dias, E. Wallace, K. Xiao, J. Heideckeet al., “Trading inference-time compute for adversarial robustness,”arXiv preprint arXiv:2501.18841, 2025. Appendix Appendix A. Sample Agent Prompts Here, we provide the sample prompts use for the baseline and hardened conditions in Figures 11- 14. F...

    Pith/arXiv arXiv 2025

  51. [2026]

    Available: https://arxiv.org/abs/2602.09222

    [Online]. Available: https://arxiv.org/abs/2602.09222

    arXiv