{"id":"a672c733-af03-4f58-9de3-9a1632214920","arxiv_id":"2606.30755","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper introduces SafeClawArena, a 406-task benchmark evaluating security failures in three Claw-like agent platforms across skill supply-chain, state exploitation, data flow, and prompt injection surfaces.","lead":"Claw-like AI agents run persistently with broad system access like always-on computers, and this paper builds a benchmark to test four attack surfaces using a systems analogy. A smart generalist might read it to see how current AI agent platforms handle real security risks that existing tests miss.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"No significant objection identified","rationale":"The reader's note that assessment is abstract-only is accurate; absent the full text, the skeptic pass cannot surface an internally inconsistent assumption or unverified derivation that would alter the UNVERDICTED verdict.","tokens_in":1834,"tokens_out":221,"duration_ms":15170,"concrete_test":"Obtain and read the full manuscript; recompute the plugin-success metric from the raw task logs in the SafeClawArena repository using the exact taint rules described in the methods section; if the 100% figure holds under independent re-execution, the headline result is internally consistent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"No load-bearing concern can be identified. The query supplies only the abstract plus a placeholder for full text; without the actual manuscript sections on benchmark construction, attack definitions, taint-tracking implementation, or experimental controls, no concrete technical weakness in the central claims (70% ASR, 100% plugin success, SeClaw reduction) can be located or tested.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims that Claw-like agents (e.g., OpenClaw) pose severe security risks due to persistent credentials, files, tools, and system-level responsibilities. Adopting a computer-systems analogy (gateway runtime as OS, Skills as applications, Plugins as loadable extensions), the authors introduce SafeClawArena: a benchmark of 406 adversarial tasks across four attack surfaces (Skill Supply-Chain Integrity, Persistent State Exploitation, Cross-Boundary Data Flow, Indirect Prompt Injection). Tasks are executed in containerized replicas of real platforms with canary-marked credentials and evaluated via automated taint tracking across nine output channels. On three platforms and five frontier LLMs, the highest attack success rate is 70%, malicious Plugins succeed in 100% of cases regardless of LLM, SeClaw reduces GPT-5.4 ASR from 70% to 22% (partly via utility-security tradeoffs), and Claude-Opus-4.6 sits near a 22% floor on all platforms. Code and data are released.","tokens_in":1901,"tokens_out":334,"duration_ms":31605,"significance":"If the benchmark construction, task validation, and taint-tracking results hold, the work is significant for importing decades of systems-security insight into agent evaluation and exposing cross-component failure modes that model-only benchmarks miss. Strengths include the containerized replicas, automated taint tracking, release of code/data, and concrete quantitative findings (70% ASR, 100% plugin success, SeClaw reduction) that are falsifiable and reproducible.","major_comments":[],"minor_comments":[],"recommendation":"accept","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their positive assessment of the manuscript, accurate summary of the contributions, and recommendation to accept. We appreciate the recognition of the benchmark's design, reproducibility, and the value of the systems-security lens for agent evaluation.","responses":[],"tokens_in":1469,"tokens_out":66,"duration_ms":22699,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that this work applies a computer-systems framing to always-on agents and measures four attack surfaces with automated tracking, producing concrete failure rates like 70% overall and 100% for malicious plugins.\n\nThey do a solid job defining the analogy (runtime as OS, skills as apps, plugins as extensions) and building 406 tasks across supply-chain integrity, state exploitation, cross-boundary flows, and indirect injection. Running them in containerized replicas of three platforms with canary credentials and taint tracking across nine channels is a clear step past single-turn model tests. The platform comparison, including SeClaw's reduction for one model, adds usable data points.\n\nThe soft spot is the complete absence of method details. The abstract states the results but says nothing about how the adversarial tasks were validated, how taint tracking was implemented without false positives, or what statistical controls were used. Without that, the 70% and 100% figures cannot be assessed. The claim that the analogy reflects real deployments also sits untested in what is shown.\n\nThis is for people working on agent security and systems-style protections for long-running AI. A reader focused on benchmarks would find the new surfaces useful even if the numbers need more backing.\n\nIt deserves peer review to check the missing implementation and validation steps.","headline":"The paper introduces SafeClawArena, a benchmark using a systems analogy for persistent agent attacks, with reported rates up to 70% ASR, but the abstract gives no validation details on tasks or taint tracking.","tokens_in":2415,"tokens_out":357,"would_cite":false,"duration_ms":21681,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Claw-like agents allow attack success rates up to 70 percent because they lack standard computer-system protections for their runtime, skills, and plugins.","keywords":["Claw-like agent security","adversarial benchmark","plugin attacks","persistent state exploitation","prompt injection","runtime mediation","taint tracking","agent platforms"],"falsifier":"Executing the 406 SafeClawArena tasks against a live production Claw-like agent and finding attack success rates that differ substantially from the reported 70 percent maximum.","tokens_in":2750,"feed_emoji":"🛡️","tokens_out":698,"duration_ms":25130,"temperature":0.7,"pith_summary":"Claw-like agents run as persistent processes with broad access to credentials, files, and services, so security failures affect the entire environment rather than isolated responses. The paper maps these agents to classical computer systems by treating the gateway runtime as an OS mediator, skills as installed applications, and plugins as privileged extensions, then checks which long-established safeguards are missing. It introduces SafeClawArena, a benchmark of 406 tasks across four attack surfaces, executed in containerized replicas with taint tracking to measure real cross-component failures. Results show malicious plugins succeed in every case regardless of the underlying model, while one hardened platform reduces success from 70 percent to 22 percent and another model already sits at that floor. The work therefore claims that current agent platforms expose the same classes of vulnerability that operating systems solved decades ago.","feed_headline":"Claw-like agents reach 70% attack success in new benchmark","feed_subtitle":"Malicious plugins succeed in every trial while platform changes cut rates only to a 22% floor on some models.","key_machinery":"The computer-systems analogy that maps gateway runtime to OS, Skills to applications, and Plugins to extensions, which directly determines the four attack surfaces and the design of the SafeClawArena benchmark with automated taint tracking.","core_discovery":"Treating a Claw-like agent as an agentic computer system whose gateway runtime mediates access like an OS, whose Skills act like user applications, and whose Plugins act like loadable extensions reveals four unprotected attack surfaces; SafeClawArena measures them with 406 tasks and finds attack success reaching 70 percent, with plugins succeeding in 100 percent of trials, while SeClaw lowers GPT-5.4 success to 22 percent and Claude-Opus-4.6 already meets that floor on every platform.","pith_inferences":["The same mapping could be applied to other persistent agent frameworks to predict which attack surfaces will appear first.","If the analogy holds, existing OS hardening techniques such as capability-based access or mandatory access control could be ported to agent runtimes with measurable effect.","The benchmark could be extended to measure utility loss when the platform adds the missing mediation layers, testing whether the observed tradeoffs are inherent or implementation-specific."],"forward_implications":["Platform designers must add runtime mediation checks comparable to an OS kernel before skills or plugins execute.","Plugin distribution channels require supply-chain verification because no model defense stops them once loaded.","Benchmarks limited to single tool calls or prompt responses will miss the persistent-state and cross-boundary failures measured here.","Model-level prompt filtering alone cannot close the gap when the platform itself grants elevated privileges.","A 22 percent floor appears across both hardened and unhardened platforms for at least one frontier model, indicating limits to current mitigation strategies."],"fun_headline_variants":["70% attack success in claw-like agent benchmark","Plugins succeed 100% across claw platforms","SeClaw reduces attacks to 22% on GPT-5.4","Claude at 22% attack floor on every platform"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The computer-systems analogy produces attack surfaces and failure modes that match those present in real Claw-like agent deployments.","fun_headline_variants_meta":{"raw":{"variants":["70% attack success in claw-like agent benchmark","Plugins succeed 100% across claw platforms","SeClaw reduces attacks to 22% on GPT-5.4","Claude at 22% attack floor on every platform"]},"model":"grok-4.3","cost_usd":0.007255,"raw_usage":{"total_tokens":3416,"prompt_tokens":813,"num_sources_used":0,"completion_tokens":64,"cost_in_usd_ticks":72549500,"prompt_tokens_details":{"text_tokens":813,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2539,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":813,"tokens_out":64,"duration_ms":32662,"temperature":1.0,"reasoning_tokens":2539,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-01T01:53:00.669215+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Executing the 406 SafeClawArena tasks against a live production Claw-like agent and finding attack success rates that differ substantially from the reported 70 percent maximum.","supporting_citations":[],"review_version":1}