{"id":"c6a5e928-d8f2-4633-a727-82690ec1dc7d","arxiv_id":"2608.09476","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"ActBench, a 600-case trajectory-based benchmark, finds that behavioral safety of cowork agents depends more on the base language model than on the agent harness.","lead":"This paper introduces ActBench, a benchmark of 600 matched benign and adversarial tasks for measuring whether AI cowork agents leak data or take unauthorized actions while completing normal work, scored from execution trajectories rather than final answers. The authors evaluate 15 language models and 6 agent harnesses over 24,000 runs, reporting wide variation in attack success across models and high vulnerability across all harnesses.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The model-vs-harness comparison is confounded: Deepseek-v4-Pro generates the attacks and is also the evaluated model (and the fixed base in RQ2), so only Deepseek-v4-Pro faces adaptive attacks while all other models and harnesses face transfer attacks; the reported 84.3% vs 20.7% ASR span does not…","rationale":"The reader's weakest_assumption identifies exactly the load-bearing weakness: Deepseek-v4-Pro is both the attack-generation model and an evaluation target, and it is also the fixed base model in the harness comparison. This is not a peripheral implementation detail; it directly undermines the paper's central empirical claim. Because only Deepseek-v4-Pro receives adaptive attacks, the RQ1 spread can be explained by attack-transfer difficulty rather than intrinsic model safety. Similarly, RQ2's small harness span may reflect that all harnesses are evaluated against attacks tuned for one particular base model, so the harness comparison is not representative of attacks that would be optimized for other models. The paper's own limitation statement calls for 'construction ablations' before causal attribution, but the headline already asserts the attribution. I concur with the rejection of the central claim as stated. I still credit the paper for releasing a large, trajectory-based benchmark with matched benign/malicious pairs, dual evidence verification, and a reproducible construction pipeline; these artifacts retain standalone value for transfer-attack evaluation. However, the specific conclusion that model choice matters more than harness choice is not supported without re-analysis that separates adaptive and transfer results and varies the attack-generation model.","tokens_in":25605,"tokens_out":3961,"duration_ms":39914,"concrete_test":"Re-run RQ1 with per-model adaptive attacks for a subset of models, e.g., Claude-Opus-4.8, GPT-5.5, Qwen3.7-max, and MiniMax-M3. Run the ActBench construction pipeline (reward-guided beam search with reflection, same budget w=3, n=2, d_max=5) separately for each target model, using that model as the reasoning model during construction and using a different model as the attack generator to avoid exact self-tuning, then measure ASR on disjoint rollouts. If Claude-Opus-4.8's ASR rises from 10.1% toward the 90%+ range under adaptive attacks, the model-vs-harness variation claim collapses. As a complementary check, fix Claude-Opus-4.8 instead of Deepseek-v4-Pro in RQ2; if the harness ASR span widens beyond 20.7%, the harness comparison is also model-dependent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that behavioral safety is primarily determined by the base model rather than the agent harness rests on RQ1 and RQ2, but both are contaminated by an attack-generation asymmetry. Section 'Construction and grading' states that 'Candidate generation uses Deepseek-v4-Pro.' Deepseek-v4-Pro is also one of the 15 evaluated models in RQ1 (Table 2, ASR 94.4%) and the fixed base model in RQ2 (Table 3). Thus Deepseek-v4-Pro's score is an adaptive-attack result: the reward-guided beam search and reflection explicitly optimize payloads against this model's behavior. Every other model in RQ1 is evaluated on attacks tuned for Deepseek-v4-Pro, i.e., transfer attacks, which typically understate vulnerability relative to adaptive attacks. The 84.3% ASR spread between Claude-Opus-4.8 and Deepseek-v4-Pro therefore conflates model safety with attack adaptivity. The same asymmetry poisons RQ2: all six harnesses are stressed with the same case set, which was constructed by optimizing payloads for Deepseek-v4-Pro (in the OpenClaw harness). A harness that would be much more vulnerable under attacks tuned for a different base model would still look similar, artificially compressing the 20.7% harness span. The disjoint-rollout statement only prevents evaluation rollouts from being used during construction; it does not address the fact that the attack generator and one evaluation target are the same model. Consequently, the paper's headline finding — 'greater variation across models than agent harness' — is not established by the reported experiments. The benchmark may still be useful as a fixed transfer-attack suite, but the causal attribution to model versus harness requires per-model adaptive attack construction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ActBench introduces a benchmark for behavioral safety of cowork agents, defined by whether an agent's execution remains within the permissions and state changes required by a benign task. The paper constructs 300 matched benign/malicious task pairs spanning 15 risk behaviors, six execution spaces, and 48 web-service APIs, and uses a reward-guided beam search with reflection-based revision and a dual evidence verification mechanism (deterministic log evidence plus LLM-based trajectory evidence) to build and grade cases. The authors evaluate 15 LLMs under a fixed harness and six cowork-agent harnesses under a fixed base model, reporting attack success rate (ASR) spans of 84.3% across models and 20.7% across harnesses, and conclude that behavioral safety is primarily determined by the base model's action selection rather than the agent harness.","tokens_in":25911,"tokens_out":9073,"duration_ms":82564,"significance":"If the empirical comparison were valid, ActBench would be a valuable contribution: the trajectory-based behavioral-safety definition, the matched benign/malicious design that keeps instructions and grading criteria fixed, the 15-behavior taxonomy with explicit propagation paths, the dual evidence verification, and the public release are all strengths. The paper also includes useful auxiliary analyses, including a β-sensitivity ablation and an execution-cost study showing that token counts and action cycles do not reproduce the ASR ordering. However, the central model-versus-harness claim is currently undermined by a confound between the attack-generation procedure and the models/harnesses being compared, and by an internal inconsistency in the reported number of evaluated models. The underlying idea is promising, but the headline conclusion is not yet supported.","major_comments":[{"comment":"The construction pipeline uses Deepseek-v4-Pro as the attack model (Section 'Construction and grading'), and the same model is one of the 15 evaluated base models in RQ1 (Table 2, ASR 94.4%) and the fixed base model in RQ2 (Table 3). The reward-guided beam search and reflection use execution feedback from this model to revise payloads, so Deepseek-v4-Pro's score is an adaptive-attack result, while every other model and harness is scored on transfer attacks. The 84.3% ASR span in Table 2 therefore conflates model safety with attack adaptivity, and the 'Finding' that behavioral safety is primarily determined by model-specific action selection is not supported by this comparison. The statement that construction and evaluation use disjoint rollouts does not remove this confound, because the attack generator has still observed the same model's behavior on the same tasks during construction.","section":"Experimental Setting, 'Construction and grading'; RQ1 (Table 2)"},{"comment":"RQ2 fixes Deepseek-v4-Pro and varies six harnesses while keeping the same case set, which was constructed by optimizing attacks for Deepseek-v4-Pro in the OpenClaw harness. All six harnesses therefore face transfer attacks tuned for a single base-model/harness pair, so the 20.7% ASR span does not measure how harnesses respond to attacks adapted to their own context assembly, memory retrieval, and tool serialization. A harness that would be substantially more vulnerable under harness-specific attack generation would not be visible in this design. The conclusion that attacks remain highly successful across all tested harnesses is accordingly limited to one attack generator and one base model, and cannot support the general claim in the abstract.","section":"RQ2 Agent Harness Effects (Table 3)"},{"comment":"Table 7 reports behavior-conditioned results for 22 models, including GPT-5.6-Sol, GPT-5.6-Terra, GPT-5.6-Luna, GPT-5.4, Grok-4.3, MiniMax-M2.5, and Deepseek-v4-Flash-0731, while the abstract, RQ1, and the experimental setting state that 15 LLMs are evaluated. If these seven models are part of RQ1, the claims of 15 models and 24,000 trajectories are incorrect; if they are not, their presence without explanation in the supplementary results makes the table uninterpretable. This inconsistency needs to be resolved before the results can be reproduced.","section":"Table 7 / 'Additional Results'"},{"comment":"GPT-5.5 is used as both the rating model and the trajectory evidence verifier, and is also one of the evaluated base models in Table 2. When GPT-5.5's own trajectories are scored, the LLM-based component of the dual evidence verification is performed by the same model under evaluation, which is a conflict of interest that could bias its AGS and ASR. The authors should either exclude GPT-5.5 from the evaluated model set, use a different verifier for its trajectories, or justify why this overlap does not affect the reported scores.","section":"Experimental Setting, 'Construction and grading'"}],"minor_comments":[{"comment":"The word 'invocate' should be 'invoke'.","section":"Abstract"},{"comment":"The x-axis labels read 'Python component weight α' but the text describes a sensitivity analysis over the log evidence weight β; the labels should be corrected.","section":"Figures 9 and 10"},{"comment":"The sentence 'Claude-Opus-4.8 reaches 0.43 and above 0.90 for the same labels' is unclear; it should specify which quantities (¬AGS and UGS) and which behavior categories are being compared.","section":"Figure 3 caption"},{"comment":"The RQ1 and RQ2 text blocks appear twice in the submitted full text (for example, the second occurrence of 'RQ1 Model Effects' after Figure 5); this duplication should be removed.","section":"Appendix / Full text"},{"comment":"The column header 'Iter.' is described in the captions as 'the malicious rollout median,' while the main text and Figure 4 use 'median iteration count' to refer to action cycles; the terminology should be aligned.","section":"Tables 2 and 3 captions"},{"comment":"The Discussion correctly states that 'paired uncertainty, guarded reruns, and construction ablations are needed' before causal attribution; this limitation should be reflected in the RQ1/RQ2 findings themselves, not only in the Discussion.","section":"Discussion"}],"recommendation":"major_revision","confidential_remarks":"The confound identified in the reader's report is real and is the main barrier to acceptance: using Deepseek-v4-Pro as both the attack generator and an evaluated target makes the model and harness comparisons transfer-attack comparisons rather than adaptive comparisons. I would be willing to review a revised version that either adds per-model and per-harness attack generation or narrows the claims accordingly. The unexplained additional models in Table 7 and the GPT-5.5 grader/evaluator overlap also need to be addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ActBench is the most complete behavioral safety benchmark for cowork agents I've seen: 300 matched benign/malicious pairs across 15 behaviors, six execution spaces, and 48 APIs, with trajectory-level evidence. The matched-pair design that preserves instructions and grading criteria, and the reward-guided beam search with reflection, are real contributions. The dual evidence verification (log + LLM trajectory) is a sensible attempt to separate realized effects from blocked calls. The benchmark is public, and the construction pipeline is described in enough detail to be reproducible in principle. The paper is also transparent about its settings and includes a useful ablation on the evidence weight.\n\nThe soft spot is not hidden, but it is load-bearing. Candidate generation uses Deepseek-v4-Pro, and that same model is evaluated in RQ1 and fixed as the base model in RQ2. So Deepseek-v4-Pro's 94.4% ASR is an adaptive-attack result, while every other model and every harness in RQ2 faces transfer attacks tuned for Deepseek. The headline span — 84.3% across models vs 20.7% across harnesses — therefore conflates model safety with attack adaptivity. The harness span may be artificially compressed because all six harnesses are stressed with payloads optimized for one base model; a harness that would be far worse under attacks tuned for another model would not show it. The disjoint-rollout statement prevents optimization leakage, but it does not fix the asymmetry. GPT-5.5 also serves as both rating model and an evaluated contestant, which is a second-order version of the same issue. The Discussion itself concedes that construction ablations are needed before attributing causal gains to individual components; that caution should extend to the model-vs-harness claim.\n\nThe consequence: the causal claim that behavioral safety is primarily determined by model-specific action selection is not established by these experiments. What is established is that a fixed transfer-attack suite produces a wide model spread and a narrow harness spread. That is still worth reporting, but it supports a weaker conclusion.\n\nFor researchers building agent safety benchmarks or doing red-team evaluations, the benchmark remains usable as a fixed suite. Anyone adopting it should treat the per-model numbers as transfer-attack measurements, not adaptive-attack measurements, and should not use the harness ranking as evidence about harness design without per-model attack construction.\n\nRecommendation: send to peer review, but require major revision — re-analyze with adaptive attacks per model or at least re-label the results honestly, and validate the trajectory verifier against something independent. The authors seem capable of doing this; the paper reads as honest about its limitations, which is why I would not desk-reject it.","headline":"A genuinely useful benchmark artifact whose headline model-vs-harness comparison is contaminated by using the attack generator as an evaluation target.","tokens_in":26550,"tokens_out":2846,"would_cite":true,"duration_ms":29080,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ActBench claims that behavioral safety in cowork agents is set mainly by the base model, not by the agent harness, and measures it from executed trajectories rather than final replies.","keywords":["behavioral safety","cowork agents","execution trajectory","prompt injection","agent red-teaming","reward-guided beam search","dual evidence verification","attack success rate"],"falsifier":"Re-run the benchmark's RQ1 and RQ2 with attacks optimized separately for each base model, for example by restarting the reward-guided beam search with every model as the reasoning model, and recompute the attack-success spans; if the across-model spread shrinks toward or below the across-harness spread, or if the harness ranking reverses under a different base model, the claim that behavioral safety is primarily model-determined would be refuted.","tokens_in":25341,"feed_emoji":"🤖","tokens_out":7637,"duration_ms":74658,"temperature":0.7,"pith_summary":"ActBench aims to measure behavioral safety of cowork agents—whether an agent's executed actions stay within the permissions and state changes a benign task requires—from execution trajectories rather than final replies. The benchmark pairs 300 benign tasks with 300 adversarial variants that inject a task-reachable payload while preserving instruction, configuration, state, grading criteria, and trusted logs, and it grades every rollout through dual evidence: log-based state verification plus LLM-based trajectory reconstruction. Controlled evaluation across 24,000 trajectories shows attack success rates from 10.1% to 94.4% when only the base model changes, and from 73.7% to 94.4% when only the agent harness changes. The paper concludes that behavioral safety is driven mainly by the base model's action selection, that benign task utility and execution length do not predict it, and that no tested harness compensates for a vulnerable base model. If this holds, agent safety evaluations must separate model and harness effects and should score executed behavior, not just refusal or final text.","feed_headline":"Base model, not harness, decides cowork-agent safety","feed_subtitle":"Attack success varies 84 points across models but just 21 across harnesses in 24,000 controlled runs.","key_machinery":"The load-bearing object is the matched test-case pair with trajectory-level grading. Each malicious case is produced from a benign case by editing only a task-reachable field in context, memory, or environment, leaving the user instruction, utility criteria, attack criteria, and trusted records fixed, while each of the 15 risk behaviors is defined by its realized propagation path through six execution spaces. Attack construction is carried by a self-evolving loop: reward-guided beam search ranks candidate edits by a geometric score $s(x)=g_a^\\alpha g_u^{1-\\alpha}$ that requires both attack evidence and task utility, reflection-based deep probing diagnoses the earliest failed checkpoint and issues a single localized revision instruction, and dual evidence verification fuses log evidence with LLM trajectory evidence so that full attack success requires both the prohibited effect and a verified propagation path. This machinery is what lets the paper claim causal attribution of safety failures and a fair model-versus-harness comparison.","core_discovery":"The core discovery is that, under controlled interventions, the choice of base model moves attack success far more than the choice of agent harness. With the harness fixed, the attack success rate spans 84.3 percentage points, from 10.1% for the safest model to 94.4% for the most exposed; with the base model fixed, it spans only 20.7 points, from 73.7% to 94.4%, across six harnesses, and every harness configuration still succeeds on at least 73.7% of malicious rollouts. The paper reads this as evidence that execution-layer controls do not compensate for a base model that authorizes prohibited actions from untrusted context, and that neither task utility (benign utility scores stay between 0.903 and 0.945) nor action-cycle counts nor token consumption explains the risk ordering. A further finding is that repeated rollouts matter: intermediate-risk models show stochastic attack activation, with pass@3 exceeding pass@1 by up to 17 percentage points, so single-rollout evaluations underestimate exposure.","pith_inferences":["Beyond the paper: the self-evolving construction loop could be pointed at a new harness or a new API set to generate fresh adversarial cases automatically, turning ActBench from a static collection into a continuous red-teaming service.","Beyond the paper: if the model-dominance result generalizes, agent safety should be reported as a function of base model first and harness second, and safety claims that name only the agent product would be underspecified.","Beyond the paper: a concrete testable system would be artifact scanning before execution followed by trajectory-level monitoring during execution, which the paper motivates but does not implement; measuring its combined recall and false-positive rate on the 600-case set would test whether the two boundaries are complementary."],"forward_implications":["Safety evaluations of cowork agents should report attack success separately for the base model and the harness, because the two are not interchangeable and a single combined score hides which component carries the risk.","Because every tested harness showed at least 73.7% attack success under a vulnerable base model, deploying a safety-focused harness without changing the base model should not be expected to close behavioral risk.","Single-rollout red-team results understate risk for models with stochastic attack activation, so pass@k or repeated-rollout estimates are necessary to compare models fairly.","Matching each malicious case to a benign twin with fixed utility criteria makes it possible to measure whether a defense or alignment intervention improves safety without breaking task completion.","Defenses at a single observation boundary are not sufficient on their own, since artifact scanning and trajectory monitoring trade off precision and recall sharply and a layered combination is the natural next step."],"supporting_citations":[{"why":"Supplies the AgentDojo benchmark over untrusted tool returns, the main security-case baseline ActBench extends to joint execution surfaces.","marker":"(Debenedetti et al. 2024)"},{"why":"Supplies WASP, a web-injection benchmark whose fragmented coverage ActBench joins with matched task configurations.","marker":"(Evtimov et al. 2025)"},{"why":"Supplies AgentHarm, the harmful-task evaluation that ActBench contrasts with trajectory-based behavioral safety.","marker":"(Andriushchenko et al. 2025)"},{"why":"Supplies Agent Security Bench's reported attack success rates and mixed attack types, motivating the need for adaptive payload optimization.","marker":"(Zhang et al. 2025)"},{"why":"Supplies MCPTox, a tool-poisoning benchmark covering one of the disjoint attack surfaces ActBench spans through 48 APIs.","marker":"(Wang et al. 2026c)"},{"why":"Supplies AgentDoG, the trajectory-context detection baseline used in the RQ3 safety-policy comparison.","marker":"(Liu et al. 2026)"},{"why":"Supplies the Llama Guard 2 model card, a content-classification defense baseline in RQ3.","marker":"(Llama Team 2024)"},{"why":"Supplies the Llama Prompt Guard 2 model card, a second content-classification defense baseline in RQ3.","marker":"(Meta AI 2025)"},{"why":"Supplies the Cisco-Skill-Scanner artifact-scanning baseline evaluated in RQ3.","marker":"(Tencent Zhuque Lab 2025)"},{"why":"Supplies the AI-Infra-Guard artifact-scanning baseline evaluated in RQ3.","marker":"(Yang et al. 2026)"}],"fun_headline_variants":["Base model, not harness, decides agent attack success","Cowork-agent safety: model choice trumps harness choice","84-point spread in agent risk comes from model, not harness","Harness doesn't fix an unsafe model in ActBench","ActBench: attack rates vary 84 pts by model, 21 by harness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the attack cases are equally hard for every model and harness being compared, even though the attacks were generated and tuned with one particular model, and that same model later anchors the harness comparison and receives the only adaptively tuned attack set.","fun_headline_variants_meta":{"raw":{"variants":["Base model, not harness, decides agent attack success","Cowork-agent safety: model choice trumps harness choice","84-point spread in agent risk comes from model, not harness","Harness doesn't fix an unsafe model in ActBench","ActBench: attack rates vary 84 pts by model, 21 by harness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000251,"raw_usage":{"total_tokens":1588,"prompt_tokens":1010,"completion_tokens":578,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":492}},"tokens_in":626,"tokens_out":578,"duration_ms":5959,"temperature":1.0,"reasoning_tokens":492,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:48:00.200476+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the benchmark's RQ1 and RQ2 with attacks optimized separately for each base model, for example by restarting the reward-guided beam search with every model as the reasoning model, and recompute the attack-success spans; if the across-model spread shrinks toward or below the across-harness spread, or if the harness ranking reverses under a different base model, the claim that behavioral safety is primarily model-determined would be refuted.","supporting_citations":[{"cited_title":"2025 , url =","cited_arxiv_id":null,"evidence_quote":"Supplies WASP, a web-injection benchmark whose fragmented coverage ActBench joins with matched task configurations."}],"review_version":1}