{"id":"c82175cb-b26d-4d34-b961-a91bece81229","arxiv_id":"2506.09600","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A policy-aware red-teaming system (CRAFT) induces policy violations in LLM customer service agents at much higher rates than generic jailbreak prompts, using a new security-focused benchmark (tau-break) built from tau-bench.","lead":"The paper introduces CRAFT, a multi-agent system that reads a customer service agent's policy and plans deceptive user messages to make the agent break its own rules. It also converts the tau-bench task benchmark into tau-break, a security benchmark, and shows that simple defenses do not fully stop these attacks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"τ-break's airline ASR labels rest on an unvalidated, single-model policy-free agent and subjective manual inspection; every headline comparison (70% vs 42.5%) is defined by these labels, so a labeling error would invalidate the central claim.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing point: the entire attack-success metric in τ-break depends on labels derived from a policy-free agent plus subjective manual inspection. If those labels are wrong, the headline number 70.0% is uninterpretable, and the comparison against baselines loses its foundation. Statistical uncertainty (n=4, no confidence intervals) is a real but secondary concern: the 27.5-point gap is large relative to the estimated standard error of roughly 5–6%, so the qualitative ranking would likely survive a proper significance test. The policy-knowledge asymmetry between CRAFT and the generic baselines is also ambiguous, but the ablation study shows CRAFT without policy knowledge (55.0%) still exceeds Emotional Manipulation (50.0%); thus the central claim does not rest solely on that asymmetry. The label-validity concern is more fundamental because it threatens the validity of the benchmark itself. The proposed check — reconstructing the labels with a different policy-free model and independent annotators — directly tests whether the ground truth is stable and objective. Should that check reveal unstable labels, the verdict would need to move from CONDITIONAL to UNVERDICTED or REJECT, since the benchmark's core measurement would be unsupported. But absent that failure, the conditional acceptance remains appropriate.","tokens_in":22917,"tokens_out":10244,"duration_ms":109758,"concrete_test":"Independently reconstruct τ-break's airline labels: run a second policy-free agent (e.g., DeepSeek-V3 instead of GPT-4o) on all 50 airline tasks, and have two annotators blind to the original labels classify each divergent action using a written rubric for 'meaningful policy violation'. Compute the overlap of the resulting ΔA sets and inter-annotator agreement (Cohen's kappa). If the selected test cases and violation labels differ substantially across policy-free models or annotators, the ground truth is unstable and the reported ASR comparisons cannot be interpreted. If the sets and labels largely agree, the label-validity concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — CRAFT achieves 70.0% ASR vs 42.5% for the τ-bench user simulation — is defined entirely by the precomputed forbidden-action labels in τ-break. Section 4 constructs these labels by running a single 'policy-free agent' (GPT-4o) on each τ-bench task, taking ΔA = Afree \\ Agold, and then 'manually inspecting' each divergent action to decide whether it is a meaningful policy violation. This labeling step is unvalidated in three ways: (1) it depends on the particular policy-free model's ability to discover a forbidden tool call for a given request; a different or stronger policy-free agent might yield a different ΔA, changing which tasks are included and what counts as success; (2) the manual inspection is subjective and no inter-annotator agreement, rubric, or independent check is reported; (3) the label is an action name (e.g., update_reservation_flights) rather than a verified environment outcome — if the action with the arguments the target agent supplies does not actually mutate the database (or mutates it in a policy-compliant way), the attack would be counted as successful without a violation. Because every ASR in Tables 1–3 and the headline comparison inherits these labels, an error here propagates to every quantitative conclusion. A conditional acceptance requires demonstrating that the label set is both complete (all meaningful violations for each request are included) and precise (every labeled action actually violates policy when executed).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CRAFT, a multi-agent red-teaming system that extracts relevant policy fragments, plans deception strategies, and advises on avoided disclosures in order to induce policy violations in task-oriented LLM agents. It introduces τ-break, a security-focused re-labeling of τ-bench airline and retail tasks, reports attack success rates for five agent models and several attack baselines, and evaluates three prompt-based defenses. The headline result is a 70.0% attack success rate (ASR) for CRAFT in the airline domain versus 42.5% for the non-strategic τ-bench user simulation, with the claim that non-strategic simulation severely underestimates real-world attack risk.","tokens_in":23200,"tokens_out":8965,"duration_ms":88787,"significance":"If the benchmark labels are reliable and the reported differences are statistically robust, the paper makes a useful contribution: it demonstrates that policy-aware red-teaming substantially outperforms generic jailbreak methods at eliciting policy violations, and that non-strategic user simulation underestimates risk. The proposed methodology for converting a task-completion benchmark into a security benchmark is reusable, and the defense evaluation addresses a practical deployment concern. The authors are transparent about limitations (small task set, synthetic environment, static attacks, full-policy assumption) and include a responsible-disclosure plan. However, the central quantitative claims currently rest on a label-construction procedure that is not fully validated and on statistics computed from very small per-task samples without uncertainty quantification.","major_comments":[{"comment":"The airline-domain ground-truth labels are obtained by running a single policy-free GPT-4o agent and manually inspecting the divergent actions in ΔA. This is unvalidated in three respects: (i) a different or stronger policy-free model could yield a different ΔA and hence a different set of tasks and different attack successes; (ii) the manual inspection has no reported inter-annotator agreement, rubric, or independent verification; and (iii) the labels are action names, so a call to e.g., update_reservation_flights that does not actually change the database state would still be counted as a success. Because every ASR in Tables 1–3 inherits these labels, the benchmark's completeness and precision must be demonstrated, for example with multiple annotators, a second policy-free model, and tool-execution checks.","section":"Section 4 and Appendix C.1"},{"comment":"All pass@k and ASR values are computed from n=4 trials per task over 20 airline tasks (80 trials per method), and no confidence intervals or significance tests are reported. For the headline comparison of 70.0% vs 42.5% (56/80 vs 34/80), the approximate 95% confidence intervals are 59–80% and 32–54%, so the difference is nominally significant, but for narrower comparisons (for example GPT-4o-mini as attacker: 71.2 vs 56.3) no such evidence is given. Report confidence intervals and a paired test (e.g., McNemar) across the 20 tasks for each method pair, and clarify how the four runs per task are made independent given the fixed seed mentioned in Section 5.2.","section":"Section 5.1 and Tables 1–3"},{"comment":"The claim 'By k = 4, even the strongest defense (Policy Reminder (Relevant)), exceeds 80% ASR' is not supported by the reported data. Under CRAFT, the pass@4 values for Policy Reminder (Relevant) are 55.0 (GPT-4o), 60.0 (GPT-4o-mini), 90.0 (LLaMA-3.3), 85.0 (DeepSeek-V3), and 90.0 (Qwen2.5-70B), which average to 76.0%. Please either correct the claim or describe the aggregation used to produce Figure 4 so that the reader can reproduce it.","section":"Section 7.1 and Table 6"}],"minor_comments":[{"comment":"The sentence 'outperforming conventional jailbreak methods such as DAN prompts, emotional manipulation, and coercive' ends with an incomplete item; 'coercive' should be 'coercive prompting' or the sentence should be rewritten.","section":"Abstract"},{"comment":"The statement 'with seed set to 10 for reproducibility' does not specify sampling temperature or how the four trials are made independent; please clarify the sampling protocol.","section":"Section 5.2"},{"comment":"The caption says 'Attack Success Rate (ASR) at various pass @ levels' but the table reports only pass@1; please update the caption to match the contents.","section":"Table 3 caption"},{"comment":"The text says 'removing either the PolicyAnalyzer or the policy knowledge itself', but Table 4 has no row for removing policy knowledge explicitly; please clarify which ablation corresponds to this statement.","section":"Section 6.1"},{"comment":"The sentence 'public release will of the work artifacts will redact the attack implementation details' is grammatically incomplete and appears to conflict with the GitHub release statement in the abstract; please rewrite.","section":"Section 9 (Ethical Considerations)"},{"comment":"The verify_user_secret tool always returns 'verified: False', so a compliant agent can never satisfy the user's request in the retail tasks; the main text should state that retail ASR measures refusal-only behavior, as this affects interpretation.","section":"Appendix F"},{"comment":"The spelling of 'DialogueExecuter' varies with 'DialogueExecutor'; please make the terminology consistent.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper's central contribution depends entirely on the reliability of the τ-break ground-truth labels. I recommend requiring a validation appendix with inter-annotator agreement, multi-model label generation, and tool-execution checks, as well as confidence intervals for all pass@k values. The discrepancy between the Section 7.1 defense claim and Table 6 should also be corrected before the paper can be considered further."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading if you work on LLM agent safety. The CRAFT decomposition into PolicyAnalyzer, DeceptionPlanner, and AvoidanceAdvisor is genuinely new, and the tau-break conversion recipe from task-completion to security evaluation is a useful idea that others will build on. The ablation showing that policy knowledge drives most of the gain is clean, and the defense experiments, though light, are a reasonable first pass. The paper is honest about its limitations, which I appreciate.\n\nThe soft spot is the airline label construction. The stress-test note is on target: the 70% ASR number and every comparison in Tables 1-3 depend on a label set derived from a single policy-free GPT-4o agent plus manual inspection. No inter-annotator agreement, no check that a different or stronger policy-free model would give the same Delta-A, and no verification that the labeled action actually produces a policy violation when executed with the target agent's arguments. That is a real validity threat, and the paper does not answer it. The retail domain is cleaner because the added authentication policy makes violations unambiguous.\n\nThe statistical concerns are real but minor by comparison. n=4 per task is thin, and the paper reports no confidence intervals. The pass@k point estimates bounce around enough that the 70% vs 42.5% gap is probably real in direction but the magnitude is shaky. That alone would not stop me from taking the paper seriously, but the label issue could.\n\nI do not think the central argument collapses. Even if the airline labels shift, the qualitative finding that policy-aware multi-agent planning beats generic jailbreaks is plausible and supported by the retail numbers and the ablation. But the exact headline number should not be trusted until the labels are validated with multiple policy-free models, an explicit rubric, and ideally released data.\n\nWho is this for? Researchers and practitioners building or red-teaming customer-service agents. It deserves a serious referee, not a desk reject. For acceptance I would require the code and data release, a validation of the airline labels, and at least some uncertainty quantification on the headline comparisons. If those are addable, this is a solid paper.","headline":"A genuinely new red-teaming framework with a plausible but under-validated benchmark; worth refereeing, but require label validation and artifacts.","tokens_in":23730,"tokens_out":2213,"would_cite":true,"duration_ms":24722,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Policy-aware red-teaming cracks policy-abiding agents 70% of the time","keywords":["LLM agents","policy adherence","red-teaming","adversarial user simulation","jailbreak attacks","multi-agent planning","tau-bench","attack success rate"],"falsifier":"Re-derive the tau-break airline labels using a different policy-free model (for example DeepSeek-V3) and have multiple independent annotators classify the divergent actions; if the forbidden-action set changes substantially or annotators disagree on what counts as a violation, the 70% attack success rate is not a stable measure of agent vulnerability.","tokens_in":22735,"feed_emoji":"🕵️","tokens_out":7571,"duration_ms":70696,"temperature":0.7,"pith_summary":"Policy-adherent LLM agents that are supposed to refuse policy-violating requests can be reliably manipulated by an attacker who reads the policy first. The paper introduces CRAFT, a multi-agent red-teaming system whose modules extract relevant policy fragments, plan false-premise \"assume that...\" lines, and advise which true details to withhold. On a re-purposed version of tau-bench called tau-break, CRAFT reaches 70.0% attack success at pass@1 in the airline domain, versus 42.5% for the original non-strategic user simulation, 50.0% for emotional manipulation, and 35.0% for DAN. The paper also tests three prompt-based defenses and finds that none eliminate the vulnerability, with attack success converging above 80% at pass@4 even under the best defense. The central message is that evaluating agent safety with cooperative user simulations or generic jailbreaks produces a false sense of security.","feed_headline":"Policy-aware red-teaming cracks policy-abiding agents 70% of the time","feed_subtitle":"A multi-agent attacker that reads the policy and hides disqualifying facts beats generic jailbreaks in customer service.","key_machinery":"The load-bearing mechanism is the division of labor among four LLM agents that together turn a policy document into a conversation script. The PolicyAnalyzer extracts exact policy fragments relevant to the user request; the DeceptionPlanner produces an \"Assume <A>\" instruction that, if accepted by the target agent, would make the forbidden request look permissible; the AvoidanceAdvisor enumerates \"Do not say <X>\" rules so the red-teamer never volunteers disqualifying facts such as a basic-economy fare or a change of plans; and the DialogueExecutor runs the multi-turn conversation using those instructions. The evaluation machinery is tau-break, which defines attack success as a forbidden tool call identified by the set difference between a policy-free agent's actions and the gold actions ($\\Delta A = A_{\\text{free}} \\setminus A_{\\text{gold}}$), manually inspected for meaningful violations, plus retail tasks with an added authentication requirement.","core_discovery":"On the paper's own terms, the discovery is that policy-adherent agents fail not because generic jailbreaks work on them, but because a red-teamer who reasons about the policy can induce specific forbidden tool calls. CRAFT decomposes the attack into four roles: a PolicyAnalyzer that copies the relevant policy fragments, a DeceptionPlanner that instructs the user to say \"Assume <A>\" to plant a false premise, an AvoidanceAdvisor that lists true statements the user must not say, and a DialogueExecutor that carries out the conversation. The success condition is a forbidden action, not task failure: the attack succeeds only when the agent executes an action like updating a basic-economy reservation or bypassing authentication. To build the benchmark, the paper re-labels 20 airline tau-bench tasks by comparing a policy-free agent's actions with the gold actions and manually flagging the divergent ones, and augments 30 retail tasks with an authentication rule whose violation counts as an attack. Stripping out the policy knowledge or the planning modules drops attack success toward the naive-user baseline, which is the paper's evidence that policy-aware planning, not generic prompting, is what carries the attack.","pith_inferences":["Beyond the paper's claims: the 70% airline ASR is measured under the assumption that the attacker knows the full policy document; if real deployments keep policies confidential, CRAFT-style attacks would likely lose some of their edge, so testing with partially inferred policies is a natural next step.","Beyond the paper's claims: because the success labels come from manual inspection of the delta set, an independent re-annotation study with multiple annotators and different policy-free models would show how much of the reported ASR is an artifact of labeling choices.","Beyond the paper's claims: the \"Assume <A>\" mechanism suggests a targeted defense the paper does not develop, namely asking the agent to verify factual premises against the database or policy before acting, rather than merely re-prompting the policy."],"forward_implications":["Generic jailbreak and prompt-injection attacks (DAN, DPI, emotional manipulation, and insistence) understate the real risk to policy-constrained agents by 20 to 35 percentage points in the airline domain.","Deploying a non-strategic, cooperative user simulator as a safety evaluation gives a misleadingly optimistic picture of policy adherence.","Prompt-based defenses can lower attack success, but none removes it; under repeated trials (pass@4) even the strongest tested defense converges above 80% ASR, so safety cannot be secured by prompting alone.","The same conversion recipe, using a policy-free agent plus relabeling or adding new policy constraints, can turn other task-completion benchmarks into security benchmarks.","Being a strong attacker and being a robust agent are separate properties; Qwen2.5-70B is among the best attackers and also the most vulnerable agent in the authors' measurements."],"supporting_citations":[{"why":"Supplies the tau-bench airline and retail environments, policies, tools, and the non-strategic user simulation baseline that CRAFT outperforms.","marker":"Yao et al., 2024"},{"why":"Defines the pass@k metric used to report attack success under stochastic sampling.","marker":"Chen et al., 2021"},{"why":"Provides the DAN jailbreak prompt used as a baseline attack.","marker":"Shen et al., 2024a"},{"why":"Provides the emotional manipulation baseline and its empathy-escalation strategy.","marker":"Vinay et al., 2024"},{"why":"Provides the direct prompt injection baseline.","marker":"Liu et al., 2023b"},{"why":"Provides the insistent prompting baseline.","marker":"Sun et al., 2024"},{"why":"Shows that induction and false-premise framing can alter model behavior, underpinning the DeceptionPlanner's \"Assume <A>\" strategy.","marker":"Jin et al., 2024"},{"why":"Prior agent-targeted foot-in-the-door attack that the paper contrasts as not explicitly modeling domain policies.","marker":"Nakash et al., 2025"},{"why":"Explains why helpfulness and compliance alignment make hierarchy prompting insufficient, supporting the paper's defense analysis.","marker":"Joselowitz et al., 2024"},{"why":"Prior observation that strong attackers are not necessarily robust agents, supporting the paper's attacker/agent asymmetry finding.","marker":"Kour et al., 2025"}],"fun_headline_variants":["Policy-reading attackers force AI agents to break their own rules","CRAFT: policy-aware red-teaming beats generic jailbreaks on agents","Adversarial policy analysis triggers forbidden actions in 70% of trials","Weaponizing policy knowledge: new red-team cracks rule-bound agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's attack-success labels rest on the assumption that comparing a policy-free GPT-4o agent's actions with the gold set, followed by manual inspection, correctly identifies every meaningful policy violation; if that labeling is incomplete or biased, all reported attack success rates inherit the error.","fun_headline_variants_meta":{"raw":{"variants":["Policy-reading attackers force AI agents to break their own rules","CRAFT: policy-aware red-teaming beats generic jailbreaks on agents","Adversarial policy analysis triggers forbidden actions in 70% of trials","Weaponizing policy knowledge: new red-team cracks rule-bound agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00077,"raw_usage":{"total_tokens":3428,"prompt_tokens":978,"completion_tokens":2450,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":2375}},"tokens_in":594,"tokens_out":2450,"duration_ms":22202,"temperature":1.0,"reasoning_tokens":2375,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:43:33.204275+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-derive the tau-break airline labels using a different policy-free model (for example DeepSeek-V3) and have multiple independent annotators classify the divergent actions; if the forbidden-action set changes substantially or annotators disagree on what counts as a violation, the 70% attack success rate is not a stable measure of agent vulnerability.","supporting_citations":[{"cited_title":"Emotional Manipulation Through Prompt Engineering Amplifies Disinformation Generation in AI Large Language Models","cited_arxiv_id":"2403.03550","evidence_quote":"Provides the emotional manipulation baseline and its empathy-escalation strategy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior agent-targeted foot-in-the-door attack that the paper contrasts as not explicitly modeling domain policies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Explains why helpfulness and compliance alignment make hierarchy prompting insufficient, supporting the paper's defense analysis."}],"review_version":1}