{"id":"8e35ee8b-0c2f-4c6e-bbad-adead0851cbb","arxiv_id":"2607.07474","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":3,"one_line_summary":"A seven-level ordinal severity scale for tool-using AI agents, computed from execution traces, reveals cases where binary attack-success-rate metrics hide dangerous cross-scope leaks and worsening tail risk.","lead":"This paper introduces a seven-level severity scale (L0–L6) for grading the harm of actions taken by AI agents during red-team testing, replacing the binary 'attack succeeded or not' metric. A smart generalist might read it because it shows how current safety benchmarks can miss dangerous agent behaviors that look safe on paper.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The L6 escalation condition requires strictly increasing per-action severity, but the canonical exfiltrate-then-delete example (L4→L3) decreases — if the oracle assigns L6 anyway, the judges' L4 scores are correct, not a blind spot.","rationale":"The reader correctly identified the attribution limitation but treated it as acknowledged and scoped out, which the paper itself does. The more load-bearing concern is an internal inconsistency in the L6 definition that the paper does not acknowledge: the stated escalation condition (strictly increasing severity) is not satisfied by the canonical example (L4→L3), yet the oracle assigns L6 to these episodes. This affects three of the paper's claims at once: (1) the judge 'blind spot' for L6 may be the judges being correct, (2) the V-C worst-case-tail-grows claim may be an artifact, and (3) the scale definition itself is internally inconsistent. The paper's other contributions — the L0–L5 scale, the V-A channel-substitution case, the V-B cross-model localization, and the general methodology — do not depend on L6 and remain sound. The paper is a useful proof of concept with released code, but the L6 definition needs to be either clarified (if escalation means something other than per-action level increase) or corrected (if the oracle is misassigning L6). A CONDITIONAL verdict reflects that the core instrument is valuable but one of the three headline cases and the judge-reliability characterization depend on resolving this inconsistency. The concrete test is straightforward to run given the released code and per-episode logs.","tokens_in":11539,"tokens_out":4643,"duration_ms":173595,"concrete_test":"Examine the oracle's L6 assignment logic in the released code. For each of the two exfiltrate-then-delete episodes scored as L6 (Table IV: GPT-4o mini, no-defense and spotlighting conditions), list the per-action severity level of every consequential step in order. If the sequence is L4→L3 (or any non-increasing sequence), the L6 assignment violates the rubric as stated in Table I and Fig. 1. Then recompute Table IV and Fig. 4 with those episodes reclassified as L4 (the peak). If the L5+L6 tail under spotlighting drops from 3 to 1, the V-C claim that 'spotlighting raises worst-case severity' no longer holds, and the judges' L4 scores on those episodes are correct rather than a blind spot.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The L6 level is defined as 'two or more completed consequential actions escalating in severity over steps' (Table I, Fig. 1). Appendix A gives the canonical example: 'the agent exfiltrates content to an external address and then deletes the sent record, a two-step chain whose severity increases across steps.' But per the tool-effect metadata in Table III, send_email to an external address is L4 (cross-scope, irreversible) and delete_file/delete_email is L3 (irreversible, local). The per-action severity sequence is therefore L4→L3 — a decrease, not an increase. If the oracle assigns L6 to these episodes despite the per-action levels not strictly increasing, two consequences follow. First, the judges scored these episodes as L4 (Section V-D, Fig. 5 bottom row), which the paper frames as a 'systematic blind spot' — but if the rubric's stated condition is not met, the judges are correct and the oracle is wrong. Second, the V-C headline claim (spotlighting raises the worst-case L5+L6 tail from 1 to 3 episodes) depends on two L6 episodes; if those are reclassified as L4, the tail is 1 L5 episode in both conditions, and the 'defense worsens the tail' claim collapses. The reader's identified concern (argument-match attribution requiring a distinguishable target) is a real but acknowledged and scoped-out limitation. This L6 inconsistency is more load-bearing because it is unacknowledged and affects the scale definition, the judge-reliability finding, and one of the three headline cases simultaneously.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"The paper introduces an action-graded severity scale (L0–L6) for evaluating tool-using AI agents under prompt injection, scoring outcomes by reversibility, scope, and privilege rather than the binary attack-success rate (ASR) used by existing benchmarks. A deterministic programmatic oracle computes severity from execution traces and the attacker's stated goal, while a panel of three frontier LLM judges grades tag-free trace summaries. Evaluated on the AgentDojo workspace suite (410 episodes across four models and two defenses), the paper identifies three cases where severity scoring exposes decisions that binary ASR gets wrong: a tool-filter defense reporting 0% ASR while still permitting a cross-scope leak (§V-A), spotlighting lowering ASR while raising the worst-case severity tail (§V-C), and severity localizing all harmful behavior to one of four models (§V-B). The judge panel reproduces the oracle with ordinal Krippendorff's alpha = 0.91 but shares a systematic blind spot for escalation chains (L6). All code, prompts, and per-episode logs are released.","tokens_in":11865,"tokens_out":1106,"duration_ms":289461,"significance":"The paper addresses a genuine gap: existing agentic red-teaming benchmarks reduce outcomes to a single bit, discarding severity information that matters for deployment decisions. The instrument is trace-grounded, portable, and inexpensive. The programmatic oracle is a deterministic function of execution traces and per-tool metadata (not a fitted model), which is a strength. The argument-match attribution rule is self-contained and does not consult the benchmark's own pass/fail verdict, avoiding circularity. The release of all code, prompts, and per-episode logs supports reproducibility. The three disagreement cases are concrete and well-illustrated. The judge-reliability methodology (measuring rather than assuming judge quality, with a stratified sample including L0 controls) is appropriate.","major_comments":[{"comment":"§III-A, Table I, and Appendix A: The L6 escalation condition requires 'two or more completed consequential actions escalating in severity over steps' (strictly increasing per-action severity). However, the canonical exfiltrate-then-delete example in Appendix A assigns send_email (external address) as L4 (cross-scope, irreversible per Table III) and the subsequent delete as L3 (irreversible, local per Table III). The per-action severity sequence is therefore L4→L3, which is a decrease, not an increase. If the oracle assigns L6 to these episodes despite the per-action levels not strictly increasing, the escalation condition as stated is not satisfied. This is load-bearing for three reasons: (1) it affects the scale definition itself; (2) §V-D and Fig. 5 report that all three judges scored true L6 episodes as L4, framed as a 'systematic blind spot'—but if the rubric's stated condition is un","section":null}],"minor_comments":[{"comment":"§V-D, Table II: The bias column shows GPT-5.5 at +0.20 and Claude Opus 4.8 at +0.30, but the MALE column shows 0.27 and 0.35 respectively. These are close but not identical; clarifying the relationship between MALE and signed bias (e.g., whether MALE is computed as absolute bias or includes additional error components) would help readers.","section":null},{"comment":"Fig. 5 caption: 'L1, L2 unoccupied in this sample' is noted, but the confusion matrices also show no L5 oracle episodes. This should be stated explicitly, as it means judge reliability at L5 is untested.","section":null},{"comment":"§IV: The model names (GPT-5.4, Claude Sonnet 4.6, GPT-5.5, Claude Opus 4.8, Gemini 3.5 Flash) appear to be fictional or placeholder names. If these are anonymized or projected names, this should be clarified; if they are real model versions, the naming convention is unusual.","section":null},{"comment":"Table III: The 'Level' column for create_calendar_event, add_calendar_event_participant, and reschedule_calendar_event shows 'L2/L4' and for cancel_calendar_event shows 'L3/L4'. A brief note explaining that the slash separates local-scope from cross-scope outcomes (already implied by the 'arg'/'state' scope column) would improve readability.","section":null},{"comment":"§V-B: The claim that 'the pattern is more consistent with safety tuning than raw capability' is speculative given n=1 suite and modest counts. The paper hedges this appropriately ('though the result rests on a single suite'), but the claim could be further softened or removed.","section":null}],"recommendation":"major_revision","confidential_remarks":"The L6 escalation condition inconsistency is the central issue. If the authors can clarify or fix the escalation definition (e.g., the condition is on trajectory-level escalation of consequence rather than per-action level, or the canonical example is revised), the paper's core contributions are sound and the revision would be straightforward. If the L6 episodes are reclassified as L4, the §V-C headline claim about the worst-case tail growing from 1 to 3 episodes would collapse to 1 L5 episode in both conditions, weakening one of three headline cases. The authors should be asked to explicitly address this before the paper can be accepted."},"author_rebuttal":{"model":"glm-5.2","summary":"The referee identifies a genuine inconsistency between the stated L6 escalation condition and the canonical example used to illustrate it. We acknowledge the error and will revise the L6 definition to match what the oracle actually computes, and will re-examine the §V-D judge blind-spot framing in light of the corrected definition.","responses":[{"response":"The referee is correct, and we are grateful for the careful reading. There is a genuine inconsistency between the stated L6 condition and the canonical example, and it must be fixed.","revision_made":"yes","referee_comment":"§III-A, Table I, and Appendix A: The L6 escalation condition requires 'two or more completed consequential actions escalating in severity over steps' (strictly increasing per-action severity). However, the canonical exfiltrate-then-delete example in Appendix A assigns send_email (external address) as L4 (cross-scope, irreversible per Table III) and the subsequent delete as L3 (irreversible, local per Table III). The per-action severity sequence is therefore L4→L3, which is a decrease, not an increase. If the oracle assigns L6 to these episodes despite the per-action levels not strictly increasing, the escalation condition as stated is not satisfied. This is load-bearing for three reasons: (1) it affects the scale definition itself; (2) §V-D and Fig. 5 report that all three judges scored true L6 episodes as L4, framed as a 'systematic blind spot'—but if the rubric's stated condition is un"},{"response":"We agree. The L6 definition in Table I, §III-A, and Appendix A currently states 'strictly increasing per-action severity over steps,' but the exfiltrate-then-delete example produces a per-action sequence of L4 (send_email to external address) → L3 (delete_file or delete_email, local irreversible), which is decreasing, not increasing. The oracle's actual implementation does not check for strictly increasing per-action levels; it checks for two or more completed consequential actions at different severity levels where the combined trajectory represents a multi-step attack chain worse than any single action. The stated condition was an error in our formalization of the concept. We will revise the L6 definition across Table I, §III-A, Fig. 1, and Appendix A to accurately describe what the oracle computes: a trajectory-level level assigned when two or more completed consequential actions at distinct severity levels form an attack chain (e.g., exfiltration followed by evidence destruction), capturing that the combined harm exceeds the peak of any individual action. We will also update the oracle description in §III-B to match the actual code, which we are releasing and which the referee can inspect.","revision_made":"yes","referee_comment":"(1) it affects the scale definition itself"},{"response":"This is a fair and important point. We accept that the §V-D framing of the judges' L6 failures as a 'systematic blind spot' is overstated given that the rubric text the judges received contained the same inconsistent L6 definition. The judges may have been failing because the escalation condition was ambiguously specified, not because of an inherent limitation of trace-only judging. We will revise §V-D to acknowledge this confound explicitly: the current data cannot distinguish between (a) judges being unable to recognize escalation chains from traces, and (b) judges being unable to apply an inconsistent rubric definition. The corrected L6 definition may also change judge behavior, and we will note that re-running the judge panel under the revised definition is necessary before drawing strong conclusions about the L6 blind spot. We will soften the claim accordingly.","revision_made":"yes","referee_comment":"(2) §V-D and Fig. 5 report that all three judges scored true L6 episodes as L4, framed as a 'systematic blind spot'—but if the rubric's stated condition is unclear or inconsistent, the judges may be failing not because of a cognitive blind spot but because the rubric given to them does not clearly define L6"},{"response":"We agree that L6 is load-bearing and that the inconsistency undermines confidence in the top level of the scale. To address this: (a) we will release the oracle source code (already promised) so the actual escalation logic is inspectable; (b) we will add unit tests for the L6 condition to the appendix, showing exactly which trajectory patterns trigger it; (c) we will re-examine the L6 episodes in our dataset under the corrected definition to confirm they still warrant L6 and report any reclassifications. If any episodes no longer qualify as L6 under the corrected definition, we will update the severity distributions in Figs. 3–4 and Tables IV–V accordingly. We note that L6 episodes are sparse (1–2 per condition), so any reclassification would affect counts but not the overall pattern of results, though we will state this honestly rather than asserting it preemptively.","revision_made":"yes","referee_comment":"(3) the escalation condition is load-bearing for the overall validity of the scale"}],"tokens_in":11193,"tokens_out":1937,"duration_ms":164071,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"The paper introduces a severity scale (L0–L6) for tool-using agent red-team episodes, computed both by a deterministic oracle over execution traces and by an LLM judge panel. The core idea — grading the worst action an agent actually took, rather than reporting a binary attack-success rate — is genuinely useful and not present in the prior work the paper cites. The three disagreement cases (Sections V-A through V-C) are concrete and each makes a real point: the channel-substitution leak through an unfiltered calendar tool is a clean example of binary scoring failing, and the cross-model localization result is informative. Code, prompts, and per-episode logs are released. The judge reliability statistics are well-reported (alpha = 0.91, all weighted kappa > 0.8), and the stratified sampling design — all non-L0 episodes plus L0 controls — is the right choice for this kind of validation. The oracle is a deterministic function of traces and per-tool metadata, not a fitted model, so the circularity burden is low. Credit where earned: the instrument is cheap, portable, and the argument-match attribution rule is a sensible design choice that keeps the oracle self-contained. The paper also honestly scopes its limitations around argument-match attribution requiring a distinguishable target, and excludes the one injection that violates this. That concern is real but acknowledged and handled. The stress-test flag about L6 is more serious and unacknowledged. L6 is defined as two or more completed consequential actions whose severity strictly increases over steps. The canonical example in Appendix A is exfiltrate-then-delete. But per the tool-effect metadata in Table III, send_email to an external address is L4 (cross-scope, irreversible) and delete_file or delete_email is L3 (irreversible, local). The per-action sequence is L4→L3 — a decrease, not an increase. If the oracle assigns L6 to these episodes anyway, the escalation condition as stated is not met. This matters for two reasons. First, the judges scored these episodes as L4, which the paper frames as a systematic blind spot — but if the rubric's own condition is not satisfied, the judges are correct and the oracle is wrong. Second, the V-C headline claim (spotlighting raises the L5+L6 tail from 1 to 3 episodes) depends on two L6 episodes. Reclassify those as L4 and the tail is 1 L5 episode in both conditions, and the 'defense worsens the tail' claim collapses. The author needs to either fix the L6 definition (perhaps the intended condition is 'two or more completed consequential actions at different severity levels' rather than 'strictly increasing') or reclassify these episodes. This is fixable but it is load-bearing because it touches the scale definition, the judge-reliability finding, and one of three headline cases simultaneously. The evaluation is small (410 episodes, sparse at the top of the scale) but the paper acknowledges this honestly. The argument-match attribution limitation is scoped out rather than hidden. No other red flags. This paper is for researchers and practitioners working on agentic safety evaluation who need more than binary ASR. A reader who works on red-team benchmarks or agent deployment decisions will get value from the instrument design and the three disagreement cases. It deserves a serious referee. The L6 issue needs to be resolved before publication, but the core contribution — a trace-grounded ordinal severity scale computed deterministically and validated against an LLM panel — is sound and novel.","headline":"Useful severity instrument for agentic red-team logs; one unacknowledged L6 inconsistency weakens a headline case","tokens_in":12567,"tokens_out":788,"would_cite":true,"duration_ms":237733,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Zero attack-success rate can still mean a cross-scope leak","keywords":["prompt injection","AI agent safety","severity scoring","red teaming","tool-using agents","attack-success rate","LLM-as-judge","agentic AI security"],"falsifier":"Find a setting where the binary attack-success rate and the severity scale agree on every deployment-relevant decision—no defense that reads as safe under ASR leaks under severity, no defense that lowers ASR raises the severity tail, and no per-model localization is hidden by aggregation. If the three disagreements the paper identifies are artifacts of this particular suite, attack family, and model set, the practical case for the instrument weakens.","tokens_in":11820,"feed_emoji":"🛡️","tokens_out":1373,"duration_ms":206087,"temperature":0.7,"pith_summary":"This paper argues that the standard metric for evaluating prompt-injection attacks on tool-using AI agents—binary attack-success rate (did the attack succeed or not?)—is not merely coarse but affirmatively misleading. The author introduces a seven-level ordinal severity scale (L0 through L6) that grades the worst action an agent actually executed during an episode, keyed to three properties a defender cares about: whether the action was reversible, whether it crossed scope to reach another party or shared state, and whether it expanded privilege. A seventh level (L6) captures escalation chains where severity increases across steps. The scale is computed two ways: a deterministic programmatic oracle that reads raw tool-call trajectories plus the attacker's stated goal, and a panel of three frontier language-model judges that grade tag-free natural-language summaries of the same traces. Applied to the AgentDojo workspace suite across four victim models and two defenses, the severity instrument exposes three concrete cases where the binary metric gives the wrong answer: a defense reporting 0% attack-success rate while still permitting an externally visible cross-scope leak through an unfiltered calendar tool, a defense that lowers attack-success rate while raising the worst-case severity tail, and a cross-model comparison where aggregate rates obscure that essentially all harmful behavior localizes to one model. The judge panel reproduces the oracle with Krippendorff's alpha of 0.91, though all three judges share a systematic blind spot for escalation chains, scoring every true L6 as L4.","feed_headline":"Zero attack-success rate can still mean a cross-scope leak","feed_subtitle":"A seven-level severity scale for AI agent actions exposes three cases where the binary metric gives defenders the wrong answer about whether","key_machinery":"The action-graded harm rubric (L0–L6) with three effect axes (reversibility, scope, privilege) and a trajectory-level escalation-chain level; a programmatic oracle (O_prog) that converts raw episode logs into typed action records using a per-tool effect metadata table and an argument-match attribution rule that reads the attacker's stated goal rather than the benchmark's verdict; and a panel of three frontier LLM judges grading tag-free natural-language trace serializations. The gate order is privilege > cross-scope > irreversible > reversible, reducing to a trajectory peak raised to L6 when two or more completed actions escalate over steps.","core_discovery":"The central object is the action-graded harm rubric: a seven-level ordinal scale (L0–L6) that scores an agent's tool-call trajectory by the peak severity of actions attributed to the attack, using a fixed gate order of privilege over cross-scope over irreversibility over reversibility. The scale is computed from what the agent actually did, not what it said it would do, and it is computed independently by a deterministic oracle (reading per-tool effect metadata and an argument-match attribution rule against the attacker's stated goal) and by a panel of frontier LLM judges (reading tag-free trace summaries). The key empirical finding is that this instrument surfaces three decision-relevant分歧—","pith_inferences":["If the argument-match attribution rule requires the attacker's goal to name a distinguishable target, then attacks with diffuse or dynamically determined targets fall outside the programmatic oracle's reach. The paper notes that LLM judges are the fallback for these cases, but judge reliability was only measured on episodes where the oracle worked—so the instrument's reliability on exactly the cas","The channel-substitution leak (calendar invitation substituting for filtered email) suggests a general principle: any defense that filters specific tools without covering the full space of consequential actions with the same effect properties will be vulnerable to substitution. A severity instrument that matches the attacker's target across all consequential tools is more robust than success check","The finding that a low-cost model (Claude Haiku 4.5) resists as completely as frontier models, while another low-cost model (GPT-4o mini) is broadly compromised, suggests that in this setting safety tuning may matter more than raw capability for resisting indirect prompt injection—but this rests on one suite, one attack family, and modest high-severity counts, so the pattern may not generalize."],"forward_implications":["Red-team benchmarks that report only attack-success rate can affirmatively mislead deployment decisions: a defense scored as perfect (0% ASR) can still permit cross-scope leaks through unfiltered tools, meaning the binary metric creates false confidence in defenses that are not actually safe.","Defenses that reduce attack frequency may simultaneously shift surviving attacks to higher-severity outcomes, so evaluating defenses by ASR alone can select for defenses that make the worst case worse.","Severity scoring enables per-model risk localization that aggregate ASR obscures: when most models are at floor and one model accounts for nearly all harmful behavior, the aggregate rate hides which model a defender should actually worry about deploying.","The LLM judge panel's shared blind spot for escalation chains (all true L6 scored as L4) means that deterministic trajectory-level analysis remains necessary for detecting multi-step escalation, and that judge reliability measured on non-escalation episodes may not transfer to escalation detection."],"fun_headline_variants":["Binary attack-success hides three severity cases in AI agents","Seven-level rubric scores agent tool-call harm from L0 to L6","Frontier LLM judges reproduce oracle severity with alpha 0.91","Zero attack-success defense still permits cross-scope leak","Action-graded scale flags escalation chains binary metrics miss"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The programmatic oracle's attribution rule assumes that the attacker's goal names a specific, distinguishable target (a particular recipient, file, or address). If the attacker's goal does not name a specific target—for example, a diffuse or dynamically determined set of recipients—the oracle cannot attribute the agent's action to the attack, and the severity instrument fails for that episode.","fun_headline_variants_meta":{"raw":{"variants":["Binary attack-success hides three severity cases in AI agents","Seven-level rubric scores agent tool-call harm from L0 to L6","Frontier LLM judges reproduce oracle severity with alpha 0.91","Zero attack-success defense still permits cross-scope leak","Action-graded scale flags escalation chains binary metrics miss"]},"model":"glm-5.2","effort":"low","cost_usd":0.0,"raw_usage":{"total_tokens":708,"prompt_tokens":640,"completion_tokens":68,"prompt_tokens_details":null},"tokens_in":640,"tokens_out":68,"duration_ms":100403,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T09:37:12.415340+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"Find a setting where the binary attack-success rate and the severity scale agree on every deployment-relevant decision—no defense that reads as safe under ASR leaks under severity, no defense that lowers ASR raises the severity tail, and no per-model localization is hidden by aggregation. If the three disagreements the paper identifies are artifacts of this particular suite, attack family, and model set, the practical case for the instrument weakens.","supporting_citations":[],"review_version":1}