{"id":"dc38085f-04fa-4d87-a68f-5830cfd3f356","arxiv_id":"2506.15253","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"RAS-Eval provides an 80-case, 3,802-attack benchmark with JSON, LangGraph, and MCP tool support for evaluating LLM agent security, reporting large attack-induced performance drops that should be interpreted cautiously.","lead":"A new benchmark, RAS-Eval, tests how vulnerable LLM agents are to attacks when they call real and simulated tools, covering 80 test cases and 3,802 attack tasks. It reports that attacks cut task completion by about 36.78 percentage points on average and succeeded 85.65% of the time in the academic scenario, though attack tests were run on only one model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Attack tasks overwrite tool inputs/outputs, so the reported ASR may measure tampered-tool propagation rather than genuine agent compromise.","rationale":"The reader's weakest assumption concerned template redundancy and single-model attack testing; those are real but secondary. The more fundamental issue is the semantics of the attack-success criterion: because the harness can write r_e directly into tool outputs, high ASR can be produced without the agent being manipulated. This concern is load-bearing because it targets the paper's central quantitative claims rather than only their generality. It is also checkable: re-annotation with an agent-centric rubric would settle it. I do not recommend rejection because the benchmark artifact, templates, and repository may support a corrected analysis, and the authors can re-score or clarify the threat model. The verdict should remain conditional, with attack re-scoring added to the required revisions. Secondary inconsistencies (abstract says 6 models, Section 3.1.1 says 8, conclusion says 7) reinforce the need for careful revision but do not change the main recommendation.","tokens_in":11356,"tokens_out":9289,"duration_ms":92612,"concrete_test":"Re-score a stratified random sample of about 200 attack tasks (balanced by mode and scenario) with an agent-centric rubric: success requires the agent to perform the attacker-intended action, such as following an injected instruction, taking a harmful tool action, or leaking data, rather than merely invoking a tampered tool and echoing its output. Compare the resulting ASR and TCR drop with the reported 85.65% and 36.78%, and report ASR separately for tool_input versus tool_output attacks. If the agent-centric ASR is substantially lower or concentrated in tool_output tasks, the reported vulnerability rates are artifacts of the success criterion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.1.4 defines an attack as mode = tool_input (kwargs replace the tool's input parameters) or mode = tool_output (return replaces the tool's output). Equations (7)-(8) then score an attack as successful when the target tool appears in the agent's final invocation sequence and the tool output contains the attacker's desired content r_e. For tool_output attacks, r_e is written into the tool response by the harness itself, so an agent that invokes the tampered tool and relays the response is counted as attacked. That measures propagation of tampered tool state, not whether the adversary achieved a security-relevant effect such as following an injected instruction, performing an unauthorized action, or exfiltrating data. For tool_input attacks, replacing kwargs assumes the attacker can directly set the agent's tool arguments, which bypasses the decision boundary that agent-security evaluation is meant to probe. Listing 2 illustrates the issue: the attack simply makes get_events_from_calendar return ['sleeping']. The headline 85.65% ASR and 36.78% TCR reduction may therefore be dominated by tasks where success is an artifact of the scoring rule rather than evidence of a genuinely exploited vulnerability. This is load-bearing because those numbers are the paper's main evidence that RAS-Eval exposes critical risks in real-world deployments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RAS-Eval, a benchmark for security evaluation of LLM agents, consisting of 80 test cases and 3,802 attack tasks across 7 scenarios and 11 CWE categories, with tools offered in JSON, LangGraph, and MCP formats and in both simulated and real execution modes. The authors report baseline scores for six LLMs, claim that scaling laws hold for security-relevant task completion, and report that attacks on the GLM4-Flash agent reduce average task completion rate by 36.78% and achieve an 85.65% attack success rate in the academic scenario. The benchmark and evaluation protocols are released publicly.","tokens_in":11579,"tokens_out":6522,"duration_ms":62814,"significance":"RAS-Eval addresses a real gap: most existing agent security benchmarks are confined to simulated environments, while this benchmark attempts to support real tool execution and multi-format toolkits. The open release of test cases, attack tasks, and tool implementations is valuable, and the failure-mode taxonomy plus CWE mapping provide a structured vocabulary for future work. If the attack-success metric is redefined and the single-model limitation is addressed, RAS-Eval could become a useful community resource for agent security evaluation.","major_comments":[{"comment":"The attack experiments were run exclusively on GLM4-Flash (the paper states 'We conducted attack task tests exclusively on the GLM4-Flash model'), yet the abstract reports the vulnerability findings as if they apply to the six evaluated LLMs. This overclaims the evidence. The attack-related claims (36.78% TCR reduction, 85.65% ASR) must either be re-run on all six models or be explicitly restricted to GLM4-Flash throughout the abstract, Section 3.3, and the conclusion.","section":"Section 3.1.1; Abstract"},{"comment":"The ASR metric counts an attack as successful whenever the attacked tool appears in the agent's final invocation sequence and the tool output contains the attacker's desired content r_e. For tool_output attacks, the harness itself writes r_e into the tool response, so the metric largely measures whether the agent propagates tampered tool state, not whether the adversary achieved a security-relevant effect such as following an injected instruction, performing an unauthorized action, or exfiltrating data. Listing 2's example (return ['sleeping']) illustrates this. For tool_input attacks, replacing the tool's kwargs assumes the attacker can directly set the agent's tool arguments, bypassing the decision boundary that agent-security evaluation should probe. The headline ASR figures therefore need a redefined metric or a separate security-impact metric.","section":"Section 2.1.4, Eqs. (7)-(8), Listing 2"},{"comment":"The scaling-law claim rests on four Qwen models only, with no reported regression equation, fitted coefficients, or per-model confidence intervals, and the adjusted R-squared is computed with one predictor on four data points. The x-axis label 'ln(ln(B))' is unexplained. This evidence is too thin for the claim that 'scaling laws held for security capabilities'; the authors should report the full fit details, add more model sizes, or substantially soften the claim.","section":"Section 3.2.2, Table 7, Figure 4"},{"comment":"The stated identity is incorrect: sum_{k=1}^{2n} C(2n,k) = 2^{2n} - 1, not 2^{2n-1}. If the intended count is over n tools with direct and indirect attack variants, the derivation should be clarified. Since this formula is presented as the basis for generating the 3,802 attack tasks, the discrepancy should be fixed and the actual generation procedure (58 templates, permutation, filtering) documented precisely.","section":"Section 2.4, Eq. (1)"},{"comment":"The average row reports TIR' = 36.59% with the parenthetical (↑36.59%), but the pre-attack TIR is 38.56%, so the value is a decrease, not an increase; moreover, the Finance row shows score increasing after attack (↑13.43%), contradicting the text that 'performance scores drop significantly across scenarios.' These inconsistencies undermine the headline quantitative claims and must be reconciled.","section":"Table 8"}],"minor_comments":[{"comment":"The tool counts are inconsistent: the text mentions 'fifteen distinct categories', Figure 1 states '18 toolkits, 75 tools', and Section 2.4 refers to '29 tools'; please reconcile these numbers.","section":"Section 2.1.1, Figure 1, Section 2.4"},{"comment":"The per-row #Test case counts sum to far more than 80; please clarify whether a single test case can be mapped to multiple CWE categories and report the distribution accordingly.","section":"Table 3"},{"comment":"The Kappa coefficient computation is underspecified: the roles of humans versus LLM annotators, the definition of Po, and the denominator n^2 are unclear; please define n and the confusion-matrix entries precisely.","section":"Section 3.2.1, Eq. (9)"},{"comment":"There are typos in the model names: 'Qwen2..5-7B-Instruct' should be 'Qwen2.5-7B-Instruct', and the same typo appears in Figure 4/Table 6.","section":"Table 6, Table 7"},{"comment":"The conclusion states that the authors 'evaluated agents powered by 7 mainstream LLMs', but Section 3.1.1 lists six models and the abstract says six; please align these counts.","section":"Section 5"},{"comment":"The abstract highlights the 85.65% ASR for the academic scenario, while the overall average ASR is 73.44%; please clarify how the academic finding should be interpreted relative to the full benchmark.","section":"Section 3.3, Abstract"}],"recommendation":"major_revision","confidential_remarks":"This is a useful benchmark resource, but the headline security findings are currently overstated: the attack results are from a single model and the ASR definition conflates propagation of tampered tool state with genuine security impact. These issues are fixable but require either new experiments or substantial re-scoping of the claims, hence major revision rather than reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: RAS-Eval is a real and reasonably careful benchmark construction, and the open-sourced tools and data are worth serious attention. But the headline numbers—85.65% ASR and 36.78% TCR drop—are not established by the paper's own definitions.\n\nWhat's new and good: the benchmark combines real and simulated tool execution across JSON, LangGraph, and MCP; it maps 80 test cases and 3,802 attack tasks to 11 CWE categories; and the failure-mode taxonomy is a reasonable diagnostic layer. The authors release code and data, which is the right way to do this kind of work. This should be a useful resource for people building and evaluating agents in tool-rich settings.\n\nSoft spots:\n\n1. The attack scoring rule is the main problem. Section 2.1.4 lets an attack replace the tool's input parameters (kwargs) or its output (return). Equations (7)-(8) give a hit whenever the targeted tool appears in the final invocation sequence and its output contains the attacker's string. For tool_output attacks, the harness writes that string into the response, so an agent that invokes the tool and relays the output is scored as \"attacked.\" That measures propagation of tampered tool state, not whether the adversary achieved a security-relevant effect. For tool_input attacks, the attacker directly sets the agent's arguments, which is a man-in-the-middle assumption, not an evaluation of the agent's decision boundary. The listed example (get_events_from_calendar returning ['sleeping']) makes this concrete. The scoring rule inflates ASR and the associated TCR drop; I can't trust 85.65% as evidence of genuine vulnerability.\n\n2. The abstract says six LLMs were evaluated; the conclusion says seven; the body says attack tests were run exclusively on GLM4-Flash. So the security results are a single-model story. That's not a fatal flaw if stated clearly, but it's currently an overstatement.\n\n3. The scaling-law \"verification\" fits a curve to four Qwen points (Figure 4, Table 7). No slope/intercept are reported, the confidence interval is only in the figure, and the sample is tiny. Calling this a \"scaling law\" for security is generous; at best it shows a monotone trend on task scores for one model family.\n\nThe combinatorial attack generation (58 templates permuted across 29 tools and 80 workflows) could also produce redundant tasks, though that's a minor worry given the open data.\n\nIn sum: the artifact is valuable, the experimental claims are not. A revision that separates \"tool-output tampering\" from \"agent takes a security-relevant action,\" and that reports attack results on more than one model, would make this a solid resource. I'd send it to peer review (major revision expected), but I would not let the current abstract stand.\n\nRecommendation: accept for peer review; request major revision. The benchmark deserves serious referee time because it fills a real gap, and the issues are fixable.","headline":"RAS-Eval is a useful benchmark resource, but its headline attack-success and TCR numbers are inflated by an attack-scoring rule that counts tampered environment state as agent compromise.","tokens_in":12104,"tokens_out":3576,"would_cite":false,"duration_ms":34705,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RAS-Eval is a real-tool benchmark claiming attacks on LLM agents succeed 85.65% of the time.","keywords":["LLM agents","security benchmark","prompt injection","tool execution","Common Weakness Enumeration","Model Context Protocol","attack success rate","scaling laws"],"falsifier":"Take 200 attack tasks from the 3,802, have a different writer re-express each attack's intent in fresh wording, and run both versions on the same agent under the same scoring rules. If the re-worded attacks reproduce the reported 85.65% and 36.78% rates, the vulnerabilities are stable; if success drops sharply, the benchmark numbers are wording artifacts.","tokens_in":11149,"feed_emoji":"🔓","tokens_out":10104,"duration_ms":100505,"temperature":0.7,"pith_summary":"This paper introduces RAS-Eval, a benchmark for measuring whether large-language-model agents can be broken through the tools they call, arguing that existing simulated-only testbeds miss these failures. The benchmark contains 80 test cases and 3,802 attack tasks mapped to 11 Common Weakness Enumeration categories, with tools implemented in three common agent-tool formats and runnable in real or simulated modes. Evaluations across seven scenarios find that attacks lower agents' task completion rate by 36.78% on average and succeed 85.65% of the time in an academic setting. The benchmark is designed as a foundation for standardized security testing of agents that act on real tools.","feed_headline":"Attacks cut LLM-agent task success by 36.78%","feed_subtitle":"New benchmark runs 3,802 attacks across 11 vulnerability classes, exposing real deployment risk.","key_machinery":"The load-bearing object is the benchmark dataset itself: each test case couples a natural-language prompt with an annotated workflow of required tool calls and an initial environment, and each attack task injects either input or output tampering at a chosen tool in a chosen workflow. The generated corpus, 80 cases with 75 tools, 58 base attack templates, and 3,802 attack tasks, is scored through task completion rate, task incompletion rate, task fail rate, and attack success rate, with failures classified into six atomic modes (partial tool omission, sequential violation, null execution, stack overflow, extraneous invocation, runtime execution fault) whose combinations produce 32 failure reasons. This structure is what lets the paper attribute performance drops to concrete agent behaviors rather than to overall model quality.","core_discovery":"The core claim is that agent security can be benchmarked against real tool execution rather than only inside simulators. RAS-Eval pairs 80 annotated workflows with 3,802 attack tasks generated by permuting 58 hand-written attack templates over the tool calls in those workflows, spanning 75 tools and 11 vulnerability classes. On the uncontaminated test cases, six models produce human-annotator agreement (average kappa 0.6499) and a scaling-law relation with model size, which the paper takes as evidence that the benchmark measures security capability. In the attack condition, average task completion falls from 61.44% to 38.84%, a 36.78% relative drop, and the average attack success rate is 73.44%, reaching 85.65% in the academic scenario. These results are the basis for the paper's conclusion that real-world deployments of current agents carry measurable, attackable risk.","pith_inferences":["A direct extension of the paper's central claim is that tool outputs should be treated as untrusted data in agent pipelines; if attackers can reach 85.65% success, validating returned content before it feeds the next reasoning step becomes a concrete defensive target.","Because only one model was run under attack, comparing all 3,802 tasks across several model families would show whether the headline rates reflect a general agent vulnerability or a particularly susceptible model.","The benchmark's dual execution modes invite a matched test of the same attacks under simulation versus real execution, which would quantify how much of the reported risk depends on real network effects, authentication, and persistent state.","Since the 3,802 tasks are combinatorial permutations of 58 templates, a semantic de-duplication analysis could reveal how many independent attacks the dataset truly contains; readers should treat the raw task count as an upper bound on independent tests."],"forward_implications":["If the benchmark is accepted, real-world agent deployments in scheduling, file handling, and database scenarios inherit a measurable vulnerability burden: post-attack task completion drops by roughly a third on average.","Attack success rate becomes a standard yardstick for defenses, so any mitigation that lowers ASR on this task set can claim a concrete improvement.","The failure-mode taxonomy turns benchmark failures into root-cause categories such as partial tool omission and extraneous invocation, giving developers a checklist for patching agent behavior.","Tool-format compatibility means one security evaluation can cover the main integration styles used to connect agents to external tools, so benchmark results transfer across deployment stacks.","The observed scaling trend among model sizes suggests that larger agents are more robust but not immune, so scaling alone is unlikely to close the security gap."],"supporting_citations":[{"why":"Supplies the simulated-environment safety baseline that RAS-Eval positions itself against.","marker":"[7]"},{"why":"Provides the LLM-emulated sandbox method for tool-risk testing that real execution is meant to surpass.","marker":"[12]"},{"why":"Offers the closest existing dynamic prompt-injection benchmark; RAS-Eval compares tool count and attack count against it.","marker":"[13]"},{"why":"Formalizes attacks and defenses in LLM agents, giving RAS-Eval an attack formalization to extend to real tools.","marker":"[8]"},{"why":"Gives a static, simulated security evaluation that motivates adding real-world execution support.","marker":"[18]"},{"why":"Defines the model-context protocol whose conformance and security gaps RAS-Eval targets.","marker":"[11]"},{"why":"Supplies the CWE vulnerability taxonomy used to label all 11 categories in the dataset.","marker":"[19]"}],"fun_headline_variants":["RAS-Eval: 3,802 attacks expose LLM agent security gaps","LLM agent task success drops 36.78% under real-world attacks","Benchmark tests 6 LLMs against real tool attacks, finds scaling law","11 CWE classes, 3,802 attacks: new benchmark for LLM agent security","Real tool calls, not simulators: RAS-Eval measures LLM attack risk"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole attack dataset is produced by permuting 58 hand-written templates across the tool sequences of 80 workflows, and only one model was attacked, so if those templates are repetitive the reported drop rates would reflect template artifacts rather than real vulnerabilities.","fun_headline_variants_meta":{"raw":{"variants":["RAS-Eval: 3,802 attacks expose LLM agent security gaps","LLM agent task success drops 36.78% under real-world attacks","Benchmark tests 6 LLMs against real tool attacks, finds scaling law","11 CWE classes, 3,802 attacks: new benchmark for LLM agent security","Real tool calls, not simulators: RAS-Eval measures LLM attack risk"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00028,"raw_usage":{"total_tokens":1647,"prompt_tokens":917,"completion_tokens":730,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":624}},"tokens_in":533,"tokens_out":730,"duration_ms":6738,"temperature":1.0,"reasoning_tokens":624,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:38:33.316327+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take 200 attack tasks from the 3,802, have a different writer re-express each attack's intent in fresh wording, and run both versions on the same agent under the same scoring rules. If the re-worded attacks reproduce the reported 85.65% and 36.78% rates, the vulnerabilities are stable; if success drops sharply, the benchmark numbers are wording artifacts.","supporting_citations":[{"cited_title":"Identifying the risks of lm agents with an lm-emulated sandbox","cited_arxiv_id":null,"evidence_quote":"Provides the LLM-emulated sandbox method for tool-risk testing that real execution is meant to surpass."},{"cited_title":"Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents","cited_arxiv_id":null,"evidence_quote":"Offers the closest existing dynamic prompt-injection benchmark; RAS-Eval compares tool count and attack count against it."},{"cited_title":"Introduction - model context protocol, 04 2025","cited_arxiv_id":null,"evidence_quote":"Defines the model-context protocol whose conformance and security gaps RAS-Eval targets."},{"cited_title":"Common weakness enumeration.Mitre Corporation, 2013","cited_arxiv_id":null,"evidence_quote":"Supplies the CWE vulnerability taxonomy used to label all 11 categories in the dataset."}],"review_version":2}