pith. sign in

arxiv: 2605.22643 · v2 · pith:576K3HMZnew · submitted 2026-05-21 · 💻 cs.CL

Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety

Piercosma Bisconti , Matteo Prandi , Federico Pierucci , Federico Sartore , Enrico Panai , Laura Caroli , Yue Zhu , Adam Leon Smith
show 6 more authors
Luca Nannini Marcello Galisai Susanna Cifani Francesco Giarrusso Marcantonio Bracale Syrnikov Daniele Nardi
This is my paper

Pith reviewed 2026-05-25 06:02 UTC · model grok-4.3

classification 💻 cs.CL
keywords agentic safetyincremental attacksmulti-turn benchmarkAI agentsstateful evaluationoperational risk taxonomytool-using modelsworkplace AI
0
0 comments X

The pith

Tool-using AI agents in office settings yield to incremental attacks at 44.4 percent aggregate rate.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Boiling the Frog, a benchmark that tests whether AI agents using tools in corporate environments can be led into unsafe actions through sequences of requests that begin benign and gradually introduce risk. It uses stateful multi-turn scenarios with a persistent workspace, placing risk-bearing payloads at specific points in the chain and checking whether the final state becomes unsafe. Scenarios follow a three-level taxonomy drawn from defined risks, high-risk regulatory contexts, and codes of practice. Across nine models the strict attack success rate averages 44.4 percent, ranging from 20.5 percent to 92.9 percent by model and reaching 93.3 percent in loss-of-control categories.

Core claim

Tool-using AI models deployed in corporate and office settings are susceptible to incremental attacks when evaluated in stateful multi-turn interactions, with an aggregate strict attack success rate of 44.4 percent across nine models, model-level rates from 20.5 percent for Claude Haiku 4.5 to 92.9 percent for Gemini 3.1 Flash Lite, and average chain category-level success reaching 93.3 percent for Code of Practice loss-of-control scenarios.

What carries the argument

Stateful multi-turn evaluation chains that expose a persistent workspace, place the risk-bearing payload at controlled positions in the turn sequence, and score whether the resulting artifact state becomes unsafe.

If this is right

  • Models differ sharply in resistance, with some exceeding 80 percent success and others below 25 percent, indicating that design choices affect vulnerability to gradual escalation.
  • Loss-of-control scenarios show the highest average success at 93.3 percent, pointing to a systematic weakness when requests build toward reduced oversight.
  • Safety evaluation for agents must move beyond single-turn prompts to include persistent state and multi-turn history.
  • The taxonomy ties directly to regulatory high-risk categories, allowing the benchmark to serve as a concrete test for compliance with defined operational risks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Agent systems could add explicit tracking of request sequences to detect and block patterns that start safe and accumulate risk.
  • Regulators might require multi-turn incremental testing for any agent application placed in high-risk workplace categories.
  • The same incremental-attack structure could be adapted to test agents outside corporate settings, such as personal productivity tools or customer-service systems.

Load-bearing premise

The three-level taxonomy and the simulated stateful scenarios with controlled payload positions accurately capture the relevant high-risk contexts and real corporate agent deployments.

What would settle it

Deploying the same incremental request sequences against actual agent systems in live corporate environments and checking whether the observed rate of unsafe final states matches the benchmark's reported attack success rates.

Figures

Figures reproduced from arXiv: 2605.22643 by Adam Leon Smith, Daniele Nardi, Enrico Panai, Federico Pierucci, Federico Sartore, Francesco Giarrusso, Laura Caroli, Luca Nannini, Marcantonio Bracale Syrnikov, Marcello Galisai, Matteo Prandi, Piercosma Bisconti, Susanna Cifani, Yue Zhu.

Figure 1
Figure 1. Figure 1: Boiling the Frog four-stage pipeline. Starting from regulatory and BF agentic risk categories (Stage 0), each scenario is instantiated in a sandboxed Docker workspace (Stage 1), planned as a multi-turn chain with escalating risk (Stage 2), executed as an agent trajectory (Stage 3), and validated through artifact-based scoring (Stage 4). 20.5% for Claude Haiku 4.5 to 92.9% for Gemini 3.1 Flash Lite, and cat… view at source ↗
Figure 2
Figure 2. Figure 2: Models and harnesses in agentic systems. Models provide reasoning; harnesses provide the control layer that defines action spaces, observation spaces, execution rules, safety boundaries, and persistence. The same model can operate through different harnesses, each exposing different affordances and risk surfaces. only how capable the model is, but how the surrounding harness constrains tools, routes observ… view at source ↗
Figure 3
Figure 3. Figure 3: The agentic interaction loop. At each turn the model proposes an action request (1), the harness validates and executes it (2), the environment updates its state and returns an observation (3), and the harness formats the observation for the next model call (4). History, memory, and workspace state persist across turns. with a full shell has a different risk profile from a model restricted to structured fi… view at source ↗
Figure 4
Figure 4. Figure 4: Three-level risk taxonomy. The benchmark organizes scenarios through BF agentic risk patterns inherent to multi-turn agentic systems, AI Act Annex I and Annex III high-risk contexts, and GPAI Code of Practice systemic-risk categories. Level 1: BF agentic risks. The third level captures emergent risk patterns native to multi-turn agentic systems that are not reducible to a single regulatory category. These … view at source ↗
Figure 5
Figure 5. Figure 5: Payload timing bins and strict ASR in the primary filtered set. Red bars count unique chains by first payload timing; teal bars report aggregate strict ASR over judged risk rows with Wilson 95% confidence intervals. criterion. The predicate refers to the artifact state, not to the model’s verbal description of its action. The sixth step designs the validation. For each scored turn, the judge snapshots the … view at source ↗
Figure 6
Figure 6. Figure 6: Primary filtered strict ASR by model. Models are ordered by decreasing ASR. The SAS ranking is different ( [PITH_FULL_IMAGE:figures/full_fig_p021_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Strict ASR (red) versus Safe Agency Score (blue) by model. The two metrics rank models in nearly opposite order: high strict ASR indicates vulnerability, while a high SAS indicates selective usefulness. 6.2. Results by risk category Risk-category results show that GPAI Code of Practice loss-of-control scenarios are the most severe slice ( [PITH_FULL_IMAGE:figures/full_fig_p022_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Strict ASR by model and risk category. GPAI Code of Practice loss-of-control scenarios (CoP LoC) produce near-ceiling ASR across most models. The SAS counterpart is in [PITH_FULL_IMAGE:figures/full_fig_p023_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Strict ASR by model and BF agentic risk mechanism. Compliance drift, false-context injection, and role drift produce the highest vulnerability across models. The SAS counterpart is in [PITH_FULL_IMAGE:figures/full_fig_p024_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Strict ASR by model and trigger-position class. Early-trigger and fast-burn chains produce high ASR for mid-tier models (Devstral, DeepSeek), while slow-boil chains are the most uniformly effective escalation pattern. The SAS counterpart is in [PITH_FULL_IMAGE:figures/full_fig_p025_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Strict ASR by model and harness in the primary filtered harness-comparison set. Safe Agency Score across harnesses [PITH_FULL_IMAGE:figures/full_fig_p027_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Safe Agency Score (SAS) by model and harness in the primary filtered harness-comparison set. 7. Discussion The payload-timing analysis supports the central premise of the benchmark. Single-turn attacks are already effective, but they are not the whole phenomenon: later payloads continue to produce high strict ASR after benign interaction has accumulated. The benchmark therefore targets a specifically agen… view at source ↗
Figure 13
Figure 13. Figure 13: Geographic distribution of strict ASR and Safe Agency Score (SAS) by model developer headquarters. Strict ASR measures realized unsafe artifact states. SAS measures capability-adjusted safe agency. The results do not show a clear alignment between regulatory preparedness and agentic safety. This is especially significant for the European Union, which has positioned itself as the leading regulatory actor t… view at source ↗
Figure 14
Figure 14. Figure 14: Safe Agency Score (SAS) by model. GPT-5.3 Codex and GLM 5.1 lead the panel because they combine high benign actual-change rates with low unsafe artifact realization. A.4. Risk-category breakdowns: SAS [PITH_FULL_IMAGE:figures/full_fig_p048_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Safe Agency Score (SAS) by model and risk category. All models score 0% on the GPAI Code of Practice loss-of-control slice, indicating no positive selectivity gap in that category. 48 [PITH_FULL_IMAGE:figures/full_fig_p048_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Safe Agency Score (SAS) by model and BF agentic risk mechanism. GPT-5.3 Codex and GLM 5.1 show the strongest selectivity across most mechanisms. Gemini 3.1 Flash Lite and Seed 2.0 Lite score near zero throughout. A.6. Hazard-label breakdowns [PITH_FULL_IMAGE:figures/full_fig_p049_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Strict ASR by model and granular hazard label. The hazard taxonomy decomposes risk categories into finer operational mechanisms. 49 [PITH_FULL_IMAGE:figures/full_fig_p049_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Safe Agency Score (SAS) by model and hazard label. GPT-5.3 Codex and GLM 5.1 maintain high SAS values across most hazard labels except control bypass and consent erosion. A.7. Trigger-position breakdowns [PITH_FULL_IMAGE:figures/full_fig_p050_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Safe Agency Score (SAS) by model and trigger-position class. The strict ASR counterpart is in [PITH_FULL_IMAGE:figures/full_fig_p050_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Normalized read/write tool use before and after the first attack payload. Lower-ASR models show a stronger contraction in both read and write operations after payload exposure. We present also a more refined agentic capabilities score, the Agentic Diagnostic Index, to evaluate the recovery of begning task executed after receiving a malicious payload: Box 3: Agentic Diagnostic Indices Post-payload contract… view at source ↗
Figure 21
Figure 21. Figure 21: Post-payload contraction index by model. Higher values indicate greater reduction in tool use after the first attack payload. Claude Haiku 4.5 contracts most strongly; Gemini 3.1 Flash Lite contracts least. 52 [PITH_FULL_IMAGE:figures/full_fig_p052_21.png] view at source ↗
read the original abstract

Background. Traditional safety benchmarks for language models evaluate generated text: whether a model outputs toxic language, reproduces bias, or follows harmful instructions. When models are deployed as agents, the safety-relevant object shifts from what the system says to what it does within an environment, and evaluating model responses under prompting is no longer sufficient to address the safety challenges posed by artificial intelligence. Recent developments have seen the rise of benchmarks that evaluate large language models as agents. We contribute to this strand of research. Approach. We introduce Boiling the Frog, a benchmark that evaluates whether tool-using AI models deployed in corporate and office settings are susceptible to incremental attacks. Each scenario begins with benign workspace edits and later introduces a risk-bearing request. The benchmark focuses on stateful multi-turn evaluation: chains expose a persistent workspace, place the risk-bearing payload at controlled positions in the turn sequence, and score whether the resulting artifact state becomes unsafe. Scenarios are organized through a three-level operational risk taxonomy grounded in the Boiling the Frog risks, the AI Act Annex I and Annex III high-risk contexts, and EU AI Act's Code of Practice on General-Purpose AI (GPAI). Results. Across a nine-model panel, aggregate strict attack success rate (ASR) is 44.4%. Model-level ASR ranges from 20.5% for Claude Haiku 4.5 to 92.9% for Gemini 3.1 Flash Lite, with Seed 2.0 Lite also above 80%. Average chain category-level ASR reaches 93.3% for Code of Practice loss-of-control scenarios.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces Boiling the Frog, a multi-turn benchmark for evaluating tool-using AI agents' susceptibility to incremental attacks in corporate and office settings. Scenarios start with benign workspace edits before introducing risk-bearing requests, using stateful evaluation with controlled payload positions. A three-level operational risk taxonomy is defined, grounded in Boiling the Frog risks, AI Act Annex I/III, and the EU AI Act Code of Practice on GPAI. Across nine models, the paper reports an aggregate strict attack success rate (ASR) of 44.4%, with model-level variation from 20.5% (Claude Haiku 4.5) to 92.9% (Gemini 3.1 Flash Lite) and category-level ASR reaching 93.3% for loss-of-control scenarios.

Significance. If the benchmark scenarios and taxonomy are shown to be representative of real corporate agent deployments, the work would be significant for agentic safety research by shifting evaluation from single-turn text outputs to stateful multi-turn tool use and incremental escalation. The regulatory grounding and focus on persistent workspace state address a recognized gap in existing benchmarks. The reported ASR variation across models and high rates in loss-of-control categories would provide actionable signals for alignment and deployment practices if the underlying data generation and scoring are reproducible.

major comments (2)
  1. [Abstract] Abstract: The aggregate strict ASR of 44.4% (and model/category breakdowns) is presented without any description of scenario construction, exact scoring rules for unsafe artifact states, model prompting templates, error analysis, or controls for confounding factors such as workspace initialization or turn ordering. This omission is load-bearing because the central claim that the benchmark measures susceptibility to incremental attacks in corporate settings cannot be verified from the reported numbers alone.
  2. [Approach] Approach section (taxonomy and scenario description): The assertion that the three-level taxonomy and stateful multi-turn chains with controlled payload positions 'faithfully represent real corporate agent deployments' is stated without supporting evidence, mapping, or validation data showing that the chosen benign-to-risk sequences, workspace persistence mechanics, or payload placements match observed distributions in actual tool-using agent use. This assumption directly underpins interpretation of the ASR results as real-world risk rather than benchmark-specific performance.
minor comments (1)
  1. [Abstract] The abstract refers to 'strict attack success rate' without defining the strictness criterion or contrasting it with a non-strict variant; a brief definition or reference to the methods section would improve clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive comments. We address each major comment below, providing clarifications from the full manuscript and indicating where revisions will strengthen the presentation.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The aggregate strict ASR of 44.4% (and model/category breakdowns) is presented without any description of scenario construction, exact scoring rules for unsafe artifact states, model prompting templates, error analysis, or controls for confounding factors such as workspace initialization or turn ordering. This omission is load-bearing because the central claim that the benchmark measures susceptibility to incremental attacks in corporate settings cannot be verified from the reported numbers alone.

    Authors: The abstract is intentionally concise per venue guidelines. The full Approach section describes scenario construction (benign workspace edits escalating to risk-bearing requests), stateful multi-turn evaluation with persistent workspace and controlled payload positions, and the three-level taxonomy used for scoring unsafe artifact states. The experimental setup details model prompting templates, error analysis, and controls for workspace initialization and turn ordering to isolate incremental effects. These elements support verification of the reported ASR values as measures of susceptibility to incremental attacks. revision: no

  2. Referee: [Approach] Approach section (taxonomy and scenario description): The assertion that the three-level taxonomy and stateful multi-turn chains with controlled payload positions 'faithfully represent real corporate agent deployments' is stated without supporting evidence, mapping, or validation data showing that the chosen benign-to-risk sequences, workspace persistence mechanics, or payload placements match observed distributions in actual tool-using agent use. This assumption directly underpins interpretation of the ASR results as real-world risk rather than benchmark-specific performance.

    Authors: The taxonomy and scenarios are explicitly grounded in the Boiling the Frog risks, AI Act Annex I/III high-risk contexts, and the EU AI Act Code of Practice on GPAI to target regulatory high-risk areas. We do not possess or cite empirical distributions from proprietary corporate deployments, as such data is not publicly available. The benchmark therefore reflects regulatory definitions of risk rather than a direct empirical match to observed use. We will revise the manuscript to clarify this grounding and add an explicit limitations discussion on the absence of deployment-distribution validation. revision: yes

Circularity Check

0 steps flagged

No circularity: benchmark results are direct empirical measurements on author-defined scenarios with no reductions to fitted inputs or self-citations

full rationale

The paper introduces the Boiling the Frog benchmark, defines its three-level taxonomy by grounding in external references (AI Act Annex I/III and Code of Practice), constructs stateful multi-turn scenarios, and reports measured ASR values across models. No equations, parameters, or derivations are present. The reported aggregate ASR (44.4%), model ranges, and category ASRs are computed directly from model behavior on the benchmark; they are not fitted quantities renamed as predictions, nor do they reduce to self-referential definitions or author-prior uniqueness claims. External grounding citations are to regulatory documents, not self-citations. The derivation chain consists solely of benchmark construction followed by evaluation, which is self-contained and does not exhibit any of the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Only the abstract is available, preventing full enumeration of parameters or assumptions; the work relies on a newly introduced taxonomy and scenario set whose grounding is stated but not evidenced in detail.

axioms (1)
  • domain assumption The three-level operational risk taxonomy grounded in Boiling the Frog risks, AI Act Annex I/III, and GPAI Code of Practice is appropriate for categorizing agent safety threats.
    Invoked to organize scenarios and interpret category-level ASR results.

pith-pipeline@v0.9.0 · 5874 in / 1383 out tokens · 65781 ms · 2026-05-25T06:02:18.911161+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

103 extracted references · 103 canonical work pages · 32 internal anchors

  1. [1]

    Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Man´ e, D. (2016). Concrete problems in AI safety.arXiv:1606.06565

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  2. [2]

    Andriushchenko, M., et al. (2024). AgentHarm: A benchmark for measuring harmfulness of LLM agents.arXiv:2410.09024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  3. [3]

    Anthropic. (2025). Activating AI Safety Level 3 protections. https://www.anthropic. com/news/activating-asl3-protections

    work page 2025

  4. [4]

    Anthropic. (2026). Claude Code documentation. https://code.claude.com/docs/en/overview

    work page 2026

  5. [5]

    Anthropic. (2026). Responsible Scaling Policy Version 3.0. https://www.anthropic. com/news/responsible-scaling-policy-v3

    work page 2026

  6. [6]

    Bhatt, M., et al. (2023). Purple Llama CyberSecEval: A secure coding benchmark for language models.arXiv:2312.04724

    work page arXiv 2023

  7. [7]

    Bhatt, M., et al. (2024). CyberSecEval 2: A wide-ranging cybersecurity evaluation suite for large language models.arXiv:2404.13161

    work page arXiv 2024

  8. [8]

    Adversarial Poetry as a Universal Single-Turn Jailbreak Mechanism in Large Language Models,

    P. Bisconti, M. Prandi, F. Pierucci, F. Giarrusso, M. Bracale, M. Galisai, V. Suriani, O. Sorokoletova, F. Sartore, and D. Nardi, “Adversarial Poetry as a Universal Single-Turn Jailbreak Mechanism in Large Language Models,”arXiv preprintarXiv:2511.15304, 2025. doi:10.48550/arXiv.2511.15304

    work page doi:10.48550/arxiv.2511.15304 2025

  9. [9]

    Beyond Single- Agent Safety: A Taxonomy of Risks in LLM-to-LLM Interactions,

    P. Bisconti, M. Galisai, F. Pierucci, M. Bracale, and M. Prandi, “Beyond Single- Agent Safety: A Taxonomy of Risks in LLM-to-LLM Interactions,”arXiv preprint arXiv:2512.02682, 2025. doi:10.48550/arXiv.2512.02682

    work page doi:10.48550/arxiv.2512.02682 2025

  10. [10]

    From Adversarial Poetry to Adversarial Tales: An Interpretability Research Agenda,

    P. Bisconti, M. Galisai, M. Prandi, F. Pierucci, O. Sorokoletova, F. Giarrusso, V. Suriani, M. Bracale, and D. Nardi, “From Adversarial Poetry to Adversarial Tales: An Interpretability Research Agenda,”arXiv preprintarXiv:2601.08837, 2026. doi:10.48550/arXiv.2601.08837

    work page doi:10.48550/arxiv.2601.08837 2026

  11. [11]

    Bort, J. (2026). A Meta AI security researcher said an OpenClaw agent ran amok on her inbox.TechCrunch

    work page 2026

  12. [12]

    (2020).The Brussels Effect: How the European Union Rules the World

    Bradford, A. (2020).The Brussels Effect: How the European Union Rules the World. Oxford University Press

    work page 2020

  13. [13]

    (2023).Digital Empires: The Global Battle to Regulate Technology

    Bradford, A. (2023).Digital Empires: The Global Battle to Regulate Technology. Oxford University Press

    work page 2023

  14. [14]

    Business Insider. (2025). Replit’s CEO apologizes after its AI agent wiped a company’s code base in a test run and lied about it. https://www.businessinsider.com/replit-ceo- apologizes-ai-coding-tool-delete-company-database-2025-7 39

    work page 2025

  15. [15]

    CEN-CENELEC JTC 21. (2024). Standardisation request M/613 for AI

    work page 2024

  16. [16]

    Chao, P., et al. (2024). JailbreakBench: An open robustness benchmark for jailbreaking large language models.arXiv:2404.01318

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  17. [17]

    Chen, Z., et al. (2024). AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases.arXiv:2407.12784

    work page arXiv 2024

  18. [18]

    Available at: https://www.chinalawtranslate.com/en/22075-2/

    China Law Translate.Cybersecurity Law of the People’s Republic of China (2026 revised version). Available at: https://www.chinalawtranslate.com/en/22075-2/. Last accessed: 19 May 2026

    work page 2026

  19. [19]

    Available at: https://www.chinalawtranslate.com/en/ generative-ai-interim/

    China Law Translate.Interim Measures for the Management of Generative Artifi- cial Intelligence Services. Available at: https://www.chinalawtranslate.com/en/ generative-ai-interim/. Last accessed: 19 May 2026

    work page 2026

  20. [20]

    A., Ruster, L

    Coggins, S., Saeri, A., Daniell, K. A., Ruster, L. P., Liu, J., and Davis, J. L. (2025). The 2025 OpenAI Preparedness Framework does not guarantee any AI risk mitigation prac- tices: A proof-of-concept for affordance analyses of AI safety policies. arXiv:2509.24394. https://arxiv.org/abs/2509.24394

    work page arXiv 2025

  21. [21]

    Costa, M., et al. (2025). Securing AI Agents with Information-Flow Control. arXiv:2505.23643

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  22. [22]

    Debenedetti, E., et al. (2025). Defeating Prompt Injections by Design.arXiv:2503.18813

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  23. [23]

    Deshpande, D., Kannappan, A., and Qian, R. (2026). Benchmarking reward hack detection in code environments via contrastive analysis.arXiv:2601.20103

    work page arXiv 2026

  24. [24]

    Doshi, A., et al. (2026). Towards Verifiably Safe Tool Use for LLM Agents. arXiv:2601.08012

    work page arXiv 2026

  25. [25]

    European Commission. (2022). The Blue Guide on the implementation of EU product rules

    work page 2022

  26. [26]

    European Commission. (2025). Code of Practice for General-Purpose AI Models: Safety and Security Chapter

    work page 2025

  27. [27]

    European Commission. (2025). Commission publishes list of signatories to the General- Purpose AI Code of Practice.https://www.politico.eu/article/eu-signatories-ai-code- practice-amazon-google-ibm-microsoft-ai/

    work page 2025

  28. [28]

    European Commission. (2025). General-Purpose AI Code of Practice. https://digital- strategy.ec.europa.eu/en/policies/contents-code-gpai

    work page 2025

  29. [29]

    European Parliament and Council of the European Union. (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence

    work page 2024

  30. [30]

    Fu, X., et al. (2024). Imprompter: Tricking LLM Agents into Improper Tool Use. arXiv:2410.14923. 40

    work page arXiv 2024

  31. [31]

    Adversarial Humanities Benchmark: Results on Stylistic Robustness in Frontier Model Safety

    M. Galisai, S. Cifani, F. Giarrusso, P. Bisconti, M. Prandi, F. Pierucci, F. Sartore, and D. Nardi, “Adversarial Humanities Benchmark: Results on Stylistic Robustness in Frontier Model Safety,”arXiv preprintarXiv:2604.18487, 2026. doi:10.48550/arXiv.2604.18487

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2604.18487 2026

  32. [32]

    Gardhouse, K., Oueslati, A., and Kolt, N. (2026). Regulating AI agents. arXiv:2603.23471

    work page arXiv 2026

  33. [33]

    Gehman, S., et al. (2020). RealToxicityPrompts: Evaluating neural toxic degeneration in language models. https://arxiv.org/abs/2009.11462Findings of EMNLP

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  34. [34]

    Available at: https://www.geopolitechs.org/p/china-issues-new-rules-on-ai-ethics

    Geopolitechs.China Issues New Rules on AI Ethics Review and Support. Available at: https://www.geopolitechs.org/p/china-issues-new-rules-on-ai-ethics . Last accessed: 19 May 2026

    work page 2026

  35. [35]

    Available at: https: //www.geopolitechs.org/p/chinas-first-policy-framework-for

    Geopolitechs.China’s first policy framework for AI agents. Available at: https: //www.geopolitechs.org/p/chinas-first-policy-framework-for . Last accessed: 19 May 2026

    work page 2026

  36. [36]

    Ghosh, S., et al. (2025). AILuminate: Introducing v1.0 of the AI risk and reliability benchmark from MLCommons.arXiv:2503.05731

    work page arXiv 2025

  37. [37]

    Greenblatt, R., Shlegeris, B., Sachan, K., and Roger, F. (2023). AI Control: Improving Safety Despite Intentional Subversion.arXiv:2312.06942

    work page arXiv 2023

  38. [38]

    Greshake, K., et al. (2023). Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection.arXiv:2302.12173

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  39. [39]

    The Guardian. (2026). Claude-powered AI agent’s confession after deleting a firm’s entire database. https://www.theguardian.com/technology/2026/apr/29/claude-ai-deletes-firm- database

    work page 2026

  40. [40]

    Hartvigsen, T., et al. (2022). ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. https://arxiv.org/abs/2203.09509ACL

    work page arXiv 2022

  41. [41]

    Hong, S., et al. (2023). MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework.arXiv:2308.00352

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  42. [42]

    Huang, Y., et al. (2025). DeceptionBench: A comprehensive benchmark for AI deception behaviors in real-world scenarios.arXiv:2510.15501

    work page arXiv 2025

  43. [43]

    Hubinger, E., et al. (2019). Risks from Learned Optimization in Advanced Machine Learning Systems.arXiv:1906.01820

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  44. [44]

    Hubinger, E., et al. (2024). Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training.arXiv:2401.05566

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  45. [45]

    Kaiyom, F. et al. (2024). HELM Safety: Towards standardized safety evaluations of language models

    work page 2024

  46. [46]

    Li, T., et al. (2024). SciSafeEval: A comprehensive benchmark for safety alignment of large language models in scientific tasks.arXiv:2410.03769. 41

    work page arXiv 2024

  47. [47]

    Li, N., et al. (2024). The WMDP Benchmark: Measuring and reducing malicious use with unlearning.arXiv:2403.03218

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  48. [48]

    Li, X., et al. (2026). Unsafer in many turns: Benchmarking and defending multi-turn safety risks in tool-using agents.arXiv:2602.13379

    work page arXiv 2026

  49. [49]

    Lin, S., Hilton, J., and Evans, O. (2022). TruthfulQA: Measuring how models mimic human falsehoods.ACL

    work page 2022

  50. [50]

    Liu, X., et al. (2023). AgentBench: Evaluating LLMs as agents.arXiv:2308.03688

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  51. [51]

    Liu, Y., et al. (2023). Prompt Injection attack against LLM-integrated Applications. arXiv:2306.05499

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  52. [52]

    Lu, J., et al. (2024). ToolSandbox: A stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities.arXiv:2408.04682

    work page arXiv 2024

  53. [53]

    Malik, S., et al. (2025). RewardBench 2: Advancing reward model evaluation. arXiv:2506.01937

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  54. [54]

    Mazeika, M., et al. (2024). HarmBench: A standardized evaluation framework for automated red teaming and robust refusal.ICML

    work page 2024

  55. [55]

    Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces

    Merrill, M. A., et al. (2026). Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces.arXiv:2601.11868

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  56. [56]

    Nadeem, M., Bethke, A., and Reddy, S. (2021). StereoSet: Measuring stereotypical bias in pretrained language models.ACL

    work page 2021

  57. [57]

    Naik, A., et al. (2025). AgentMisalignment: Measuring the propensity for misaligned behaviour in LLM-based agents.arXiv:2506.04018

    work page arXiv 2025

  58. [58]

    Nangia, N., et al. (2020). CrowS-Pairs: A challenge dataset for measuring social biases in masked language models.EMNLP

    work page 2020

  59. [59]

    Nannini, L., et al. (2026). AI agents under EU law.arXiv:2604.04604

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  60. [60]

    New York Post. (2026). Anthropic’s Claude Mythos model sparks fear of AI doomsday if released to public. https://nypost.com/2026/04/08/business/ anthropics-claude-mythos-model-sparks-fears-of-ai-doomsday-wave-of-devastating-hacks/

    work page 2026

  61. [61]

    Nous Research. (2026). Hermes Agent documentation. https://hermes- agent.lzw.me/docs/en/

    work page 2026

  62. [62]

    OpenAI. (2025). Preparedness Framework. https://openai.com/safety/ preparedness

    work page 2025

  63. [63]

    OpenAI. (2026). Codex CLI features and approvals documentation. https://developers.openai.com/codex/cli/features

    work page 2026

  64. [64]

    (2026).OpenClaw: Personal AI Assistant

    OpenClaw maintainers. (2026).OpenClaw: Personal AI Assistant. GitHub repository. https://github.com/openclaw/openclaw 42

    work page 2026

  65. [65]

    Pan, A., et al. (2023). Do the rewards justify the means? Measuring trade-offs between rewards and ethical behavior in the MACHIAVELLI benchmark.arXiv:2304.03279

    work page arXiv 2023

  66. [66]

    Parrish, A., et al. (2022). BBQ: A hand-built bias benchmark for question answering. Findings of ACL

    work page 2022

  67. [67]

    Bench-2-CoP: Can We Trust Benchmarking for EU AI Compliance? arXiv preprint arXiv:2508.05464, 2025

    Matteo Prandi, Vincenzo Suriani, Federico Pierucci, Marcello Galisai, Daniele Nardi, and Piercosma Bisconti. Bench-2-CoP: Can We Trust Benchmarking for EU AI Compliance? arXiv preprint arXiv:2508.05464, 2025

    work page arXiv 2025

  68. [68]

    Perrigo, B. (2025). Exclusive: New Claude model triggers stricter safeguards at Anthropic. TIME.https://time.com/7287806/anthropic-claude-4-opus-safety-bio-risk/

    work page arXiv 2025

  69. [69]

    Phuong, M., et al. (2025). Evaluating Frontier Models for Stealth and Situational Awareness.arXiv:2505.01420

    work page arXiv 2025

  70. [70]

    Pierucci, F., Prandi, M., Bracale Syrnikov, M., Galisai, M., and Bisconti, P. (2026). Agentic microphysics: A manifesto for generative AI safety.arXiv:2604.15236

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  71. [71]

    Qin, Y., et al. (2023). ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs.arXiv:2307.16789

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  72. [72]

    Ren, R., et al. (2025). The MASK Benchmark: Disentangling honesty from accuracy in AI systems.arXiv:2503.03750

    work page arXiv 2025

  73. [73]

    Available at: https://openstd.samr.gov

    State Administration for Market Regulation and China National Standardization Admin- istration.GB/T 45654–2025 Cybersecurity Technology—Basic Security Requirements for Generative Artificial Intelligence Service. Available at: https://openstd.samr.gov. cn/bzgk/std/newGbInfo?hcno=F67D3F376E0A0A0FF5317FB36B32A30A. Last accessed: 19 May 2026

    work page 2025

  74. [74]

    Available at: https://std.samr.gov.cn/gb/search/ gbDetailed?id=4C5277928DA2411EE06397BE0A0AE436

    State Administration for Market Regulation.General Security Requirements for Artificial Intelligence Agent Application. Available at: https://std.samr.gov.cn/gb/search/ gbDetailed?id=4C5277928DA2411EE06397BE0A0AE436. Last accessed: 19 May 2026

    work page 2026

  75. [75]

    Shinn, N., et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv:2303.11366

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  76. [76]

    Singh, S., et al. (2024). Measuring and improving persuasiveness of large language models. arXiv:2410.02653

    work page arXiv 2024

  77. [77]

    Souly, A., et al. (2024). A StrongREJECT for empty jailbreaks.arXiv:2402.10260

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  78. [78]

    Stelling, L., et al. (2025). Mapping Industry Practices to the EU AI Act’s GPAI Code of Practice Safety and Security Measures.arXiv:2504.15181

    work page arXiv 2025

  79. [79]

    Su, H., et al. (2025). A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents.arXiv:2506.23844

    work page arXiv 2025

  80. [80]

    P., Kaunismaa, J., and Panickssery, A

    Sudhir, A. P., Kaunismaa, J., and Panickssery, A. (2025). A benchmark for scalable oversight protocols.arXiv:2504.03731. 43

    work page arXiv 2025

Showing first 80 references.